Jeff Atwood explains why Vista uses so much memory. "You have to stop thinking of system memory as a resource and start thinking of it as a a cache. Just like the level 1 and level 2 cache on your CPU, system memory is yet another type of high-speed cache that sits between your computer and the disk drive. And the most important rule of cache design is that empty cache memory is wasted cache memory. Empty cache isn't doing you any good. It's expensive, high-speed memory sucking down power for zero benefit. The primary mission in the life of every cache is to populate itself as quickly as possible with the data that's most likely to be needed - and to consistently deliver a high 'hit rate' of needed data retrieved from the cache."
To view parent comment, click here.
To read all comments associated with this story, please click here.
Member since:2005-07-10
Wow did you totally miss the point. I never said Linux didn't have a disk cache, in fact if you actually read the article it was talking about the SuperFetch technology that tries to minimize disk cache.
And that is what I was referring to the actual process that minimizes the use of disk cache. But in order to understand the comments I made on the article you actually have to read the article.
And the very fact that you had to insert the words disk cache, to go off on a tangent about a sentence you took out of context. Shows me you had no intention of actually arguing the point of the article.
Edited 2007-02-28 22:57
Member since:2005-07-08
First, I can't possibly figure out how that "it" I replaced could refer to SuperFetch or any similar technology because Linux doesn't have anticipatory page-in mechanism. Not in 2.4, not now, and probably not in the near future. In addition, memory is not a cache for paging space, so any attempt to associate SuperFetch with using memory as a disk cache is simply wrong.
SuperFetch has nothing to do with the disk cache, since it doesn't have anything to do with the filesystem. On the contrary, it is a virtual memory mechanism. It performs page-ins from the paging space before the virtual pages are referenced based on previously observed trends, replacing least-recently-used pages in memory with hopefully soon-to-be-used pages from paging space. SuperFetch doesn't change the fact that memory should always be full, nor does it create a magic way to page something in without paging something out. It doesn't minimize paging, either. In the best case scenario, it performs paging earlier than it would have otherwise occurred. In anything but the best case, it generates more paging operations than necessary.
I don't understand why you take offense to my comment. Anyone can argue the point of the article, but not that many people actually have the knowledge to decide for themselves whether the whole premise of the article is full of crap. I assumed that the article must be about a disk cache, since that's the only way the teaser makes sense. Memory is not a cache for paging space! When a page gets copied into memory, its blocks are freed on the disk's paging space. When filesystem data gets cached, it persists on the backing store. The whole premise of the article is wrong. So why should I argue it?
Member since:
2005-07-08
I don't even think Linux had [a disk cache] when Windows XP first came out because they were still dealing with the 2.4 kernel.
That's the funniest thing I've read so far today. Linux and every other major UNIX-like system back to at least SVR4 and beyond has had a disk cache that consumes any otherwise unused physically-resident virtual memory allocated to the kernel. Now, to be fair, Linux 2.4 implemented disk caching through the buffer cache, which uses disk blocks as the fundamental unit of storage. In Linux 2.6, the disk cache is implemented mostly through the page cache, which uses larger and more well-behaved (with respect to alignment) virtual memory pages as the fundamental unit of storage. The buffer cache is still used for caching inherently block-sized storage objects such as inodes and other filesystem metadata objects.
The kernel does have a hard limit on virtual address space with comes into play on 32-bit systems (or 64-bit systems running a 32-bit OS). This limit must be set no later than boot-time (usually it is set at install-time) and depends on how much physical memory you have installed on your system. Basically, if you have 4GB of memory installed, and you're running a 32-bit OS, the kernel can only allocate 1-3GB of this memory no matter how small the memory requirements of userspace might be. If it's set to allocate 3GB, you run the risk of starving your userspace applications of virtual memory, causing them to crash. This was a bug that some users hit while running the 32-bit Vista installer on systems with 3GB+ of memory.
These techniques are not new. There are classic OS textbooks that describe these mechanisms in great detail. In fact, I'm sure that any Windows OS that had support for NTFS must have also featured a disk cache, since the basic design consideration for NTFS is that reads should usually be satisfied from the disk cache. When the disk cache hit-rate drops, read performance on NTFS becomes unacceptable, nearly as bad as FAT32.
The fact of the matter is that physical memory is a resource. It's obviously not just a cache. Even on systems with pervasive support for pageable kernelspace (i.e. AIX), some portions of physical memory may never be paged to disk. But more importantly, the system must make the best use of its resources on all levels of the storage hierarchy. For example, local disk storage should be used as a cache for network resources such as NFS volumes. If the NFS volume is large enough, the cachefs volume should expand to fill your unused and unreserved space. Obviously some space will need to remain reserved for paging, dumping the kernel (if so configured), and for emergency maintenance by the superuser. But any other unused space is wasted if there's uncached data on a more distance level of the storage hierarchy.
Edited 2007-02-28 22:33