Linked by Thom Holwerda on Mon 2nd Nov 2009 23:20 UTC
ZFS has received built-in deduplication. "Deduplication is the process of eliminating duplicate copies of data. Dedup is generally either file-level, block-level, or byte-level. Chunks of data - files, blocks, or byte ranges - are checksummed using some hash function that uniquely identifies data with very high probability. Chunks of data are remembered in a table of some sort that maps the data's checksum to its storage location and reference count. When you store another copy of existing data, instead of allocating new space on disk, the dedup code just increments the reference count on the existing data. When data is highly replicated, which is typical of backup servers, virtual machine images, and source code repositories, deduplication can reduce space consumption not just by percentages, but by multiples."
Thread beginning with comment 392473
To view parent comment, click here.
To read all comments associated with this story, please click here.
To view parent comment, click here.
To read all comments associated with this story, please click here.
RE[2]: I skimmed the article...
by Tuxie on Tue 3rd Nov 2009 14:15
in reply to "RE: I skimmed the article..."
Member since:2009-04-22
HTPC / media servers: if you have lot's of DVD rips of TV shows, then you could save several hundred MBs with the intro/outro credits being duduped alone.
Err, no. There is no way the intro/outro scenes are going to be byte-by-byte-identical in the encoded data for different episodes even if they look identical to the eye. Even if nothing else is, the timestamp metadata for each frame is going to differ.
RE[3]: I skimmed the article...
by Laurence on Tue 3rd Nov 2009 14:31
in reply to "RE[2]: I skimmed the article..."
Member since:2007-03-26
Err, no. There is no way the intro/outro scenes are going to be byte-by-byte-identical in the encoded data for different episodes even if they look identical to the eye. Even if nothing else is, the timestamp metadata for each frame is going to differ.
I guess that depends on the codec used.
I thought many MPEG codecs didn't have a timestamp as such and used a form of encoding that allowed an MPEG file (be it a video container file or an MP3 audio file) to be chopped in to parts at any random point and each of the parts can still play individually (much like the myth about worms ability to be chopped up and each part becoming alive)
Besides, your point is only valid for shows that have a pre-opening credits teaser rather than those (typically older) shows that always opened with music and credits.
Member since:
2007-03-26
This feature will be better suited for servers than home PCs, but that doesn't mean that such facility couldn't be useful for some home users:
HTPC / media servers: if you have lot's of DVD rips of TV shows, then you could save several hundred MBs with the intro/outro credits being duduped alone.
media professionals: Granted ZFS isn't coming to OS X now, but if you're a media professional (music, graphics, etc) and want to keep back ups of your projects then you may well have several files with similar contents as the art took shape (much like backed up lines of code in a CVS repository)
that all said, I'd be a touch cautious about jumping in and dedup'ing your file system on consumer grade hardware unless you were confident with your hardware and I'd still recommend weekly scrubs to highlight data degradation before it rots your data completely