Linked by Thom Holwerda on Mon 2nd Nov 2009 23:20 UTC
ZFS has received built-in deduplication. "Deduplication is the process of eliminating duplicate copies of data. Dedup is generally either file-level, block-level, or byte-level. Chunks of data - files, blocks, or byte ranges - are checksummed using some hash function that uniquely identifies data with very high probability. Chunks of data are remembered in a table of some sort that maps the data's checksum to its storage location and reference count. When you store another copy of existing data, instead of allocating new space on disk, the dedup code just increments the reference count on the existing data. When data is highly replicated, which is typical of backup servers, virtual machine images, and source code repositories, deduplication can reduce space consumption not just by percentages, but by multiples."
Thread beginning with comment 392522
To view parent comment, click here.
To read all comments associated with this story, please click here.
To view parent comment, click here.
To read all comments associated with this story, please click here.
RE[5]: I skimmed the article...
by Laurence on Wed 4th Nov 2009 08:39
in reply to "RE[4]: I skimmed the article..."
Member since:2007-03-26
Well, why don't you just try it for yourself?
diff <(head -c 100000 file1.avi) <(head -c 100000 file2.avi)
This will compare the first 100000 bytes of file1.avi and file2.avi.
diff <(head -c 100000 file1.avi) <(head -c 100000 file2.avi)
This will compare the first 100000 bytes of file1.avi and file2.avi.
Unfortunately I'm moving house in a couple of days so my DVDs are packed away - thus I can't rip and diff. (though if anyone else is able to perform this test then i'd love to see the results)
....So I'm going to have to take your word on the differences being there.
However, (and going back to deduping for a moment) if understood the article properly, then the credits don't have to by byte for byte exact as the dedup looks at the bytes themselves rather than the whole MB block of bytes.
Thus there only has to be enough similar grouped bytes for a space saving to occure.
So unless MPEG compression uses some kind of random hash to encode it's data, then surely the very fact that the A/V is the same (timestamp or not) must mean that there are SOME similar bytes that can be grouped and indexed?
RE[6]: I skimmed the article...
by Kebabbert on Wed 4th Nov 2009 12:02
in reply to "RE[5]: I skimmed the article..."
Member since:2007-07-27
ZFS dedupes at block level, not byte level.
For synchronous, I think you can just move your old ZFS data to a new ZFS filesystem which has dedup=on, and then your data will be deduped. Or, move your data off ZFS and move back to a dedup=on ZFS filesystem.
For asynchronous, Jeff bonwick (ZFS architect) says it is needed if you dont have enough CPU and RAM. Then you can dedupe in the night, when no one uses the server. This functionality is needed for legacy hardware. But todays modern hardware, CPU and RAM will be enough. And it will only be better in the future. Hence, asynchronous dedupe is not important with modern hardware. It's role will diminish. Why focus on something legacy? ZFS is top modern and state of the art. No need for asynch. ZFS can dedupe in real time, it requires not that many CPU cycles.
Edited 2009-11-04 12:03 UTC
Features
Linked by Thom Holwerda on 05/18/13 21:33 UTC
Why does Google get so much credit in the technology industry? Why, despite the company's many obvious failings, do many geeks and enthusiasts still hold a somewhat positive view on the all-knowing technology giant? A specific talk at Google I/O this week provides the answer.
Linked by David Adams on 05/16/13 4:23 UTC
OSNews is a sponsor of this year's O'Reilly OSCON in Portland, Oregon, USA. A lucky OSNews reader will win a free three-day pass, including two tutorials days. To win the pass, post a comment on this story saying something about Open Source Software or OSCON. We'll pick a winner at random next week. If you don't have an OSNews account, you may email us your entry. Part of the conference is the 9th annual Open Source Awards, and today the 16th is the deadline for nominations. If you'd like to nominate an outstanding open source contributor, do it here. Read on for more information about OSCON. Update: The 20% discount code for OSNews readers is "OSN."
Linked by Thom Holwerda on 05/11/13 21:41 UTC
"Windows is indeed slower than other operating systems in many scenarios, and the gap is worsening." That's one way to start an insider explanation of why Windows' performance isn't up to snuff. Written by someone who actually contributes code to the Windows NT kernel, the comment on Hacker News, later deleted but reposted with permission on Marc Bevand's blog, paints a very dreary picture of the state of Windows development. The root issue? Think of how Linux is developed, and you'll know the answer.
Linked by Thom Holwerda on 05/08/13 14:22 UTC
This is one of those news items that's fun to write, fun to read, fun to comment on, and where no one will be able to say anything unkind. It's all just one big ball of awesome fluffiness. TuneTracker, the BeOS radio automation software, has just released something very special: TuneTracker System 5, the first version designed entirely and specifically for Haiku. In fact, it actually includes Haiku in the software package. Better yet, TuneTracker also unveiled several system-in-a-box products - which have Haiku and TuneTracker pre-installed.
Linked by Thom Holwerda on 05/02/13 15:28 UTC
Exactly twenty years ago, a document was published that played a huge role in establishing the web as we know it today. Twenty years later, and this simple and straightforward document is proof of an irrefutable fact: while closed technologies can change markets, open technologies can change the world.
Linked by Thom Holwerda on 04/29/13 21:06 UTC
Oh multitasking. That staple of computing that got thrown out the window with many modern smartphones. We got some rudimentary thing in its place - but even as multitasking on phone and tablets improves, its user-visible side remains cumbersome. Windows 8 has a neat implementation, and now it's time Android follows in it footsteps.
Linked by Thom Holwerda on 04/24/13 22:24 UTC
They're here! Whether that excites you or not remains to be seen, but the Galaxy S4, which will most likely become the best selling Android smartphone of the year by a huge margin, has been reviewed by all the major sites, and there's lots of interesting conclusions in there - although I think most of you will get the gist.
Linked by Thom Holwerda on 04/18/13 11:21 UTC
Nokia has posted its quarterly results for the first quarter of 2013, and just like the quarters that came before, there's not a whole lot of good news in there. The rise in Lumia sales still can't even dream of making up for the sales drop in Symbian phones, and when broken down in versions, the sales figures for Windows Phone 8 Lumias in particular are very disappointing. In North America, Nokia is getting slaughtered.
Linked by Thom Holwerda on 04/16/13 9:29 UTC
The day has finally come: Google has started shipping Google Glass to the lucky few early adopters. Now that it's shipping, Google has also unveiled a lot more about the API and the specifications of the device itself. While the company had already given out substantial details at earlier occasions, there are still a few surprises here.
Linked by Thom Holwerda on 04/15/13 22:44 UTC
Almost exactly three years ago, I wrote about why OSNews was no longer OSNews: the alternative operating system scene had died, and OSNews, too, had to go with the times and move towards reporting on a new wave of operating systems - mobile, and all the repercussions that the explosion of smartphones and tablets have caused. Still, I was wondering something today: why aren't we seeing alternative operating systems on mobile?More Features »
Sponsored Links
Member since:
2009-04-22
Well, why don't you just try it for yourself?
diff <(head -c 100000 file1.avi) <(head -c 100000 file2.avi)
This will compare the first 100000 bytes of file1.avi and file2.avi.