http://www.osnews.com/story/17442/Google_Helps_Terabyte_Data_Swaps Exploring the Future of Computing en-us Copyright 2001-2009, David Adams adam+nospam@osnews.com Tue, 10 Nov 2009 11:10:20 GMT http://www.osnews.com/images/osnews.gif http://www.osnews.com http://osnews.com/thread?219292 http://osnews.com/thread?219292 Beware the USPS compression format for this data transfer! Wed, 07 Mar 2007 18:19:00 GMT donotreply@osnews.com (fretinator) Comments http://osnews.com/thread?219305 http://osnews.com/thread?219305 As the old saying goes, nothing can beat the bandwidth of a truck full of hard drives Though maybe someday we'll get something faster someday. Wed, 07 Mar 2007 18:39:00 GMT donotreply@osnews.com (MightyPenguin) Comments http://osnews.com/thread?219333 http://osnews.com/thread?219333 Would something like Sun's ZFS aide in this data transfer or is it strictly trying to get a protocol to transfer the files from one site to another? In which case you'd need some hardware to do route the packets, like carrier grade or whatever its called. Wed, 07 Mar 2007 19:27:00 GMT donotreply@osnews.com (TaterSalad) Comments http://osnews.com/thread?219362 http://osnews.com/thread?219362 Unrelated problems.

ZFS, being a filesystem, is unrelated to data transfer. I'm sure it could be useful to the project in otherways, though: ext3 filesystems, for example, have a size limit in the neighborhood of 8-16 TB. You could probably use some kind of logical volume manager to concatinate a bunch of filesystems together, but why do that if ZFS can manage such datasets withought breaking a metaphorical sweat?

Of course, the article says nothing about the actual technology Google is using for these "hard drive systems," and I can't recall off the top of my head what the state of ZFS on Linux is (is it working now through FUSE?).

As for the problem at hand--transfers of enormous datasets--it's really just a Google implementation of the old proverb "never underestimate the bandwidth of a station wagon full of backup tapes speeding down the highway." Wed, 07 Mar 2007 20:45:00 GMT donotreply@osnews.com (zbrimhall) Comments http://osnews.com/thread?219366 http://osnews.com/thread?219366 You can't route 120 TB of data from one university to another. You just have to ship it on hardware. Wed, 07 Mar 2007 20:50:00 GMT donotreply@osnews.com (Ford Prefect) Comments http://osnews.com/thread?219390 http://osnews.com/thread?219390 Mr DiBona, open source program manager at Google, said the team was inspired by work done by Microsoft researcher Jim Gray, who delivered copies of the Terraserver mapping data to people around the world. Wed, 07 Mar 2007 21:41:00 GMT donotreply@osnews.com (CrazyDude0) Comments http://osnews.com/thread?219507 http://osnews.com/thread?219507 sneakernet for the win. Thu, 08 Mar 2007 02:11:00 GMT donotreply@osnews.com (sn0n) Comments http://osnews.com/thread?219593 http://osnews.com/thread?219593 bandwidth indeed. By the latency sucks. Thu, 08 Mar 2007 10:09:00 GMT donotreply@osnews.com (jessta) Comments http://osnews.com/thread?219641 http://osnews.com/thread?219641 The Checksum is in the mail! Thu, 08 Mar 2007 14:43:00 GMT donotreply@osnews.com (fretinator) Comments http://osnews.com/thread?219660 http://osnews.com/thread?219660 "sneakernet for the win."

Both in terms of bandwidth and cost.

http://www.codinghorror.com/blog/archives/000783.html

I think the economic value of a network is based on its latency, not its bandwidth. It's just much more difficult to measure the former. Thu, 08 Mar 2007 16:35:00 GMT donotreply@osnews.com (chaosvoyager) Comments http://osnews.com/thread?219881 http://osnews.com/thread?219881 Correction. You'd use LVM to join a bunch of physical volumes together into a logical volume. You'd still have to put a filesystem on the logical volume. Fri, 09 Mar 2007 04:33:00 GMT donotreply@osnews.com (Mathman) Comments