Linked by Thom Holwerda on Sat 25th Dec 2010 17:38 UTC
OSNews, Generic OSes First of all: the entire OSNews team would like to wish you a very merry Christmas. Even if you're not religious, there's always porn on the internet, right? Anywho, these wishes are a bit tardy, but that's because I've been fighting a battle with my computer the past few days trying to find a way to record Minecraft footage so I could make a Christmas wish from inside my creations - a losing battle, so it would seem. So, for Christmas, I have two OSNews Asks items for you to ponder. First, help me record Minecraft footage. Second, and this is of more practical use to myself and probably others as well, help me to set up an automatic backup solution that backs up the contents of one folder on an external hard drive to another external drive.
Permalink for comment 454772
To read all comments associated with this story, please click here.
RE: Backup
by djaddison on Sun 26th Dec 2010 05:59 UTC
djaddison
Member since:
2010-12-26

My personal files amount to around 700 GB. These files include personal music recordings, photography, lots of art related files, code, etc...

I'm also extremely paranoid about loosing data / silent corruption / bit rot / drive failure / etc. I keep about 4 copies of the 700 GB data set.

With that much data and number of copies to maintain, I've tried several backup solutions [block level / file level] and what I found was that they didn't verify file transfers. Run md5deep after any of the backup ops and you find most software fails silently. I found this out after validating files of an external hd that was loosing data to either bit rot or some other failure.

Even when the software work "normally" and didn't report any problems I found that about approximately 13 files would become corrupt on transfers of the whole data set.

So, I would second the notion of verification. Even if the verification is done as a post operation.

K. Enough of the background.
I'm currently using SyncBackPro to sync devices on the pc and ChronoSync on the mac.

The reason I use syncback
- runs at start / in the background
- runs sync op on the insertion of a external drive
- really slow [either hash / crc] compare that can be turned on as needed
- drive aliases. aka run when 'backup' drive is inserted.
- allows for bi-directional syncs
- text to speech that can be setup to announce starts / stops / fails
- only copies what's needed
- verifies by copy, then re-reading the src/dest and comparing. This extra read of the src catches faulty src.

I'm currently using it to shuttle/backup my school data. I setup the initial profile about two years ago and haven't looked back. I plug in my usb key and it transfers everything. And the best [geeky] part is that when I insert the key, I hear the voice "key copy initiated", and "key copy complete" when it completes. Makes me think I have one of those sci-fi computers that talks to me. The software works well for my purposes.

That said, I also have some custom python scripts that I use for mass copies and verification. The scrips are broken out into copy / generate md5 list / verify. They could be used as the basis for an automated system. If you're interested, I wouldn't mind opening them up. Email me if you are.

I never got a chance to test it, but Unison also looked interesting. Sadly, school started and I didn't have time to read more about it / investigate.

Reply Score: 2