Monthly Archives: March 2014

RSync: Backing up your media without tears

I’m sure I’m not unique in using a couple of older laptops (a Macbook and a PC) for my various professional needs. Both run Adobe’s Creative Cloud apps with no problems; One is home to a modern Microsoft Office distribution for a couple different legacy jobs I need to perform occasionally and has a SQL Server/.NET development environment; the other is my Apache/PHP/MySQL environment. I have laptops because I grab work time when and where I can, and that is frequently on the road and with people around me. I create an invisible work perimeter around me in the form of a music library and headphones.

Yeah, I have given in to the temptations of iCloud, and do manage my purchases that way- it’s nice to have those available no matter what. But it’s only a small piece of my vast media empire. These machines have older drives, too; I’m waiting for the SSD price/capacity ratio to hit that magic decimal place. So in order to provide some multi to my media, I bring along an external drive. You know how it goes- now I have media in three different places: in the cloud and on two different portable drives.

Since I work in the development world, I tend to forget that I am sometimes The User, and need some professional guidance. I had this fab idea that I needed a NAS in my home network mix, so it can do most of the heavy lifting so my drives can retire and be primary and secondary backup sources. I impulsively grabbed a Seagate Control NAS while idly browsing at Best Buy the other day. Sure, I read the mixed reviews mentioning the slow transfer times. But no one was reporting you couldn’t stream stuff once it was set up. So I prepped myself to deal with days of slow copies from the external drives to the Seagate.

It was even more miserable than I anticipated. Directly connecting the external drives to the Seagate’s USB port proved to be the worst. I was seeing transfer rates of 1MB/s. At that rate, it was going to take a week. But it worse than that- the Seagate became completely unresponsive. Clearly neither Seagate nor Windows could handle a massive queue. Chunking it up was better- the OS and Seagate at least remained responsive- but the rate was no better, and I really didn’t want to copy 500GB 1 gig at a time. Interestingly I was getting 50 times the rate of transfer when copying over the wireless network- rates at about 50MB/s.

OS X, or rather *nix in general, provided a solution. The desktop support tech in me realized the best way to do it was to eliminate the U-factor entirely and let a program that specializes in large file transfers handle the packet negotiation. The usual OS “explore” method (whether *nix or Windows) just isn’t built for the task. It bombs out, becomes responsive, and doesn’t have a very automatic way of dealing with duplicates. The user is not always available to fix or retry. Robocopy, I would like to point out, is a strong contender for this. But only my Macbook reads both external drives, so I went for the built-in Rsync utility. It’s very easy to use; merely sync the Seagate with each drive, thus creating a single drive that has one copy of each media file from the two drives.

It is trivially easy to use the –ignore-existing switch to provide exactly that sort of situation. And in OS X, all you have to do is navigate to the source and destination folders, then drag them over the Terminal in order to add their paths to the command, which ended up looking something like this:

rsync –ignore-existing –recursive -verbose /Volumes/source/audio /Volumes/destination/audio

The -verbose switch is important for your peace of mind, as you can see RSync chunking through the files, calmly and consistently until the job is complete. Granted, at 50MB/s it took a couple of days to complete, but it did not die and it did not max out the CPU or RAM. It quietly sat in the background doing its thing in a Terminal window, and I was able to work on the machine without any grief. I I highly recommend this utility for moving large files across networks.

[Tweet “RSync: Backing up your media without tears”]