Over the course of the last few months, I’ve tested a number of online backup solutions, and found them all lacking. Some are disappointingly constructed and others seem feature-poor, but no matter how well they were designed they all share a single fatal flaw: consumer broadband in the United States is insufficient for backing up the dozens of gigabytes that an average user requires.
Still, the fact that broadband is the problem is progress, given my past experiences with backing up my data. I used to find that backup solutions were expensive or complicated to implement — a decade ago, I used the completely bewildering Retrospect and a prohibitively expensive tape backup system to back up my files — or tried to, anyway. The setup was unwieldy enough that I ran backups erratically, at best, and an erratic backup is not much better than none at all.
Better, Cheaper, Faster
Today, thanks to the advent of cheaper storage and remarkably improved consumer backup software, I back up my data with spotless regularity: each and every night to two 1-terabyte external hard drives that cost just US$70 each. Apple’s Time Machine does incremental backups to one, and the indispensable, must-have utility SuperDuper! clones the complete contents of my internal hard drive to the other.
It works like a charm — except of course, for the one glaring gap in this strategy: there’s no off-site redundancy. Fire or theft would leave me as helpless as any less-conscientious computer user, rendering all my self-congratulatory local backups worthless.
Up, Up, Upstream and Away
A whole crop of Internet-based backup solutions promises to remedy this situation, and in theory any of them should work. I’ve tried Mozy, Carbonite and CrashPlan. Each installs some sort of desktop software component on your hard drive which facilitates the progressive copying of your data to the service’s storage center ‘in the cloud,’ as the kids like to say. To varying degrees you can see that they could work, but they just don’t.
Each of these services has its own pluses and minuses, and some are decidedly quirkier than others. But not one of them is cleverly designed or implemented enough to get over the limitations imposed by my cable modem’s paltry upstream bandwidth, even though Time Warner Cable assures me that I’m subscribed to the fastest service they offer.
Heavy Clouds Ahead
After running the services for several days each, I found that it would take me about a month of uniterrupted upload time to even finish an initial backup of all my critical files. And even if I managed to finish that, incremental backups will inevitably lag behind the rate of change that occurs on my computer. Heaven forbid that a catastrophic event actually spurs me to call on a full backup; the time it would take to restore my system would be impractical.
(At least one service, CrashPlan, is clever enough to offer to send new users a physical hard drive onto which they can copy their initial backup, allowing further backups to be incremented over the cloud. In the event of a major data loss, they can send a hard drive back with the latest data, too. Both are clever workarounds, but either adds considerable expense to the cost of the service. They’re also a bit of a cheat in my book, since they’re not actually cloud computing. Though, hey, if they work for you I would’t blame you one bit for using these methods. For me, even the incremental backups would still be too slow and unreliable.)
What’s so frustrating about the situation these online backup players are in is that there’s no good reason that these services can’t work. Each one has more or less done everything they need to provide a truly useful functionality that makes complete sense. The only thing that’s holding them back is the pathetic state of consumer broadband in America; it’s simply not up to the task, even though the task, while nontrivial, would seem to be fundamental to a healthy data infrastructure. What’s more basic than allowing consumers to upload their files, especially for something as critical as data redundancy? It makes you wonder, really, what other kinds of businesses and services are being handcuffed by the broadband we have in this country.
+
You might check out Backblaze.
When my wife and I had our first daughter this year, I got serious about offsite backups of irreplaceable media.
I think Mozy and other services throttle your uploads on their end, but Backblaze lets you determine the throttling on your end. I was able to upload about 60 gigs in a matter of a few days, and since then it has backed up any daily changes in the wee hours of the morning with no trouble.
Also, I think they’ll mail you either DVDs or a drive of your data, plus you get web access to your files.
At $5 a month for an unlimited amount of data, it’s pretty cheap. I haven’t been using it for more than a few weeks, but so far I’ve been pretty impressed.
Have you tried dropbox.com? I like their syncing ability, so its only mirroring when something updates on your local drive. And if you accidentally delete something on the local, a version is preserved up ‘in the cloud’
Hi Khoi,
I’m in complete agreement with you, but with one distinct difference – the US is positioned far better for this sort of thing than my home nation of Australia.
When you have caps on upload/download in the tens of GB a month even business models such as Netflix are dead in the water.
Just to put some perspective on it.
Cheers,
Joel
I highly recommend checking out Dropbox. I agree that most cloud-based backup services are a slog, but Dropbox is snappy and generally has a pleasant “just works” vibe to it. I use it on my Mac at home and my PC at work and both implementations work equally well. Here’s my referral link which gives you a free 250MB of extra space (disclosure: I get free space too): link.
Matt and Eric G.: Thanks I already use Dropbox. It’s terrific. But I’m talking about backing up dozens if not hundreds of Gigabytes. Dropbox is impractical for that.
Jason: I’ll take a look at Backblaze, but I’m not even sure the ability to throttle on my end will make much of a difference. I need to upload about 400 GB of data.
@Khoi: are you attempting to create an offsite clone daily or a versioned history daily?
I would take the following approach (though you are welcome to content my concepts if you’re creating 400GB+ data a day):
– versioned history on a daily basis backed up locally via time machine + in the cloud with service of your choice
– cloned copy of computer locally daily
– cloned copy of computer taken off site physically periodically (eg: weekly).
Without a dedicated fiber link (and the costs entailed with it) you’ll be struggling to get a clone backup daily onto a cloud host.
I have a three tiered system.
1) Dropbox for my Documents. It’s under 2GB but it still includes all of my Word files from 1998 to now and all of the PDFs of articles I’ve been collecting in grad school and it accessible on iPhone anytime, anyplace. Unfortunately, I don’t think this will be as useful for you, since graphic design files are much bigger than text files.
2) Time Capsule for the whole HD. Doubles as a router. Not perfect (copying can noticeably slow the system a little), but pretty good.
3) External HD offsite. Every once in a while, I SuperDuper my hard drive onto an external, then take that and leave it at the office. Sure, if the town is hit by a meteorite, it’ll be lost with the TC, but it’s not too big a risk.
jungledisk.com. just adopted it last week, seems to be working quite well so far.
Even a measly 384kbps DSL uplink can back up about 1GB per day if you let it run overnight. I use the open-source rsync utility to back up to a computer I keep at work for offsite backup, from 2AM to 8AM, and it would take at most a few days to catch up with a particularly busy photo shoot. If there is a really big bump in my backup set, I just bring the backup PC home to sync over Gigabit Ethernet, and take it back the next day.
I switched from DSL to a 45Mbps symmetric service from a metro Ethernet provider in the Bay Area called Web-Pass. They do an end-run around the telco-cableco duopoly by using newer apartment buildings’ inside wiring to carry 100Mbps Fast Ethernet to the roof, where they use microwave backhaul to their data center. There may be equivalent options in New York.
Let me second the recommendation of Backblaze. I have over 200GB stored. Took 4 or 5 days for the initial backup, but after that it’s been pretty seamless. I do have fios though, which may have helped on the speed end.
What are you using for your 1TB external hard drives? $70 seems like a great price; most 1TB drives I’ve seen run closer to $100.
I think the reason why consumer broadband is good for downstream but has poor upstream is best described with the dancing pigs problem: “Given a choice between dancing pigs and security, users will pick dancing pigs every time.”
The dancing pigs is watching youtube/having excessive downstream.
I’m using 2 external hard drives with Time Machine. Every month I rotate the drives and store the offline drive in an offsite location.
I use Backblaze too. Great service. My initial upload was close to 500GB, and it took a month, but the incrementals are very quick.
I’ve got a time machine disk and a super duper disk as well, so I finally feel good about my backups!
A third vote for Backblaze. The install is an unobtrusive preference pane. It took a long time — two months — on a relatively poor connection for the initial upload of 200GB+, but now I don’t even notice that it’s working away in the background.
Use an extra computer around the house to run os x server. Set it up as an open directory master, and them migrate your machines to network home folders with mobility. Then, two things happen.
The first is that every time your computer syncs, it does so in a way that you could log in to another machine at the house if your primary dies, but you’d still have your latest backup.
Secondly, you can arrange the server to handle offsite backing up. The problem with the above mentioned services and large data is that it isn’t practical to leave your primary machine to archive everybday. By offloading this responsibility to the local server, it can be backing up 24×7 without affecting your workflow. My initial backblaze archive took 4 months, but it was worth it.
Yes, your initial backup will take about a month. But after that, I’ve been surprised at how little data changes on a daily basis. From my experiences, only about one or 2 GB needs to be uploaded each day. I work mainly with text documents, so your experience would be different if you’re working with video or photos. For what it’s worth, I also use Backblaze.
Maybe the problem lies not with the back up services but with your stingy broadband provider. We have Verizon’s FIOS service and upstream speeds are considerably better than we had a cable modem from Comcast. When last I tested it, we were getting upload speeds of 4,365 kbps (see link) which allowed me to upload my initial backup in a week or so of overnight dumps.
I can second the convenience of erin’s approach of offloading the backup process to a second computer, and also highly recommend portable home folders. While my backup hasn’t finished yet (backblaze starts to get a little wonky with 2+tb of stuff), I have never had to think about it interrupting my day to day.
Also, if you are using a laptop, I would suggest creating a partition of your laptop drive, so that time machine can use that. perfect for when you are on the road and don’t have room or a power plug to use an external drive
Khoi,
The average up-speed of an American line is 4.8mbps, while Japan on the other hand would have access to 60+mbps. (See: link)
A high-speed infrastructure is required in order to successfully backup over broadband.
I m not sure how viable it would be, but you could backup to a local server which could then be accessed via the internet wherever and whenever you liked. If done right, it would be fast, safe and secure.
I use two La Cie USB 2.0 external hard drives. Using Time Machine, I cycle one from home to work and back about 8 times a year but, after doing this for a few years, one forgets to do it sometimesЁ
Speaking of LaCie, has anyone tried their Wuala service?
Koi,
Couldn’t agree more with you here. I do freelance work and consulting for small businesses, and having hundreds of gigs to move around simply isn’t efficient. Dropbox is awesome for keeping important documents in sync, but you really need that business class bandwidth for the real data.
{ Justin }
Khoi – good post. I use Time Machine & Superduper just as you do. For offsite backups I use an Amazon S3 account paired with the excellent Jungle Disk app. You might want to check into that.
I’ve tried Mozy and had the same trouble. I never really felt any progress was being made. I had it for about 12 months, and it never went past the initial upload. Most of the apps have very bad design. They do very little to communicate with the user and make them feel safe and secure. I guess that’s technology companies. They think they’re in the bit storage business, but their actually in the ‘making people feel safe’ business. They do the former pretty well, but the latter terribly.
I’ve tried Backblaze more recently and it’s a bit better. But year, you do need to leave your computer on overnight for a month or so (especially if you’ve got 400GB) in order to get the full backup.
I managed mine in a phased way. First I only checked my personal and business documents to be uploaded. When they were all synced, I checked my designs to be uploaded. When they were done, I added in my music. I’ve made sure not to upload any video I have on my HD as I generally don’t want to keep it.
This staged approach really helped me feel safe with the service. It’s a shame such a feature isn’t built in to the software.
We use JungleDisk. The initial backup took about a week, day to day it takes about 20 minutes. Most of my work is done on the web, so at some point in the lifecycle everything I do touches the cloud, there is always a copy somewhere. It might not be the one I want all the time, but at least all is not lost.
A full blown backup from the web seems a little impractical now, but I do use it as a last resort in the event of a total catastrophe of which I’ve had several recently. In those episodes, I have been able to retrieve what I needed.
This reminds me of the first wave of video on the web. Back around 99/2k. Everyone was trying to do streaming video, but it was still painfully frustrating for most users. As soon as the bandwidth opened up, we got YouTube. Upstream bandwidth will come, eventually.
Great article. I am wrestling with these issues now. I now use an external HD and Time Machine. Once a month I create bootable clone with SuperDuper! but I get lazy about taking it off site. But I’m wary of cloud solutions due to speed and security.
I’ll put another vote in for JungleDisk – brilliant at picking up where it left off if your broadband drops off, or if you need the bandwidth for something else momentarily… I’ve got around 110gig up on Amazon now, works fine.
I recommend Amazon S3 and Jungle disk as well, has been very good for me, seems to be pretty fast. I have tried other services but like Amazon because its pay as I use.
Its funny how some people completely skip through an article then decide to offer stupid solutions such as dropbox or ask questions about what you are trying to do when it was clearly outlined in your post, ugh.
Dropbox is a horrible solution for backing up and it is quite impractical.
It seems like you have the right back up solution down, just have to endure the super long initial upload time.
pathetic state of consumer broadband in America
For some reason this jumped into my head when I read that.
Photos. Music. Project files. Those are the 3 things I’d never want to loose—in that order.
So I attach a portable Western Digital Passport hard drive to my Mac and back up the photos with time machine. Same with my music. And I use a separate one for project files.
It’s fast, efficient, portable, and cost-effective.
And I can sleep at night.
When you say you use SuperDuper every day to clone it does that mean you have to erase the HD each time and then clone?
And the cloned drive is bootable I guess?
What brand external HD’s you using?
I have friends who use LiveDrive and rave about it.
This is a great post. Thanks Khoi and everyone else for the input.