sexta-feira, novembro 24, 2017

Amazon Cloud Drive Shutting Down Unlimited Storage: Outrage galore!

My own motto when it comes to security: "There are two kinds of bikers: those who already did fall and those who will."

I was outraged by the cut of the QUASI-FREE Amazon storage space. I checked how much data I had stored and I have used a whopping 4Tb. 

Nope. Not gonna happen.

Amazon offered "unlimited" storage to paying users then acted surprised when people started using it "creatively". They could have remedied that by implementing a 2TB limit or whatever for paying users and left everyone on the free tier with a 25GB limit, but they chose not to.

That's fine, it's their prerogative, but to make out that somehow Amazon hands were tied is somewhat misleading. Welcome to the world of money. Cloud storage was inevitably to hold its trusting customers to ransom one day...

People often say "you don't need to keep high res originals unless you're going to print large copies", but I think that's a bit misleading. Maybe I'm a geek, but there are times when I want to zoom in to, say, a street sign and work out where I took an old picture (non-GPS), or look at the time on someone's watch, or perhaps crop it to just a single face to print a portrait or use it as a profile image; and if I only have a low res copy, that can be difficult or impossible. I never know what use I might have for a photo in the future, so I always want to keep the full-size originals just in case. Having originals makes a big difference if you want to crop and edit photos. Also, screen resolutions are never going to go down. The days when your 640 x 480 pixel photo filled a screen have long gone....

I have paid great heed to Schofield's Law for many years: "It's not IF your hard drive fails, its WHEN...".

For people who don't know:
  • Schofield's First Law of Computing states that you should never put data into a program unless you can see exactly how to get it out;
  • Schofield's Second Law of Computing states that data doesn't really exist unless you have at least two copies of it, i.e., two copies that don't have changes automatically synced between them and aren't stored in the same physical location.
  • Schofield's Third Law of Computing states that the easier it is for you to access your data, the easier it is for someone else to access your data.

NB: Roobags' addendum to Schofield's Laws: nothing exists without a link.

Personally I use a combination of local and cloud backup - best of both worlds. Sounds as if Amazon had an unsustainable business model. I work for a company which is (very) big in the area of Cloud services, and you need huge investment to provide the level of performance and resilience which customers expect these days. 

It amazes me that businesses are happy to entrust their most sensitive data to faceless corporations in other countries, where there is no way to verifying their integrity. The potential for industrial espionage and government snooping is vast and even more so after the Snowden revelations. May be ok for consumers, but I wouldn’t go anywhere near it for business. Sensitive data should be kept local to the company. Google Drive might be secure, but frankly I'd be surprised if they didn't harvest the data stored by its users, looking for some way to either make money from it or give the juicy bits to some US government agency. Folks should take a look at this tech conference showing exactly how the NSA and GCHQ are keeping track of your data. Just Google "30c3 To Protect and Infect" and follow the youtube link. If you think data is safe in the cloud or any data for that matter. You owe it to yourselves to educate yourself on what is really going on with your privacy and data safety.

And please, for the love of God, don't confuse backup with cloud storage. Cloud storage is great, but it's not a backup solution. Most of services listed are synchronisation solutions. Delete data from one place and its gone from all of them. More than twenty years of working IT (damn I feel old saying that!) tell me that most data loss is caused by user error, not hardware failure. You need a backup solution that lets you get to a file that was there yesterday, but has gone today (e.g. accidentally deleted by the kids) to protect your data (Schofield's Second Law of Computing). Cloud providers often don't explain clearly enough, is the difference between backup, archiving and sharing.

  • Backup: keep an immutable copy of your data as it existed at a certain time, in such a way so to be safe from catastrophic errors in the original.
  • Archiving: offload your data to somewhere safe, though possibly less convenient to use, in order to free space on your primary device.
  • Sharing: enable access to your data from multiple devices, or from multiple users.

A lot of grief would be avoided if the various providers explained the differences, and how to achieve them. 

You're also entrusting your data to a third party, they can go bust, get taken over, change their terms and conditions at a whim, get hacked, decide what was once free will now cost you. If you must use the cloud, make sure the data is encrypted for transit and decrypted again locally, which gives at least some protection… Not only the privacy issue but the viability of these companies as a going concern. Many internet service companies do not make a profit and rely on venture capital, some do not even have a actual business model that makes sense. After all, growth of the user base alone does not magically make one a successful business if they are not paying customers.

If one's data is important then it simply does not make sense to entrust that data to a company that may not even exist in a year's time.

My Modus Operandi to solve all these issues follows with some comments to help you along.

Offline backups are essential. My worst experiences have been with creeping data corruption either by, say, a failing hard disk or by malware. Deleting a file or saving changes that you shouldn't have made permanently are also common problems if you are using a system that syncronises. At home, in terms of the most important reason for doing backups, i.e. data loss due either to disk failure or a cloud provider going bust, a NAS configured as RAID 1 is a backup because if one disk fails, there is another copy. If you then attach a USB hard drive (of the same disk size as the NAS RAID 1) to the NAS, then that is a further backup. I accept that this does not cover you in the event of a fire or theft.  For that I'll be using an offsite backup (I use ShadowProtect). I remember a few weeks after I had started working at a company many years ago one of my co-workers said:

"What does this message mean "Tape Full. Insert next tape." ? "

"How long have you been getting it?" I asked, innocently.

"Oh, months and months".

Human error is a bitch! (apply Schofield's Second Law of Computing when we want to prevent data corruption; One wrong byte and the file may not be readable).

The local backup of my NAS drive means I have instant access, but if there were ever a house fire or the drive(s) failed, I would still have the cloud as a fallback (cloud backups are also managed automatically by the NAS). Back in the day, I remember paing US$90 for two years worth of access to 2Tb (1Tb as backup space and 1Tb as a sync drive) via IDrive, which used fully encrypted storage and transfer. The biggest downside to cloud storage is not space, but the bandwidth needed to make it effective. Unless you are on a direct fibre connection, even fast broadband is way too slow for routine use of large cloud storage due to the woefully poor upload speeds you generally get, since ISPs seem to fixate on download speed only. Roll on the day when affordable direct fibre to the home was widely available. Hence, you really do need a local solution for day-to-day storage.

Don't forget that even if you are careful about not deleting files that are synced, they can be corrupted - and the corruption will then replace the data in the backup destination. This is particularly important if you are worried about the ransomware viruses that encrypt your data and then demand money to unlock them - if you only have 1 backup, then the chances are the encrypted files will be backed up and you won;t be able to give the hackers the finger.

NB: I've been a Synology aficionado for a long time, but beware. Simple NAS has a lot going for it if you want backups you don't have to plug in or remember to start. However you might need to read those drives if the device fails. One possible route out might be to mount the drive in a USB attached enclosure as I did when I had a blackout with my NAS, then tried accessing it as an external drive using a bootable USB stick running Linux. Another might be to try adding a file system driver to a windows PC, a utility that allows Windows to read Linux formatted drives.

Bottom-line: The primary market for Cloud storage is the large corporate market, not the individual. For companies, Cloud storage simply has to be attractively priced compared to the cost of provisioning and operating their own data centre. Storage is cheap now a days, If you are careless enough to leave your documents in the cloud, when the cloud disappears taking your docs with it, that's what you get for taking the easy/cheap way out. Clouds move, shift and disappear, whether in the sky or on the Web. If you want to rely on something that isn't guaranteed to stick around and not under YOUR control, then you deserve whatever you get!

Bottom-Line 2: After setting up a backup procedure, be sure to actually restore or recover some backed up files to be quite sure the procedure works. It's not enough simply to use the backup software's "verify" procedure. You don't need to do this every time unless the backup is mission-critical (e.g., no more backups have been made). I'm also not a fan of backing up only important files, what I'd describe as "intelligent backup"; a full image of everything is better if you can afford the space and time. Someone I was advising back in the day, decided to rationalise the full tape backup I recommended, doing it rarely. He set up an "intelligent" procedure to back up all files and directories he considered important. Then two drives failed simultaneously (power issues), and he discovered he had no backup, a combination of an "intelligent" backup procedure that didn't do everything expected, and lack of a full file system image. This was largely resolved, by sending the failed discs to a disc drive recovery company with a clean room, but it cost thousands and took several days.

Bottom-line 3: If you encrypt the data, make sure you have good backup for the key to unlock it.


  • Starting downloading all the stuff from Amazon Clouddrive to a staging area:
(3 machines downloading my stuff)

  • My new 3 Tb HDD

  • My HDD disk (2 bays) with my new 3 Tb disk

  • My Synology NAS hooked up to my HDD Caddy

If you don't want to follow-up the "prescriptions" above, you can back up all your work on clay tablets. You can't be too careful...

2 comentários:

Bookstooge disse...

Yeah, there is no free lunch. I've dealt with enough "free" programs that became non-free so I no longer trust any of that stuff.

Of course, my boss's son is taking over the tech side of things and he's so used to cloudstuff that he's moving our backup to cloudstuff. I'll be doing a harddrive backup in December and then we're switching over. I have to admit, I'm just waiting for something catastrophic to happen. But hey, whatever. Its no longer my balliwack...

Manuel Antão disse...

Sure. To each his own... the problem is always in the long run...