The Era of Simple, Fast, Incredibly Cheap Cloud Storage

This article first appeared in CloudTweaks. Reprinted with permission.

Data storage, like other commodities such as bandwidth, electricity, or simple computer power, underpins nearly everything we do in information technology. Whether it’s a computer game, an international banking system, or a TV show, I can’t think of a single application that doesn’t depend on storing data. However, most of the companies that sell storage don’t like to think of storage as a commodity. That’s especially true in the more nascent market for cloud storage.

The market leaders – Amazon, Google, and Microsoft – all have their own ways of doing things and their own incompatible APIs that make it difficult or impossible to switch from one vendor to another. Even their pricing models are designed to punish you if you want to take your data and go home because while it’s free to put your data in, they charge you to take your data out. To make matters more confusing, they offer a bewildering array of tiers such as S3 standard, Glacier, Reduced Redundancy, Infrequent Access, Nearline, Coldline, and on and on. At a recent Gartner conference, a storage analyst recommended having a full-time analyst to determine what data should go in what tier.

High prices, competing standards, and incompatible interfaces are typical of the early stages of any commodity. Let’s take electricity, for example. In the late 1880s, Thomas Edison and Nikola Tesla were embroiled in a “War of the Currents.” Edison developed direct current (DC), and Tesla, later joined by George Westinghouse, believed that alternating current (AC) was the future. Of course, any motor or device made for one would not work with the other. AC eventually won out, though we still have both 110v in the US and 220v in Europe.

But, for most purposes, electricity is electricity and you can plug almost anything into any outlet in the world and get cheap, reliable power. A more recent example in the IT industry is networking. Today, we take networks for granted. When you buy a new computer, it comes with Wi-Fi and Bluetooth. They work everywhere. But for years, we went through many competing “standards,” starting with ARPANET in 1965, SNA, ProNET, IBM Token-Ring, Ethernet, eventually leading us to the modern day TCP-IP and Internet.

Dial-up Days

As standards emerge, prices drop. When prices drop, new things become possible. My first connection to the Internet was an AOL 1200 bit-per-second dial-up. You could drink your cup of coffee while you waited for a picture to paint on your screen. At $10/month, I was getting roughly 100 bps per dollar. Today, I can buy a pipe to the Internet that offers 1 billion bps for $100 per month, or 10 million bps per dollar–a 100,000-fold decrease in price! As the price of bandwidth dropped, unimaginable new products emerged. Who would have dreamed in the AOL days that in just a few short years we would be watching high-definition movies on demand via the Internet?

Similar to other commodities over the years, the price of storage has also dropped to the point that your wristwatch can store more data than most computer centers 40 years ago. And with super-cheap bandwidth, the prices for cloud storage are heading in the same direction. Today, you can buy cloud storage from Amazon, Microsoft, and others for between 2-3 cents/GB/month. With the introduction of Wasabi, you can find storage that is 80% cheaper and six times the speed compared to what was introduced only a few years ago. As time goes on, you can expect these prices to continue to decline as the cost of raw storage and bandwidth continues to drop.

Storage Tiers

One obvious effect of lower cost and high speed is that it obviates the need for so-called “tiers” of storage. If you have a product that is both cheaper than Amazon’s least expensive storage (Glacier) yet faster than their fastest storage (S3), why would you need or want everything in-between? You don’t have multiple electrical outlets in the wall of your office for good, so-so, and crummy electricity!

Some people think that the cost of storage will go to zero. When you plot the trajectory, it looks like that, but it’s important in this circumstance to use a log scale, not a linear one. Unless you can convince Seagate and Western Digital to give away their disk drives, the cost of storage is not going to zero. But it is continuing to decline at somewhere between 20-30% annually, with no end in sight. It’s getting so cheap that the old preoccupations with storage cost are disappearing. It wasn’t long ago that my IT director would scold me for using up too much storage for my email. The old idea of cleaning up no-longer-needed files seems to be dying as we approach an era where everything can be saved forever. There comes a point when it is just no longer worth the time and effort to clean up old storage. I call it the “attic effect.” You stick your old furniture and memorabilia up in the attic and, since there is really no cost to keeping it there, you never find time to clean it out.

As the price of storage drops, the usage increases. So, the absolute amount of money we spend on storage probably stays constant (and even increases!) because the utility of that information increases exponentially with the amount of data stored. Big Data has shown us that if we have large data sets to analyze, we can make important discoveries, improve efficiencies, and better manage our organizations. Even at the consumer level, cheap storage means we have the ability to go from 2-megapixel cameras to 12-megapixel cameras, and so on and so forth.

As we enter into this new era of cheap, super-fast, and highly reliable cloud storage, I predict the following will happen:

1. Cloud storage will become a commodity, like electricity or bandwidth. It will be there for whatever you need to store.
2. It will be a one-size-fits-all product. Except for edge cases, it will be “storage is storage.”
3. On-premises storage will not keep pace with the cost of large-scale cloud storage, and, for the most part, it will vanish or be reduced to short-term, high-speed applications.
4. A handful of vendors will scale up and compete in the market with compatible, interchangeable storage. Those that provide high quality at the lowest price will win.
5. And, finally, vendors like Amazon that offer more than 100 different cloud products will be vulnerable to specialists picking off parts of the cloud ecosystem that they can do better. Digital Ocean is doing this on the compute side, and Wasabi is doing it on the storage side. Both are thriving. Big companies can’t be best at everything.