The Amazonian Gambit

Ahhh, vacation! Time to do the things I’ve been wanting/needing to do for a while. First up is upgrading my OS. Next up, is gathering requirements for my next “super money making idea” – the first 5 of which entail using amazon.com for my storage, using amazon.com for for my server space, using amazon.com for my queue processing, using amazon.com for my payment processing agent, and using amazon.com for my on-the-fly database. Some of you are probably saying, “Why are you doing that kind of stuff on your vacation?”, the rest of you are saying, “Amazon.com does all of that?”

A lot of people hear amazon.com and immediately think: online bookstore. Very few of them hear amazon.com and think: purveyors of computing’s future. In addition to building a business to allow you to buy just about anything you want to online, Amazon has quietly (well to most people it’s been quietly) built a very enterprise-ready web service back-office infrastructure that allows anyone to:

  • access an unlimited amount of storage capacity [S3];
  • build flexibly structured databases that change contexts as my business requirements change (no pesky schemas required) [SimpleDB];
  • access an unlimited amount of computing power (why do I need to buy a huge server or large number of LAMP boxes when I can instantly instantiate one when I need it, then throw it away – persisting my data in S3 and SimpleDB thank you very much – when my need subsides) [EC2];
  • store data to and pull data from a simple message queue for connecting applications on multiple systems (When my queue gets too full, instantiate a machine. When the queue is emptied, turn off the machine. Just like a sump pump!) [SQS];
  • use their payment processing engine to accept and manage payments. [FPS]Amazon runs the servers, manages the backups, and makes sure the availability and scalability are aligned with your enterprise’s expectations. You just need to figure out how best to interface with all of these resources. Are you a Windows shop? Are you a Unix shop? Amazon doesn’t care. Just hit their API with whatever technology you use. For those of you still struggling with the definition of Web 2.0, how amazon.com has implemented their API is one of the basic tenets of that definition.

While all of the services have business applicability, S3 is probably the most approachable – having applications even for individual users. For about a year now, I’ve been using Amazon’s S3 (Simple Storage Service) as my offsite backup solution. Every night at 3am, all of the day’s work get moved into the cloud. Every month, it pulls 10 random items out of the cloud so I can test the viability of the backup. Clockwork. In the beginning, it was a little bit of a pain to manage as most of the mainstream file transfer and backup tools were still built on the concept of FTP, SFTP, etc. – so I had to use a little ingenuity to automate the process of getting data to my cloud. Recently however, more and more vendors have updated their file transfer clients to include dealing with S3 buckets just like any other file system. S3 is becoming a standard.

All of this flexibility and power will cost you fractional pennies per transaction, per query, per gigabyte, or per machine hour. If your needs are small, your cost is small. As you succeed, amazon.com succeeds. This kind of “pay for it as you need it” model is perfect for startups (low entry cost) as well as businesses that are very cyclical with regards to the ebb and flow of the demand for their services.

The solutions and applications are limited only by the imagination. However, in the early stages of its existence, a certain amount of courage, visionary thinking, and appetite for exploration will be additional requirements (as well as the underlying business need). While there are some business cases and success stories already in play, Amazon’s web services are by no means a fully vetted and business proven solution framework…yet.

Amazon Web Services developer accounts are free. Go to amazonaws.com for more information. Oh, yes – remember a couple of posts ago when I was talking about “staying ahead of the curve” and “catching the next big wave”? That wave is parallel processing and it’s called: Erlang. Amazon’s new SimpleDB is built on it. Part of my vacation will be spent building something cool with that too. We’ll talk about it next time.