Cloud infrastructure providers like Amazon are putting out the technology that the enterprise and SaaS providers need to move beyond testing the waters and take advantage of the Cloud today. The latest, and most important from the data storage perspective, is Amazon’s Elastic Block Store, or EBS.

Over the years we’ve witnessed a shift to hosted IT infrastructure where all the issues surrounding the physical plant are consolidated and managed by a specialist service. In the past six months we've witnessed the incredible rate at which cloud computing has really taken off and is now allowing businesses to shed the problems of ordering, racking and maintaining servers and disk storage systems.

The public cloud is now knocking down the barriers to a broader business audience that has seen the advantages of “pay as you go” IT and not having to build or rent another data center. Why do that when you can instantly spin up 10, or 1,000 virtual server instances at a fraction of the cost? Cloud infrastructure providers, like Amazon, are putting out the technology that the enterprise and SaaS providers need to move beyond testing the waters and take advantage of the cloud today. The latest, and most important from the data storage perspective, is Amazon’s Elastic Block Store, or EBS.

Datasets, Throughput and Snapshots
In short, EBS is a SAN (Storage Area Network) in the cloud that works with Amazon’s existing Elastic Compute Cloud (EC2) and Simple Storage Service (S3). One hurdle for many businesses has been the data storage and throughput limits for each instance. Now you can allocate a disk volume of 1GB to 1TB from what is a virtually endless SAN in the cloud, and attach it to an instance running in EC2. The volume is stored on redundant disks and has a lifetime that's separate from any instance on which it is mounted. This is important, as previously the data was lost when an instance was no longer used. Now you can unmount it, and later remount it on another instance. We’ll look at how to get very large datasets using EBS into the cloud.

Another benefit of EBS is taking advantage of the snapshotting feature. You can snapshot a volume to S3, where it is stored with the redundancy and durability of all objects on S3. Moreover, successive snapshots are incremental providing a very powerful and efficient backup capability for volumes.The ability to take snapshots is a complex feature, but RightScale provides some cool scripts to make it even easier to freeze all data access while the snapshot is taken to ensure that the data on the snapshot is consistent.

The RightScale Dashboard supports all the features of EBS and offers a number of additional features such as configuring volumes to automatically be attached to servers when these launch and track the ancestry of a volume or snapshot. What does EBS enable? In short: traditional processing on large datasets and reliable storage for many servers. But let's look at these two areas one-by-one.

Amazon Web Services are designed for scale. EC2, S3, SQS, and SDB are ideally suited for building large systems that process huge data volumes. The catch has been that they are geared towards modern service oriented systems using a non-relational database like Amazon SDB, and thrive on large numbers of simple servers (EC2). Business users have more traditional applications, such as relational databases, that require large datasets stored in a file system with a POSIX interface. While an EC2 X-large instance comes with about 1.4TB of local disk space, it is difficult to use in a production system. Populating the disk with data at boot time can take hours and backups, replication and restoring the data in case of an instance failure are all sore points. For up to 100GB the timescales are workable, but beyond that it gets difficult.