Private Cloud Provisioning Templates

One of the primary benefits of a cloud computing environment is the increased automation. The Provisioning Service is perhaps the core mechanism to deliver this. To better understand the kinds of things we might orchestrate, take a look at the following template. You'll notice that it takes on the same format as Amazon's CloudFormation. This example launches a load balancer as part of our LB-aaS solution for a Eucalyptus cloud:

{

"ToughTemplateFormatVersion" : "2011-03-01",

"Description" : "Launch Load Balancer instance and install LB software.",

"Parameters" : {

"AvailabilityZone" : {

"Description" : "AvaialbilityZone in which an instance should be created",

"Type" : "String"

},

"AccountId" : {

"Description" : "Account Id",

"Type" : "String"

},

"LoadBalancerName" : {

"Description" : "Load Balancer Name",

"Type" : "String"

}

},

"Mappings" : {

"AvailabilityZoneMap" : {

"msicluster" : {

"SecurityGroups" : "default",

"ImageId" : "emi-FF070BFE",

"KeyName" : "rarora",

"EKI" : "eki-3A4A0D5A",

"ERI" : "eri-B2C7101A",

"InstanceType" : "c1.medium",

"UserData" : "80"

}

}

},

"Resources" : {

"LoadBalancerLaunchConfig": {

"Type": "TOUGH::LaunchConfiguration",

"Properties": {

"AccountId" : { "Ref" : "AccountId" },

"SecurityGroups" : { "Fn::FindInMap" : [ "AvailabilityZoneMap", { "Ref" : "AvailabilityZone" }, "SecurityGroups" ]},

"ImageId" : { "Fn::FindInMap" : [ "AvailabilityZoneMap", { "Ref" : "AvailabilityZone" }, "ImageId" ]},

"KeyName" : { "Fn::FindInMap" : [ "AvailabilityZoneMap", { "Ref" : "AvailabilityZone" }, "KeyName" ]},

"InstanceType" : { "Fn::FindInMap" : [ "AvailabilityZoneMap", { "Ref" : "AvailabilityZone" }, "InstanceType" ]},

"EKI" : { "Fn::FindInMap" : [ "AvailabilityZoneMap", { "Ref" : "AvailabilityZone" }, "EKI" ]},

"ERI" : { "Fn::FindInMap" : [ "AvailabilityZoneMap", { "Ref" : "AvailabilityZone" }, "ERI" ]}

}

},

"LoadBalancerInstance" : {

"Type" : "TOUGH::EUCA::LaunchInstance",

"Properties" : {

"AccountId" : { "Ref" : "AccountId" },

"AvailabilityZone": { "Ref" : "AvailabilityZone" },

"LaunchConfig" : { "Ref" : "LoadBalancerLaunchConfig" },

"Setup" : {

}

}

},

"RegisterLoadBalancerInstance" : {

"Type" : "TOUGH::ElasticLoadBalancing::RegisterLoadBalancerInstance",

"Properties" : {

"AccountId" : { "Ref" : "AccountId" },

"LoadBalancerName" : { "Ref" : "LoadBalancerName" },

"Instance" : { "Ref" : "LoadBalancerInstance" }

}

},

"Setup" :{

"Type" : "TOUGH::EUCA::Parallel",

"Operations" : {

"TrackLoadBalancerInstance" : {

"Type" : "TOUGH::EUCA::TrackInstance",

"Name" : "LoadBalancerInstance",

"Properties" : {

"AccountId" : { "Ref" : "AccountId" },

"InstanceId" : { "Fn::GetAtt" : [ "LoadBalancerInstance", "InstanceId" ] }

}

},

"InstalLoadBalancerSoftware" : {

"Type" : "TOUGH::ElasticLoadBalancing::InstallLoadBalancerSoftware",

"Properties" : {

"AccountId" : { "Ref" : "AccountId" },

"IP" : { "Fn::GetAtt" : [ "LoadBalancerInstance", "PublicIp" ] }

}

}

}

}

},

"Outputs" : {

"PublicIP" : {

"Description" : "PublicIP address of the LoadBalancer",

"Value" : { "Fn::GetAtt" : [ "LoadBalancerInstance", "PublicIp" ] }

}

}

}

The JSON format can be a bit difficult to read if you're not familiar with it. Amazon and others now have UI's that facilitate the creation of the templates. In this example, there are a few items worth noting:

1. The template accepts input variables and returns information at the end of execution

2. The orchestration automates a series of tasks (launches a bare image, installs LB software, tracks the progress, configures the software, registers the newly launched instance, etc.)

3. The templates treat the cloud concepts (availability zones, cloud services, etc.) as first-order concepts in the syntax.

Keep in mind that the orchestration scripts can be multiple levels deep. This example was a simple one just to launch a load balancer. A more complicated orchestration would initiate multiple orchestration templates.

In the coming months, we'll be releasing a series of templates designed to orchestrate the provisioning of many common applications. The provisioning templates will fully leverage the power of the cloud (auto scale, auto recover, auto-snapshot, auto balance, etc.)

Private Cloud Provisioning & Configuration

Cloud provisioning has focused on the rapid acquisition and initialization of a new server, disk or some other piece of infrastructure. Provisioning a single piece of infrastructure is now quite easy. Provisioning an entire set is much more complicated. In addition to the setup of a single piece of equipment, it's necessary to understand the dependencies between elements. In some cases, certain infrastructure components must be launched before another element or configuration data from one item needs to be used in a third element. Getting it all right is a difficult task and is a major cause of system failures. An approach to solving the problem is to consider the Deployment Fidelity, that is, the degree to which a deployment is able to fully describe it's architecture and configuration in a digitally precise manner.

Historically, application architects have used Word documents and Visio diagrams to depict the relationship between their software modules and the hardware infrastructure that would host them. Deployment Fidelity deals with accurately describing a set of computing resources and their relationship to each other. Organizations that embrace high fidelity will digitally describe their software and hardware topology: what type of hardware, operating systems, memory, infrastructure services, platform services, etc. and pass the digital description to the cloud provisioner for execution. The business value is two-fold. First, the high fidelity description reduces the chances of manual error, especially during hand-off. Second, the automation of the provisioning task reduces the deployment time and associated costs (e.g., sysadmins running individual scripts, testers waiting for new environments, etc.)

To increase the Deployment Fidelity, the relationships between elements must be captured. For instance, if an application server uses a relational database, the link between the two is recorded and configuration variables (such as IP addresses) are noted. If the server has an outage, a replacement can be auto-launched with the same configuration information. As the complexity of an application increases (load balancers, web servers, app servers, multiple databases, message queues, pub/sub, etc.) the need to keep a digital description becomes extremely important in order to reduce the chance of errors during deployment.

From an organizational perspective, there are two highlights: 1. The deployment architect can describe their proposed solution with complete fidelity - no misinterpretation. In addition, if there is an issue, the changes to the architecture can be captured in version control, just as if it was another piece of software code. 2. The sysadmin or release engineer can take the provisioning script and easily create a new environment (i.e., replicating Dev to Test, etc.)

Today, MomentumSI is announcing the release of two new services that orchestrate the provisioning of complex application topologies and then provide the configuration information:

The Tough Provisioning Service provides equivalent functionality found in Amazon's CloudFormation and is API/Syntax compatible with their offering.

The Tough Configuration Service integrates the most popular configuration management systems into the private cloud. Use your choice of Chef or Puppet to create configuration scripts and then expose them as enterprise grade services (secure access, multiple node delivery, guaranteed transmission, closed loop feedback, etc.)

Our solution brings this functionality to your private cloud by complementing your existing investment in VMware or Eucalyptus.

For more information, see Tough Solutions.

Are Enterprise Architects Intimidated by the Cloud?

Are Enterprise Architects Intimidated by the Cloud?

EA's are often the champion of large change initiatives that span multiple business units. If they're not on board - we've got problems.

Here's why I ask the question:

1. It's my perception (perhaps incorrect) that the EA leadership typically doesn't come from a background in infrastructure architecture. It's been my observation that the EA's who tend to get promoted usually have a background in business or application architecture. These people are often hesitant to enter deep discussions on CPU power consumption, DNS propagation, VLAN decisions, storage protocols, hypervisor trade-offs, etc.

2. Most people have agreed that the cloud can be viewed as a series of layers. You can attack it from top (SaaS) or bottom (IaaS). Quite frankly, there isn't *that much* architecture in SaaS (other than the secure connection and integration). That leaves IaaS as the starting point - which takes me back to point #1 - IaaS intimidates the EA team - - meaning that they're relying on the I.T. data center operations team (and localized infrastructure architects) to define the foundational IaaS layers which will serve PaaS, Dev/Test, disaster recovery, hadoop clusters, etc.

Any truth here? Leave a comment (moderated) or send me an email either way: jschneider AT MomentumSI DOT com

Cloud.com offers Amazon API

The most recent version of Cloud.com is now offering a 'bridge' for the core AWS EC2 services:

"CloudBridge provides a compatibility layer for CloudStack cloud computing software that tools designed for Amazon Web Services with CloudStack.

The CloudBridge is a server process that runs as an adjunct to the CloudStack. The CloudBridge provides an Amazon EC2 compatible API via both SOAP and REST web services."

The functions they support include:

Addresses

AllocateAddress

AssociateAddress

DescribeAddresses

DisassociateAddress

ReleaseAddress

Availability Zones

DescribeAvailabilityZones

Images

CreateImage

DeregisterImage

DescribeImages

RegisterImage

Image Attributes

DescribeImageAttribute

ModifyImageAttribute

ResetImageAttribute

Instances

DescribeInstances

RunInstances

RebootInstances

StartInstances

StopInstances

TerminateInstances

Instance Attributes

DescribeInstanceAttribute

Keypairs

CreateKeyPair

DeleteKeyPair

DescribeKeyPairs

ImportKeyPair

Passwords

GetPasswordData

Security Groups

AuthorizeSecurityGroupIngress

CreateSecurityGroup

DeleteSecurityGroup

DescribeSecurityGroups

RevokeSecurityGroupIngress

Snapshots

CreateSnapshot

DeleteSnapshot

DescribeSnapshots

Volumes

AttachVolume

CreateVolume

DeleteVolume

DescribeVolumes

DetachVolume

Although this list represents the core features of EC2, it doesn't yet cover the upper layers (CloudWatch, Auto Scale, etc.) or the PaaS offering (SNS, SQS, etc.) Regardless, I'm excited to see more emphasis being placed on supporting the AWS standard. It's easy for people to say that IaaS standards don't matter. However, if you're the guy building software on top of IaaS, they matter a WHOLE lot.

Cloud.com is a solid piece of software that has achieved success in the service provider market. To date, they haven't pushed too hard in the enterprise. Their decision to embrace the AWS API is a good one - and is complemented with their decision to use the pieces of OpenStack in their software where appropriate. This idea seems to be getting more traction. I'm hearing more and more people talking about OpenStack like it's a drawer that you reach into and grab out the components that you want - - rather than a holistic platform. I'm not sure if that's what the OpenStack team was shooting for but it's interesting to see guys like Cloud.com being open to leveraging the bits and pieces that they find useful.

The commoditization of scalability

Last week, I had an interesting discussion with a product owner at an ISV. We discussed his offering; it was core plumbing-middleware-kind-of-stuff. When I asked about how he differentiated his offering from others on the market the answer was that they scale better. Our discussion moved from what he was doing to what I was up to and without trying to be coy I said, "We enable the commoditization of scalability". What I mean by this is that we help our customers adopt public and private clouds that know how to auto scale applications (and much more).

Of course, ISV's have always used non-functional attributes like availability, scalability and security as competitive differentiators in their offering. These capabilities are now being provided as features in the IaaS fabric. The next generation products coming from ISV's will need to redesign their solution on top of cloud infrastructures like Amazon, Eucalyptus, vCloud Director, Cloud.com, OpenStack and Nimbula. It will no longer be acceptable for an ISV to march into a customer and demand a block of servers to run their proprietary clusters. They will be expected to be able to allocate computer resources from the IaaS common pool. In addition, the ISV's will need to differentiate on attributes other than those provided by the IaaS fabric.

This change will affect the corporate I.T software development department as well. I've witnessed several I.T. groups design highly scalable architectures. Usually, the I.T. personnel aren't educated to perform this kind of work and either the project fails or delivery costs are very high. I believe that the I.T. departments that invest in IaaS will be able to significantly reduce the cost to design, deploy and operate highly scalable systems. It might be premature to declare the commoditization of scalability, but I truly believe we are witnessing the most significant step towards that goal in my 20 year career.