Service Oriented Enterprise

Will Amazon Support Linux Containers?

2011-12-03T08:08:00.001-05:00

Early on, Amazon EC2 was recognized as the leading IaaS provider because of their ability to easily provision new virtual machines with a variety of configurations (size, speed, attachments, etc.) Virtual machines are a powerful, yet simple tool for engineers to use but they come at a price (a performance hit). At MomentumSI, we've been pondering if Amazon would ever support Linux Containers in their cloud.

When asked, "Will Amazon Support Linux Containers?" Raj comments, "Would love it. We may see a type of instance whichallows containers on it. You will have to take the whole machine and not justa container on it. That way AWS will not have to bother about maintaining the hostOS. Given the complexities I think it will be a lower priority for Amazon andas it may be financially counterproductive; they may never do it."

Tom comments, "I doubt it. While I'm one of, if not *the*, biggestproponent of linux containers, the business reasoning still lags the technicalreasoning. Intel, for instance, would *hate* such a move. Why? They spent a tonof money on virtualization at a chip level, which becomes a non-issue incontainers (no hardware gets shared at the metal, rather, it's all one kernelfor all containers). So, while it would be a great thing to see, the businessmarket simply doesn't support this at this point, other than for folks likePixar or other compute heavy folks.

What I *would* bet on is that AWS internally switches tosome container based systems. For instance, ElasticMapReduce is far better offin a container world than in a VM world. Easier to maintain, direct access to'cpu speed' and no need to virtualize access to disks -- it's all just there(even ISCSI ends up better in containers -- no 'vm to hypervisor' networktranslations)."

Amazon will likely be forced into one of three positions:

1. Delivering sub-optimal platform performance on VM's (current state)

2. Supporting Linux Containers behind the scenes but not giving customer access to it.

3. Delivering Linux Containers to customers and dealing with a whole new set of technical headaches.

I'm more optimistic than my counterparts on the likelihood of #3. My reasons are simple: First, Amazon has done what they needed to do to satisfy customer needs. Second, I think they'll need to do it to remain competitive with companies like Rackspace. As developers move from "needing a vm" to "needing a platform" (database, app server, etc.), Amazon will be pressed to expose a more highly performant layer to platform developers. One thing my associates and I agreed on is that we will not likely see containers in 2012... perhaps 2013?

Is Cloud Foundry a PaaS?

2011-11-22T19:03:00.001-05:00

I've been asking some people in the industry a real simple question, "Is Cloud Foundry a Platform as a Service"?

The obvious answer would seem to be "yes" - after all, VMware told us it's a PaaS:

That should be the end of it, right? For some reason, when I hear "as-a-Service", I expect a "service" - as in Service Oriented. I don't think that's too much to ask. For example, when Amazon released their relational data service, they offered me a published service interface:
https://rds.amazonaws.com/doc/2010-07-28/AmazonRDSv4.wsdl

I know there are people who hate SOAP, WS-*, WSDL, etc. - that's cool, to each their own. If you prefer, use the RESTful API: http://docs.amazonwebservices.com/AmazonRDS/latest/APIReference/

Note that the service interface IS NOT the same as the interface of the underlying component (MySQL, Oracle, etc.), as those are exposed separately.

Back to my question - is Cloud Foundry a PaaS?

If so, can someone point me to the WSDL's, RESTful interfaces, etc?

Will those interfaces be submitted to DMTF, OASIS or another standards body?

Alternatively, is it merely a platform substrate that ties together multiple server-side technologies (similar to JBoss or WebSphere)?

Will cultural pushback kill private clouds?

2011-11-22T08:45:00.001-05:00

Derick Harris asks the question, "Will cultural pushback kill private clouds?" His questioning comes from a piece provided by Lydia Leong where she notes that many enterprises have fat management structures and aren't organized like many of the leaner cloud providers.

I tend to agree with the premise that the enterprise will have difficulties in adopting private cloud but not for the reasons the authors noted. The IaaS & PaaS software is available. Vendors are now offering to manage your private cloud in an outsourced manner. More often than not, companies are educated on cloud and "get it". They have one group of people who create, extend and support the cloud(s). They have another group who use it to create business solutions. It's a simple consumer & provider relationship.

Traditionally, there are three ways things get done in Enteprise IT:
1. The CIO says "get'er done" (and writes a check)
2. A smart business/IT person uses program funds to sneak in a new technology (and shows success)
3. Geeks on the floor just go and do it.

With the number of downloads of open source stacks like OpenStack and Eucalyptus, it is apparent that model #3 is getting some traction. My gut tells me that the #2 guys are just pushing their stuff to the public cloud (will beg forgiveness - not asking for permission). On #1, many CIO's are hopeful that they can just 'extend their VMware' play - while more aggressive CIO's are looking to the next generation cloud vendors to provide something that matches the public cloud features in a more direct manner.

There are adoption issues in the enterprise. However, it's the same old reasons. Fat org-charts aren't going away and will not be the life or death of private cloud. In my opinion, we need the CIO's to make bold statements on switching to an internal/external cloud operating model. Transformation isn't easy. And telling the CIO that they need to fire a bunch of managers in order to look more like a cloud provider is silly advice and a complete non-starter.

Measuring Availability of Cloud Systems

2011-08-12T07:06:00.003-05:00

The analysts at Saugatuck Technology recently wrote a note on "Cloud IT Failures Emphasize Need for Expectation Management". One comment caught my attention:

"Recall that the availability of a group of components is the product of all of the individual component availabilities. For example, the overall availability of 5 components, each with 99 percent availability, is: 0.99 X 0.99 X 0.99 X 0.99 X 0.99 = 95 percent."

I understand their math - but it strikes me odd that they would use this thinking when discussing cloud computing. In cloud environments, the components are often available as virtualized n+1 highly available pairs. If one is down, the other is taking over. In a non-cloud world, this architecture is typically only reserved for the most critical components (e.g., load balancers or other single-point-of-failures). It's also common to create a complete replica of the environment in a disaster recovery area (e.g., AWS availability zones). In theory, this leads to very high up-time.

Let me put this another way... I currently have 2 cars in my driveway. Let's say each of them has 99% up-time. If one car doesn't start, I'll try the other car. If neither car starts, I'll most likely walk over to my neighbors house and ask to borrow one of their two cars (my DR plan). You can picture the math... in the 1% chance that car A fails, theirs a 99% chance that car B will succeed, and so on. However, experience in both cars and in computing tells us that this math doesn't work either. For instance, if car A didn't start because it was 20 degrees below zero outside, there's a good chance that car B won't work start - and for that matter, my neighbors cars won't start either. Structural or natural problems tend to infect the mass.

I wish I could show you the new math for calculating availability in cloud systems - but it's beyond my pay grade. What I know is that the old math isn't accurate. Anyone have suggestions on a more modern approach?

OpenShift: Is it really PaaS?

2011-08-11T07:16:00.002-05:00

Redhat recently announced an upgraded version of OpenShift with exciting new features including support for Java EE6, Membase, MongoDB and more. See details at:

https://www.redhat.com/openshift/blogs/whats-new-in-openshift-august-2011

As I dug through the descriptions, I found myself with more questions than answers. When you say Membase or MongoDB are available as part of the PaaS, what does this really mean? For example:

They're pre-installed in clustered or replicated manner?
They're monitored out of the box?
Will it auto-scale based on the monitoring data and predefined thresholds? (both up and down?)
They have a data backup / restore facility as part of the as-a-service offering?
The backup / restore are as-a-service?
The backup / restore use a job scheduling system that's available as-a-service?
The backup / restore use an object storage system that has cross data center replication?

Ok, you get the idea. Let me be clear - I'm not suggesting that OpenShift does or doesn't do these things. Arguments can be made that it in some cases, it doesn't need to do them. My point is that several new "PaaS offerings" are coming to market and they smell like the same-ole-sh!t. If nothing else, the product marketing teams will need to do a better job of explaining what they currently have. Old architects need details.

It's no secret that I'm a fan of Amazon's approach of releasing their full API's (AWS Query, WSDL, Java & Ruby API's, etc.) along with some great documentation. They've built a layered architecture whereby the upper layers (PaaS) leverage lower layers (Automation & IaaS) to do things like monitoring, deployment & configuration of both the platforms and the infrastructure elements (block storage, virtual compute, etc.) The bar has been set for what makes something PaaS - and going forward, products will be measure based on this basis. It's ok if your offering doesn't do all they sophisticated things you find in AWS - but it's better to be up front about it. Old architects will understand.

Private Cloud Provisioning Templates

2011-04-26T06:06:00.003-05:00

One of the primary benefits of a cloud computing environment is the increased automation. The Provisioning Service is perhaps the core mechanism to deliver this. To better understand the kinds of things we might orchestrate, take a look at the following template. You'll notice that it takes on the same format as Amazon's CloudFormation. This example launches a load balancer as part of our LB-aaS solution for a Eucalyptus cloud:

{

"ToughTemplateFormatVersion" : "2011-03-01",

"Description" : "Launch Load Balancer instance and install LB software.",

"Parameters" : {

"AvailabilityZone" : {

"Description" : "AvaialbilityZone in which an instance should be created",

"Type" : "String"

"AccountId" : {

"Description" : "Account Id",

"Type" : "String"

"LoadBalancerName" : {

"Description" : "Load Balancer Name",

"Type" : "String"

}

"Mappings" : {

"AvailabilityZoneMap" : {

"msicluster" : {

"SecurityGroups" : "default",

"ImageId" : "emi-FF070BFE",

"KeyName" : "rarora",

"EKI" : "eki-3A4A0D5A",

"ERI" : "eri-B2C7101A",

"InstanceType" : "c1.medium",

"UserData" : "80"

}

"Resources" : {

"LoadBalancerLaunchConfig": {

"Type": "TOUGH::LaunchConfiguration",

"Properties": {

"AccountId" : { "Ref" : "AccountId" },

"SecurityGroups" : { "Fn::FindInMap" : [ "AvailabilityZoneMap", { "Ref" : "AvailabilityZone" }, "SecurityGroups" ]},

"ImageId" : { "Fn::FindInMap" : [ "AvailabilityZoneMap", { "Ref" : "AvailabilityZone" }, "ImageId" ]},

"KeyName" : { "Fn::FindInMap" : [ "AvailabilityZoneMap", { "Ref" : "AvailabilityZone" }, "KeyName" ]},

"InstanceType" : { "Fn::FindInMap" : [ "AvailabilityZoneMap", { "Ref" : "AvailabilityZone" }, "InstanceType" ]},

"EKI" : { "Fn::FindInMap" : [ "AvailabilityZoneMap", { "Ref" : "AvailabilityZone" }, "EKI" ]},

"ERI" : { "Fn::FindInMap" : [ "AvailabilityZoneMap", { "Ref" : "AvailabilityZone" }, "ERI" ]}

}

"LoadBalancerInstance" : {

"Type" : "TOUGH::EUCA::LaunchInstance",

"Properties" : {

"AccountId" : { "Ref" : "AccountId" },

"AvailabilityZone": { "Ref" : "AvailabilityZone" },

"LaunchConfig" : { "Ref" : "LoadBalancerLaunchConfig" },

"Setup" : {

}

"RegisterLoadBalancerInstance" : {

"Type" : "TOUGH::ElasticLoadBalancing::RegisterLoadBalancerInstance",

"Properties" : {

"AccountId" : { "Ref" : "AccountId" },

"LoadBalancerName" : { "Ref" : "LoadBalancerName" },

"Instance" : { "Ref" : "LoadBalancerInstance" }

}

"Setup" :{

"Type" : "TOUGH::EUCA::Parallel",

"Operations" : {

"TrackLoadBalancerInstance" : {

"Type" : "TOUGH::EUCA::TrackInstance",

"Name" : "LoadBalancerInstance",

"Properties" : {

"AccountId" : { "Ref" : "AccountId" },

"InstanceId" : { "Fn::GetAtt" : [ "LoadBalancerInstance", "InstanceId" ] }

}

"InstalLoadBalancerSoftware" : {

"Type" : "TOUGH::ElasticLoadBalancing::InstallLoadBalancerSoftware",

"Properties" : {

"AccountId" : { "Ref" : "AccountId" },

"IP" : { "Fn::GetAtt" : [ "LoadBalancerInstance", "PublicIp" ] }

}

"Outputs" : {

"PublicIP" : {

"Description" : "PublicIP address of the LoadBalancer",

"Value" : { "Fn::GetAtt" : [ "LoadBalancerInstance", "PublicIp" ] }

}

The JSON format can be a bit difficult to read if you're not familiar with it. Amazon and others now have UI's that facilitate the creation of the templates. In this example, there are a few items worth noting:

1. The template accepts input variables and returns information at the end of execution

2. The orchestration automates a series of tasks (launches a bare image, installs LB software, tracks the progress, configures the software, registers the newly launched instance, etc.)

3. The templates treat the cloud concepts (availability zones, cloud services, etc.) as first-order concepts in the syntax.

Keep in mind that the orchestration scripts can be multiple levels deep. This example was a simple one just to launch a load balancer. A more complicated orchestration would initiate multiple orchestration templates.

In the coming months, we'll be releasing a series of templates designed to orchestrate the provisioning of many common applications. The provisioning templates will fully leverage the power of the cloud (auto scale, auto recover, auto-snapshot, auto balance, etc.)

Private Cloud Provisioning & Configuration

2011-04-24T07:28:00.007-05:00

Cloud provisioning has focused on the rapid acquisition and initialization of a new server, disk or some other piece of infrastructure. Provisioning a single piece of infrastructure is now quite easy. Provisioning an entire set is much more complicated. In addition to the setup of a single piece of equipment, it's necessary to understand the dependencies between elements. In some cases, certain infrastructure components must be launched before another element or configuration data from one item needs to be used in a third element. Getting it all right is a difficult task and is a major cause of system failures. An approach to solving the problem is to consider the Deployment Fidelity, that is, the degree to which a deployment is able to fully describe it's architecture and configuration in a digitally precise manner.

Historically, application architects have used Word documents and Visio diagrams to depict the relationship between their software modules and the hardware infrastructure that would host them. Deployment Fidelity deals with accurately describing a set of computing resources and their relationship to each other. Organizations that embrace high fidelity will digitally describe their software and hardware topology: what type of hardware, operating systems, memory, infrastructure services, platform services, etc. and pass the digital description to the cloud provisioner for execution. The business value is two-fold. First, the high fidelity description reduces the chances of manual error, especially during hand-off. Second, the automation of the provisioning task reduces the deployment time and associated costs (e.g., sysadmins running individual scripts, testers waiting for new environments, etc.)

To increase the Deployment Fidelity, the relationships between elements must be captured. For instance, if an application server uses a relational database, the link between the two is recorded and configuration variables (such as IP addresses) are noted. If the server has an outage, a replacement can be auto-launched with the same configuration information. As the complexity of an application increases (load balancers, web servers, app servers, multiple databases, message queues, pub/sub, etc.) the need to keep a digital description becomes extremely important in order to reduce the chance of errors during deployment.

From an organizational perspective, there are two highlights: 1. The deployment architect can describe their proposed solution with complete fidelity - no misinterpretation. In addition, if there is an issue, the changes to the architecture can be captured in version control, just as if it was another piece of software code. 2. The sysadmin or release engineer can take the provisioning script and easily create a new environment (i.e., replicating Dev to Test, etc.)

Today, MomentumSI is announcing the release of two new services that orchestrate the provisioning of complex application topologies and then provide the configuration information:

The Tough Provisioning Service provides equivalent functionality found in Amazon's CloudFormation and is API/Syntax compatible with their offering.

The Tough Configuration Service integrates the most popular configuration management systems into the private cloud. Use your choice of Chef or Puppet to create configuration scripts and then expose them as enterprise grade services (secure access, multiple node delivery, guaranteed transmission, closed loop feedback, etc.)

Our solution brings this functionality to your private cloud by complementing your existing investment in VMware or Eucalyptus.

For more information, see Tough Solutions.

Are Enterprise Architects Intimidated by the Cloud?

2011-04-05T06:06:00.002-05:00

Are Enterprise Architects Intimidated by the Cloud?

EA's are often the champion of large change initiatives that span multiple business units. If they're not on board - we've got problems.

Here's why I ask the question:

1. It's my perception (perhaps incorrect) that the EA leadership typically doesn't come from a background in infrastructure architecture. It's been my observation that the EA's who tend to get promoted usually have a background in business or application architecture. These people are often hesitant to enter deep discussions on CPU power consumption, DNS propagation, VLAN decisions, storage protocols, hypervisor trade-offs, etc.

2. Most people have agreed that the cloud can be viewed as a series of layers. You can attack it from top (SaaS) or bottom (IaaS). Quite frankly, there isn't *that much* architecture in SaaS (other than the secure connection and integration). That leaves IaaS as the starting point - which takes me back to point #1 - IaaS intimidates the EA team - - meaning that they're relying on the I.T. data center operations team (and localized infrastructure architects) to define the foundational IaaS layers which will serve PaaS, Dev/Test, disaster recovery, hadoop clusters, etc.

Any truth here? Leave a comment (moderated) or send me an email either way: jschneider AT MomentumSI DOT com

Cloud.com offers Amazon API

2011-04-04T05:37:00.002-05:00

The most recent version of Cloud.com is now offering a 'bridge' for the core AWS EC2 services:

"CloudBridge provides a compatibility layer for CloudStack cloud computing software that tools designed for Amazon Web Services with CloudStack.

The CloudBridge is a server process that runs as an adjunct to the CloudStack. The CloudBridge provides an Amazon EC2 compatible API via both SOAP and REST web services."

The functions they support include:

Addresses

AllocateAddress

AssociateAddress

DescribeAddresses

DisassociateAddress

ReleaseAddress

Availability Zones

DescribeAvailabilityZones

Images

CreateImage

DeregisterImage

DescribeImages

RegisterImage

Image Attributes

DescribeImageAttribute

ModifyImageAttribute

ResetImageAttribute

Instances

DescribeInstances

RunInstances

RebootInstances

StartInstances

StopInstances

TerminateInstances

Instance Attributes

DescribeInstanceAttribute

Keypairs

CreateKeyPair

DeleteKeyPair

DescribeKeyPairs

ImportKeyPair

Passwords

GetPasswordData

Security Groups

AuthorizeSecurityGroupIngress

CreateSecurityGroup

DeleteSecurityGroup

DescribeSecurityGroups

RevokeSecurityGroupIngress

Snapshots

CreateSnapshot

DeleteSnapshot

DescribeSnapshots

Volumes

AttachVolume

CreateVolume

DeleteVolume

DescribeVolumes

DetachVolume

Although this list represents the core features of EC2, it doesn't yet cover the upper layers (CloudWatch, Auto Scale, etc.) or the PaaS offering (SNS, SQS, etc.) Regardless, I'm excited to see more emphasis being placed on supporting the AWS standard. It's easy for people to say that IaaS standards don't matter. However, if you're the guy building software on top of IaaS, they matter a WHOLE lot.

Cloud.com is a solid piece of software that has achieved success in the service provider market. To date, they haven't pushed too hard in the enterprise. Their decision to embrace the AWS API is a good one - and is complemented with their decision to use the pieces of OpenStack in their software where appropriate. This idea seems to be getting more traction. I'm hearing more and more people talking about OpenStack like it's a drawer that you reach into and grab out the components that you want - - rather than a holistic platform. I'm not sure if that's what the OpenStack team was shooting for but it's interesting to see guys like Cloud.com being open to leveraging the bits and pieces that they find useful.

The commoditization of scalability

2011-04-02T07:48:00.002-05:00

Last week, I had an interesting discussion with a product owner at an ISV. We discussed his offering; it was core plumbing-middleware-kind-of-stuff. When I asked about how he differentiated his offering from others on the market the answer was that they scale better. Our discussion moved from what he was doing to what I was up to and without trying to be coy I said, "We enable the commoditization of scalability". What I mean by this is that we help our customers adopt public and private clouds that know how to auto scale applications (and much more).

Of course, ISV's have always used non-functional attributes like availability, scalability and security as competitive differentiators in their offering. These capabilities are now being provided as features in the IaaS fabric. The next generation products coming from ISV's will need to redesign their solution on top of cloud infrastructures like Amazon, Eucalyptus, vCloud Director, Cloud.com, OpenStack and Nimbula. It will no longer be acceptable for an ISV to march into a customer and demand a block of servers to run their proprietary clusters. They will be expected to be able to allocate computer resources from the IaaS common pool. In addition, the ISV's will need to differentiate on attributes other than those provided by the IaaS fabric.

This change will affect the corporate I.T software development department as well. I've witnessed several I.T. groups design highly scalable architectures. Usually, the I.T. personnel aren't educated to perform this kind of work and either the project fails or delivery costs are very high. I believe that the I.T. departments that invest in IaaS will be able to significantly reduce the cost to design, deploy and operate highly scalable systems. It might be premature to declare the commoditization of scalability, but I truly believe we are witnessing the most significant step towards that goal in my 20 year career.

Providing Cloud Service Tiers

2011-03-16T03:33:00.007-05:00

In the early days of cloud computing emphasis was placed on 'one size fits all'. However, as our delivery capabilities have increased, we're now able to deliver more product variations where some products provide the same function (e.g., storage) but deliver better performance, availability, recovery, etc. and are priced higher. I.T. must assume that some applications are business critical while others are not. Forcing users to pay for the same class of service across the spectrum is not a viable option. We've spent a good deal of time analyzing various cloud configurations, and can now deliver tiered classes of services in our private clouds.

Reviewing trials, tribulations and successes in implementing cloud solutions, one can separate tiers of cloud services into two categories: 1) higher throughput elastic networking; or 2) higher throughput storage. We leave the third (more CPU) out of this discussion because it generally boils down to 'more machines,' whereas storage and networking span all machines.

Higher network throughput raises complex issues regarding how one structures networks – VLAN or L2 isolation, shared segments and others. Those complexities, and related costs, increase dramatically when adding multiple speed NICS and switches, for instance 10GBase-T, NIC teaming and other such facilities. We will delve into all of that in a future post.

Tiered Storage on Private Cloud

Where tiered storage classes are at issue, cost and complexity is not such a dramatic barrier, unless we include a mix of network and storage (i.e., iSCSI storage tiers). For the sake of simplicity in discussion, let's ignore that and break the areas of tiered interest into: 1) elastic block storage (“EBS”); 2) cached virtual machine images; 3) running virtual machine (“VM”) images. In the MomentumSI private cloud, we've implemented multiple tiers of storage services by adding solid state drives (SSD) drives to each of these areas, but doing so requires matching the nature of the storage usage with the location of the physical drives.

Consider implementing EBS via higher speed SSD drives. Because EBS volumes avail themselves over network channels to remain attachable to various VMs, unless very high speed networks carry the drive signaling and data, a lower speed network would likely not allow dramatic speed improvements normally associated with SSD. Whether one uses ATA over Ethernet (AoE), iSCSI, NFS, or other models to project storage across the VM fabric, even standard SATA II drives, under load could overload a one-gigabit Ethernet segment. However, by exposing EBS volumes on their own 10Gbe network segments, EBS traffic stands a much better chance of not overloading the network. For instance, at MSI we create a second tier of EBS service by mounting SSD on the mount points under which volumes will exist – e.g., /var/lib/eucalyptus/volumes, by default, on a Eucalyptus storage controller. Doing so gives users of EBS volumes the option of paying more for 'faster drives.'

While EBS gives users of cloud storage a higher tier of user storage, the cloud operations also represent a point of optimization, thus tiered service. The goal is to optimize the creation of images, and to spin them up faster. Two particular operations extract significant disk activity in cloud implementation. First, caching VM images on hypervisor mount points. Consider Eucalyptus, which stores copies of kernels, ramdisks (initrd), and Eucalyptus Machine Images (“EMI”) files on a (usually) local drive at the Node Controllers (“NC”). One could also store EMIs on an iSCSI, AoE or NFS, but the same discussion as that regarding EBS applies (apply fast networking with fast drives). The key to the EMI cache is not so much about fast storage (writes), rather rapid reads. For each running instance of an EMI (i.e., a VM), the NC creates a copy of the cached EMI, and uses that copy for spinning up the VM. Therefore, what we desire is very fast reads from the EMI cache, with very fast writes to the running EMI store. Clearly that does not happen if the same drive spindle and head carry both operations.

In our labs, we use two drives to support the higher speed cloud tier operations: one for the cache and one for the running VM store. However, to get a Eucalyptus NC, for instance, to use those drives in the most optimal fashion, we must direct the reads and writes to different disks,– one drive (disk1) dedicated to cache, and one drive (disk2) dedicated to writing/running VM images. Continuing with Eucalyptus as the example setup (though other cloud controllers show similar traits), the NC will, by default, store the EMI cache and VM images on the same drive -- precisely what we don't want for higher tiers of services.

By default, Eucalyptus NCs store running VMs on the mount point /usr/local/eucalyptus/???, where ??? represents a cloud user name. The NC also stores cached EMI files on /usr/local/eucalyptus/eucalyptus/cache -– clearly within the same directory tree. Therefore, unless one mounts another drive (partition, AoE or iSCSI drive, etc.) on /usr/local/eucalyptus/eucalyptus/cache, the NC will create all running images by copying from the EMI cache to the run-space area (/usr/local/eucalyptus/???) on the same drive. That causes significant delays in creating and spinning up VMs. The simple solution: mount one SSD drive on /usr/local/eucalyptus, and then mount a second SSD drive on /usr/local/eucalyptus/eucalyptus/cache. A cluster of Eucalyptus NCs could share the entire SSD 'cache' drive by exposing it as an NFS mount that all NCs mount at /usr/local/eucalyptus/eucalyptus/cache. Consider that the cloud may write an EMI to the cache, due to a request to start a new VM on one node controller, yet another NC might attempt to read that EMI before the cached write completes, due to a second request to spin up that EMI (not an uncommon scenario). There exist a number of ways to solve that problem.

The gist here: by placing SSD drives at strategic points in a cloud, we can create two forms of higher tiered storage services: 1) higher speed EBS volumes; and 2) faster spin-up time. Both create valid billing points, and both can exist together, or separately in different hypervisor clusters. This capability is now available via our Eucalyptus Consulting Services and will soon be available for vCloud Director.

Next up – VLAN, L2, and others for tiered network services.

Auto Scaling as a Service

2011-03-14T03:47:00.003-05:00

The Tough Auto Scaling Service is our offering to enable the automated scaling of an application tier at runtime. System data collected by a monitoring service provides the intelligence to provision or deprovision resources according to SLA's. Out of the box, our service uses our own Tough Monitoring Service, however, since we're using the de facto standard (Amazon Web Services), you can plug in any implementation that is AWS compatible.

The auto scaling service works by defining an 'auto scaling group'. This identifies the kind of service which will shrink or expand based on system load. The most common use case for auto scaling is for the Web tier where additional Web servers are added on the fly to respond to heavy loads. Auto scaling can also be used on stateful tiers but extra attention must be spent on managing the state replication mechanisms (clustering, etc.)

As new servers are provisioned to respond to the load request, they can be added to a dynamically programmable load balancer. This enables in-bound application traffic to be evenly divided across the array of virtual servers identified in an auto scaling group. Conversely, when the load returns to normal levels, the virtual servers are taken out of the load balanced pool allowing a graceful shutdown. To enable this scenario, we're using our Tough Load Balancing Service, but once again, customers can use any AWS compatible load balancer to perform this operation.

One of the key concepts of cloud computing is the concept of 'elasticity'; another is 'automation'. The Auto Scaling Service brings these two concepts together and applies them toward the compute side of the world to provide three key benefits:

1. Increased success rates on Service Level Agreements - The system auto scales to meet SLA's

2. Higher utilization rates - Unused virtual servers are released back to the pool

3. Reduced operating costs - Predefined policies automate activities that previously would have been human intensive tasks.

Combined, these three benefits make auto scaling a critical component of any private / hybrid cloud environment. It's also worth pointing out that the auto scaling service is a fundamental building block to enable other scalable services such as Platform Services (PaaS).

Non-Invasive Cloud Monitoring as a Service

2011-03-09T03:36:00.004-05:00

The Tough Cloud Monitoring solution is our next generation offering targeting virtualized workloads, as well as PaaS services, housed in either traditional data centers or private cloud environments.

By monitoring, we mean 'health and performance' monitoring of infrastructure and platforms. Our service provides the traditional statistical information: CPU utilization, disk I/O, network traffic, etc. This begs the question, "why does the world need yet-another monitoring solution?" Quite frankly, we were surprised that there weren't better options available on the market. So, once again, we started from scratch with a new design center:

1. Make it massively scalable and highly available

Some of our customers currently have 1,000's of virtualized work loads operating and it is clear that the next generation service providers will have 10's of thousands running. Our design needed to easily scale itself from both a data collection perspective and burst storage. We bit the bullet and designed a solution from scratch to use Apache Cassandra at the core. This enabled us to leverage it's built-in cross-data center peer replication schemes and dynamic partitioning. In addition, Cassandra was a good fit for us because it was designed to accept very fast (stream oriented) writes of data.

2. The monitors should be non-invasive and agent free

Being non-invasive is always a good goal; it makes it easier to collect data on targets without having to install additional software on the machine (which can be a real problem when you already have lots of machines running in production). Knock-on-wood, but so far, we've been able to deliver all of our monitors completely out-of-band. No need to install Ganglia, collectD, etc. on hundreds/thousands of boxes...

3. The monitors should support a standard, service oriented API

In building our early private clouds, we were surprised to see that most of the system monitoring tools were "closed" systems. They collected the data but didn't make it easily available to other systems; they were designed to deliver the information to humans in HTML. This was a non-starter for us since the new world is about achieving higher levels of system automation (not human tasking). Naturally, we went with the de facto standard Amazon Web Services and the CloudWatch API. Our solution delivers full compatibility with CloudWatch from a WSDL, AWS Query and command line perspective. This makes it real easy for the monitoring data to be consumed by other services like Auto Scale.

4. Use a consistent model for IaaS and PaaS

By supporting the AWS service interface model, we inherited this feature. Just as CloudWatch monitors services like their Elastic Load Balancer and Relational Data Services, we'll be providing similar support for internal PaaS platforms.

We believe that we have achieved all of our design goals. The Tough Cloud Monitor is available today for traditional data centers, private clouds or service providers.

Tough Load Balancing as a Service

2011-03-08T03:35:00.003-05:00

Last week, MomentumSI announced the availability of our Tough Load Balancing Service along with a Cloud Monitoring and Auto Scaling solution.

The concept of load balancing has been around for decades - so nothing too new there. However, applying the 'as a Service' model to load balancing remains a fairly new concept, especially in the traditional data center. Public cloud providers like Amazon have offered a similar function for the last couple of years and have seen significant interest in their offering. We believe LB-aaS offers an equivalent productivity boost to traditional data centers, private cloud customers or service providers who want to extend their current offerings.

Our design goals for the solution were fairly simple - and we believe we met each of them:

1. Don't interfere with the capability of the underlying load balancer

The LB-aaS solution wraps traditional load balancers (currently, software based only) to enable rapid provisioning, life-cycle management, configuration and high availability pairing. All of these functions run outside of the ingress / egress path of the data. This means you do not incur additional latency in the actual balancing. Also, our design enables us to snap in various load balancer implementations. Our current solution binds to HAProxy and Pound for SSL termination. Based on customer demand, we anticipate adding additional providers (e.g, F5, Zeus, etc.) Our goal is to nail the "as-a-Service" aspect of the problem and to be able to easily swap in the right load balancer implementation for our customers specific needs.

2. Make life easier for the user

I was recently as one of my enterprise customers speaking with an I.T. program manager. She commented that her team was in a holding pattern while they ordered a new load balancer for their application. Her best guess was that it was going to take about 5 weeks to get through their internal procurement cycle and then another 2-3 weeks to get it queued up for the I.T. operations people to get around to installing, configuring and testing it out. When I told her about our LB-aaS solution (2-3 minutes to provision and another whopping 5-10 minutes to configure), she just started laughing... and made a comment about necessity being the mother of all invention.

3. Deliver an open API

Delivering an open API was an easy decision for us. We went with the Amazon Web Service Elastic Load Balancer API. We maintained compatibility with their WSDL as well as providing command line capabilities and the use of their AWS Query protocol. As the ecosystem around AWS continues to grow, we want companies to be able to immediately plug into our software without code-level changes.

4. Don't cause pain down the road

We've seen some companies put software based load balancers into their VM image templates. We see this as last-years stop-gap solution. The lack of device-specific life-cycle management leads to configuration drift and no service-oriented interface means you can't use the load balancer as part of an integrated solution pattern (like auto-scale). Let's face it, the world is moving to an 'as a service' model for some good reasons.

Again, the Tough Load Balancing Service is available today and can easily work in current data centers, private clouds or service providers.

Separating IaaS into Two Layers

2011-03-02T05:34:00.004-05:00

For some time now, I've been watching cloud architects consider their strategy for deploying wide-scale Infrastructure-as-a-Service (IaaS). Many of my friends are quick to draw the standard Gartner cloud stack (SaaS, PaaS followed by IaaS). And although I think this is a simple way to look at the layers, it can be dangerous if that's where the conversation ends.

I'd like to suggest that we consider at least two distinct IaaS layers:

Some people call the first layer, "Hardware-as-a-Service". It primarily focuses on the 'virtualization' of hardware enabling better manipulation by the upper layers. This was the core proposition of the original EC2. There are some great vendors in this space like Eucalyptus, Cloud.com and VMware. Cool projects are also emerging out of OpenStack which many of the aforementioned companies hope to adopt and extend.

The second layer is the 'automation layer'. It focuses on providing convenience mechanisms around layer 1 services. This includes everything from making multi-step human tasks more easily accomplished through orchestrations, to closed-loop systems akin to the problem defined in autonomic computing. The core elements delivered in layer 2 includes self-inspection, self-healing, self-protection and resource optimization. These are some pretty powerful concepts. So powerful in fact, that it often makes sense for consuming technologies to bind to layer 2, rather than directly to layer 1.

We're starting to see this layered approach unveil itself at Amazon. Services like Elastic Beanstalk focus on integrating many of the lower layer building blocks into an easy to consume bundle, while also delivering several of the autonomic properties. It's pretty cool stuff. But, it's only cool if you actually use it. I loved that Amazon started off real low in the stack (EC2 servers) and worked their way up. It was fundamentally the right way to rethink the problem. The downside is that many engineers are now overly comfortable using the original atomic elements when they need to be looking harder at the new convenience layers (e.g., CloudFormation, Elastic Beanstalk, etc.)

The announcement we made yesterday regarding custom implementations of Amazon CloudWatch, Elastic Load Balancer and Auto Scaling for private cloud demonstrate our commitment to this approach. We're also big believers in industry standards. In my younger (and more naive) days, I would have preached about 'open standards' over 'industry standards' but I've sat in on too many industry conference calls listening to vendors with agendas bicker over standards only to wait years to get a solution which was designed by a committee. When it comes to cloud standards, I'll gladly let those younger (or more patient) than I fight those fights. Until then, we're backing the de-facto standard, AWS. And to those who say "standardized API isn't important", I'll have to kindly disagree ;-)

Amazon CloudFormation Exceeds Expectations

2011-02-25T04:51:00.003-05:00

Today, Amazon released their latest offering, CloudFormation. Simply put, CloudFormation is the service we've all been waiting for. The entire topology of an application can be described including the images, storage, security, load-balancing, auto-scaling, databases, messaging and more. It's the glue that holds it all together.

CloudFormation provides some UI screens to allow developers & release engineers to easily describe the makeup of their applications. Under the covers, the description is turned into a structured template. This template can then be sent to the AWS provisioning engine which understands the dependency chain - and launches each component in the precise order.

In standard AWS fashion, the template descriptions are available for developers to review or to create from scratch. Once a template is created, CloudFormation provides an API for developers to call which will take the template as input and execute it. It's great to see Amazon continue down the path of not only providing UI's but also making the functions available as services.

At MomentumSI, we've been anticipating the launch of this service. It pulls together all of the piece-parts which Amazon has been developing over the years. Finally, the picture can be painted on how Amazon can be used for complete application solutions. I tip my hat.

Defending the Private Cloud

2010-11-23T07:04:00.007-05:00

Phil - You know I love hyperbole as much as the next guy, but come on... discrediting the private cloud?? (and to those who aren't aware - I've corresponded with Phil for just over 7 years and have a sincere respect for him... but that doesn't mean I won't rip into his posts ;-)

Phil writes:

"Back in January, I made a controversial prediction that private clouds will be discredited by year end. Now, in the eleventh month of the year, the cavalry has arrived to support my prediction, in the form of a white paper published by a most unlikely ally, Microsoft."

A whitepaper from Microsoft is the cavalry? Wow - it must have been written by Bill Gates himself! Or... a couple noobs with MBA's and banking backgrounds... But to be fair, the paper rocks. It's dead on. It says that the cloud model is a good one - and that *eventually* more and more applications will be a good fit for the large scale public clouds.

The cool stuff described in the paper is evolving; it will take time (like a decade). That said... let's take a look at the realities that my clients live with on a daily basis:

1. ALL (not some) of my large clients have a mixed computing environment including some combination of AIX, Solaris and Z. NONE (not some) of the public cloud providers have options for supporting all of these environments. I know, you're thinking to yourself... well, they should just port the applications to Linux/Wintel and all would be good. However, in the vast majority of the cases, the applications are packaged software and my clients have little influence over the vendors who own them. So, to be clear - a significant portion of the applications are not targets for the current large scale cloud providers (like Amazon, Microsoft, etc.)

2. Most applications are data intensive and coupled together. This presents a problem when you want to move applications from your internal data center to a public cloud. I compare it to pulling out a paper-clip from your desk drawer only to find it bound to a bunch of other paper-clips. Enterprise applications are often glued together, with either low latency requirements between them or requiring large amounts of data to be moved between them (not good if you have remote data centers with thin pipes and ingress/egress fees.) **Phil, it's about Loose Coupling ;-)

3. Hardware and software provisioning times in the enterprise are embarrassing. The amount of time/money that is wasted waiting for new environments to be procured, stood up, tested, secured, etc. would astound you. The pain is real TODAY - and waiting a decade for a public cloud to be able to support the half-dozen hardware platforms, operating systems, COTS licenses, etc. you need to perform integration testing on isn't an option.

4. Mankind didn't suddenly change in 2010. It turns out that wholesale moves from one computing model to another is not in the corporate DNA. Enterprises who excel at mitigating risks are taking incremental steps to the cloud. First, they're interested in finding out simple things like "how will my business critical application perform if we virtualize it?" or... "If we moved our data intensive application off of our vertically scaled mainframe onto a horizontally scale commodity compute (share nothing) architecture - - will it still perform?" You see... enterprise I.T. has lots of unknowns around cloud architectures. It will take some time for them to understand the basics. Once they answer the architectural questions, figuring out who hosts it is rather simple problem (price, service & reliability).

Private cloud is a natural stepping stone. Most I.T. professionals that I have met do not understand the architectures, processes and operating models (regardless of public or private). Pushing naive people to a public cloud where their mistakes will be hidden by a magically elastic service interface is *not a good idea*. Trust me... it shows up when they get the bill. Instead, I wholeheartedly recommend a stepwise approach to learning about horizontal scaling, sharding, MapReduce, BigData, multi-tenant services, etc. in an environment where they can observe actions and outcomes.

MomentumSI Partners on Private / Hybrid Cloud

2010-08-25T10:01:00.002-05:00

Why self-service private cloud?

Improved agility — Deployment cycles shrink from months to minutes, making IT far more responsive to business lines and other internal customers.
Reduced capital expense — Utilization of hardware capacity improves dramatically due to elastic provisioning and de-provisioning of services.
Reduced operating costs — Software infrastructure and provisioning processes are standardized and automated. Control is decentralized to decentralized constituents.
Reduced risk — The controlled cloud provides an alternative to rogue deployments to the public cloud. The ability to move workloads between deployment environments (physical, virtual or cloud) avoids platform lock-in.

The core technologies and services within the MomentumSI self-service private/hybrid cloud platform include:

newScale service catalog — www.newScale.com
rPath release automation — www.rPath.com
Eucalyptus cloud automation — www.Eucalyptus.com

For full details, visit: http://www.momentumsi.com/solutions/Cloud1.html

Current challenges for Application Performance Engineering

2010-06-08T07:43:00.001-05:00

Application performance engineering is a discipline encompassing expertise, tools, and methodologies to ensure that applications meet their non-functional performance requirements. Performance engineering has understandably become more complex with the rise in multi-tier, distributed applications architectures that include SOA, BPM, SaaS, PaaS, cloud and others. Although performance engineering ideally should be applied across the lifecycle, we’re seeing more factors that unfortunately push it into the production phase, typically to resolve problems that have already gotten out of hand. That clearly a tougher challenge, so how did we get to this point?

In the client-server past, performance optimization was something that folks in the IT department typically figured out through trial and error. Developers learned to write more efficient database queries, database administrators learned to index and cache, and system administrators monitored CPU and memory to upgrade when needed.

As application architectures started to get more complex, the dependencies increased and it was harder for one team track down problems without chasing their tail. More organizations adopted something that was previously only used by enterprises with highly scalable, reliable mission critical applications – the performance testing lab. Vendors like Mercury created popular load testing tools like LoadRunner, and organizations invested millions in lab hardware and software in an attempt to recreate production environments that they could control for testing purposes.

Unfortunately, these performance labs became very difficult to cost justify. First, it always seemed to take too much time and money to setup the realistic test environments you’d like, particularly as apps became more distributed. Next, projects were often already behind schedule when it came time to test, and so lab times often had to be cut short. Factors like these minimized the lab’s value, but the real killer was the high maintenance costs for all that hardware and software, along with the data center and staff.

This put many IT organizations in a tough spot. With limited means to perform system-wide performance testing, and the inclusion of more SaaS/PaaS/cloud services in their architecture, they had to make due with whatever subsystem level performance testing they could get. After that, its finger-crossing and resigning yourself to further optimization in production.

Unfortunately, production can be a very frustrating place to try and optimize performance, particularly when you have performance problems and growing complaints from customers, partners, etc. It’s in these pressured environments where you need true performance engineers that follow a methodical and systematic end-to-end approach. Performance bottlenecks can reside in a myriad of places in highly distributed architectures, and you need to follow a disciplined methodology to analyze dependencies, isolate problem areas, and then leverage the best of breed tools to trace, profile, optimize, etc each of the tiers and technologies in the application delivery path. This takes a lot of skill and expertise.

In short, the challenges faced by today’s application performance engineer in production settings is a far cry from the client-server days of in-house tuning and experimentation. We expect that the role of Performance Engineer will grow in importance as SOA, BPM, cloud, and SaaS/PaaS implementations increase, and until more viable pre-production system performance testing options are available to rise up to the challenge.

SOA Manifesto

2009-10-23T23:07:00.002-05:00

Here's a few quick links:

The MomentumSI SOA Manifesto from 2007:

http://schneider.blogspot.com/2007/10/enterprise-soa-manifesto.html

Our discussion forum on the 2009 SOA Manifesto:

http://www.SOAManifesto.com

The 2009 SOA Manifesto:

http://www.SOA-Manifesto.com

The Case for Expropriated Reuse

2009-06-21T16:24:00.001-05:00

Expropriated Reuse is a form of reuse that focuses on the here and now. The goal isn’t to define some new service and hope for ‘accidental reuse’ or even to put forward a case for ‘planned reuse’. Instead, it’s the act of going out and finding redundant code that already exists across multiple systems and turning it into a single shared service. I’ll repeat that: Find redundant code and refactor out the common elements into shared services. This ALWAYS results in multiple consumers (or mandated consumers, if you prefer).

I’ll give a quick example. Over time, company X had ‘accidentally’ created 5 software modules that existed inside of larger applications that all did some form of ‘quoting a price to customers’. This led to 5 teams maintaining it, 5 sets of computer hardware, etc. Expropriated Reuse is the act of going to the applications and cutting out the common elements and turning them into one shared service. Note that this is different than application rationalization or application consolidation that tends to use a nuclear bomb to deal with the problem. We’re recommending sniper firing with a bit of additional precision bombing.

The reasons that we do this are mostly financial. We want to reduce the amount of code that has to be maintained and operated. We want to reduce our costs. There are plenty of other reasons like increased quality, time-to-market, etc. but I’m done with those softies. The case is cold hard cash. Show me how to save money or go away.

IMHO, the new SOA agenda is about expropriated reuse. The SOA Program must actively identify opportunities to make the enterprise software portfolio more efficient and less costly. Just like in city planning exercises we must acknowledge the needs of the community over the needs of the few. And I’m in agreement with Clinton that ‘we should never waste a good crisis’. Reduced I.T. budgets have created a ‘crisis of efficiency’ in virtually all of our clients. The imperative is to find ways to reduce budgets in the short term and over the next 3-5 years.

To quote Todd Biske, “… a common analogy for enterprise architecture these days is that of city planning. … (does) your architecture have more parallels to the hot political potato of eminent domain? Are you having to bulldoze applications that were only built a few years ago whose residents are currently happy for the greater good? What about old town? Any plans for renovation? If you aren’t asking these questions, then are you really embracing enterprise SOA?”

I’ve been wondering: Should all SOA programs that do not have the authority to issue an order of ‘eminent domain’ on the software portfolio be shut down? Does your SOA program have a hunting license to go find inefficiencies / duplicate coding and to issue an order of eminent domain on that code? Can you imagine what our country would look like if we couldn’t issue an order of eminent domain to capture land for our highways, bridges or railroads? Can you imagine if we didn’t have the ability to implement ‘easement by necessity’? Consider your town/neighborhood and think about the following: Railroad easements, storm drain or storm water easements, sanitary sewer easements, electrical power line easements, telephone line easements, fuel gas pipe easements, etc. What a mess it would be. The crisis of budgets must draw out the leaders. If you haven’t already been issued a hunting license, it’s time to go get one.

So I’ll answer my own question. If you’re SOA program is responsible for watching the blinking lights on your newly acquired SOA Management tool, or making sure that people enter their ‘accidental’ services into your flashy registry, I’ll recommend that they shut you down. This is a waste of time. The SOA group must be given the imperative and authority (a hunting license) to find waste in the enterprise and to destroy it. SOA isn’t about policing people on WS-Standards or similar crap – it’s about saving your company millions.

The Case for Planned Reuse

2009-06-21T16:22:00.001-05:00

In my last post, I argued that the concept of ‘accidental services’ or ‘build it and they will come’ is a bad idea – because … they typically don’t come. Services that are created with a very specific consumer in mind are typically limited in capability, scope and result in limited reuse.

The MomentumSI Harmony method suggests that service analysis be performed on the first consumer’s needs as well as potential consumers that aren’t in the immediate scope. This is easier said than done. How do you identify the requirements of a service if you have ‘phantom consumers’?? The short answer is that there are techniques that involve looking at UI models, process models, data models and other artifacts that will give you insight into the domain. The result is a list of potential consumers and a plan for their eventual consumption. The point is that there are techniques to help organizations define services according to a plan – and doing so leads to increased reuse and a better software portfolio.

Again, Planned Reuse is most effective when you’re working in a new domain and you don’t already have a bunch of conflicting/overlapping software that exists. The immediate project might call for an ‘Order Service’, but you know that the service will eventually be called by the Web eCommerce system, the call center software, the B2B gateway, etc. Those projects aren’t in scope – but you consider their needs when designing the service.

This is all fine, but what happens when you’re analyzing a service for an immediate project that clearly should be called by existing projects/software? This is the case for Expropriated Reuse.

The Case Against Accidental Reuse

2009-06-21T08:20:00.004-05:00

"Accidental Reuse" is a term that I've been throwing around a lot lately. In layman's terms, it means, "If you build it, they will come." This notion has been disproved in virtually every field (except in the Field of Dreams) - and I suggest it is even more unlikely in the field of software engineering.

It has been my observation that most software engineering teams suffer from a bad case of 'not invented here' syndrome. We work in a discipline that is not regulated and the certifications are a joke at best. Most programmers don't have engineer training - and most architects have learned from doing and observing. In short, there are good reasons for one team to not fully trust the output of another team. We also have the issues of human entitlement. It's been my observation that a generation of software developers has been created who feel that they are entitled to 'not be bored' and 'deserve cool problems'. Once again, we have people who feel the need to create new stuff - not reuse.

I could go on - but I'm guessing there is no need. Anyone who has been in the industry has seen the problems. This leads back to my original point - accidental reuse is a bad idea. And here, I'm as guilty as the guy sitting next to me on the last 20 panels where I've been on telling the world to 'build it and they will come'... design for reuse... create services... register the services... and they will come. Apologies.

Most SOA report cards that I get still show metrics on: how many services there are, how many consumers there are or how many times the service has been called. These are fine metrics but in my humble opinion leave behind one very important metric: how many unknown consumers use the service? (accidental reuse, planned reuse & expropriated reuse)

Now it gets interesting. My aggregate data shows that most services are built with one consumer in mind and almost 80% of all transactions go through the original consumer. Less than 2% go through 'accidental channels'. Of the 2%, most of those were for externally facing systems (B2B) where advance knowledge of consumption is limited or they were in 'technical' or 'entity' services where they offered limited functionality and in some cases, limited return.

The remaining 18% go toward 'planned reuse' or 'expropriated reuse'. This number is too low - but this is the opportunity. More on this later...

Coming next:

- The Case for Planned Reuse

- The Case for Expropriated Reuse (right of eminent domain)

New Blog on Cloud Computing

2008-12-20T09:45:00.002-05:00

SOE isn't dead but my primary go-forward blog will be:
http://cloudv.blogspot.com

Nomination for Federal CTO

2008-11-07T18:45:00.002-05:00

From the Barack Obama campaign site:

"Obama will appoint the nation's first Chief Technology Officer (CTO) to ensure that our government and all its agencies have the right infrastructure, policies and services for the 21st century. The CTO will ensure the safety of our networks and will lead an interagency effort, working with chief technology and chief information officers of each of the federal agencies, to ensure that they use best-in-class technologies and share best practices."

I nominate Tim O'Reilly as the first Federal CTO. Do I hear a second??

Is Amazon Competing with RightScale?

2008-10-24T07:39:00.001-05:00

It looks like RightScale is going to be a 'multi-cloud management suite', see;
http://www.allthingsdistributed.com/2008/10/using_the_cloud_to_build_highl.html#comment-2061

16 Corrections on Cloud Computing

2008-10-23T16:43:00.002-05:00

In March of 2008, RedMonk analyst, James Governor, submitted his list of "15 Ways to Tell if it's not Cloud Computing".

The consultants at MomentumSI have found 16 corrections:

Real World SOA

2008-10-20T20:07:00.003-05:00

It was my pleasure to be a guest on the Real World SOA Podcast with David Linthicum. Take a peek:

http://weblog.infoworld.com/realworldsoa/archives/2008/10/my_conversation.html

Topics:
- How do we help organizations with SOA?
- What is the purpose of a SOA methodology?
- What is the next big thing???

Talking to the Business about SOA

2008-09-03T15:02:00.003-05:00

Recently, there's been more chatter about how (or if) you should talk to the business about SOA. Yesterday, I sat in on the SOA Consortium conference call where this was the main theme. Interestingly, the moderator posed the questions and a couple participants were quick to respond... "we don't talk to the business about SOA..." The moderator took it in stride and started down the path of business and I.T. alignment - and once again the participants pushed back. Not to be dissuaded the moderator went down the BPM path. Once again, the participants pushed back. The group commented that, "talking SOA is too abstract for the business" and there was a "need to talk about business specific functionality vs just high level Agility and Change".

After attending this call, I stumbled on to Joe 2.0's blog post on the exact same topic! Even funnier was that he was quoting Jean-Jacques Dubray (JJ), who I had a 3 hour phone call with on the subject just days earlier. JJ had commented, "My experience is that the key people that you have to focus all your energy on are the developers, architects, business analysts, QAs and operations."
Joe 2.0 goes on:

Dubray says SOA is a “pure IT problem.” But in this era of the online collaborative organization, when we rely on technology for every aspect of our business, are there really any “pure IT” problems?

I don't want to split hairs... but IMHO, the answer is, "yes, some problems are just I.T. problems". Sure, I.T. problems, like HR or garbage collection, may bubble their way up to become a business problem, but at the end of the day I.T. has to figure out how to do their job and go do it. When the janitor picks up the trash in my office they do it in the most efficient way they know how. They don't ask 'the business' if they should do it efficiently - they just do it. When did I.T. become such wussies?

I'm a big believer in talking to business about whatever they want to talk about... Inventory Visibility? Love it. Customer Loyalty? Love it. New Product Introduction? Love it. That said, I believe it's I.T.'s responsibility to bring technology solutions forward. Most business people understand things like forms, window, graphs, reports, etc. They understand visual deliverables (not invisible deliverables like WSDL's). I think that is why we're seeing the most successful SOA shared service centers adopting capabilities around Rich Composite Applications, Mashups and other edge-of-the-enterprise development capabilities. They engage with the business about business problems and then use mashups and other techniques to quickly demo/prototype/build solutions that their users can relate to.

If you're looking for inspiration on this process, I'm happy to recommend a book on the subject, Mashup Corporations.
The authors do a great job of walking the readers through a fictional company. As business problems are encountered, they introduce Web 2.0 and mashup solutions. Prototypes are put together and the concepts are tested out. SOA is discussed as the 'efficient way' to make it happen. Again, they didn't talk to the business about SOA (or even services)!

PaaS Enables New ROI

2008-09-01T18:44:00.005-05:00

If you haven't already checked out Amy Shuen's book, "Web 2.0: A Strategy Guide", you should grab a copy; it's worth the read. Amy discusses the trends around Web 2.0 in the clearest, most concise manner I could have hoped for. Enough bragging about her book - one of her diagrams inspired me to think about the effects that PaaS has on the Enterprise I.T. development model.

Amy pointed out that a version of the Long Tail lives in the I.T. application development world. Certain business problems (ie, order management) have a very real and significant value proposition; these systems are often purchased from ISV's. The next set of applications often have slightly less of an ROI and are often built by the I.T. custom development group. In many cases these are departmental applications or add-on's to the procured systems. Recently, new SaaS solutions are finding their way into the enterprise because they fulfill point-requirements and have a low-cost of entry.

In the past, this left lots of business problems in the hands of shadow I.T., or power-users. But all too often new systems concepts were taken to the I.T. review board and turned down because they didn't project an adequate ROI. The return on some of these systems may have been rewarding, but the initial investment (hardware, infrastructure licenses, long development cycles, etc.) drove down the overall ROI to the point where the idea was rejected. These systems are prime candidates for PaaS, where the initial investment is significantly decreased by the pre-hosted, pay-by-the-drink model. Once again, hosted platforms will likely be the key enabler of long tail opportunities.

The long tail model of reviewing new system requests is an interesting method for I.T. governance and planning committees to consider. It is my belief that if enterprise organizations fail to meet the needs of the long tail, they will be met by other 3rd party providers who will be all to willing to help!

Services, Mashups & Cloud: Puma

2008-08-31T16:09:00.004-05:00

Imagine for a moment that Olympics took place in China – and all the world came out to watch. You’re with a shoe company called Puma, who for many years hid in the shadow of branding giant Nike. But this Olympics you made some interesting bets, including a big bet on a Jamaican sprinter named Usain Bolt. This distinguished athlete proceeds to win the gold and to destroy the world record – and to your delight holds up a pair of your shoes. What a dream.

For a brief period you own the consumer. Search engines are bombarded with “Usain Bolt” and even “Puma runner”. Your online store is blasted with orders but now every outlet that carries ‘hot tickets’ wants to resell the Usain Bolt Limited Edition shoes. Orders are coming in from channels you’ve never even heard of. New channels, new orders, new customers are abound and new demands are being placed on your I.T. environment.

The world of API’s (or services), mashups and cloud computing are all upon us. As many businesses continue cut costs and hunker down for a ‘wanna-be’ recession, others are innovating and driving new sustainable revenue sources.

Services, Mashups and Cloud drive business opportunities for those who seek a competitive advantage.

Services, Mashups & Cloud: Photosynth

2008-08-31T15:54:00.002-05:00

Imagine for a moment that a Microsoft research group finished a beta version of a project that they had dubbed ‘Photosynth’. Their pet project was to allow users to create panoramic virtual environments by using regular 2-D digital cameras. Users would merely take lots of adjacent pictures where the edges of one photo would overlap with the next. The photos would then be submitted online to a set of computers that would use artificial intelligence to transform the individual digital pictures into a panoramic viewing environment that could be witnessed by anyone who had the Internet and a browser.

For this to be interesting a few things would be needed. First, we would need lots of people who could contribute pictures (user generated content). Second, we’d need a whole bunch of computers to act as one to crunch on digital image matching algorithms. Third, we’d need the ability to host the image online and allow the proud publishers to pass around copies of their new site's URL to all of their friends, encouraging them to become publishers as well. Imagine now, that the this happened and that the traffic to the site increased by 140,000% in just 7 days. Imagine that.

The world of API’s (or services), mashups and cloud computing are all upon us. As many businesses continue cut costs and hunker down for a ‘wanna-be’ recession, others are innovating and driving new sustainable revenue sources.

Services, Mashups and Cloud drive business opportunities for those who seek a competitive advantage.

Services, Mashups & Cloud: Shelfari

2008-08-31T15:12:00.002-05:00

Imagine for a moment that an online ‘virtual book store’, called Shelfari, created a widget that displayed pictures and descriptions of the widget owners favorite books. These widgets could then be embedded inside of blogs and social networks so that people could see what their friends or influencers were reading. We’d have to assume that the widget was viral in nature, allowing consumers to copy the widget and republish it with their own content at their own site. And like all things viral, the network of users grew at an exponential rate. The widget became so successful that Amazon.com decided it was time to acquire the company. The press related to the activity drove additional traffic to the Shelfari site to the point where response times were so long that new users couldn’t even sign up.

What if – what if widgets drove traffic and viral widgets drove huge traffic? What if personalization drove conversion rates and Web API’s enabled financial transactions? What if we could mashup content and community to drive commerce? What if the hardware automatically scaled to meet the needs of the user community?

The world of API’s (or services), mashups and cloud computing are all upon us. As many businesses continue cut costs and hunker down for a ‘wanna-be’ recession, others are innovating and driving new sustainable revenue sources.

Services, Mashups and Cloud drive business opportunities for those who seek a competitive advantage.

Joe McKendrick 2.0

2008-08-31T12:13:00.003-05:00

Joe McKendrick is one of my favorite bloggers over at ZDNet. However, I've found some of his stuff to be a bit hum drum. That is, until I found his 'secret' posting site...

Joe 2.0 (like Jeff 2.0), is spending much more time thinking, writing and talking about all of the reasons that you did SOA in the first place. Want to check out some good reading?

Mashups: So Easy a Caveman Can Write Them?

and

Managing Data in the Clouds

Fundamentally, SOA has progressed from debating about ESB's, governance styles, REST vs. SOAP, etc. to a discussion on how to use the services to create new business functionality through mashups, while delivering on low cost platforms (cloud and PaaS). Joe 2.0 gets it.

Are you 2.0?

Amazon Sends Death Blow to I.T. Data Center

2008-08-22T10:51:00.002-05:00

The time of death for the corporate data center was pronounced at 11:03EST, August 21st of 2008. Forensic pathologists continue to investigate the death, however, it is known that the patient had stability issues, persistent hemorrhaging and had been considered terminal.

Thousands of companies have mortally wounded data centers. Unfortunately, CIO's lacked a viable replacement for this archaic agility anchor. In essence, they've been forced to keep them on life-support. Billions of dollars have been pumped into various unsuccessful attempts to heal the patient. Billions more have been spent on 'Data Center Hospice'. Jeff Schneider, MomentumSI CEO commented, "When it becomes it was clear that the patient will die, corporations shift their thinking to 'reducing the pain'. This largely involves moving labor intensive data center tasks to low-cost offshore facilities."

Although some debate remains if the data center is actually dead. Schneider stated , "Yes, the data center has a heart beat, however the beat coincides with the I.T. family beating on the heart with their fists, insisting that the patient isn't dead."

Congratulations to Amazon on delivering the last critical element of their Elastic Cloud Computing offering.

The 7 Dirty Words of SOA

2008-08-17T11:29:00.005-05:00

I recently had a conversation with one of my good friends at IBM. He had mentioned that IBM is doing a better job of differentiating between "Business SOA" and "I.T. SOA". Interesting... what did he mean? After drilling him with questions it came down to this:

- I.T SOA is about running a more efficient Information Technology group

- Business SOA is about creating new information products and capabilities

This resonated with me. It sounded very similar to the conversations I've had with people at SAP. They have positioned SOA as a technology enabler to their large suite of business applications. They sell business solutions. And as one SAP insider put, "... and of course those solutions are Service Oriented... it's 2008!"

These conversations were music to my ears. IBM and SAP are both back to attacking business problems and are making the assumption that SOA is in place and acts as the enabler.

In Business SOA we need to think about a new set of "Business SOA Patterns":

- How do we deliver new business products through new channels?

- How do we deliver more/better information to our distributors, retailers and consumers?

- How will consolidated/shared information lead to a tighter supply chain?

It's been too many years of people talking about ESB's and other I.T. focused trivia. With that said, the 7 Dirty Words of SOA are:

Loose Coupling

Abstraction

Reuse

Autonomous

Discoverability

Composability

Interoperability

Yes, it's time to move past "I.T. SOA"...

New CEO at StrikeIron

2008-08-15T13:36:00.002-05:00

I just noticed that Richard Holcomb is new CEO at StrikeIron, see:
http://www.strikeiron.com/company/management.aspx

StrikeIron has been pioneers in the 'Data as a Service' space for some time. Congratulations!

RFP's for Service Oriented Architecture

2008-07-31T10:27:00.004-05:00

At MomentumSI, we keep a close eye on the various RFP's that are issued relating to SOA. Yesterday, one caught my attention, and I'd love to hear from the blogging community on what they think.

The issuer is the Rhode Island Administration of State Courts, RFP/LOI #: B2008011. This is an active/open RFP titled, "Service Oriented Architecture Implementation". Naturally, an initiative that MomentumSI would be interested in...

I encourage others to take a look at the RFP - I don't mean to single this particular agency out, it's just a concrete example that I believe will facilitate a conversation.

Bloggers, what I want to know is do you think this is a good SOA RFP?
http://www.purchasing.ri.gov/RIVIP/ExternalBids/Judicial/JudicialBids/B2008011.PDF

Speak up - I want to hear your thoughts!

(For the record, MomentumSI will not be bidding on this RFP)

SOA Governance Priorities

2008-07-29T08:54:00.003-05:00

I've been sending out emails and making some phone calls asking a simple question. How are companies prioritizing the various aspects of SOA Governance, using the following categories:

A. Plan Time Governance - Ensuring that business initiatives aligned with key I.T. services, avoiding silos while gaining cross organizational funding, etc.

B. Design Time Governance - Utilizing registries, repositories, validation frameworks, etc. to promote best practices in schema and contract design enabling reuse, search, verify, etc.

C. Pre-Release Governance - Validating solutions meet pre-production needs (highly available, secure, fast response times, new version doesn't break existing consumers, etc.)

D. Run Time Governance - Ensuring that the services in production are monitored, adhere to SLA's, leverage run time policies, etc.

E. Infrastructure Governance - Verify that the infrastructure used (ESB's, Registries, Mediation Devices, etc.) adhere to the corporate standards.

I've got quite a bit of feedback (thank you). For the most part, people feel that B, D and E are getting the attention, while A and C are not.

Here's where I'm seeing people spend time/money:
(B, D, C, E, A)

Here's my personal opinion on where they SHOULD spend time/money:
(A, C, B/D, E)

Here's my take:
- SOA Governance = A
- B,C & D aren't actually SOA Governance, they fall under the umbrella of "Service Lifecycle Management"
- E is a form of EA Governance (applied to the SOA domain)

Most companies have completely failed in "plan time governance". If you're going to do "transformational SOA", this is a recipe for disaster.

Companies that choose not to tackle Plan Time Governance are doing something other than SOA. I like to call it SOB. More on this later...

Generational Platforms

2008-07-27T09:37:00.002-05:00

Programming COBOL on the mainframe was the preferred platform of the generation before me. Don't get me wrong, I did my fair share. I did it, but I remember thinking that it was "my fathers platform".

I was a PC guy. When it was time to develop some new piece of software, the PC and personal OS was my starting point. I programmed in 14 different languages mostly client/server or monolithic clients running on Unix, Windows, OS/2, etc. It was obvious for me to see the benefits of this platform yet it surprised me that the last-gen often didn't get it.

In the late 1990's, I began working with people who were slightly younger than I. These people assumed that any new software built would be browser based. The Web was their platform. I remember thinking that the big thing was really Java, virtual machines, distributed computing, security sand-boxes, etc. The Web was limited in capability. Initially, I fought it - but then I got it. I laughed a bit about how I was getting old.

I'm now seeing a new paradigm emerge, once again, initiated by a younger generation. This group believes that software starts with social networks like Facebook and MySpace. It's less about the function of the software and more about the communities, relations and communications that they enable. Again, the last gen is challenged to understand the importance of this jump and many old-timers will not make the transition.

Evidence is also appearing to suggest that a second shift is occurring in parallel. That is, pervasive computing seems to have finally crossed the chasm. The iPhone has blazed a trail for a new generation applications available whenever and wherever you happen to be located. GPS enabled features brings a whole new dimension to the platform enabling scores of location based offerings.

This blog is about SOA - more specifically it is about services that enable productivity. It's often easy to get stuck in the old traps of debating stuff that doesn't really matter (REST vs. SOAP, do ESB's suck?, etc.) The exciting stuff that is going on in computing is related to the next generation platforms. The new generation doesn't care about the SOA debates - their debate is on geo-location services, cross-SaaS provisioning, etc. In essence, it isn't about SOA - it's about the next generation of services that enable pervasive computing on a social network. All of this other chatter is just last generation fools talking about stuff that only obsolete people care about.

Multiple ESB Example

2008-07-25T11:39:00.004-05:00

David Linthicum accurately assesses my feelings:

http://weblog.infoworld.com/realworldsoa/archives/2008/07/esbs_on_trial.html

"I think Jeff may be a bit grumpy from even having to respond to this. However, once again, as I mentioned in my previous post, I'm looking for case studies and details that prove why I'm wrong, and thus why this is "silly advice." Thus far, it's been out there for 24 hours, and I've gotten zero responses. "

Now, I'm even grumpier for having to respond twice.

In Dave's own blog, within hours, of his post a gentleman responded,

FedEx Delivery is driven largely from a TIBCO-based platform. Kinkos is similarly built upon webMethods. Then they merge. It is simply not rational to think that we cannot provide FedEx with a solution to service orchestration across these 2 systems. That's the business need and what we have to solve for.

So FedEx has Tibco and WebMethods. This is interesting because my friends at SAP have told me that FedEx is a huge customer, hence, they'll probably be using the NetWeaver ESB functions as well. And when I last visited FedEx, they had purchased the BEA AquaLogic stack. My guess is that an Oracle account representative either has or will be talking to them about the virtues of Fusion Middleware and their migration plan. My guess is that they probably also have some DataPower boxes sitting around - and a few of the developers have probably downloaded Mule and are testing it out as well.

I know of no company who isn't facing the exact same issue. I'm surprised that Dave didn't run into this when he was at Mercator or Saga/Software AG. Not picking on Dave here, but it's funny, people always talk about being 'business driven, not I.T. driven', but then they make recommendations to rip out huge pieces of infrastructure to make "I.T Cleaner" because in the long run it will help business. Again, in my opinion, this is silly advice.

Linthicum on ESB's

2008-07-18T14:03:00.003-05:00

Several people have sent me the latest Linthicum article on the back channel. People are asking if I agree...

http://weblog.infoworld.com/realworldsoa/archives/2008/07/are_esbs_hurtin.html

Dave comments:

First, if there is indeed "enterprise architecture" and an "enterprise architect" then the different divisions should not be using different ESBs, or even an ESB for that matter.

Companies are buying ESB's, will continue to buy ESB's, will buy products that are packaged with ESB's and will acquire/merge with other companies that have ESB's. They aren't going away - so yes, large companies will have to figure out how to deal with having more than one...

Dave moves on to his second point:

Second, considering that my first point is correct (which it is), why the heck are you attempting to integrate these integration engines when they should perhaps be displaced altogether.

In an overconfident, if not pompous manner, Dave declares himself correct. He then wants to know why a single dictator doesn't come in and end-of-life various products across business units, geographies and missions. Well, it might be a good idea for I.T. but may not be in the best interest of the business.

Personally - I see this as just plain silly advice...but perhaps, I misunderstood his point.

Silo Analysis: Key to SOA Success

2008-07-10T11:13:00.005-05:00

At virtually every conference I attend, I hear someone declare that we need to "destroy the silos", and that SOA is the enabler. Loosely, I think of a silo as a system or information repository that solves a problem for some portion of the enterprise, while leaving other areas responsible for solving the same or similar problems on their own. Silos causes duplication of systems, business rules and data. They require multiple levels of integration and cleansing, resulting in increased maintenance and operational expenses. In general, silos are bad.

After working with a handful of organizations, it occurred to me that these companies had not performed "Silo Analysis". Almost every large organization has a number of silos that exist for a variety of reasons including:
- I.T. funding occurred by business area and each area bought/built their own (silo) systems
- Mergers or acquisitions caused duplicate (silo) systems
- Poor visibility or planning across the enterprise led to unintentional silos

The rumors that SOA can help are true. But before rushing down the path, organizations need to ask some important questions:
- What are our silos?
- Is there a good reason for the silos?
- Which silos do we want to end-of-life?
- Which silos can we practically kill (politics, investment, etc. )
- Will SOA lead to "silo services"?

Silo Identification
The first step to this process is to identify what silos you currently have. As I mentioned earlier, silos often mimic organizational reporting and funding structures. Here are some easy ones to look for:
- Corporate Sectors / Divisions / Business Units / Departments
- Sales Centers (Asia, Europe, Americas, etc.)
- Facilities (warehouses, manufacturing locations, distribution centers, etc.)
- Delivery Channel (Web, kiosk, mobile, etc.)

It is common for these structures to be embedded within each other as well. For instance, Acme Corporation might have duplicate CRM systems for each SBU, but also have additional CRM systems in the Asia locations of each SBU. In essence, the silo analysis has to look at combinations of the structures.

Silo Justification
Not all silos are bad. There are often good reasons why silos exist. Unique business requirements may require unique I.T. solutions. It is important to understand WHY the silo system exists but be careful not to let the 'unique business requirements' become a universal excuse for duplicate systems.

Silo Targeting
Once you've found silo systems which could be rationalized to a single service, you have to decide if their is political will power to "de-silo" the area. In many cases, if not most, it is very difficult to retire systems. The upfront cost of retiring a system is often a burden that an organization will attempt to defer for as long as possible, typically stating there are "higher priorities". The bottom line is that there will always be higher business priorities but I.T. must make the case for retiring the old.

Often the goal isn't to retire the systems but merely to create a single point of entry. In such cases, I.T. can create front end interfaces that access multiple legacy systems which cross silos. Here, we aren't retiring systems, rather we're 'bridging the silos' by standardizing the access point allowing consumers to get a single view.

Preventing Silo Services
Service Orientation offers a great mechanism to decrease the number of silos and mitigate the damage of the silos which can not be eliminated. From a rationalization perspective, SOA offers the ability to create a single "Master Service" that acts as the single source of truth for logic AND data. However, many organizations are not realizing this vision. As many have noted, services continue to be built in an ad hoc fashion resulting in JBOWS (Just a Bunch of Web Services). In almost every case JBOWS are merely silo services that were created because no one funding authority had the charter to perform silo analysis and justify a non-silo solution.

Although the 'governance' and 'cross-silo funding' problems are barriers, I believe a more tactical barrier remains. I.T. leaders are failing to teach their organizations about the silos. What are the silos? Why do they exist? Which ones are we chartered to knock down? Which will be tactically bridge? And when we do knock them down, how will we advertise that a service isn't a silo service, but rather a Master Service that crosses the divide? And perhaps most important, how do we prevent 'silo creep' in the future. Today, most software professionals have no concept of 'attaching' their unique business logic to existing services.

The Bottom Line
SOA offers a great solution to the silo problem but it will take significant political power to pull it off. Even if you are able to remove some of the silos, you'll need a plan to prevent unwanted new ones. Silo Analysis must be ingrained in the mind of every business analyst, architect, developer and governance professional. Failure to teach our next generation will lead to more silos - they'll just be written in Ruby instead of Java...

Mindreef acquired by Progress Software

2008-06-26T17:22:00.002-05:00

Wow - how did I miss this one??

Mindreef was acquired by Progress Software!

Quietly, Progress added another set of tools to their growing SOA portfolio. Through this acquisition, Progress picks up new SOA testing at quality management solutions. Congratulations to Frank Grossman the rest of the Mindreef team.

And again, I've updated the SOA Acquisition List:
http://www.momentumsi.com/SOA_Acquisitions.html

Iona acquired by Progress

2008-06-26T15:42:00.002-05:00

Progress Software announced the acquisition of Iona.

This move continues to round out their SOA offerings by adding registry/repository, data services, message brokering and CORBA middleware. However, as others have pointed out, this leaves Progress with significant overlap in the ESB area. I'm confident that we'll be hearing from the Progress leadership on how they intend to resolve the redundancies that exist between Artix, FUSE, SOAPStation and SonicESB.

Naturally, here is the updated SOA Acquisition List.

Gartner: Force.com Case Study

2008-06-11T14:24:00.002-05:00

Highlights from the Force.com Case Study presenation inlcude:

MS client/server developers will quit when you move to SaaS. Hire web developers to replace them. Those people then implement innovative ideas rapidly on the SaaS platform.

Don't underestimate the need for change management training in this or any software rollout.

Developers productive within two weeks.

Easy to train up people in India to be productive on the platform.

Myth is that data on SaaS platforms is not available. Customer downloads all data from all SaaS providers onto their SAn nightly.

Very excited about their release of SAML. Seems to be standard most vendors are coalescing around.

Gartner: SOA Design Patterns

2008-06-11T14:19:00.001-05:00

As you know design patterns describe recurring solutions to common problems in software design. And recognizing and implementing these patterns is an essential part of successful Service Oriented Design. Below are 5 SOA design patterns that Gartner analysts have observed:

1. Multichannel Applications
2. Composite Applications
3. Business Process Orchestration
4. Service Oriented Enterprise
5. Federated SOA

Multichannel Application
SOA is a perfect fit for multichannel applications. This pattern separates the back-end business logic from the front-end logic and delivers full application functionality to a maximum number of users from various channels in a minimal amount of time while reusing the same exposed services.

Strategic Planning Assumption: By 2008, more than 66% of new midsize and large interactive applications will be designed to support multichannel access, up from less than 33% in 2007.

Composite Applications
Services used in composite applications may be new service implementations, fragments of old applications that were adapted and wrapped, or combinations of the above. Two types of integration technology are essential for effective operation of a composite SOA environment: 1) integration technology behind the service interfaces, helping users wrap and adopt various pre-SOA applications; and 2) integration technology helping users assemble and monitor transactions from services.

Strategic Planning Assumption: Through 2012, the majority of SOA-style applications will be interactive composite applications.

Business Process Orchestration
Business process management (BPM) suites are the tools devoted to implementing such SOA-based multistep processing flows. The BPEL standard is often used to document the designed metadata flow model. The metadata repository ("meta-database") is used to manage the proceedings of the business process model at runtime. Some steps of the process are implemented by calling
SOA services. Other steps require human intervention.

Strategic Planning Assumption: By 2009, more than 75% of SOA applications will implement some sequencing control outside of code of the service implementations, via external BPM technology.

Service Oriented Enterprise
The SOA-based enterprise model is a step beyond composite applications. Here, all applications are perceived as components of one integrated whole. No new application is created in isolation. Each application is built from reusable components that are available for use not only in their initially intended context, but also by other clients in other contexts. Essentially, the integrated composite enterprise consists not of applications, but rather business components — each an asset of the entire enterprise.

Strategic Planning Assumption: By 2010, more than 85% of enterprises will have combined their application integration and SOA management tools and organizations.

Federated SOA
The fundamental idea behind federated SOA is to logically split the enterprise into semi-independent SOA domains (for example, reflecting the enterprise organization in terms of subsidiaries, Business Units or departments), each with its own specific SOA infrastructure, governance processes and SOA Center of Excellence. Domains are then federated (that is, integrated — usually, but not necessarily, after the fact — to enable inter-domain sharing of services) through appropriate interoperability infrastructure, governance processes and organizational settings. "SOA federation" is the process of enabling a federated SOA by establishing the
proper technical, governance and organizational capabilities.

Strategic Planning Assumption: Few large organizations are able to establish a singular architectural blueprint for their entire IT. The best practice is to endorse domain independence and allow them to differ in technology and architecture in exchange for agreement to synchronize interoperability protocols and transports. Mergers and acquisitions are clearly candidates for Federated SOA.

Posted by Tony Cook

Gartner Update: The People Side

2008-06-11T09:05:00.000-05:00

Massive changes coming:

IT organizations are becoming change saturated. Both home and work. In next 20 years will experience the world going through peak oil production of all time.

• Medicare/Medicaid big change. Baby boomers drawing down

• Social security

• Subdivisions/developments gated by water resources.

• Question. As a manager, how do I manage the people side of these kinds of changes.

Framework of change:
Why, what, who? Are we changing.

If we can’t get the people to achieve the change then we can’t be successful.

The gating factor on change is no longer technology (has it ever been? Chris) – it is people’s ability to handle/absorb change.

If our success in technology is gated by our ability to manage the people side of change then we had better figure it out. It was very disappointing to see in the target audience here at the Gartner conference how little effect current change management initiatives have had.

Key points are that as we knew, Executive sponsorship of change is vital. However executive sponsorship isn’t just “show up for the kickoff and sign the checks”. It really must embody active involvement. Even more worrying is that, again in the audience at the session, that most attendees don’t have a “structured approach”, nor proper communication plans.

It has been our experience that change management is viewed by the technical teams as “touchy feely, unnecessary, gets in the way.” Changing that perception is a change management in its own right. That being recognized, the whole “Awareness, Desire, Knowledge, Ability and reinforcement” cycle. The presenters thesis is that all change goes through this cycle. So if we attempt to encourage the technical teams just by giving them Knowledge, we are entering the cycle too early. We need to find ways to get Awareness and Desire enshrined.

Keynote...

2008-06-10T10:06:00.001-05:00

Notes from Roy Schulte’s session on Event Processing.

The big key here is that Gartner are positioning EDA and Client Server (Request/Response) as subsets of SOA. We use many of the same concepts (standards based, abstracted, etc.) in both of the models. Event Driven Architecture is, however, better placed for situational awareness. Much of what Roy had to say was around the handling of Complex Events – typically described as aggregations of simpler events. He also made the key observation that an event stream can, of course, kick off a number of action streams. Using an airline example where as a result of some event data being received a whole raft of operational processes is triggered – from scheduling catering to preparing the ramp.

It is also clear from Roy’s remarks that even though we had the message oriented movements of the 90s, Event processing is still emerging. The usual innovators (financial institutions especially) have jumped onto this paradigm because of a combination of regulatory needs and some major competitive advantage in an environment when market analysis in milliseconds is now the norm.

BAM – Business Activity Monitoring leading to “real time BI” or genuine situational awareness is fast becoming a real possibility when event processing approaches are applied.

Gartner Coverage!

2008-06-10T09:45:00.001-05:00

For all the MomentumSI coverage at the Gartner show, please visit our corporate blog:

http://www.momentumsi.com/blogs.php

Gartner Conference - SOA

2008-06-08T08:23:00.002-05:00

MomentumSI will be sponsoring and participating at Gartner’s Application Architecture, Development & Integration Summit taking place this week in Orlando, Florida. This is a significant milestone for SOA as it represents the 10th anniversary of the AADI Summit…the underlying theme being “How to deliver value in a fast-changing SOA environment”.

Our Enterprise Architecture Solutions team will be on the ground at the summit and interacting with the architects, application managers and analyst in attendance. We will be posting daily blogs regarding the hot topics encountered and engaged discussions across SOA, application and Web infrastructure… stay tuned!

The Worst SOA Presentation I've Ever Seen

2008-06-07T07:55:00.002-05:00

Jean Jacque Dubray is outraged. He declares:

"I watched Jim (Webber) and Martin's (Fowler) presentation tonight before I went with the kids see Kung Fu Panda. The two furious have given a presentation that's got to be by far the worst presentation on SOA I have ever seen, it is not even pathetic, it reached the level of Sadness. If you wanted to turn off customers you couldn't do it better. "

I watched it. It sucks - but I've seen worst. Jim and Martin keep talking about why we shouldn't buy an ESB. The early ESB's were largely just JMS products and proprietary. In addition, the vendors positioned the ESB as the center of the software universe rather than a valuable component that has a specific function. However, the concerns I voiced five years ago on this subject have largely been addressed. Regardless, I'll agree with Jim and Martin that the ESB has the potential to do harm if over used or used incorrectly (but this is true for just about everything).

Here's my beef with the presentation. Other than the lame attacks they did on the acronym SOA, they failed to present a solution to the problem they introduced: Attacking Silos of Systems. Jim and Martin avoid this issues completely. They basically say "use our stuff from 7 years ago and you'll be fine". This includes refactoring, continuous integration, test-driven development, dynamic languages, etc. Let's be clear, these are excellent concepts that all moderns software development groups should use (if they aren't already...) but it has virtually nothing to do with the problem at hand.

If you look closely at the pitch, they fail to discuss how to not create multiple silos in the enterprise, let alone how to remove the ones that exist today. Their approach is "program better" and "increment". Fowler is an icon of modern computing and the concepts that he championed in the late 90's were brilliant. However, this stuff is crap. The promotion of "use a dumb network" and "use agile techniques" are things SOA guys agreed on 8 years ago. It's a "no shit, Martin..." kind of thing.

Now, tell me how you're going to solve the silo problem. Attacking SOA with your Same Ole Agile crap is a non-starter. These guys will have to address the real problem and quit pushing their old books. Life sure would be easy if we could all just "do REST" and then call SOA a success.

McGovern on EA-Joke

2008-06-01T08:42:00.004-05:00

In a recent post, James "fully insured" McGovern disputes my criticism of enterprise architecture. Mostly, he was disappointed that I went after Zachman, suggesting that EA's just don't care about that anymore (which I agree). I'll add that many of them feel that FEAF and TOGAF are also a bit too 'fluffy'.

But unlike me, James seems content with the state of his Visio and Powerpoint tooling, suggesting that he and his colleagues are doing just fine.

And on the subject of "silo funding", James states, "Funding models should never look like enterprise architecture and besides they are managed by two different entities..." I'm not sure what that means. My point is that the funding model should align with your business strategy. If the strategy is to provide a seamless experience for a customer as they cross through the silos, then the funding should reflect that priority. Without proper funding, EA's become security guards with a flashlight and no gun.

BUT - the thing that struck me the most about his post was that he never actually defended the enterprise architecture discipline. This could be that he feels that it doesn't need defending. Alternatively, he could have his own frustrations with the discipline. So James, which is it?

WOA wasn't important to me last week

2008-05-31T08:31:00.002-05:00

I spent last week working with a couple of my enterprise customers. The time was focused around the Shared Service Center (formerly known as the SOA Center). Here are the big issues that we worked on:

- Improving the help desk for production implementation problems related to shared services

- Improving the ability to perform root-cause analysis for service exceptions

- Getting funding to pay for a better staging environment

- Changing the way we engage with large I.T. programs so that they don't get bombarded by 5 different CoE's

- Changing our 'service discovery' process to make it easier to project ROI for shared services

- Identifying a new set of processes that support organizations will have to follow if the business application requires high availability: "Gold, Silver, Bronze SLA"

- Modified and communicated changes to our "Service Architecture Document" in order to provide consistency in documentation detail

In my world, SOA refers to the organizational politics necessary for an I.T. group to perform their job efficiently.

SOA has evolved from 'web services' to 'an architectural style' to 'an enterprise architecture framework' to 'a specialized I.T. lifecycle and supporting organizational design' - to 'all of the above'.

If WOA wants to encompass the aforementioned activities - I want in. Until then, WOA is just a cool thing for analysts, press and marketers to talk about. Debating the HTTP verbs might seem like 'real architecture' to some but I am genuinely concerned that a 'shiny object of misdirection' is taking our eye off of the real problems.

WOA wasn't important to me last week and it won't be next week.

Is Your Programming Paradigm HOT or NOT?

2008-05-26T10:17:00.003-05:00

Is your programming paradigm Hot or Not????

Well, you might not agree with my observations, but here's what I'm seeing:

Real Time
Let's start at the bottom of the stack: Real Time. Due to an increase in consumer devices, wireless networks, etc. we are seeing additional requirements for new low level programming. I wouldn't call it 'explosive demand', but the area does seem to be growing.

3GL
Surely, I'll get beat up here... but that's life. I am of the opinion that the amount of 'straight 3GL' coding continues to decrease. With the maturity of other paradigms people don't need to do as much old fashion hand coding as they used to. The bulk of this work has moved up a layer in the stack (3.5 GL / App Dev).

Application Development
These days more and more people write their applications on top of some stack (Java EE, open source frameworks, .Net, etc. Typically, they are using a combination of a 3GL with DSL extensions (JSP, ASP, etc.) In recent years, we saw the big application platforms (ERP, CRM, etc.) get (mostly) migrated to one of the aforementioned stacks.

Package App Customizations
The ERP, CRM and other packaged app vendors have increased their market share (as a percent of the enterprise footprint). This has moved traditional application development to more of a 'customization' model. Power users or application configurators use a workbench to add fields to a screen, modify reports, grant user access and other basic operations. New modules or complex changes are still typically done with more of an Application Development model except that the developer is expected to use the 'business framework' that comes with the system.

Platform as a Service
Although PaaS remains in it's infancy, it is growing at a rapid pace. From a language perspective, we seem to be seeing three options:
- The first option is to create a new language (like Apex) that is PaaS friendly (multi-tenant, sandbox, governed, etc.)
- The second option is to take a current programming language and throw exceptions with people call potentially harmful features (like Google did with their Python impl)
- The third option is to let the developer program in any language they want and let them deploy it on a VM. Here, developers are typically charged for CPU time and the VM is the sandbox, thus they don't really care what you do.

In general, we're seeing more and more development move up the stack. On occasion, we have to rework the stack, so we drop back down into lower layers and then work our way back up. Ultimately, the stacks get figured out and dollars (and work) flow into the layer of the stack that is most productive.

Why Enterprise Architecture is a Joke

2008-05-24T08:32:00.003-05:00

Enterprise Architecture is a joke. And I don't say that lightly. My goal isn't to pick fights with enterprise architects - quite the contrary. I've got huge bets on EA - my time - my career - my money. My goal is to improve it.

Anyone who has been in our industry for any period of time has heard the jokes about EA... "EA's are the guys who program in PowerPoint." Despite valiant efforts to mature the discipline by groups like IASA, the OMB, The Open Group, The Zachman Institute as well as individuals like Ambler, the discipline remains fragmented and often unproductive.

In my opinion, there are several reasons why the discipline has not matured more quickly:

1. Zachman pioneered, than stagnated. I believe that the single largest reason that EA is a joke can be linked back to the pioneer. This pains me to say, but many in our industry were patiently waiting for better stuff to come from Zachman and it just didn't happen.

2. Bad Application Architects got promoted to be bad Enterprise Architects. Although this isn't a universal truth, I've witnessed my fair share of it. Those who can't architect do PowerPoint.

3. Silo Organizations promote Silo Funding. Many EA's never had a chance. They live in organizations that fund everything according to business silo's. Then, the EA is expected to bridge the silos with nickle and dime funding. Their inability to perform Herculean change (multi-channel, master data, cross-organizational BPM, master SOA services) has many of them designated as cops with no gun, just a good flashlight.

4. The Tooling Sucks. Modern EA tooling is complete pile of crap. It is designed and written by a generation who is out of touch with the needs of modern I.T. groups.

5. Unconnected Models. Expanding on #4, our models (and tools) do not sufficiently flow from one model to the next. Software development is often explained as a series of model transformations from concept to design to construction, where each stage adds additional fidelity. We currently have a huge hole between EA and the downstream constituents.

Today's enterprise architects have been given the equivalent tooling as programmers in the 1960's. I feel like I'm bashing developers who were handed punch-cards, told to program in assembler and then scolded for their lack of productivity.

The good news is that most EA's are providing significant value despite their handicaps. The great news is that smart people have identified the problem and are actively working on the solution.

Key WOA Concepts

2008-05-20T11:49:00.002-05:00

Nick Gall clarifies a few key concepts of web architecture around the URI:

On Thu, May 1, 2008 at 9:09 AM, Gregg Wonderly wrote:
> I think that there are two distinct uses of URIs. There are URIs that are never
> intended to be typed by people (or at least don't need to be), and there are
> URIs that only people will type.

I couldn't disagree more strongly. We should be doing everything in our power to eliminate the distinction between human understandable and machine understandable interfaces. Thus suggesting URIs for humans be different from URIs for machines is a step in the wrong direction.

There are three important reasons to make interfaces for humans and machines as similar as possible:

Debugging: you never know when a person needs to look under the covers to see what's going wrong. Why do you think the Internet and Web emphasize ASCII text-based protocols even though they are far less efficient than binary and will rarely be seen by most people

"Show source": human readable interfaces generally and human readable URLs specifically make it easier for developers (and even occasional scripters like me) to learn from one another

Serendipity: TBL and RTF emphasize that the web is designed/engineering for serendipity. Thus you never know (and shouldn't try to guess) which URIs people will or won't want to use directly.

Our goal should be to minimize the distance between UI and API. Assuming that people will see and use all URIs is a big step in that direction.
-- Nick

Spot on, Nick! I love the notion of 'minimizing the distance between UI and API'...

See: http://tech.groups.yahoo.com/group/service-orientated-architecture/message/10245

SOA Acquisitions List

2008-05-12T08:00:00.002-05:00

I've updated the list of SOA Acquisitions:
http://www.momentumsi.com/SOA_Acquisitions.html

The new transactions were:
- Cape Clear to WorkDay
- LogicLibrary to SOA Software

I also added Sonoa Systems as a new target.

SOA Software Acquires LogicLibrary

2008-05-12T07:16:00.002-05:00

Today, SOA Software announced the acquisition of LogicLibrary, a leading provider of software asset & reuse management. The terms of the acquisition were not disclosed.

SOA Software is best know for their integrated SOA Governance platform. The key word here being "integrated". SOA Software has both design time and run time platforms that share policy information seamlessly. For anyone who has felt the pain of integrating stand-alone registries and repositories with policy enforcement points, mediation points and web service management tools, you'll appreciate the beauty of having an integrated solution.

SOA Software is realizing a grand vision: unified governance and management of SOA across the Enterprise Lifecycle. The acquisition of LogicLibrary will give them significant depth in managing design and development assets. LogicLibrary has deep integration in IDE's and SCM's. In addition, it extends their reach into enterprise architecture by capturing reference architectures and development policies.

Through this acquisition, SOA Software is demonstrating the depth and breadth of their vision. At this point, it is unclear if the competition fails to see the vision or merely fails to execute on the vision. Regardless, SOA Software has raised the bar (again) before many of the competitors have caught up with the last bar-raising that they performed.

SOA and WOA Comparison

2008-04-28T08:49:00.006-05:00

Several peopled (too many to name) have recently compared SOA and WOA. Unfortunately, virtually all of the comparisons are really more of an analysis of 'Web Services vs. REST/POX/RSS/AJAX'. For those of us who 'do SOA' for a living, this drives us crazy. We fought for years to get people to quit thinking about SOA and Web Services as being the same thing. Now we have both analysts and press undoing the progress that we had made. Ugh.

Ok guys - steal this:

Please consider extending this little framework as a way to do proper comparisons. If you feel the need to add rows, I encourage it. If you don't like the contents of a cell - change it, but for the love of God, please quit comparing Web Services and REST and calling it "SOA versus WOA".

As for the cute one liners: "WOA is the SOA that works"... "SOA is the internal cloud" :-) You guys kill me. This is excellent nonsensical dribble.

However, the idea of 'building your WOA on your SOA' caught my attention. My deciphering of this statement is "use the EA & Governance practices to ensure business alignment, proper sharing and quality while also using lightweight protocols to drive barrier-free consumption and composition." Assuming this is what was meant... I'd agree! Perhaps another post on this subject...

Targeted Markets for PaaS Providers

2008-04-10T08:21:00.004-05:00

Yesterday I wrote that the Google PaaS offering was not suited for the Enterprise primarily due to the language of choice (Python). This had me scratching my head. The people at Google aren't stupid - they know where the money is... why would they make this decision. And then it occurred to me.

Google is betting that the Enterprise will spend more money on SaaS than they will on PaaS. As Gartner said,

“Ease of use, rapid deployment, limited upfront investment in capital and staffing, plus a reduction in software management responsibility all make SaaS a desirable alternative to many on-premises solutions, and they will continue to act as drivers of growth.”

Google has made a bet that they can lure start up companies and next generation developers to their platform by choosing a dynamic language that facilitates mash-ups. And by doing this they will be the preferred platform for many next generation SaaS companies. But which ones?

The Google platform doesn't really offer any 'enterprise grade' functions. What it does offer is simple access to the Google world of social networking, ads, etc. In its current form the platform is best suited as a platform to enable SaaS for large audience applications (like social networking).

PaaS providers are still trying to figure out their target markets. Companies like Coghead are chasing the SMB market, hoping to attract customers who want to avoid setting up their own hardware and paying expensive programmers. SalesForce is clearly going after the enterprise ISV's. We're quickly getting to the point where each of these players will have to announce their target market to the world. The 'one size fits all' model won't last.

Enterprise PaaS

2008-04-09T14:00:00.003-05:00

The recent PaaS announcements from Google and Amazon have grabbed the attention of the enterprise customer. Utility computing, clouds, pre-integrated platforms and business services are all sexy topics. But these same companies also realize that there are huge hurdles to adopting these concepts. They're big ships - and they don't turn easily.

How do I get PaaS?
Companies will ask how they can take advantage of this incredibly disruptive computing phase. I'm in agreement with Michael Nygard who feels that "everyone will want one". See:
http://www.michaelnygard.com/blog/2008/02/a_cloud_for_everyone_1.html

Michael discusses the typical progression of high tech products:

Very expensive. Only a few exist in the world. They are heavily time-shared, and usually oversubscribed.
Within the reach of institutions and corporations, but not individuals. The organization wants to maximize utilization.
Corporations own many, as productivity enhancers, some wealthy or forward-looking individuals own one. Families time share theirs.
Virtually everyone has one. To lack one is to fall behind. No longer a competitive advantage, the lack of the technology puts one at a disadvantage.
Invisibility. Most people have or use several, but are not aware of it.

Michael goes on to say that he feels that Cloud Computing is currently at Stage 1 but it won't be long before large enterprises want their own. I'd argue that several enterprises already have their own cloud or utility computing environment. However, not many of them have a pre-integrated, platform sitting on the cloud ready for I.T. customers to use. There's a big difference between virtualized hardware and offering a software computing platform.

If you need to store data using the Amazon or Google offering, the options are clear. They have a couple services available - just grab the one you want and use it. To accomplish the same task in your average enterprise I.T. shop, you'd call up your enterprise architect, get a copy of the Technical Reference Model, identify the applicable elements, and then go to an infrastructure group to try and get them loaded so that you can test to see if they will work for their project. Ugh!

It will take a rebellion for enterprise I.T. to change but this act might be sooner than you think. Developers are already using Amazon and will quickly be messing with Google. They'll be telling management that they want to use this stuff because it is quick and easy to get going. (It's the same reason why people wanted off the mainframe and eventually ignited the client/server era).

In my opinion, Enterprise PaaS is inevitable. Organizations will not be able to switch to a purely outsourced model but will look for hybrid solutions, including creating their own PaaS. The PaaS model of the future will have to embrace multiple PaaS vendors (dare I say federated Paas). But, of course, the major infrastructure vendors will offer a PaaS-in-a-Box that mimics their own hosted model (think Oracle/BEA, IBM, SAP, Microsoft).

Who will win?
It's too early to say. Early indications are that it isn't Google. Their decision to go "Python-Only" is an extremely strong statement. Personally, I can't think of a single enterprise that considers Python part of their strategic computing platform (nor do they have armies of trained Python developers). The winner should be IBM, but they tend to think everything is a consulting problem. SAP isn't exactly known for their technology... Amazon has done some pretty cool stuff. Normally I would have discounted them, but who knows? Salesforce has an early lead but they don't have much of a footprint in the enterprise. Oracle/BEA and Microsoft both have interesting prospects in this space. If I were a betting man (which I am), I'd throw my chips on these guys.

Microsoft's 10 Year Old SOA Strategy

2008-04-02T06:43:00.002-05:00

If you aren't familiar with Microsoft's SOA Strategy, I'll sum it up for you:

BizTalk + WCF + Vaporware = SOA

Yesterday, I attended a Microsoft SOA event to get briefed on their strategy. To say the least, I found myself disappointed. After 3 hours of showing slides and demo's I finally concluded that Microsoft's SOA efforts to date have been a failure.

At the heart of their strategy is BizTalk. It's the one SKU that they can actually sell that is related to SOA. And if you listen to Microsoft, you'll be told that virtually all SOA paths lead to BizTalk. If memory serves me, BizTalk was released in Beta in 1998 and became generally available in December of 2000. This isn't exactly a new SOA product.

Windows Communication Foundation (WCF) is a fairly new component that was introduced in the last couple years as a pluggable framework to enable the Web service protocols.

The third part of the equation is Oslo. This mythical beast is basically the next version of a bunch of products which will help build on the vision of Software Factories. Oslo will not be 'released' as a unit, but rather each individual product will get released on its own timeline.

Robert Wahbe, the Microsoft executive in charge of the Connected Systems vision, must either be sitting on some elaborate game plan which involves a well-kept secret to acquire some actual SOA products, or he's in serious trouble. It is clear that Microsoft no longer has the killer instinct that it had years ago. But even then, the lack of results in this area must stand out like a sore thumb.

JBoGS and PoPS

2008-03-23T14:48:00.006-05:00

I noticed that Nick Malik and others have stepped up the call to stamp out "JBoWS" (Just a Bunch of Web Services). Not me. After some soul searching, I've determined that JBoWs is the natural first step that an organization takes on the path to service orientation. It's not that it's right or wrong, it's just a stepping stone.

Since the industry likes buzzwords and acronyms, I'll add to the mix. It has been my observation that the next step is to take the 'wild' services and govern them. I've been calling this, 'JBoGS' or (Just a Bunch of Governed Services).

JBoGs is the natural extension of JBoWS. Services continue to be funded in a project (and often silo manner) but are designed, built and operated according to modern governance concepts. With JBoGS, a company will most likely have some type of registry / repository solution, lifecycle governance and runtime management infrastructure and practices in place.

This is an excellent step after JBoWS and it isn't too hard. People like to say, "You can't buy SOA". Well, that's true... but for the most part you can buy JBoGS. Moving from JBoWS to JBoGS requires some infrastructure, a SOA lifecycle and governance practices, all of which can be bought from vendors like MomentumSI, IBM, SOA Software and Progress.

JBoGs might sound like a derogatory term, but it isn't. I applaud the companies that are getting experience in building and governing services. But let's get real, that's not "SOA" - it's another stepping stone.

I'm throwing out another term: "Patches of Planned Services" (PoPS). Here, we're aligning the enterprise architecture with the Services. In essence, we're performing urban planning for communities of services. This is the 'planning' view of SOA typically thought of as a top-down approach. Notice that I used the word 'patches'; I didn't use a term like "Enterprise-Wide Planned Services" - the fact is, no one (who keeps their job) will do this across the enterprise. We'll cut up domains (or patches) and plan one area at a time.

The Evolution
We have several customers who are at the JBoGS stage right now. Most companies want to get their 'wild services' governed before they move to more ambitious goals. 2008 appears to be the start of the PoPS era. Companies that have matured their JBoGS are now looking for SOA to support business critical processes like order-to-cash. This means that they now have to roll out multiple services (not 1 or 2, like in JBoGS). This is the driving force for creating planned communities of services which are typically aligned to business processes. Finally, eh?

You'll notice that in the diagram above I imply that there is something after PoPS. There is - I know it, but the future remains cloudy. In the past, I've predicted things like: organizational alignment to SOA, alignment to business strategy, external service networks, and a host of other great ideas. Bottom line is that I don't know, and most likely no one else does either. Don't worry about it. For now, keep working on the other stepping stones!!!

SOA Acquisitions (updated)

2008-01-28T13:07:00.000-05:00

I've updated the list...

http://www.momentumsi.com/SOA_Acquisitions.html

I have 16 companies still listed in the "producer" category (target companies). I'm guessing we'll see two more go in 90 days.

Facebook Terms

2008-01-06T20:47:00.000-05:00

You probably saw where Scoble was temporarily suspended from Facebook. It made me go back and look at their legal terms. The one that threw me for a loop was:

You can not...
"use automated scripts to collect information from or otherwise interact with the Service or the Site;"

I must be missing something. How can a software platform like Facebook not allow "automated scripts to... interact with the Service or the Site"??

Just like moving from ISV to SaaS requires a change in attitude, so does the move from a 'web application' to a 'platform'.

Enterprise SaaS Must Stay Up

2008-01-06T07:31:00.000-05:00

This is the kind of thing that gives me the hebejebees:

As soon as you start putting "production data" in a system, it must stay up. In this case, Spock clearly identifies that the system isn't 'Production', but is in 'Beta'. The difference between Beta and Production in SaaS is really just one of expectations. If Spock were to drop all of our data we'd be pretty upset; in my book that looks more like production than beta.

I want a SaaS provider to prove to me that they know how to add features without taking the system down. That is, even in beta, I expect the system to stay up. I expect that they'll do a behind the scenes data migration to make things right.

Balanced Views of SOA

2008-01-02T11:31:00.000-05:00

I hate adding new terms, but I think we're hurting ourselves by constantly overloading the term SOA. IMHO, we would be better off if we formalized the "The Balanced Views of SOA". By "views" I'm referring to the old "4+1" type concept, see: http://tinyurl.com/2qed42 or http://tinyurl.com/2pmpmt

Premise: SOA is a collection of techniques which can be understood by observing a solution set from several different vantage points, or Views. The views can be divided into three primary categories:
1. Service Portfolio Views
2. Individual Service Views
3. Consumer-Service Views

The Service Portfolio Views focus on treating services as business assets residing in a portfolio. The role that most likely uses this view is the Enterprise Architect. Example views might include:
Business Priority View, Service Pipeline View, Process View, Portfolio Investment View, Consolidation/Rationalization View, Information Model, etc. These views help in the prioritization and planning process and to keep things organized (think Metropolis).

Individual Service Views focus on describing a single service. This view is most likely used by analysts, developers and testers to create a new service. Example views might include: Service Interface View, Service-Component View, Service-Deployment View, etc. Note that these views most closely resemble the traditional 4+1 concepts.

Consumer-Service Views focus on describing the relationship between services and the consumers (at plan-time, design-time, provision-time and run-time). The roles that would leverage these views include SOA administrators, product managers and configuration managers. Example views might include: Consumer/Service Dependency View, Policy Views, SLA Views, etc. These views help people to keep composite solutions from breaking due to incompatible versions, capacity problems, etc.

================

The power of SOA is that it is the first model that crosses these three areas. It has been my observation that the companies that balance these three broad views of SOA are the most successful.

Is SOA "predominantly an enterprise architectural style"? Well, it sure is if you're an enterprise architect! I understand that the evangelists in this group want to push the importance of the Service Portfolio Views. But, I'm going to pull a "Mark Baker" and become religious about the "Balanced Views of SOA" - that's my New Year's Resolution :-)

=== This is a repost from the Yahoo SOA Group

Top 100 SOA Predictions for 2008

2007-12-30T18:46:00.000-05:00

Here are my top 100 predictions for the SOA community:
#1 - The incredible value of SaaS is realized and buyers want in
#2 - The buyers realize they need enterprise SOA to effectively pull off SaaS

#3 - #100 are irrelevant.

Updating the SOA Scorecard: SCA Services

2007-12-19T08:50:00.000-05:00

For almost 5 years now, we've been advising clients on their SOA scorecard. In the early days of SOA adoption, customers often measure their success by 'releasing services'. This is a great metric for beginners. As the program progresses the shift moves to their utilization. Services are only valuable if they are actually used. From this perspective, we recommend looking at the consumer-to-service ratio as the primary test. The secondary test looks at message counts; that is, are the services actually being called.

Unfortunately, neither of these tests actually look at the value that software brings to the business. We've been encouraging our customers to spend more time looking at the business proposition that the service reflects. Business strategy is based on those activities and assets which are considered 'Sustainable Competitive Advantages' or SCA. These are the things that allows a company to be successful in their business year after year. Strategist have various means of identifying and describing the SCA's. Moore's 'core vs. context', and 'value chains' are often applied to help an organization understand where they need to be competitive.

Many organizations have moved beyond SOA pilots and now have scores of services running in production. However, many of these same organizations do not have the truly important services available. The fixation to increase the number of services often overrides the commonsense notion of providing business-valued services.

I've been encouraging customers to look at the three C's of business: Customers, Commerce and Channel. All three of these domains are typically excellent candidate for service enablement with high business value. Obviously, there are other domains that might be more important for your business which should be prioritized higher. The point is that you need to create a priority list of services based on the business that you're in. Service enabling non-valued added aspects of the business might look good on an old fashion SOA scorecard, but it no longer passes the smell test!

Amazon SimpleDB Launches

2007-12-14T07:46:00.000-05:00

Amazon Web Services has launched their latest offering, "Amazon SimpleDB":

"Amazon SimpleDB provides a simple web services interface to create and store multiple data sets, query your data easily, and return the results."

SimpleDB sits on top of their hosted computing layer (EC2), proving an integrated cost model for those who are already EC2 customers. From a developer perspective, SimpleDB looks more like a big 'hashtable in the cloud' than a traditional RDBMS. The database is not relational and doesn't support SQL for data definition or data manipulation (DDL/DML). Instead, they have chosen to use a 'schemaless' system for data definition (name/value pairs) and for data manipulation (insert, update, delete) they have chosen to use simple REST style verbs (PUT, GET, DELETE, CREATE, QUERY).

The service is specifically designed for small payloads (not big BLOBS). For larger files, they recommend that you use AWS S3. They also recommend that you only run 'real time queries'. In fact, query execution time is limited to 5 seconds which prevent users from running very large requests. Although some might disagree with this decision, I'm a full supporter of this model. However, organizations will still need to perform two vital operations: 1. Export all data 2. Perform complex BI style queries. It is currently not clear how these items will be supported. The system doesn't seem to support the notion of 'change data capture' which would allow changed records to be sent to a separate analytics engine, nor am I finding a mechanism to easily load or unload a data store which would be a real issue if you time out after 5 seconds...

SimpleDB is in beta which implies that features have been frozen while bugs are ironed out. My primary concern is that they may have accidentally designed "TooSimpleDB". I love the idea of keeping it simple, but they may have overdone it.

Replace SOA Governance with Expertise?

2007-12-12T11:32:00.000-05:00

Owen Pettiford takes a swipe at SOA Governance:

The reasons for this conclusion stems from the drug induced high I was on just after the operation when it occurred to me that I had not just survived the biggest operation of my life because someone "governed" it well (sure it was important that the right people were there and that they had the right tools) – I survived because I had EXPERTS working on me who had TRAINED for YEARS.

Well, I partially agree with Owen. Let's not forget that much of the doctors training was on the best practices (sanitary environment, cross-checks, standard procedures, etc.)

SOA Governance isn't a magic bullet. But if it is implemented correctly it will prevent (some) people from doing (some) stupid stuff (some) of the time. And I'm ok with that.

He goes on to say:

Enterprise SOA will be achieved through expertise, passion and shared vision.

You could replace "Enterprise SOA" with just about anything and you'd be correct. The problem is that large organizations are filled with people who are non-experts who don't have passion and lack shared vision. It's that simple. Good SOA Governance makes the assumption that statistically, the average person in your organization is, well, average!

See:
http://pettiford.blogspot.com/2007/04/governance-is-not-answer.html

SDLC and Work Patterns

2007-12-11T16:09:00.000-05:00

Nick Malick stumbled upon a real common misconception. In a recent blog post he mentioned that a SOA SDLC must be more iterative:

First, a comparison: Waterfall looks like this:
Waterfall: Plan --> Envision --> Design --> Develop & Test --> Deploy
Agile: Plan --> Sprint --> Sprint --> (occasionally) Deploy --> Sprint --> Deploy

A SOA SDLC looks more like this:

Plan --> Sprint (Process and User Experience) --> Sprint (Process & Services) --> Deploy --> Sprint (P&UX) --> Sprint (P&S) --> Deploy
What Nick is describing is the use of Work Patterns within an SDLC. It is common for people to accidentally combine these two distinct concepts.

Here's the difference...
- SDLC focuses on the phases, activities and artifacts
- Work Patterns focus on the selection of phases, iterations and workflows

It's hard to describe but take a look at this and I think you'll see what I mean:
http://www.usdoj.gov/jmd/irm/lifecycle/table.htm

See section 13.2 for an example. I do agree that running a serial SOA SDLC is not a good thing. I was glad to see Nick mention User Experience. Usually, we'll talk about the Consumer SDLC and aligning it with the Service SDLC.

Good Stuff!

Ruby on Rails picks REST, will you?

2007-12-10T07:24:00.000-05:00

Just out...
http://weblog.rubyonrails.org/2007/12/7/rails-2-0-it-s-done

ActionWebService out, ActiveResource in

It’ll probably come as no surprise that Rails has picked a side in the SOAP vs REST debate. Unless you absolutely have to use SOAP for integration purposes, we strongly discourage you from doing so. As a naturally extension of that, we’ve pulled ActionWebService from the default bundle. It’s only a gem install actionwebservice away, but it sends an important message none the less.

This is an interesting question: When do you have to use SOAP? or WSDL? The answer is that rarely do you "have" to use it. The two most common reasons for using it are
1. It's the corporate standard (someone tells you to)
2. You have some software that only speaks SOAP

I'm a fan of using the smallest amount of protocol to accomplish the job at hand. REST is about simplicity - simple to design, simple to consume. And simplicity introduces scalability of human consumers. That is, it doesn't require reading gobs of white papers on the subject to understand it.

The funny thing is I'm also a fan of protocol extensions. The big idea behind the WS-* stack was that it didn't require you to use all of the WS-* stuff at once; you were able to pick and choose the elements you needed. You could use the basic messaging stuff and throw in a bit of security. If you don't want security, don't use it. You need transactions, hell throw that in... What the REST approach seems to be telling the Web Services world is that even the light stack (WS-I Basic Profile) is too fat. And I agree.

Clearly, I'm not alone on this. Plenty of people have pointed figures at the WS-I and called them ugly names. However, I haven't picked up anything from the WS-I organization suggesting that they realize that their lightweight stack is too fat. Instead of working on an even slimmer profile (call it the WS-SuperLight Profile), they are adding new WS-Fat stuff to the mix. I can't make it any clearer than this: WS-* is dead unless they create a lighter weight protocol.

True RESTifarians will point out that REST isn't really about a 'lighter stack' but rather the genius of REST is in the tenets of the disertation. I mostly agree but I think that there are still variations of the tenets that need to be explored. For example, some might argue that if a new profile was created that Relax NG might be strongly considered as an element.

Regardless, it is clear to me that the WS-I is too fat and about to have a heart attack. But rather than going out and exercising, the leadership team seems devoted to going out and eating a couple pounds of cheese cake. WS-* will collapse if measures are not taken - and soon.

The Next Phase of SOA

2007-12-08T18:50:00.000-05:00

I've spent the last couple days at the Gartner Application Development and Integration Conference in Las Vegas. As you might guess, the majority of the show was really about Service Oriented Architecture.

Like many SOA events, there were plenty of sessions on SOA governance, strategy, quality and implementation technologies. New this year was a strong focus on Web 2.0 as the consumers of the services.

Despite all of the good sessions, there was a void around "Information Modeling for SOA". The process of analyzing and designing an information model that mimics the business is a huge issue at every client that I work with. Many of our clients have moved past the initial planning and setup phases of SOA and are out designing and building services. Companies are realizing that having pockets of service teams (working in silos) leads to a collision of service semantics.

Nick Gall, Gartner Guru, commented that this year everyone seemed to be on the same page. The vendors, consultants, analysts and buyers were all in agreement about the steps you take to roll out SOA, the kind of infrastructure you need, etc. In essence, we've been doing it long enough now that there are well known success patterns that are rarely disputed. Nick was also quick to point out that we're not home yet. Like me, Nick feels that Information Modeling for SOA remains a deadly issue for many organizations. Whew! I was so glad to hear that I wasn't alone out there.

He drew the analogy to the database world. Today, most I.T. savvy people know the basics about how to design databases. However, it is a much smaller number of people who are experts in database design - the kind of people who can tell you the difference between 4th normal form and Boyce-Codd normal form. In SOA, we lack the equivalent set of rules for modeling services. In SOA, most people don't even know the equivalent of 1st normal form. Guys like Thomas Erl have thrown out the basics of service design, but it is hardly enough to avoid the pitfalls.

And the issue is much larger than just designing a single loosely coupled service. The real issue is designing a portfolio of services without creating redundancies or conflicts in the message model. Imagine taking the message models of all of the services across a domain (or set of domains) and reconciling their semantics. This view is takes the service out of the 'semantic silo' and begins to introduce an enterprise vocabulary. And like all data, relationships will exist. Here is where it gets interesting. Many people have focused on 'decoupling clients and services' (which is great), but have completely forgotten about decoupling the data model (or service model if you prefer). Pull out a large ER-model and look at all of the relationships between tables. When you resurface the data models as business or data services those relationships will still exist.

The collection of services and their message models is what we call the Enterprise Service Model (ESM). The science of creating the ESM is centered around "Information Modeling for SOA". More to come on this...

Service Oriented Analysis - Harmony

2007-11-17T11:19:00.000-05:00

Alex Rosen of MomentumSI recently introduced our method for Service Oriented Analysis. Check this out for a great overview.

BEA Systems or BEA Consulting?

2007-11-15T19:33:00.000-05:00

Today, BEA announced their 3rd quarter results:
BEA reported third quarter total revenues of $384.4 million, up 11% from last year's third quarter. BEA reported third quarter license fees of $134.8 million, down 1% from a year ago, and services revenue of $249.6 million, up 18% from a year ago.

249 million in service revenue represents approximately 65% of the overall revenue, well beyond the norm. As if BEA wasn't facing enough challenges, they'll now have to answer the very difficult question of valuation. Should BEA be valued as a product company, a services company or a hybrid.

According to Google Finance, their current Price-to-Earnings is 46.31. This is out of whack relative to competition (ORCL, TIBX, etc.) and is even farther away from the valuations of service companies like Accenture, who P/E is 18.6.

Assume for a moment that speculators weren't holding up the BEA stock awaiting an ORCL acquisition; what would happen? What if... Wall Street called BEA on their revenue split and began to value them as a service company? If you were to use consulting valuation multipliers, you might see BEAS price drop to $7 per share, immediately erasing several billion dollars of market cap.

BEA is in a tough position, they need to show year-over-year growth and strong earnings. They were able to accomplish this by creatively finding revenue. This strategy may pay off in the short term, but this quarter over quarter pattern is looking less like a stop-gap measure and more like a business model.

Looking back 10 quarters...

"If it walks like a duck and quacks like a duck..."

Disclaimer - Do not make investments based on this information.

Power to the People, SOA Style

2007-11-11T11:06:00.000-05:00

Guerrilla SOA advocate, Jim Webber comments, "One by one your services will be stripped from the clutches of enterprise architecture and governance teams and returned to the (business) people."

Enterprise architecture is responsible for creating SOA principles, methods, policies and infrastructure. They do this to ENABLE project teams, not to OWN the services. Often EA will work with 'zone architects' to help identify conceptual services so that the 'just build it guys' don't recreate stuff that already exists. I'm not sure where Jim got the idea that EA owns the services... perhaps that is common in Europe?

What I don't like about Jim's comments is the "us against them" mentality that he is provoking. Is EA an evil organization plotting to destroy the business? The enterprise architects that I work with know that there is no way that they could "own" the thousands of services in a large Enterprise. However, it is common for EA to champion "enterprise services" like customer and product. I think that Marty Brodbeck of Pfizer clearly demonstrated the need for creating a certain set of Master Services when he described the relationship between SOA and MDM at the InfoWorld conference.

Someone asked me if I disliked 'Guerrilla SOA' and I told them, "I have no idea if I like it or dislike it because it has no shape or form. Right now it's just a funny name to a concept that implies applying agile principles to SOA." I'd suggest that the advocates put some additional thought around the concept.

Jim Webber didn't refer to his concept as 'Chaotic SOA'. This implies that there are rules. However, I have no idea what they are. My guess is that Guerrilla SOA will end up looking much more like Enterprise SOA than Chaotic SOA.

Obviously we can't just yell, "power to the people!", although it sounds cool and is a lot easier than thinking it through :-)

The Enterprise SOA Manifesto

2007-10-25T08:08:00.000-05:00

Recently, the SOA camp has entered into a period of self-reflection and evaluation. Through this endeavor, the camps are dividing. There are those who feel that SOA aligns more closely with the concepts of enterprise architecture and there are those who feel that success can only be driven by more agile, quick hit wins. The optimal answer is likely somewhere in the middle... an approach that blends the best of EA with the best of Agile.

In the vein of the "Agile Manifesto", I'll post my position and invite my colleagues to offer their ideas.

First, it is worth defining Enterprise SOA. The definition that has evolved from working with our customers is,

Enterprise SOA is:
An Enterprise IT Strategy that encompasses a set of business, process, organizational, governance and technical methods.

It enables business agility through the use of loosely coupled services that are used as building blocks to develop composite applications that can be reused and recombined to address changing business priorities.

Complementing this definition is a set of supporting philosophies. Note that these philosophies are about Enterprise SOA, not just 'service oriented architecture', which has already been covered.

The Enterprise SOA Manifesto

Communities over Silos
Enterprise Architecture focuses on creating portfolios of integrated software to fulfill a business mission. Although application silo's can often be created more quickly, the long term process of integrating silo's of logic, data and policy create spaghetti architecture, demoralize teams, stagnate innovation and increase long term maintenance costs. Communities (or application portfolio domains) should adhere to enterprise standards while each zone tailors the localized rules and regulations.

Balanced Planning over The Extremes
Enterprise SOA attempts to balance 'planning' versus the extremes (too little/too much). The popularity of the Agile movement was largely a knee-jerk reaction to the frustrations with "waterfall planning". Enterprise SOA blends long term planning with tight iterations. Think, "planning in the large, agility in the small".

Governed Delivery over Ad-hoc Delivery
The enterprise must prioritize the needs of the many over the needs of the few. Applications must be architected to fit into an ecosystem of applications. This will require adherence to guidelines and policies on technical standards and software processes, employed to protect the long term interests of the community.

Sharing and Reuse over Building from Scratch
Portfolios of applications will have many common functional requirements. An implicit non-functional requirement in Enterprise SOA is to design for sharing and reuse where appropriate.

Business Priorities over the Enterprise SOA Manifesto
I.T. systems are either a reflection of the business today or a projection of where the business is heading tomorrow. The I.T. approach must not become a religious battle fought at the expense of the business. On occasion, it will be in the best interest of the business to violate the principles of the Enterprise SOA Manifesto for the purpose of 'doing the right thing' for the business. Appropriate planning, governance and leadership should make this the exception, not the rule.

JackBe Closes $9.5 Million in Growth Funding

2007-10-24T08:05:00.000-05:00

JackBe, a leading provider of mashup software, has closed on a new round of funding. This is a Series C round and should provide them the resources to continue building out their platform and sales engine.

I had the opportunity to speak with JackBe CEO, Luis Derechin, earlier this week. First, I have to publicly congratulate him on running (and finishing) his first marathon! I'm sure there are plenty of parallels to be drawn with running his business...

A couple interesting topics were discussed in our meeting. The first was the inevitable maturation of the market from widgets to platforms. Like many RIA vendors, JackBe initially focused on AJAX widgets and presentation layer accessories. Their business has grown into server-side mashup engines capable of integrating data from a variety of sources (databases, legacy systems, Web Services, etc.) and transforming the transports, payloads and data into easy-to-consume formats for Web 2.0 developers.

I believe that the mashup layer will quickly become an essential element in User Experience Reference Architectures (UE-RA's). The focus of older portal technologies is 'on-the glass integration'. Here, disparate data sources are pulled together as a set of portlets (or windows). Conversely, the Mashup Layer focuses on what I've been calling 'before-the-glass integration'. This enables new functionality to be inserted between consumers and providers (think clients and services), aimed at providing mediation and transformation services 'just in time' based on the context of the consumption.

Because this software is positioned squarely as a presentation layer-to-service go-between, it has the unique advantage of also enabling wide scale governance through policy visibility and control mechanisms. Think of it this way. Today we have 'mash-ups' tomorrow we'll have 'crash-ups'. Service interfaces will change, as will formats and protocols. This places new challenges on software configuration management and the overall governability of the Web 2.0 solutions.

We saw a similar trend in the SOA world where vendors who provided intermediaries had to make a decision. Are you the source of governance policies or the target? This is a fundamental question that each vendor will have to resolve. Said another way, will your tagline be "Mashup Governance" or "Mashup Enablement". That said, a key attribute of Mashup Enablement will be providing Policy Enforcement Points (PEP's), fed by governance vendors.

Over the last 5 years, many of us SOA-guys have put extensive thought into every aspect of the 'service provider' aspect, while neglecting the 'service consumer'. However, an acknowledgement that more advanced UE-RA's are needed will hopefully draw out other Enterprise Architects to surface a more in-depth discussion on this very important topic.

Quit the Clowning

2007-10-23T08:13:00.000-05:00

Joe McKendrick is confused. In his latest blog post, "Enterprise SOA Concept Falls out of Favor", he implies that 'enterprise SOA is failing' which couldn't be farther from the truth. I believe that his bad information comes from people who don't know SOA, don't do SOA and in some cases, caused the mess that SOA cleans up.

In my position, I have visibility into HUNDREDS of SOA programs. Many are active clients - many are past - many are just companies that I've had the privilege to speak with. And through my partners, (the leading SOA infrastructure companies), I have excellent visibility into their historical sales and future demand pipeline. Between these vehicles, I am able to see commitments related to SOA strategy, training, governance, architecture, infrastructure, organizational design, change management, packaged application enablement, service integration and composite application development. That said, I feel like I'm in an excellent position to comment on what is really going on with regard to SOA.

First, SOA is nowhere near the mythical Gartner beast called, "The Trough of Disillusionment". And I think it's funny that people keep trying to force us in that phase so that we can then say we've moved past it. Nope. We're not even close.

Second, the idiots that are running around yelling "guerrilla SOA" have to be put in their place. Many of these individuals are the ones responsible for silo-oriented thinking in the first place. They proposed small (agile) projects where we captured just enough requirements to begin coding and releasing. Guess what? This style of development doesn't jive with the concept of shared services. It is the cause of the problem, not the solution.

We love to compare EA to city planning (plotting out neighborhoods, identifying common infrastructure, etc.) Yes - this requires using your brain and creating a plan. Enterprise SOA involves long term planning coupled with short term results.

Don't get me wrong - I'm a fan of 'controlled agile'. The rules of agile by themselves are so incredibly destructive to large organizations that they have done immense harm. Shouting 'agile manifesto' at people who have to build real 'application communities' is a non-starter.

I am proud of the companies that I work with who take the time to think about what SOA means to them. They plan their community. They consider the common infrastructure. They create policies and rules for their citizens. They identify practices to build the structures (reference architectures). And once they've figured out how to build a community they go do it. They don't do it for the entire enterprise all at once - that isn't what 'enterprise SOA' means. Instead, they partition their enterprise into a set of communities and attack them, often in parallel.

Let's be clear. Enterprise SOA is, by far, in the strongest position it has EVER been in. The jokers who feed columnists bad information need to go away. But this won't happen on its own - it requires the columnist to call them the clowns that they are.

Consolidation in the Software Industry

2007-10-12T07:23:00.000-05:00

In June of 2003 I noted that, according to Yahoo Financials, Microsoft was larger than the next 40 largest software/service companies:

Quest Software, Inc. + Tibco Software, Inc. + Documentum, Inc. + Take-Two Interactive + Activision, Inc.+ Hyperion Solutions Corp. + Sybase, Inc. + Macromedia, Inc. + RealNetworks, Inc. + Business Objects + Cognizant Technology Sol. + Red Hat, Inc. + Satyam Computer + J.D. Edwards & Company + Autodesk, Inc. + CSK Corporation + Network Associates, Inc. + Konami Corporation + Compuware Corporation + Trend Micro Incorporated + Cognos Incorporated + Mercury Interactive Corp. + VeriSign, Inc. + Citrix Systems, Inc. + Cadence Design Systems + BMC Software, Inc. + Amdocs Limited + BEA Systems, Inc.+ Synopsys, Inc. + Siebel Systems, Inc. + Check Point Software Tech + PeopleSoft, Inc. + Infosys Technologies Ltd.+ Symantec Corporation + Adobe Systems Incorporate + Intuit Inc. + Electronic Arts Inc. + VERITAS Software Corp. + Computer Associates + SAP AG + Oracle Corporation

As I scanned this list, it occurred to me how many of them have been acquired...

Documentum --> EMC
Veritas --> Symantec
Hyperion Solutions --> Oracle
Macromedia --> Adobe
Business Objects --> SAP
J.D. Edwards --> PeopleSoft / Oracle
Network Associates --> McAffee
Mercury Interactive --> HP
BEA Systems --> ??TBD - Oracle??
Siebel --> Oracle
PeopleSoft --> Oracle

It is interesting to look at the list of companies that didn't get acquired. Let's remove the non-enterprise I.T. companies (entertainment software, semiconductor, consulting). The odd balls include: Sybase, Tibco, Cognos, Adobe, Citrix, BMC, Red Hat and Computer Associates. In my opinion, these companies need to find a home - and soon.

Companies that have had significant growth since my last post include:
VMware, Inc. and salesforce.com, however you should note that their PE's are 269xTTM and 1,245xTTM - and in my humble opinion - this is just a tad bit inflated! At some point these companies are going to have to increase their top line numbers. Regardless, we've entered into a period of massive consolidation where few new companies can truly shake the market.

SAP TechEd - We Have SOA (for real this time)

2007-10-03T21:07:00.000-05:00

I've been attending the SAP TechEd in Las Vegas conference for the last few days. Overall, I'm impressed with the reality of the story that SAP has put together. In the past, SAP had a SOA-by-PowerPoint story. Today - they have real products and guidance.

At the core of the SAP SOA story is the Enterprise Service Repository (ESR). It is actually a combination of both registry and repository. The registry is a UDDI 3.0 implementation and has been tested to integrate with other registries such as Systinet. But the bulk of the work is in their repository. Unlike other commercial repositories, the first thing to notice is that SAP's is pre-populated (full, not empty). It contains gobs of information on global data types, schemas, wsdl's and similar artifacts relating to the SAP modules.

Both the registry and the repository are designed to embrace service metadata that is housed by SAP as well as service information that might be in other platforms (IBM, Microsoft, etc.) And although the registry will do metadata interchange with other registries, we're not so lucky with the repository. Apparently the current version is designed to be a single instance across the entire enterprise. Of course, this isn't realistic in most organizations due to mergers & acquisitions, etc.

SAP has also customized the TOGAF enterprise architecture to meet the needs of large customers. I was hoping to learn more on this but the speaker failed to show up (bummer). However, it was clear that SAP is relying on IDS Sheer to supply EA modeling tools.

I felt like SAP really got it. Unfortunately, I felt like most of the people at the conference didn't. The gap between vendor and customer has grown significantly. Many of the SAP customers have avoided technology topics, favoring black box implementations of large monolithic applications. SAP is introducing a fundamental change that will take time to sink in. They are doing a great job of evangelizing and providing community guidance but this is a huge leap for many of the 'ABAP' programmers of today.

A Watched Pot Never Boils - SOA Style

2007-09-30T18:21:00.000-05:00

Literally - I was trying to boil some water today. While staring at it, I began thinking, "My God this is taking forever!". A Watched Pot Never Boils. Sadly, too many of my life experiences end up relating to SOA...

Some journalists and bloggers have been critical to the success of SOA. Personally, I view it as a "watched pot". Impatiently staring at an SOA effort and constantly asking, "How many services do we have?" or "What's the ROI?" - doesn't really help.

SOA is a long term change to I.T. - it will take time. I've seen HUGE changes take place in some of the world's largest corporations. I've also seen companies with poor I.T. leadership whine about how they failed. Hell, I could have told them that they would have failed at about anything they tried.

I guess what I'm trying to say is that I don't think that the journalists / bloggers should ask EVERYDAY, "are we there yet?" It's annoying and not helpful. And the LOSERS who threw in the towel - well, they're losers.

Budgeting for SOA

2007-09-20T14:42:00.000-05:00

Well, it's that time again - I.T. teams around the nation are hunkered down working on their 2008 budgets. All of them are asking the same questions: How much do I need? What will it be spent on? Where can I cut? Unfortunately, I can't answer all of these questions in a blog post, but I can provide some investment categories to consider:

1. SOA Foundation - A good SOA Foundation program is typically a 3-6 month process utilizing internal and external resources. The initiative rolls together a number of common deliverables: SOA Strategy & Roadmap, SOA Methodology Updates, SOA Reference Architecture, Standards Development and SOA Governance Planning. This will typically cost about $200-400k from external resources, and will eat up significant time of 2-4 internal resources.

2. SOA Infrastructure Realization - Over the last several years, software vendors have been perfecting their SOA solutions. Most large organizations evaluate and acquire their software in 4 different stages, often a year apart.

Stage 1 software is typically a combination of Registry, Repository, Mediation and Web Service Management. For an enterprise license this is typically $750k - 1.25, depending on the size of the organization.

Stage 2 is typically a combination of security and integration. This often includes SOA firewalls and other edge devices. For integration, most organizations are looking at ESB's, orchestration engines and service engines / adapters. Plan on $400-800k for Stage 2.

Stage 3 is typically a combination of EA and Advanced Integration. Organizations that don't have EA modeling tools are quickly bringing them in house and are laying the foundation for their process, service and information modeling needs. Advanced Integration is usually a combination of Data as a Service tools (EII, MDM) as well as legacy host integration. Stage 3 can be quite expensive depending on what all you need. If you're starting from scratch, plan on $1 to 2.5 million for all of it.

Stage 4 is primarily focused on using the services. This includes client side platforms (AJAX, Web 2.0, Next-Gen Portal, Composite Application Tools, etc.) as well as the quality tools to verify the services. From a quality perspective, organizations are buying SOA testing platforms as well as asset governance tools. Comparatively, this area is a bit cheaper, plan on $250 to 800k.

3. SOA Governance Team - After the SOA strategy and roadmap have been defined, the next step is to create an organization that moves SOA forward. Activities include: Program Management, Service Portfolio Management, SOA Infrastructure Architect, SOA Infrastructure Administrator, Service Product Management and typically a couple SOA consultants to engage in active projects and review deliverables. Again, this is typically a combination of internal resources and external SOA consultants. Total combined costs are typically in the $.5 to 1 million range. Also, expect this team to provide significant guidance in the selection of the SOA Infrastructure and to mature the documents defined in the SOA Foundation Program.

4. EA Domain Analysis - All of the expenses on SOA planning, infrastructure and governance is wasted if you don't actually DO SERVICES. It surprises me how many organizations forget this point. Domain analysis is typically performed by enterprise architects who have a strong background in process modeling and service design. They pick a domain area (Sales, Supply Chain, etc.) and perform Process Reengineering, Process Modeling, Service Identification, Service Analysis and Composite Application Requirements Gathering. It is common for these activities to be driven by a Global Process Reeningering effort, or by Application Rationalization / Consolidation efforts. These efforts vary significantly in size, but rarely can you analyze an enterprise domain for under $300k. It is typical for an organization to be analyzing multiple domains in parallel. Most EA teams I've met with are already fully utilized, have minimal budget and don't have time to perform this work. Plan on either adding permanent members to the team or bringing in on-demand external resources.

5. SOA Training and Change Management - Unfortunately, we're not born with SOA skills, we must be trained. Managers, analysts, architects, developers, QA professionals and operational support personnel must all be trained in their piece of the solution. Even after training, you'll find that some people didn't make the change (Silo Oriented Architects), and you will need to provide some degree of change management to either get them on the right page or get them out of the way. Most organizations that we're dealing with are sending 100-300 people to training and are sending IT leaders to conferences. I'd plan on $150k to 400k for getting the teams up to speed.

5. SOA Build and Integration Teams - As organizations continue business as usual, they are constantly bringing in new packaged applications as well as building new systems for their business customers. Going forward, these systems will be sent through the SOA Governance Center. In some cases, the Governance team will determine that they shouldn't be services and will pass them through. In other cases, the systems will be required to adhere to the Governance standards. Packaged applications will often require 'service enablement' and 'service oriented integrations'. New systems will require 'SO-analysis, design, construction and testing'. If you think that your offshore teams will be building your first services, well, you're probably wrong. Service design and construction, like anything new, will most likely be done by in-house teams or on-shore development centers. After the art of SOA Build and Integration is turned into more of a repeatable discipline, you should strongly consider moving this to your favorite commodity development center. For planning purposes, I recommend that you first get a non-service orieneted cost using your internal estimating scheme. Then add on an extra 20-35% to turn the the software into hardened, reusable shared services.

I've said it before and I'll say it again... SOA Transformations cost big bucks, take years to complete but in the end are worth the investment. I hope that this off-the-cuff analysis is valuable to you. Again, it is impossible for me to provide anyone with accurate advice without knowing their situation. I'm currently running around the country helping large organizations put together their precise 2008 SOA budgets. If you need an extra hand looking at your situation - feel free to reach out to me: e-mail me

Terry White on SOA

2007-09-19T19:59:00.000-05:00

I just caught a great comment that Terry White, EDS Fellow, left:

A good approach is to develop a taxonomy that is easily understood by all involved. We recently developed an Enterprise Architecture that was SOA based. It didn’t really look that much different from any other Enterprise Architecture that is modular and effectively layered.

In developing this architecture we involved people from both the IT and business areas of the company. When we first started the context was purely business process – re-engineering and decomposition. We moved them to a services view for looking at the business processes. Then we added the IT “application” details and show how business services would be supported by application services. We decomposed the existing application landscape and re-ordered it to align with the business services.

Business processes have owners, or people responsible to manage them as assets. Applications have owners. Now the move is for services, both business and IT application, to have owners and be managed more like products. One potential problem I noticed is that management tends to count the number of applications supporting their business and work to reduce that number. When we move to services and managing services, there is an automatic increase in things you are managing because before these were all just application functions buried within the application. Aggregating the services can help but it is a bit of a mind shift.

I couldn't agree more. This is very similar the process that we do at MomentumSI. I'm glad to hear that others are having success as well.

Using SOA to Align I.T. with Itself

2007-09-13T09:49:00.000-05:00

After much consideration, I've come to the conclusion that SOA is best suited to facilitate "I.T. and I.T. Alignment" (not Business and I.T. Alignment). That is to say, SOA (from an enterprise architecture perspective) is better suited to align internal I.T. efforts with other internal I.T. efforts. This might sound like common sense (and hopefully it is), but SOA is fundamentally about sharing common logic and data while facilitating accurate and complete client side consumptions.

I had a conversation with a gentleman the other day. I'll paraphrase his comments... he asked me to imagine an enterprise without computers or software systems. Instead, it had one Filing Room that people went to when they needed to store or retrieve data.

At the front of the Filing Room were people working the Service Counter to fulfill your requests. In the back of the room were Filing Clerks who kept the filing system organized.

In this model it was assumed that the people in the Filing Room did a good job of organizing their files, as to ensure that when a customer asked for "all customers", they didn't have to go to 5 different Filing Cabinets. It was the responsibility of the Filing Clerk to facilitate Master File Management. The Chief Filing Officer was responsible for making sure that the Filing Cabinets stayed organized and on occasion were reorganized.

The "Service" in SOA is the new filing cabinet. Our SOA Governance Teams will work the front counter taking requests and also verify that they filing clerks do their job correctly. They must ensure that the portfolio of filing cabinets stay organized and avoid duplicate filing systems. And ultimately, the CIO must be held responsible for the state of the Filing Room.

SOA is not a holistic EA framework, but it will provide the taxonomy and organizational structure to become the foundation for a single enterprise system of services.

SOA Consolidation - Updates

2007-07-21T14:08:00.001-05:00

I finally updated the SOA consolidation list:
http://www.momentumsi.com/SOA_Acquisitions.html

The clock is ticking on some of the smaller firms...

Software M&A Rumors

2007-07-14T08:08:00.000-05:00

I noticed two items of importance this week:
1. The Dow hits an all time high
2. Rumors of VERY LARGE M&A activity in the I.T. space are off the chart

I'll avoid perpetuating any of the rumors that are going around; talk is cheap. What seems evident is that both I.T. buyers and providers seem bullish on the idea of massive consolidation. Our industry has been in constant flux for a long time, exhibiting all the signs of an immature industry.

Despite our strong economy, many of the preeminent brands continue to bleed money while failing to create a differentiated product portfolio. The buying community has rewarded platform commoditization and 'design to standardization'. Cloners and imitators have shown their ability to rapidly re-create innovator's products using open source and competitive pricing models.

Most large ISV's have already executed on a significant number of acquisitions and have had to become experts in integrating their own products. As their ability to refactor, integrate and generally absorb new products grows, so does their desire to capture additional market share through acquisition.

The unanswered question is where is this all leading? I'm confident that we'll see a few models emerge. However, the one that seems obvious is the one-stop shop. One company provides hardware, software infrastructure, packaged applications and professional services. With the growing popularity of SaaS, we can anticipate that multi-tenant hosted application support will also be a key element. The other trend that can not be discounted is the use of offshore resources to provide low cost labor.

I believe that massive consolidation is both inevitable and necessary. Information and technology providers must reach the next level maturity. They must be able to provide end-to-end solutions and take full responsibility.

Service Lead Management

2007-06-25T10:00:00.000-05:00

When a customer or prospect inquires about a product or service we call this a "lead". Typically the information given in the initial call is not enough to understand what the customer hoped to achieve. Therefore it is necessary to go back and talk to them about their goals and plans. Today, modern software enables this process. For instance, if you go to the MomentumSI web site, I know you visited within 30 seconds. Via our web marketing automation system (and a network of cookies), I am informed about visits.

A similar system is needed in the SOA world. The SOA registry / repository must be viewed first and foremost as a shopping catalog and should employ modern techniques for capturing visits, searches and users. The service librarian must view all hits to the catalog as "leads". The process of following up on these leads is considered "Service Lead Management".

Here is a sample follow up
- Who are you? What department? What project?
- What kind of service were you looking for? (let me help you look)
- Do you want me to tell you more about Service X? (you found one)
- We don't have Service X. Do you want me to talk to the Service Portfolio Manager about adding one to the Shared Services Group?
- etc.

Remember, when you first get your Service Catalog going, it will be EMPTY. The goal is to find out what people need and to determine if they are good candidates for shared services. Kill "Empty Registry Syndrome"!

Using the Zachmann Framework to Manage Schneider Oriented Architecture

2007-06-18T07:02:00.000-05:00

If you haven't seen Bill Zachmann's attempt to throw water on the SOA fire, it is worth a read.

Bill basically says that SOA has been overhyped, is the same-ole architecture and that it is like CORBA, etc. He goes on to say that the new thing is XML and you can already do that in Microsoft .Net. He also states that, " SOA is a matter of good, modular, object-oriented design..." No Bill, SOA is a matter of good Service Oriented Design.

I'm at a loss for words. I can understand his distaste for the amount of attention that SOA is getting. He actually states that he doesn't like: "IT guru firms that peddle high-priced snake oil as "expert advice" and use high-sounding, yet vague and obscure terminology to cloak the utter banality and limited practical value of what -they're saying." My God Bill - have you looked in the mirror? At least we didn't name it after ourselves!!!

This article, written in the Microsoft Redmond Journal, is the biggest bunch of shit I've seen written since Nicholas Carr put pen to paper.

Advanced SOA Governance - Negotiation

2007-06-07T13:50:00.000-05:00

As we split our monolithic applications into right-sized units of work (services) and pull them back together again with composite applications, we realize that the sharing of assets introduces benefit (subsidized costs, consistent logic, single source of the truth, etc.) but it also introduces new pain, namely change management.

In the last few years, most large enterprises have moved to some variation of the three tiered architecture. Here, the system typically has 4 or 5 layers or tiers:

The stateful interactions that monitor the business processes have been factored out of the business logic, and distributed joins (EII) have become first-order citizens of the architecture.

More advanced Service Oriented Enterprises have adopted this tiering model as the heart of their technical service taxonomy.

This diagram depicts the relationship that services have with the composite applications that consume them. As you can see, some services are required by more than one composite application. This is a core feature of SOA - the sharing of assets across the enterprise to solutions that need them.

Modern SOA Governance practices focus on passing the 'service baton' from one group to the next while verifying that best practices are adhered to and that the baton remains in motion. This movement throughout the software life cycle is managed according to new service/operation requests as well as for change requests. The Service Life Cycle Governance and the Evolution Governance become the primary framework for monitoring change.

The needs (features and functions) of a service will vary by the consuming application. The teams that deal with each software life cycle will need to be aware of the multiple masters (or owners) that they must serve. And on occasion, we can expect that the masters will have disagreements. They will disagree on what it should do, who pays for it, when it should be rolled out, etc. These questions are at the heart of SOA Governance!

As you can see, consuming applications need to approve, prioritize, fund, plan and communicate the changes to shared services. The Service Oriented Enterprise realizes that the services are an enterprise asset, shared for the benefit of the organization. And each business or process owner will, and should, argue for their needs. Ultimately, the group needs to come together to resolve the differences.

SOA will force more occasions where departments and business units will need to find a common ground. I.T. shops have had the need to negotiate for shared infrastructure in the past. If you move forward with SOA, this activity increases significantly. There are no magic answers to SOA Governance. My only recommendation is to make sure that the tools, processes, roles and committees are in place to make the negotiation process as efficient as possible. Said another way, a competitive advantage for the Service Oriented Enterprise is the ability to efficiently negotiate differences and take action.

SOA is an I.T. Strategy

2007-06-01T06:51:00.000-05:00

I've read some stuff lately where people have called SOA a "business strategy". I'll go on a limb and estimate that in 99.99% of the cases where SOA is being used, it isn't a business strategy but rather an I.T. strategy. And you know what? That's ok. There is absolutely nothing wrong with acknowledging that SOA is about making I.T. better. The better I.T. works, the better it can serve the business.

The notion that everything I.T. does is "about the business" is just a little bit silly. Sure, at some secondary or tertiary level it's all about the shareholder. But let's get real... buying rackmount servers is about a more efficient I.T.; standardizing on one or two operating systems is about creating a more efficient I.T - - and sharing common application logic and data across the enterprise is about creating a more efficient I.T.

SOA is first and foremost an I.T. strategy. If you need money from the business to fund SOA, then talk to them using words they understand. But don't kid yourself - this is an I.T. problem and you have to clean up your own mess. If you've created a silo-oriented, sphaghetti integrated, inefficient architecture that slows down your ability to server your internal customers then it's your problem to clean up the mess. Whatever you do, do not insult the business by telling them that you've got a new business strategy called SOA.

Did the SOA Community Fail You?

2007-05-30T07:12:00.000-05:00

Yesterday, I had an interesting discussion with another SOA veteran on the subject of barriers to successful SOA. The conversation led to a fairly obvious conclusion that several parties aren't pulling their weight. The parties we discussed included: the Venture Capital Community, Standards Bodies, I.T. Analysts, the Press, the Academics, SOA Infrastructure Vendors, Consulting and Training Vendors, Packaged Application Vendors and Design Tooling Vendors. Here is my personal view on their success:

Venture Capital Community
The VC's did their job. They invested early; many of the investments were made in the days following 9-11 which was a tough time for anyone to invest, but they did it. And when they did, they funded deep - but not too deep. It wasn't crazy dot-com investment. As their investments grew up, they did a good job helping them to find parent companies. On a scale of 1 - 10 stars, I'll give the VC a solid 10 on SOA.

Standards Bodies
Many will argue that a key enabler of SOA was the creation of Web Service standards. The creation of SOAP, WSDL, UDDI and the WS-* stack has been both a blessing and a curse. Often the concepts of SOA are tied to implementations and the limitations of those implementations are then attached to SOA. This is a tough one for me to score. IMHO, the real standards work was done behind closed doors at IBM and Microsoft and then handed to the standard bodies for cleaning and revising. In some cases the standards bodies did more harm then good. Overall, I'll give them 6 stars.

I.T. Analysts
The I.T. analysts got on the SOA bandwagon pretty early. Overall they did a good job of covering an extremely broad subject. Boutiques like ZapThink, Burton Group, CBDI and Macehiter Ward-Dutton showed precision analysis and deep insight. While Gartner, Forrester and IDC all did a sincere job covering the space, their insights often lagged the blogs and mainstream media. I'm giving the I.T. Analysts 8 stars.

The Press
The press has been overly generous on the topic of SOA. Most of the large publications have a regular column on SOA and even an SOA blog. Some, like Infoworld, have even held SOA conferences. I think that the reporting has been fair, but often light. Maybe I'm too old, but I remember back when magazines would do more lab testing and comparing products. This just doesn't seem to happen very often anymore. Now, we vote on 'hot products'; how lame is that? If the press would bring back the labs (or more case studies), I'd give them a solid 10. But lacking this I have to give them a 7.

The Academics
I have no expectations for the academics to do anything. And in my opinion they haven't. They're just where I like them. Hence, they get a solid 10.

SOA Infrastructure Vendors
Infrastructure such as: XML Appliances, ESB's, Orchestration Engines, Smart Intermediaries, Web Service Management/Monitoring, Registries and Repositories became mainstream in the SOA era. For the most part, I was impressed with the ingenuity that the startup's demonstrated in forging their products. And even some of the big guys like IBM showed that they have game. I'd love to give these guys a solid 10 but I can't. It's because of the big lie. These vendors have told customers that their products work with each other and for the most part they don't. HP/Systinet promoted the Governance Interoperability Framework as a solution, but in my personal opinion, they completely failed to deliver. And the rest of the vendors have sat on their lazy asses acting like the problem doesn't exist. I'm giving them 7 stars. How ironic is it that SOA vendors lose 3 stars for having a lack of interoperability?

Consulting and Training Vendors
As a consulting and training vendor - this is a hard one to do. There are a few companies like MomentumSI that I believe are doing a good job of helping organizations make the transition to SOA. The problem is that there are only a few doing a good job and a very large percentage who are not. Consulting companies should be able to quickly deliver SOA best practices, methodology adjustments, job descriptions, project mentoring and deep knowledge on tough subjects like governance and change management. For the most part, the big guys (Accenture, BearingPoint, CSC, EDS, SAIC, WiPro, etc.) are still very immature. There are a few notable exceptions, in my opinion Infosys, CGEY and IBM Global Services are all making sincere attempts to build out their SOA competency, with IBM leading the pack. I'm going to have to give this category 6 stars.

Packaged Application Vendors
First, as long as this category is called 'packaged apps' and not 'packaged services' I'm probably going to bash it. I remember back when we used to talk about the end of 'big bang implementations'. This was the idea that you wouldn't spend 3 years installing and configuring a huge monolithic application like SAP. Instead, you'd do it one service at a time. Ha! The packaged app companies were quick to get on the SOA bandwagon. They realized that this paradigm offered a genuine threat to their model and were quick to say that they embraced it. The question is, "embraced what?" That they'd web service enable BAPI's and hide the ABAP code? You've gotta be joking? And there is a reason why people refer to the Oracle strategy as conFUSION. Without a doubt, the best thing that these guys have done is made promises about SOA indicating to customers that SOA is the future. Yes, because of their marketing efforts I am forced to give this category 2 stars.

Design Tooling Vendors
In every paradigm change, we require new tools to facilitate the new architecture. In past generations we saw wonderful products like Rational Rose, ER/Win and TogetherJ rise to the occasion. I am so disappointed in the lack of progress in this category. The EA modeling vendors have demonstrated 'pompous ignorance' and failed to deliver anything resembling modern SOA tooling. Notational standards for service models remain in their infancy. I firmly believe that the lack of SOA modeling standards and tools will defer the adoption cycle. This category deserves zero stars.

Now the burden is on the buyer. Did the SOA Community fail you or will you fail it? It is the responsibility of every buyer to push the community to do a better job. Money talks.

Talking to the Business about SOA

2007-05-20T08:22:00.000-05:00

I often recieve the question, "How do you go about talking to the business about SOA?"

Here's my advice. Don’t lump all non-I.T. functions into a single bucket called the business! We must know our audience and create a message that they can relate to.

FINANCE
There are many definitions of SOA, but my favorite is to describe SOA as an 'I.T. economic model focused on cost savings through increased utilization of existing enterprise assets'. First and foremost, shared services should be viewed as a subsidized and highly leveraged asset that lends itself to supply and demand economics.

From a slightly different perspective, we can say, “SOA is an I.T. investment model that allows you to view your enterprise I.T. assets from a portfolio perspective enabling precision investing.” Here, the focus is on the granularity of the investment. By breaking large, monolithic investments into smaller units we have increased our ability to specify and evaluate individual investments.

PROCUREMENT
In the past, it was difficult for I.T. to divide up large systems into smaller units and have each smaller unit sourced individually. In order to do this successfully, you need to be able to specify how they will reintegrate by using common standards. Recently, I.T. has made a significant advance in these standards and we are now able to employ a multi-sourcing strategy for very large systems. In fact, buying large pre-integrated systems that are proprietary is now considered a worst practice.

LEGAL
I.T. has finally learned that we need to specific software requirements using precise terminology. We’re now using “digital contracts” to specify the functionality of the systems that we will build or buy. SOA makes I.T. focus on knocking out the contract before we start doing the work. This will allow us to hold internal and external parties accountable for their work products. We’ve also adopted a model for incorporating Service Level Agreements so that on-going satisfaction can be reviewed.

CORPORATE DEVELOPMENT & GROWTH
Corporate development groups must evaluate the risk and costs of performing acquisitions. Integrating the I.T. systems of acquired companies is generally considered a high risk and a costly endeavor. SOA enables the newly acquired systems to be leveraged in a shorter time frame facilitating business and system integration. In essence, SOA is the I.T. strategy for enabling mergers and acquisitions.

The assumption here is that the I.T. department is identifying ‘master services’ and ‘master data’. Newly acquired I.T. systems are rolled underneath these services.

PROCESS IMPROVEMENT
Most mature SOA programs have adopted a ‘process driven’ approach to business and service analysis. By analyzing business capabilities, processes and activities, SOA services and operations can be discovered. This enables I.T. to promote (and often fund) the use of formal process analysis. In addition, since services have been broken into smaller units of work, we now have greater visibility into each unit. This is a key enabler of business process management and monitoring. Now, process improvement specialists can utilize a ‘process driven, service oriented’ approach to monitoring the business and recommending improvements.

THE P&L OWNERS
At the end of the day, someone owns the P&L. This could be the CEO or a business unit head. Organizational leaders need to hear:
- You have a new I.T. strategy that benefits the entire company.
- The strategy can be successful through incremental investments but it will require an initial out-of-the-gate investment.
- If successful, you’ll be able to significantly reduce I.T. costs without sacrificing results.
- The entire I.T. and software industry is heading down this path, so eventually we’re going to be impacted by this software model. The only question is do we want to invest now to get ahead of the curve.

Unlike previous I.T. paradigms, SOA is not about changing out your hardware or creating a new ubiquitous user interface. For the most part, SOA is invisible to the users. This paradigm is harder to explain than others but if done correctly, it's easier to justify.

Try to avoid lumping all non-I.T. personnel into a unit called, “the business”; it’s not ‘us and them’. Know your audience and what they understand and value. SOA is a complex model that has many advantages. The key is to know which advantage to pull out and when!

Composite Service Analysis

2007-03-06T19:50:00.000-05:00

A common scenario that I see is the case where an organization wants to build a new application with the intent of 'finding services' in the implementation of the project. This is common because current I.T. governance favors 'project funding'. This begs the question, what does the process look like for capturing requirements for a silo-oriented application in a service oriented, mashup, process driven world?

First and foremost, we must recognize that we are not in the object oriented world where we captured requirements as if there were only one type of logic. We live in a world of domain specific languages and self describing interfaces. Capturing requirements as if we didn't know that kind of input our downstream partners need is just plain silly (and obsolete).
A simplified view of the new process looks like this:

Today, we leverage our downstream partners (UI, design, architecture) to help drive full and useful requirements. The analyst leverages the UI/Mashup/Composite specialist to create mockups, prototypes and even working user interfaces. The UI's are then presented back to the business users to help validate the requirements. UI's typically are good at conveying a single users role in a larger process but fail to paint the big picture. To bring breadth to the table, the analyst will call on a process and integration specialists to clarify process steps, human workflow and even the flow of information between legacy systems. In most cases, the 'composite services', 'integrations' or 'process logic' calls atomic services. This is where fine grained business rules, calculations and constraints are housed. The anlayst will leverage the skills of a Service Designer to stub out or mock up an interface (WSDL) along with a dummy implementation.

Our goal is to use a combination of rapid prototyping and extreme programming to quickly create releases that can be presented to the business community for feedback. Modern analysis embraces the following key concepts:
1. It acknowledges that some functions will be shared (as services)
2. It embraces various types of logic (presentation, process, data, etc.) and documents those requirements in a manner that makes sense to the people who have to build it
3. It leverages the techniques we've learned in spiral, iterative, RAD and agile methods limit risk and involve the user

Remember - SOA is only a piece of the puzzle.

Service Domain Analysis

2007-03-02T08:09:00.000-05:00

Today, most organizations have someone who owns an application. The ownership is often split between two areas. One group will own the budget and control of features while the other group owns engineering and staffing decisions. In the service oriented world this will likely stay the same. One potential difference is the scope of ownership. With the popularity of mashups and composite applications new customer facing systems are being designed around the needs of the user rather than around a specific domain. Many of these systems are 'process oriented' and cross several business domains. This makes it hard to determine who owns it.

In a service oriented enterprise, you have a 'owners' of services and 'owners' of composite applications. The service owners are domain focused (manufacturing, sales, etc.) while the composite application owners will be process (or problem) focused. Service owners focus on creating consistent logic and having 'a single version of the truth'. Composite app/mashup owners work with the end users to understand their day-to-day computing needs. Their job is to deliver the right information at the right time to their users. They leverage the service teams to provide the right information.

Today, many organizations have not split their groups into 'service teams' and 'client teams'. This typically requires a re-org and as I've mentioned before is a SLOW process. In the meantime, I'm suggesting the following process:

In this scenario we have a project analyst who is working with a business user to determine their needs. They will employ a variety of techniques to uncover the requirements. The analyst will often be trained in rapid prototyping (Web 2.0 style) or work with others with that skill set. They will also use Service Oriented Analysis techniques to identify potential services and scope them out. The analyst will document their findings in a form different than legacy silo systems. They will use a combination of Service Cases and Operation Cases. These cases are used to fully describe that portion of the system which would be shared and controlled as a service. The portfolio analyst works with the project analyst to verify that the candidates are valid (reusable, distinct, etc.) and create a placeholder for the service, know as a Slot or a Service Slot.

There are a variety of techniques for performing Service Oriented Analysis. At MomentumSI, we utilize a set of practices known as Harmony™. In the coming months, we'll be discussing these practices in detail.

Starter SOA, Part 3

2007-03-01T08:07:00.000-05:00

So far we've discussed a few concepts that organizations can take on without too much pain:
1. Get the project disciplines to talk with the enterprise guys about SOA
2. Create a SOA Steering Committee to start a dialogue between enterprise disciplines

Number 3 is a bit harder for some organizations. It deals with money, or the lack thereof. The heart of the problem is that the funding process in most I.T. organizations still doesn't facilitate the SOA paradigm. In the past, systems were generally funded as a standalone 'application' that had one owner. Going forward, funding will be broken into:
- Shared Services
- Composite Apps / Clients
- Non-Shared Services (services with one consumer/client; temporarily dedicated)
- Shared infrastructure (reg/rep/esb/wsm/etc.)
- Non-Shared Infrastructure (service container, app server, etc.)

Within each of the aforementioned areas, the dollars will be broken down by the various support groups (engineering, operations, etc.)

The SOA Steering Committee must be capable of recommending financial models that support the future paradigm. Most organizations already have some mechanism to make a request for shared investments. The SOA initiatives must work together to create a standard model for joint investment as it deals with services. Initially, the request looks like an exception to the rule but after time shared investment will become the norm and non-shared investments will be critically reviewed.

Starter SOA, Part 2

2007-02-28T10:47:00.000-05:00

Many organizations seem to hit the brick wall on SOA when one of four things happen:
1. Their isn't a critical mass of SOA enthusiasts
2. No one will pay for SOA infrastructure
3. No model for funding or governing shared services exists
4. Doing SOA right requires a reorganization of I.T.

Simple SOA is primarily directed at the 4th problem. Most I.T organizations have groups that "own applications" or dare I say, they "own monolithic, tightly coupled applications" - they don't own "shared services" (business or technical). Doing SOA right usually requires a new I.T. reporting structure but this can be a SLOW process.

So, the second concept I'm promoting is the use of an SOA Steering Committee.

As I mentioned in my last post, an early (and easy) step to take is having project teams leveraging the enterprise disciplines. Application architects should be working with their counterparts in EA, etc. Once the project teams begin communicating with the enterprise disciplines, the natural progression is to get all of the enterprise disciplines talking to each other.

The agenda for a typical SOA Steering Committee is:
10 minutes - The status of the SOA Roadmap: what is done and what isn't
20 minutes - A review of any new artifacts or SOA deliverables
20 minutes - A discussion on problems that the project teams are encountering
10 minutes - Recommended changes to the SOA Roadmap

It's a simple meeting that is cross-discipline. The team uses a program plan (SOA Roadmap) to keep a list of initiatives visible. The roadmap remains 'agile' and reacts to findings in the field. A fundamental goal of the team is to address issues that pop up quickly and to take decisions back to their respective teams.

In the beginning the team will likely meet often (every 2-4 weeks) and as the program matures the team will probably meet quarterly or until stabilization is realized.

Starter SOA

2007-02-27T11:24:00.000-05:00

Many large organizations are trying to figure out their new processes around SOA. I've been involved in a handful of these efforts. Typically this will involve a cross-discipline team who reviews current processes (RUP, ITIL, COBIT, etc.) to determine what must change in a service oriented world. Although I fully support this model, I've also noticed that this is a LONG and difficult road. This got me thinking... how do you do "starter SOA"?

Of course, I'm less interested in the technology/architecture side of SOA and I'm more interested in the people/process side. We have literally hundreds of modified SDLC processes that depict changes to support SOA. It's great stuff. But I've determined that it is too much to convey the basic idea of what we need to get done. This led me to create a simple picture about what I believe is at the heart of the SOA issue.

The basic idea that I'm suggesting is this:

1. Don't roll out all of the updated disciplines at once - pick a few key disciplines and insert them into the process.

2. Attack three specific areas: Portfolio Mgmt, Enterprise Architecture and Information Management.

Again, this won't solve all of your SOA problems but it will solve many of the problems related to 'Master Service Management' and basic SOA Governance. As this simple process becomes part of the regular work stream you can begin bringing in more and more disciplines like Operations, BPM specialists, Shared Service Groups, etc. My primary warning is don't wait until you have ALL your ducks in a row to get started - it may never happen.