Designing Web Service API's

A while back I blogged that we need to learn lessons from the Salesforce.com web services API... the implication that I made was that they did it wrong. Since then, others have poked at it as well:
CBDI
Phil Windley

Recently I had to design some web service api's for various types of services. One client required a read/only ws api that proxied data from their database. In addition, they wanted all insert/update/delete to go through the business tier.

My design turned out interesting - for the WS proxy api to the database, it looked very close to the SF.com api.
read( xyz)

For the ws api for the business services, it looked very much like Amazon:
AddEmployeeRequest( xyz)

Different services will require different invocation and granularity models. I think the SF.com got the api correct if you merely want to front-end a database. It is a bad design if you want to front-end business services. And the vice-versa is true with the Amazon way.

I'm going to try to be more careful when I blog service oriented designs... they are obviously context sensitive. One mistake that I see made is people saying that web services should be designed as 'course grained'. Well, you sure need course grained if you're going across a network with latency, but 'fine-grained' is just fine if you know that all your calls are In-Proc. Which, is an out-of-the-box function of the Microsoft WSE-2. My point is that we (bloggers) must be careful not to report 'generic' solutions without putting them in context - I'm as guilty as the next guy.

Job Openings - Web Service Consultants

Momentum Software has recently opened several new positions in our Web Services Practice:

Title: Web Services Architect
The Web Services Architect is responsible for:
- Translating business requirements into technology solution
- Designing the service network (security, directory, queues, management, adaptors, routers, balancers, etc.)
- Identifying vendor solutions
- Leading small teams of implementation consultants
- Some pre-sales support activities, including estimating jobs and candidate architectures for proposals
- Knowledge transfer to client teams

The successful candidate will have:
- 10+ years experience in software, 4+ years in architecture
- Very strong background in WS space (primary specs from WS-I, OASIS and W3C)
- Strong background in J2EE, .Net or both
- Strong background in distributed computing (CORBA, DCE, etc.)
- Strong background in non-functional requirements (quality attributes)
- Ability to document architectures
- Great interpersonal and communications skills (written and verbal)
- Able to travel within U.S. greater than 70%

Title: Web Services Consultant
The Web Services Consultant is primarily responsible for implementing service-based solutions. Activities include:
- Consulting on application and enterprise re-designs
- Front ending legacy applications with web services
- Designing and implementing new services, including service frameworks
- Coding JAXM, JAX-RPC, JAXB, .Net IIS Web Methods, GXA & WSE (Java, C# or both)
- Designing XML Schemas, schema to Bus Obj maps, schema to Er maps

The successful candidate will have:
- 4+ years experience in software, 1+ years in WS related activities
- Strong background in J2EE, .Net or both
- Very familiar with base technologies (XML, SOAP, WSDL, UDDI, JAX*, WS-*)
- Great interpersonal and communications skills (written and verbal)
- Able to travel within U.S. greater than 70%
- Orchestration development is a plus

These are new positions - note that traveling is mandatory. Please indicate your desired role (Architect or Consultant). Along with your resume indicate your compensation requirements, citizen status and ability to start. Send your resume to : careers@momentumsoftware.com

Web Services Choreography Requirements 1.0

The W3C published their "requirements" document for web services choreography:
http://www.w3.org/TR/2003/WD-ws-chor-reqs-20030812/

This just might be the biggest piece of crap that I've ever had the misfortune of reading. I'm not just saying that because I think that the W3C should get out of the business of 'business process languages' - I mean it.

Seriously - look at the document. Remember it is a requirements document. I've read it twice now - for the mercy of spec readers everywhere - can someone please kill this working group???

The Mindreef Team

Currently, I do not use the Mindreef software.

I was considering using it so I checked out their home page. And, like many people, I went to the 'team' page to see if I knew any of the guys working on it. I didn't, but they had a great team picture:



Wow. These guys look like geeks. I mean that as a compliment. I am much more likely to download and use their software after previewing the team photo. This seems silly doesn't it? The fact is, we've all worked with great engineers - and generally speaking they... kind of look alike. Now, Graham Glass may be the exception to the rule:



Anyway, if you have a 'mindreef' looking team, it may be in your best interest to post the pictures :-)

I'll go drink some coffee now - I'm sure I have something slighly more insightful to say than just this...

WSEE

Anne Thomas Manes wrote a piece on a topic near & dear to me, see:

http://www.sys-con.com/webservices/articleprint.cfm?id=627

The only thing she didn't do was give the set of services a name... like... uh... Web Services Enterprise Edition (WSEE).

WS-I Basic Profile 1.0 has been approved

All -

I am pleased to announce that the WS-I Basic Profile 1.0 has been approved as final by the WS-I membership.
Congratulations to the Working Group for a job well done.

cmk

Apache Geronimo: A waste?

I don't follow the open source J2EE stuff very much.The decision for the Apache group to write yet-another J2EE implementation has me exhasuted.

I personally feel that the J2EE programming model is out dated. Yea... it was good in its day - but the service based model is here and J2EE is "last-gen". A platform architecture that is hard-baked to three tiers is not going to survive.
When I saw that Apache was going to re-write the entire J2EE stack, I felt really sad. It was so disappointing that a group would attempt to rewrite the same last-gen stuff that has already been written a dozen times over.

I just wish they would consider doing something NEW, rather than yet-another-impl.

Service Oriented versus Service Based

I found myself in a debate with some people around the term 'service oriented'. I fought that the term referred to the architectural pattern - that is, the triangle - producer, consumer, directory. My worthy opponent fought that most people that are building service based architectures are leaving out the directory. He went on to argue that if in the majority of the cases people were only using 2 of the 3 legs of the triangle, that we shouldn't refer to it as a 'service oriented' architecture. I agreed. We actually threw out the term Service Based to refer to a set of architectural patterns that revolve around services.

Thus:
'Service Oriented' refers to the triangle pattern.
'Service Based' refers to the set of patterns that leverage services (service oriented, pipe & filter, pipe-to-pipe, etc.)

It is clear that confusion still exists in this area... more work neeeded.

Java Pet Store as a BPEL Orchestration

Recently, I've been working on rewriting the classic "Java Pet Store" application as a BPEL orchestration. It has been a great exercise to understand how one might develop entire applications using a service based model. The application ties together a variety of services (Presentation Services, Logic Services, Data Manipulation Services, Data Persistence Services, etc.)

Now, to make the effort fun, I decided to use an all .Net orchestration engine (OpenStorm) and to utilize services written in Java (on the J2EE model). What I am finding is:
1. The granularity of the J2EE model is currently too fine grained to be mapped directly to a service based architecture
2. The J2EE programming model is "location aware", that is, it has been designed with 'tiers' in mind
3. The concept of "Location Transparency as an Aspect" is still in infancy
4. The UML Sequence Diagram may be used to create a candidate architecture for a service based application
5. MOF, or some other meta data facility MUST be a key component of the fabric

I'll dive into each of these items later.

Can it be done? Yes.
Will it be 'clean'? Hmmm... parts of it will, parts of it won't.
Will it be fast? I think it will be fast enough, but hopefully TSS won't benchmark it ;-)
Will it be agile? Absolutely.

Consulting in Web Services is up...

I don't need Gartner to tell me that more companies are buying consulting around web services - I just ask my sales team. And the news is good. More and more companies are calling us to kick off engagements.

The majority of what we are seeing right now is actually driven by the web service vendors. Companies that supply web service platforms, networks, security, transformation and management platforms have been asking us to help with client engagements. This is great news. It means that the product vendors are selling their products and the engagements are large enough to bring in hired guns.

We are also seeing new engagements where we are helping companies:
- Define a roadmap for becoming 'service oriented'
- Evaluate web service vendors & packages
- Integrate systems using web service technology
- Define corporate standards, policies and procedures around web services
- Educate internal staff on web service technologies

The most interesting area where we are getting involved is in helping companies create whole new methodologies around what we call a "Business Driven Architecture". Here, we discuss business strategy and measurements. This moves into identifying key business processes that they would like integrated / automated. We then spend time creating new artifacts to describe Digital Process Execution. From here, we create a service network to act as a substrate. Lastly, we 'service enable' legacy applications and orchestrate them to support the desired 'to be' business process.

It really feels good to give customers a BPM solution that is built the *right* way.

Invasive vs. Non-Invasive Architectures

must a fabric be invasive?
can bpel avoid it?

If we standardize the concerns and remedies and interfaces, can we commoditize the fabric?

The Service Fabric

Here is my first attempt at defining a Web Services Fabric:

Definitions
Concerns are the non-functional requirements (ilities if you will) - Scalability, Availability, etc.
Remedies are architectural components that attempt to resolve the concerns (clusters, etc.)

---
Every architecture has multiple components (database, presentation server, etc.)

Every architectural component has concerns (scalability, availability, security).

And every concern has at least one remedy (scalability is remedied by load balancing).

Every remedy has its own concerns (the load balancer must be secure).

And the concern to a remedy has at least one remedy (the load balancer will use secured authentication)

---
When the remedies to concerns leverage other remedies within the architectural circle, a consistent architectural fabric is created.

Service oriented architectures have the potential for creating such a fabric.

A service fabric is an integrated set of services that utilize each other to satisfy their architectural requirements.

Webify comes out of stealth

Webify, a company headed by Manoj Saxena (formerly of Exterprise; acquired by CommerceOne) has come out of stealth mode.

The company vision statement states:
"The promise of the Internet is an open e-business platform where companies can do business spontaneously with anyone, anywhere, anytime. Business Services Networks fulfill that vision.

We envision every company publishing standards based business services and processes that customers and business partners can discover and self-provision into their own business processes with a few clicks of a mouse. Companies will build on each other's services, creating new loosely coupled applications and industry-transforming, network-centric business models."

Webify is Austin based - and a neighbor of mine. I had a chance to speak with Manoj a few months back and I think he gets it. It is interesting to see that he is bypassing the 'chasm' theory though. I'll have to ask him why he is attacking FiServ, Health & MFG all at once.

It is also interesting to see how much they are basing their products around BPEL (and J2EE). I anticipate this to be a trend for some time to come. Webify has also had to harden the current IBM implementations to include early versions of reliability, security, isolation, etc.

I look forward to following their progress.

Blue Titan in WSM Fabric Space

I had heard of Blue Titan from their web advertisements but had lumped them into the 'web service management' space along with AmberPoint and friends. Their more recent product literature describes their offering in the 'web service fabric' space - which really is a distinct category.

According to the Blue Titan literature, the 'fabric' facilitates the following functions:
- Register web services
- Associate policies with services (security, management & monitoring)
- propagate policies to endpoints for enforcement
- Capture endpoint runtime execution information, report data
- Feed statistics from endpoints back into the process, repeat as needed

Now, I haven't used the product - but it looks like it sits on top of Web Logic and leverages an Oracle database. This means that it is more than just 'a set of libraries'. It has a library that a service designer would incorporate into their services and it has servers. It is a full blown closed-loop web service management enforcement product.

So, why would they call it a 'fabric'? My best guess is that unlike other products the service management substrate must be built into the software that you are building. It is not an add-on (servlet filter) or a bump on the network (promiscuous sniffer). By providing a library, they are able not only to listen and monitor, but also to control and resolve. Assuming I got the product concept right - it sounds like a good idea.

This raises an interesting question, "what is a service fabric?" I'll attempt to answer that later, for now I think it is safe to say that a fabric must be integrated into the code / services that the developer is writing. It isn't a passive bump or a container. It is an integrated library. Can the library depend on other well known servers / services (or must it be fully encapsulated)? In my opinion the answer is that it can (and often will) have outside dependencies that will be described via WSDL.

For now, I am going to call the Blue Titan offering a 'fabric based web service management platform'. This will distinguish them from the passive bumps and those that monitor via the run-time container (vm, app server, etc.). It is also clear that 'web service fabric' is too open-ended. The Blue Titan offering has some overlap with the TME GAIA, but I would not consider the two direct competitors.

Defining the ESB (Enterprise Service Bus)

The ESB or Enterprise Service Bus seems to be the acronym of the quarter. I've been debating some ways to describe the ESB, here are some of my attempts:

"It is a means for a message queue company (like Sonic and Spirit) to make it seem like they do web services when they really spent the majority of their time doing JMS based queues."

or

"It is a cool sounding buzz-phrase that will lure not-so-smart software developers towards a Java API that sounds like it has something to do with web services. On occassion, they will re-describe the JMS API via WSDL!!!"

or

"Queuing companies realized that the message queue was essential in an asynchronous, document based service oriented network - thus, they created a cool phrase that steals from three predecessors ("Enterprise" from J2EE, "Service" from SOA, and "Bus" from either CORBA or hardware buses)."

Just kidding of course... but I do think it is a dumb term. For goodness sake, A BUS? Has anyone ever looked at the topology of a service network? :-)

Web Services Package for Mathematica

See:
http://www.wolfram.com/news/webservices.html


May 29, 2003--Wolfram Research is inviting users to try the beta version of its new Web Services Package, which allows users to interact with web services in Mathematica. With Web Services Package 1.0, Mathematica users can dynamically access new data and functionality located on the network, creating an automatic extension to the Mathematica environment. The release of Web Services Package is yet another demonstration of Wolfram Research's commitment to providing freely available, cutting-edge technology and support to fulfill the needs of the Mathematica community.

Commentator comments on BPEL

Kendall Grant Clark made some comments on BPEL. I think he has some valid points around the fact that BPEL isn't the only player or final player in the Web Service Based Integration space. See:
http://webservices.xml.com/pub/a/ws/2003/07/08/ws-deviant.html

WSE 2.0 Technology Preview Released

See:
http://microsoft.com/downloads/details.aspx?FamilyId=21FB9B9A-C5F6-4C95-87B7-FC7AB49B3EDD&displaylang=en

"WSE 2.0 Technology Preview builds on the security, routing, and attachments capabilities with new features including a policy framework, enhanced security model, message-based programming model, and support for multiple hosting environments. "

Databases & SOA

Sean McGrath & John McDowall have touched on a subject that is important to me - the use of the dbms in the soa. Yes, one of my first jobs was working for an RDBMS vendor on the mainframe (Must Software - Nomad). Everything we did was based on the relational model. Everything COULD be done via a database (if we wanted to). Hell, you want two applications to share data?? Stick it in the database!!! We preached Boyce-Codd normal form, we used terms like "sargable predicates" and "shippable selects" - we were cool.

Well, we ended up misusing the database and creating ER-models that glued everything together. Later, the middleware people (CORBA, EJB) figured out how to front-end these meshed data-spaces with services, but they didn't actually un-screw the data normalization problem. The fact is that we use surrogate keys and auto-gen id's because they are efficient for the dbms and they are easier to use in our SQL. However, generally they are vendor specific and implementation specific. We all know that "Data Coupling" is the exposure of internal data to the external world, leading to the coupling of two distinct systems (failure to encapsulate internals). Want to see something funny? Take a look at the sForce web services api and see how they dealt with the issue. Ask yourself these questions:
1. Did they expose internal database Id's to the outside world?
2. Do they provide any queryable artifacts to access the constraints (e.g., I need a Customer record before I can insert a Customer Order record)
3. Do they provide ER-Diagrams (data view) or some new service view (activity / sequence) diagram?

After you look at these questions, you can begin to see some of the problems that will be surfacing in providing external access to internal data. The art of service enabling the db is still early - let's learn from sForce and advance the art.

Web Service Frameworks (WSF)

I have a strong interest in web service frameworks. In general, I am of the opinion that the WSF is where MS & IBM wil have their next major battle. So, they HAVE to agree on a certain sub-set of functionality (those concerns which must be remedied at the protcol level), but beyond protocols, the field is open and the battle will begin shortly.

Right now, Sun has a set of api's that they call J2EE (JMS, JNDI, etc.) They expose each of these api's in a "java only" format. Microsoft has a similar set of api's on the .Net side. In my opinion, we will quickly be moving to the exposure of technical architectural elements as web services (web services front ending LDAP, web services front ending databases, etc.) We will also see web services frameworks for horizontal domains (accounting, crm, etc.) and vertical domains (insurance, travel, etc.) Each of these horizontal and vertical frameworks will have to pick a substrate layer to work with.

One of the more interesting web service frameworks is GAIA from TME. I like the fact that TME knocked out a quick impl of a soap engine (glue) and are quickly moving into the substrate. They are looking at the world from a set of concerns (scalability, availability, deployability) and identifying remedies (balancing, resilience through redundancy, etc.) Many of the problems that GAIA takes on are concerns that either were done in proprietary hardware (cisco stuff) or concerns that were handled in a monolithic app server (availability through clustering). I am a huge fan of re-addressing these remedies via a consistent web services approach. Using a balancing web service, or a caching web service inside of an orchestration makes a lot of sense.

Other frameworks have popped up, but at a higher level. One such framework is the sForce web services api from salesforce.com. The sForce api is early. It provides some real basic services (insert, update, delete, change password, logon, etc.) You can really view this framework as a "data service" framework - and from this perspective, it does the job. Their choice to use a significant amount of abstract data types (records & maps) facilitates versioning and adding new data fields easily, but has the trade-off of potentially introducing typing errors at run-time. Sidenote: This framework looks a lot like the old MS My Services (aka Hailstorm).

Web service frameworks are now emerging. It shouldn't be long before MS & IBM throw one (or several) out. It will be nice to move beyond protocol crap (ws-boring, service substrate) and into value-add.