The SOA Manifesto: Business Value and Architecture First

The overloaded paradigm matrix of heterogeneous understandings of “what SOA really is” demanded a consolidation – given a plethora of different guidelines, principles and best practices. Even if you look at the closed Thomas Erl universe of SOA books it is a daunting task for the busy architect to grasp the essence and derive well-defined corridors for his project.

Thus, a group of SOA representatives from vendors gathered with analysts and book authors in Rotterdam last week and worked on the core values and rules for Service Oriented solutions. On Friday 23rd of Oct. they presented their result of hard, sometimes tough collaboration work at the end of the SOA Symposium which is an aggregation of the principles on the basis of a hard earned consensus of some of the main SOA thought leaders: The SOA Manifesto (www.soa-manifesto.org).

 

 

 

 

 

 

The debate has started

As you could have anticipated, a heated debate has started in blogosphere already with flames rising high – some pro, many very loudly against the manifesto as well as against SOA in general. Opposing many of the complaints in the posts, I think it is good that the manifesto was written by both vendor representatives and independent thought leaders: Both could not get their individual beliefs published but they needed to find a compromise – and from what I observed they worked very hard on finding that compromise – so unlike many say, the manifesto is not a vendor’s view! It states clearly you cannot buy SOA through purchasing an ESB! On the other hand, it is not a hard core REST proponents view as well. What I like is that the result is still not lame. For me, the following aspects bring real value by their emphasis on aspects that distinguish a successful SOA initiative from a flawed one.

The SOA Manifesto emphasises

diverse social and cultural aspects that should be respected. This notion demands thorough management of change

diverse solutions based on the context. I personally believe this is key and that along this notion the term SOA will diversify in the next years. Similar to the “forces” in good old object design pattern, the context should determine designs, tools and the degree of virtualization! The decision should be justified along criteria such as requirements for agility, reuse and criticality and the organizational level for which the solution is built: You might not need in ESB in some integration optimization for a local department. On cross departmental / enterprise level it could be well justified

business value first

understanding the conflict between the interest of the project manager and the interest of the strategic goals of the overall enterprise (the problem is that the latter often has no stakeholder!)

a strong architectural standpoint – not tools – at the center of successful service orientation

loose coupling and separation of concerns (oddly the manifesto is never directly mentioning both, but they are the underlying principles that shine through the manifesto).

What I miss is an even stronger argument that Service Orientation implies organizational change – Service Orientation is the antidote against silo thinking! This is hinted at in “Recognize that SOA ultimately demands change on many levels.” – unfortunately the levels (technical and organizational) are not mentioned.

The SOA Manifesto is another important starting point

The SOA Manifesto has the potential to frame a constructive discussion on how to shape the future of enterprise computing: SOA (or whatever we will name the ongoing evolution from objects to components to services to whatever, maybe services that have event based interfaces and are used by more and more agile processes) will further evolve and architects will learn to understand when best to apply which degree of loose coupling – we don’t need SOA and loose coupling everywhere! I see Service Orientation as an architectural pattern that only applies when certain forces are prevailing. Thus I appreciate that the manifesto does not ask to “SOAficate” the whole IT. Some applications are to stay untouched with all their point to point connections and it should be recognized that loose coupling comes with a price tag. 

A short discussion with the group

To delve a bit into the text and its making process, here are some thoughts that stem from discussions with my good friend Clemens Utschig-Utschig, as well as with Dirk Krafzig, Nicolai Josuttis, Dave Chappel – all ground braking book authors, Anne Thomas Manes – “SOA is dead – long live services” and some of the other authors. The conversation took place right after the presentation of the SOA Manifesto on Friday evening in the exhibition hall.

After reading the printed manifesto, I congratulated the group for the simple and easy to understand language and this compliment was well received, you could see that this was a major concern and goal. Yet, I said, for me the only sentence hard to grasp was the third value:

Intrinsic interoperability over custom integration

What actually is “intrinsic interoperability”?

I learned from our discussion that a service has intrinsic interoperability if it is easy to use, it’s semantic is clear and there is as little integration hassle (Adapters, Transformations etc.) as possible. Anne added that a REST based service has better intrinsic interoperability than a WSDL based service because the technology footprint is smaller. I would agree, but regard REST applicable in other context than WSDL, so an architectural decision should take non functional requirements into account as well.

To deeper understand intrinsic interoperability Clemens later emphasized the importance of a well established canonical data model that is used across services and further conventions that make the service easy to understand and use.

… and what is a service that needs custom integration?

Asked another way: What is a not so well-behaved service? Here, Anne dismissed extrinsic services that are not easily reusable: A typical example is a proprietary API, either of a packaged application or of a legacy system that is exposed through a Web Service layer. This design tightly couples consumers of services to implementation details and thus needs lots of further custom integration if a consumer needs that bit of functionality: Data must be mapped, implementation-level IDs must be resolved etc. Anne referred to such services on top of existing APIs as “lipstick on a pig”.

I like that metaphor as I regard a respective “right-click service” based on existing APIs as one of the most fundamental anti patterns in naïve “SOA” projects and as one of the main architectural root courses for failure of solutions in the long-term because it does nothing to prevent from a badly maintainable system landscape and lead to a tightly coupled design. This shows that with the sole application of WS-Standards nothing is achieved, you just have a new interface technology (WSDL, SOAP instead of CORBA, EJB, FTP or COBOL copy books) on top of the old, often, not always, messy systems. Such pseudo services, if directly exposed for public usage are the latest “Emperor’s New Clothes” – transparent “Ugly SOA” wardrobe on the still naked king.

A way to deal with API-Service is to understand them as “Private Services” – implementation details of Business Oriented “Public Services”, the latter focusing on the consumers and its needs, completely abstracting from underlying proprietary data schemes, programming languages, platforms etc. The trick is to never expose those details of Private Services, but translate them to the needs of the public interface behind a service facade. I think this “Public/Private Service Pattern” serves well the discussed value Intrinsic interoperability over custom integration.

And now: Best Wishes!

I wish for the SOA Manifesto to serve as a guiding foundation for many proposals and solutions in SOA space – similarly as the Agile Manifesto served beautifully well for many to start thinking Agile.

Further Reading

• An excellent start for digging deeper into the manifesto is
  Stefan Tilkov’s Comments on the Manifesto.
• Peer ACE Director Mike van Alst looks back at SOA Symposium and comments on the Manifesto
• Clemens Utschig-Utschig’s architectural and personal notes on the manifesto
• Peter Paul on the SOA Symposium, the exorcism of the evil SOA and the SOA Manifesto
• This critique from Jordan Braunstein is constructive and thoughtful
• Jürgen Kress gives an overview on presentations at the SOA Symposium and features a picture of the SOA Manifesto group in discussion
• In German: Deep analysis by book author Nicolai Josuttis
• In German: An interesting, very critical post from Ralf Westphal

OOW SOA Gems: Larry explains SOA and BPM

In his keynote Larry Ellison nicely explains SOA in the context of Fusion Applications. His point is that customers do not want to replace exisiting apps, but integrate with them. He even used the word BPEL . He made a very interresting point: A packaged application is no longer closed, but through usage of SOA middleware at its core opens itself up to integrate internally with other custom or third party applications.

Larry as well showed a nice slide on the BPM architecture with processes orchestrating services. He used this to show that Fusion Application is not only customizable through integrating with other applications via SOA services but as well through changes at the business process layer.

My main conclusion is that with Fusion Apps a packaged application is a white box where they used to be black boxes. This can open up tremendous opportunities for integrators when they offer creating a custom application based on Fusion Application using BPEL, ADF, SCA etc. creating the UIs, workflows and processes the customer needs integrating with existing systems he has in place.

OOW SOA Gems: SCA is the J2EE of SOA

Here are some takeaways from Greg Pavlik’s and Clemens Utschig’s outstanding and inspiring presentation yesterday at Oracle Open World on key architectural aspects of Oracle SOA Suite 11.

Clemens and Greg spoke about many aspects from which I share some talking points on Service Component Architecture – SCA.
SCA is still in the standardization process, probably it will take until 2010 to be a final OASIS Standard. Greg made the analogy that SCA is to integration what J2EE was to Java programming. The key umbrella artifact in SCA, the “Composite” is analogous to an EAR in J2EE.

SCA Eventing

WSDL is a push based service interface and the SCA standard currently is built around classic WSDL based services; Thus, SCA has no publish subscribe (pub sub) functionality. Oracle realized this as a limitation. Thus, to broaden usage of SCA Oracle introduces events into SCA as another endpoint paradigm to the “Event Delivery Network” (EDN). I think this is a smart move – a lightweight means to communicate from applications with BPEL processes is requirement I often ran into at projects. To make sure events are part of the standard asap, Oracle works with the standard committee on “SCA Eventing”. Of course, customers can leave it out when they want to stick to the standard while it is evolving.

SCA Eventing and loose coupling

Designed the right way, SCA events can further enable decoupling/loose coupling when compared to WSDL based endpoints because events free from the need for explicit wiring between services: You just exchange lightweight events with a minimum of technical communication overhead. For example, a WSDL based service can have multiple WSDL parts, while events are limited to just one part.

SCA beyond “SOA Suite classic”

Clemens showed Java Spring exposed in a SCA composite and now Oracle looks to create SCA layers for the integration platform Tuxedo and for C based logic!

I think this is great because more and more SCA becomes the core logic exposure, easily exposing Tuxedo or C based pieces of logic as “first class citizens in SCA world” – and, if carefully exposed through well designed service interfaces – in a SOA world.

Clemens showed all concepts live, so you could feel the need in the room to open JDeveoloper and play with 11g.

Tips for your initial SOA project

I was asked for advice at a SOA project that starts soon based on Oracle middleware. This is what I came up with as my most critical things to consider at the start of a project. This list is by no means complete and you are welcome to add your top 5.

- It is all too easy to underestimate the efforts of creating a initial working prototype and the reference architecture and then run into large delays because things that seemed so easy just don’t work the way anticipated or the solution proves to be just too slow! So make sure you have the time for try and error until the reference architecture stabilizes and you have solutions for every critical aspect! If this is one of your first projects, don’t get into stress from the beginning due to delivery dates but take some weeks to get a solid reference implementation.

- It is amazing how some things on configuration side are much less intuitive and more complicated than anticipated. Examples for areas where you can spend days understanding the nitty gritty details are
- configuration to optimize performance
- asynch services
- timeouts
Thus it is advisable to have at least one very technical and experienced expert who has deep configuration skills in the team, either from the middleware vendor, from a system integrator or from your own team. Always strange things happen, especially in the beginning and you never know, when you are not very experienced, whether it is a programming mistake or a bug. Fast cycles of understanding, resolving, knowing how to search bugs and solutions in the documentation and fixing it is key.

– for skills in the team: Check you have this type of main skills: Architectural / design, technology and tool expertise.
Make sure you have advanced XML skills: XML Schema, WSDL, especially namespaces, XPath, BPEL. Everybody who is exposed to BPEL and XML is lost without deep understanding because of the untyped, declarative nature of XML things go wrong late after compilation and are hard to track down: “80% of the problems are namespace problems”

- Make performance tests on performance critical aspects early. XML is potentially large and can blow up memory and performance easily. XQuery and Database are means to deal with that. And, even way faster is good old STAX parsing. Again, it takes time to find the bottlenecks and feasible solutions.

- determine main architectural attributes such as the level of loose coupling and make sure everybody buys into them and you map designs to these attributes and guiding principles. This determines efforts and tools.

- don’t be too much in love with tools. Rather have less tools: The complexity rises with each tool and platform drastically and you might end kicking out tools that seemed fantastic but cause problems in the concert of collaborating tools. This is a hard earned lesson

- On change management and project culture many peers have written profoundly, this is important but too broad for this short checklist.

You want to sell SOA? Speak COBOL!

If we are lucky, we “SOA Architects” are faced with a rather modern deployment of existing applications, based on platforms such as Java, .Net or Oracle Forms. Here, the upgrade to a Service Oriented Architecture is a natural and gradual one, programmers and architects of the existing systems are used to think in terms of components, which are close (or predecessors) to services. Thus, you can quite easily sell SOA to the stakeholders of this type of environment as a mere “upgrade” to the latest technology. For example, establishing an automated workflow by introducing BPEL on top of “servicized” EJBs will provide quick wins. On developer and architect level you have people who are thinking in terms of interfaces, have XML skills and an interest in learning the next new thing. It is this type of environment we see SOA maturing and succeeding.

Yet, where SOA gets really interesting, in broad enterprise application landscapes, things are most often different. The Mighty Host is at the center of everything. Some years ago I read an assessment that 80% of the business critical code in the world’s large enterprises and government ITs is still written in COBOL. I don’t think this number has drastically shrinked since!

To approach host / COBOL developers with some SOA slides easily makes you look like some kind of freaky alien who speaks in front of humans: They know what they are doing because they are doing it that way for many many years – and it works!

The only question remains whether you are seen as a harmless alien, someone they can just ignore because he has not enough influence to actually change something. Or whether you are a recognized as a potential threat to the way IT always has has worked here. Then fear is what you will have to deal with. Many people in this environment – developers, architects, even business people live in the culture of job controls and COBOL copy books where, if you need some data from an other system you will find an easy ad hoc way. We have to deal with it: People in this environment don’t want change and will have very good points in why everything is good as it is.

Technically, SOA and mainframes can come together, the implementation culture, language and the needed skills are worlds apart.

To address this it is important to understand the audience you talk to, the culture, the language – and you should be able to speak this language. Ideally, you show them that you can solve problems in their way too, for example by creating some job control based solution. This way gaining trust, reducing alienation and establish a real relationship between people who understand each others way of thinking.

Then you can start to think about how to sell loose coupling to people who embrace point to point connections…

Meet Thomas Erl, Dirk Krafzig and many more @SOA Symposium in Rotterdam

I am looking forward to the SOA Symposium! We will give a preview together with Thomas Erl to the upcoming book “Next Generation SOA and meet David Chappel and Dirk Krafzig. This symposium features the creme de la creme in today’s SOA world, I am also happy that a hero of the 90ies, Grady Booch is there too! I am curious what he has to say on SOA…

Our Magazine on SOA is out (links with German content)

One week ago our SOA Magazin “SOA Spezial: Ready For Change” has been published. This is a compendium of a series of SOA articles that our little cross company group – Clemens Utschig, Torsten Winterberg, Berthold Maier, Bernd Trops and myself have written over the last year. These articles have been published in the German JavaMagazin every month. They deal with topics such as SOA Project Categories, Loose Coupling, some more technical such as Transactions and some more business oriented such as Change Management towards SOA (from our “guest author” Natsuko Hara). All articles have now been gathered into one nice magazine that you can buy at Newspaper shops throughout Germany, Austria and Switzerland:

SOA Spezial

 

 

 

 

 

 

 

This link contains a (German) interview with us authors:

http://it-republik.de/jaxenter/news/SOA-Spezial-Ready-For-Change-051440.html

and here is what Torsten writes about the mag:

Torstens blog entry on the mag

Now, whenever I am at a train station or airport, I look for our nice mag and position it in front of all others (Just kidding)

We will bring some to Oracle Open World, where you can have a look.

Software AG buys IDS Scheer

While it is great to celebrate events such as Oracle SOA Suite 11, the actual impulse to kick start this blog was a shock: Software AG buys IDS Scheer – the company that creates the business process / Enterprise Architecture (EA) platform “ARIS”. ARIS is as well the basis for Oracle’s cool Business Process Management (BPM) offering “BPA Suite”, with its unique approach to align business process models and executable (BPEL) process in a closed loop.

In June 2006 BPA Suite was still a vision and I had memorable discussions with the responsible managers at Oracle Headquarters on Oracle’s plans for BPM platform. I was at a EDS project at that time that investigated ways to close the business / IT gap through model – code round tripping.

IDS Scheer was under discussion for a takeover for long time, I had IBM, Oracle, SAP on my list, now, surprise surprise, Software AG…

I follow Paul Harmon who said in his book “Business Process Change” that BPM tools will be mature and able to fully deliver to their promises by 2015. It will be very interesting to see what Oracle will do now and until then.

What is needed is a tool that allows to navigate from high level, abstract, not automated process models down to models of executable processes and, from there, allows to drill into deployed processes and statistics on their instances and their behavior. And from EA models that depict the system landscape down to the actual physical systems to get a “life” picture of your overall application landscape.

SCA and Loose Coupling

In Oracle SOA Suite 11 it is easy to put everything into a SCA Composit that makes a SOA solution – BPEL processes, integration logic, business rules, human interaction – you name it.

The question architects should consider is whether conceptually all these belong tightly bundled together.

The answer depends on the desired degree of loose coupling.

If you want to add a SOA-Layer on top of a fixed application it may be feasible to wire integration aspects in the new “Mediators” with business process logic in BPEL. This tightly coupled design may be fine when the integrated application (or a set of applications) will stay for the foreseeable future.

Yet, how would a design look like if you want to achieve loose coupling, where processes orchestrate Business Services whose implementation is easily exchangeable? Then the business aspects in processes and business rules should be isolated from integration logic that is close to end systems.

Then it becomes possible to exchange the underlying applications and the affected integration logic with no changes to the orchestrating process logic in your business related SCA Composites.

Such integration logic could translate the exchanged data into the aplication data dialect and knows their protocols. Following the separation of concern principle such logic is very different to the more business related aspects in highler level BPEL business processes and and should be deployed in distinct chunks. This means business aspects and low level technical aspects do not belong in the same SCA composite.

Following this distinction, for integration aspects you could create own SCA composites, one for each orchestrated Business Service. Or, if you have more demanding integration requirements, consinder creating fassades to end systems that are based on services developed in the “new” Oracle Service bus (OSB – ex BEA Aqualogic Service Bus).

OSB belongs now to Oracle SOA Suite, yet the integration into one holistic IDE and console that consolidates SCA, mediator, BPEL and OSB is one of the major things to look for in future releases of Oracle SOA Suite

Fireworks for SOA Suite 11!

Oracle SOA Suite 11 is a major step for Oracle being a major platform to base a SOA on. I remember first talks on one integrated look and feel and deployment at Oracle HQ some three years ago.

There is an attribute of SOA Suite 11g I really like which is easily overlooked among the overwhelming wealth of new features: Although SCA adds yet another technology to the acronym zoo in modern middleware, it rather not increases complexity, but decreases it. Through SCA as a bundling technology it becomes easier to understand an assembly of the constituents of a composite such as a BPEL process, a human interaction, some rules and routing logic. In earlier versions those have been scattered over JDeveloper and we had to figure out through clicking around how they fit together. Now these puzzle pieces of a composite are graphically grouped in one view, so it becomes way easier to grasp what is actually going on and get started. This reduction of complexity continues during deployment and administration, you always look at one bundle, a SCA composite. This reduction of complexity promises to help adoption of SOA based on Oracle Fusion Middleware.