Tag Archives: Integration

Integration

Continuous Delivery at bol.com

Last month two of our software engineers Mihaela Tunaru and Mary Gouseti were invited to give a presentation of how continuous delivery is done at bol.com. The presentation gives a good insight in the state of continuous delivery at bol.com from a software engineering perspective.

In case you want to know more from the operations perspective check Mayfly on GitHub and the presentation below. Maarten Dirkse gave a talk Docker your user stories using Mayfly.

Mayfly is a development platform built by bol.com. Mayfly speeds up your service development by wrapping your scrum user story code in containers, testing it in an isolated, production-like environment and automatically enforcing your Definition of Done.

New Whitebook article (Dutch) – Integratie maar dan lean

lean integrationRecently my latest Whitebook was published, called ‘Integratie maar dan lean‘. The article focusses on solid tips to move your integration practise to a more lean version.

It’s in Dutch. International readers can use Google Translate. An English translation can be provided on request. Please leave a comment with your email. Have your integration lean 😉

Gartner Magic Quadrant for SOA Infrastructure Projects

In June gartner published it’s Magic Quadrant for Application Infrastructure for Systematic SOA Infrastructure Projects. Due to the nature of SOA initiatives the selection of technologies and products aimed at supporting the implementation of the SOA infrastructure is done upfront. The resulting platform is shared among SOA applications and other integration initiatives in the enterprise.

To address the need for SOA infrastructure vendors typically have “SOA suites” of “SOA platforms”. These package products like:

Magic Quadrant for Application Infrastructure for Systematic SOA Infrastructure

.
SOA infrastructure magic quadrant
All open source vendors are in the visionary quadrant. In some cases their offerings are more modern than the Leaders’ products, since they are hardly burdened with backward compatibility issues. However these vendors are constrained by their small size or sometimes inconsistent execution.

In general the open source platforms are less expensive and easier to implement and deploy. However their offerings are generally less comprehensive than the Leaders’ offerings. If these offerings fit your requirements this could be an easy-to-use/low-cost SOA infrastructure for your organisation. The open source platforms are a strong technology offering.

Other recent Magic Quadrants for SOA and integration

Service Bus definition

While preparing guidelines for the usage of the Oracle Service Bus (OSB) I was looking for a definition of a Service Bus. There wasn’t one on my blog yet (more posts on integration) so i decided to use the following and share them with you.

Forrester Service Bus definition

From 2009 Forrester has used this one:

An intermediary that provides core functions to makes a set of reusable services widely available, plus extended functions that simplify the use of the ESB in a real-world IT environment.

Erl Service Bus definition

Thomas Erl offers the following description of a Service Bus::

An enterprise service bus represents an environment designed to foster sophisticated interconnectivity between services. It establishes an intermediate layer of processing that can help overcome common problems associated with reliability, scalability, and communications disparity.

An Enterprise Service Bus is seen by Erl et al as a pattern. That is why it is even more important to share what that patterns is. Later on I’ll also shortly describe the VETRO pattern. Also a very useful pattern to use when comparing integration tools or developing guide lines.

Erl Enterprise Service Bus pattern

On the SOA patterns site we learn that an enterprise service bus represents an environment designed to foster sophisticated interconnectivity between services. The Enterprise Service Bus pattern is a composite pattern based on:

  • Asynchronous Queuing basically an intermediary buffer, allowing service and consumers to process messages independently by remaining temporally decoupled.
  • Service Broker composed of the following patterns
  • Data Model transformation to convert data between disparate schema structures.
  • Data Format transformation to dynamically translate one data format into another.
  • Protocol bridging to enable communication between different communication protocols by dynamically converting one protocol to another at runtime.
  • Intermediate routing meaning message paths can be dynamically determined through the use of intermediary routing logic.
  • With optional the following patterns: Reliable Messaging, Policy Centralization, Rules Centralization, and Event-Driven Messaging. Also have a look at slide 12 etc of the SOA Symposium Service Bus presentation.

VETRO pattern for Service Bus

The VETRO pattern was introduced by David Chappell, writer of the 2004 book Enterprise Service Bus.

  • V – Validate: Validation of messages eg based on XSD or schematron.
  • E – Enrich: Adding data from applications the message doesn’t originate from.
  • T – Transform: Transform the data model, data format or the protocol used to send the message.
  • R – Routing: Determine at runtime where to send the message to.
  • E – Execute: You can see this as calling the implementation.

We also used this pattern to compare Oracle integration tools and infrastructure. It can be very well used while choosing the appropriate tools for a job and deciding on guidelines on how to use these tools.

Book review: Do more with SOA integration

Book cover: Do more with SOA IntegrationRecently I read Do more with SOA integration that was published December 2011. This book is a mash-up of eight earlier published works from Packt, including Service Oriented Architecture: An Integration Blueprint, Oracle SOA Suite Developer’s Guide, WS-BPEL 2.0 for SOA Composite Applications with Oracle SOA Suite 11g, and SOA governance. More details on this title:

Target audience according to the publisher:

If you are a SOA architect or consultant who wants to extend your knowledge of SOA integration with the help of a wide variety of Packt books, particularly covering Oracle tools and products, then “Do more with SOA Integration: Best of Packt” is for you. You should have a good grasp of Service Oriented Architecture, but not necessarily of integration principles. Knowledge of vendor-specific tools would be an advantage but is not essential.

My thoughts

My assumption is that most people won’t read the around 700 pages of this book cover to cover. In my view it is a good reference book to get a solid introduction to SOA and integration in general.

To deepen you knowledge on real world scenario’s there a good examples eg given in the chapters on Extending enterprise application integration and Service oriented ERP integration. The first gives an example of of BPEL orchestrating various web service exposed on ERP systems (SAP, Siebel) using EAI (TIBCO, webMethods). This sample includes an example of centralized error handling. The latter shows an integration of PeolpleSoft CRM 8.9 and Oracle Applications 11g using BPEL 10g. The ideas and mechanismes of the integration will also hold in the 11g version.

Chapter 14 on SOA Integration a Scenario in detail, offers another example on how to use Oracle SOA technology (10g again) to integrate legacy systems into a more modern application landscape. It does a thorough job.

The chapter on Base Technologies has parts that are based on the Trivadis Integration Architecture Blueprint. Beside that it offers a good introduction on transactions, JCA, SCA and SDO. Their fundamentals are well explained without getting too technical. So should you be looking for coding examples on these topics, there are other great sources.

When reading about XML for integration I noticed that it answers questions we get from our customers on a regular basis like: How to design XSDs – XML Schema Definitions. Questions on when to use a type or an Element, chose targetNamespace or XMLSchema as the default namespace, the number of namespaces to use. These are all well adressed in the book.

Where on the other hand a complete view on the following statement could fill at least a whitepaper:

Adopt and develop design techniques, naming conventions, and other best practices similar to those used in object-oriented modelling to address the issues of reuse, modularization, and extensibility. Some of the common design techniques are discussed in the later sections.

The chapter on loose coupling offers an example of how to achieve this using the Oracle Service Bus. It is hard to overrate the importance of loose coupling since a lot of both the technical and the business advantage rely on whether or not this loose coupling is achieved.

Bottomline

As a reference this is a good starting point to learn about SOA and integration in general. It could be more consistent on some details and with the great BPEL and BPM tooling these days I wouldn’t implement processes in an ESB. Of course there is a good chapter (12) with an eaxmple of using both BPM and BPEL. As mentioned before it has some great illustrative examples of real world scenarios. The bottom line is that I would recommend this book to people looking for a reference on SOA and integration.

Cons:
Some text seems a little dated.

Pros:
Good description of SOA and integration in general; practical ; solid introduction on the XML stuff, transactions, JCA and SCA; nice real world integration examples.

Additional reviews

If you’re interested in other reviews on this book, visit the ADF Code Corner blog by Frank Nimphius, AMIS blog by Lucas Jellema, or this SOA / BPM on Fusion Middleware blog by Niall Commiskey.

Selected software development trends

About 2 months ago InfoWorld published 11 programming trends to watch. I’d like to share three with you since they are close to home for me:

  1. No code is an island
  2. Bandwidth is no longer free
  3. Energy is no longer free, either

No code is an island

Having worked in integration project for almost a decade the idea that there is little code living on an island is not strange to me. However InfoWorld points out that besides that more and more software developer are creating products to enhance other products

Our code is living increasingly in ecosystems. Many PHP programmers, for instance, create plug-ins for WordPress, Drupal, Joomla, or some other framework. Their code is a module that works with other modules.

The same goes for development for mobile devices that rely increasingly on modules or apps created by others, whether they run on the device or in the cloud. This increases the demand for stable interfaces and contracts. Besides that the requirements for availability and scalability will weigh in heavier.

An urge for lean programming

Or create programs that deliver value in an efficient way. New releases of software programmers tend to demand always more resources (just a small example). The cost of keeping a computer plugged in has never been an issue. It never mattered how much energy your rack of servers sucked down because the colo just sent you a flat bill for each box.

The Cloud trend tends to make cost more transparent. Some of the clouds — like Google App Engine or Amazon S3 (example) — don’t bill by the rack or root password. They charge for database commits and queries. This adds a new perspective for software developers. We might need to start thinking about the cost of each subroutine in euros, not in lines of code, function points or milliseconds of execution time.

On the consumer side more and more ISPs adding bandwidth caps and metering. To a software developer this means that optimizing bandwidth consumption when designing apps is becoming imperative. Besides the cost issue this will also be needed because of the customer experience (loading speed etc).

Lean, agile and SOA reading list of 2011

Since this blog is also dedicated to sharing resources that are valueable to me I decided to share my reading list of 2011 with you.

Lean Integration: An Integration Factory Approach to Business Agility


A great best practices book on integration. The first part provides description of the business value of Lean. It introduces the core concepts. As a manager that doesn’t need all the details you could just read this part and you can get a good grasp of the ideas presented.

The second part translates the lean principles from the world of manufacturing to the world of systems integration. It has great case studies that shows the principles applied in a real world context.

Part three of the book provides a “how to” guide. This can be used as a reference and as such is a great desk-top reference manual. This book is great and a must read for all technology and business practitioners and innovators.

Web Service Contract Design and Versioning for SOA

Great reference (not a book that I read front to back) on Web Service Design from Thomas Erl and his co-authors. This book focuses exclusively on the contract part of the service. Due to the depth it is a extensive resource to use besides others. The book is filled with extensive examples on how to meet the goals of SOA properly using contract design.

Via the site of the publisher and on iTunes are additional service design podcasts by the authors of the book. Could be a great resource to start with.

The Back of the Napkin (Expanded Edition): Solving Problems and Selling Ideas with Pictures


This is a great book on problem solving, extremely useful and in a sense thought provoking. It structures problem-solving into a six by five visual codex. This makes sense; you can literally see the evolution of the thought processes and the development of the insights take shape through the pages. Fun read as well.

Industry Data Models, Processes and Architectures

Recently while listening to OTN ArchBeat podcasts, a panel discussion on Reference Architectures (part 2 and part 3) I was thinking back to some pieces I wrote on industry data models and processes that I didn’t share with you yet. There a some similar argument to using these and reference architectures.

The value of reference models whether it contains data models, standardized messages, processes of a reference architecture, is or should be in a faster time to market and better quality of the solution.

Handle with care

What makes it hard to achieve this value, is the fact that these models contain always far more than is needed. That can be considered a waste. Even the parts that are not used still require attention while implementing and maintaining. This incurs work to understand the complex model, hide the details you don’t need, and customize and extend the parts you need.

Implementing a reference model requires spending time to determine how and to what extend this model meets the needs of your business? That is typically something you have to discover for yourself. It is where the majority of the time is spend! If you don’t go through the effort of understanding your business requirements, you are missing understanding of how the business can and should use the model. That makes it very hard to determine the value of the end solution to the business.

When using a reference models you should be aware that your business is not average. In some shape or form it delivers value to your customers in away a reference model doesn’t provide. Reference models should be used with care your business deserves.

Lean Integration Presentation

Just uploaded the presentation I gave at the Seminar “Lean & Agile IT: beter resultaat, betrokkenheid en IT volwassenheid” (Dutch) on Lean Integration. Besides the aspect of getting a lean process to create integrations we also focused on how integration is lean in the sense that it can create value.