Tag Archives: Service Bus

Article on integration infra components published in OTech magazine

article OTech magazine - integration infra componentsDuring Oracle Open World 2013 OTech magazine was launched. OTech is a new independent magazine for Oracle professionals. The magazine’s goal is to offer a clear perspective on Oracle technologies and the way they are put into action. As a trusted technology magazine, OTech Magazine provides opinion and analysis on the news in addition to the facts.

My article in OTech magazine one of the most frequently asked questions is how to pick the right integration infrastructure component to solve the problem at hand.

Download the fall issue of OTech magazine.

Service Bus definition

While preparing guidelines for the usage of the Oracle Service Bus (OSB) I was looking for a definition of a Service Bus. There wasn’t one on my blog yet (more posts on integration) so i decided to use the following and share them with you.

Forrester Service Bus definition

From 2009 Forrester has used this one:

An intermediary that provides core functions to makes a set of reusable services widely available, plus extended functions that simplify the use of the ESB in a real-world IT environment.

Erl Service Bus definition

Thomas Erl offers the following description of a Service Bus::

An enterprise service bus represents an environment designed to foster sophisticated interconnectivity between services. It establishes an intermediate layer of processing that can help overcome common problems associated with reliability, scalability, and communications disparity.

An Enterprise Service Bus is seen by Erl et al as a pattern. That is why it is even more important to share what that patterns is. Later on I’ll also shortly describe the VETRO pattern. Also a very useful pattern to use when comparing integration tools or developing guide lines.

Erl Enterprise Service Bus pattern

On the SOA patterns site we learn that an enterprise service bus represents an environment designed to foster sophisticated interconnectivity between services. The Enterprise Service Bus pattern is a composite pattern based on:

  • Asynchronous Queuing basically an intermediary buffer, allowing service and consumers to process messages independently by remaining temporally decoupled.
  • Service Broker composed of the following patterns
  • Data Model transformation to convert data between disparate schema structures.
  • Data Format transformation to dynamically translate one data format into another.
  • Protocol bridging to enable communication between different communication protocols by dynamically converting one protocol to another at runtime.
  • Intermediate routing meaning message paths can be dynamically determined through the use of intermediary routing logic.
  • With optional the following patterns: Reliable Messaging, Policy Centralization, Rules Centralization, and Event-Driven Messaging. Also have a look at slide 12 etc of the SOA Symposium Service Bus presentation.

VETRO pattern for Service Bus

The VETRO pattern was introduced by David Chappell, writer of the 2004 book Enterprise Service Bus.

  • V – Validate: Validation of messages eg based on XSD or schematron.
  • E – Enrich: Adding data from applications the message doesn’t originate from.
  • T – Transform: Transform the data model, data format or the protocol used to send the message.
  • R – Routing: Determine at runtime where to send the message to.
  • E – Execute: You can see this as calling the implementation.

We also used this pattern to compare Oracle integration tools and infrastructure. It can be very well used while choosing the appropriate tools for a job and deciding on guidelines on how to use these tools.

Book review: Do more with SOA integration

Book cover: Do more with SOA IntegrationRecently I read Do more with SOA integration that was published December 2011. This book is a mash-up of eight earlier published works from Packt, including Service Oriented Architecture: An Integration Blueprint, Oracle SOA Suite Developer’s Guide, WS-BPEL 2.0 for SOA Composite Applications with Oracle SOA Suite 11g, and SOA governance. More details on this title:

Target audience according to the publisher:

If you are a SOA architect or consultant who wants to extend your knowledge of SOA integration with the help of a wide variety of Packt books, particularly covering Oracle tools and products, then “Do more with SOA Integration: Best of Packt” is for you. You should have a good grasp of Service Oriented Architecture, but not necessarily of integration principles. Knowledge of vendor-specific tools would be an advantage but is not essential.

My thoughts

My assumption is that most people won’t read the around 700 pages of this book cover to cover. In my view it is a good reference book to get a solid introduction to SOA and integration in general.

To deepen you knowledge on real world scenario’s there a good examples eg given in the chapters on Extending enterprise application integration and Service oriented ERP integration. The first gives an example of of BPEL orchestrating various web service exposed on ERP systems (SAP, Siebel) using EAI (TIBCO, webMethods). This sample includes an example of centralized error handling. The latter shows an integration of PeolpleSoft CRM 8.9 and Oracle Applications 11g using BPEL 10g. The ideas and mechanismes of the integration will also hold in the 11g version.

Chapter 14 on SOA Integration a Scenario in detail, offers another example on how to use Oracle SOA technology (10g again) to integrate legacy systems into a more modern application landscape. It does a thorough job.

The chapter on Base Technologies has parts that are based on the Trivadis Integration Architecture Blueprint. Beside that it offers a good introduction on transactions, JCA, SCA and SDO. Their fundamentals are well explained without getting too technical. So should you be looking for coding examples on these topics, there are other great sources.

When reading about XML for integration I noticed that it answers questions we get from our customers on a regular basis like: How to design XSDs – XML Schema Definitions. Questions on when to use a type or an Element, chose targetNamespace or XMLSchema as the default namespace, the number of namespaces to use. These are all well adressed in the book.

Where on the other hand a complete view on the following statement could fill at least a whitepaper:

Adopt and develop design techniques, naming conventions, and other best practices similar to those used in object-oriented modelling to address the issues of reuse, modularization, and extensibility. Some of the common design techniques are discussed in the later sections.

The chapter on loose coupling offers an example of how to achieve this using the Oracle Service Bus. It is hard to overrate the importance of loose coupling since a lot of both the technical and the business advantage rely on whether or not this loose coupling is achieved.

Bottomline

As a reference this is a good starting point to learn about SOA and integration in general. It could be more consistent on some details and with the great BPEL and BPM tooling these days I wouldn’t implement processes in an ESB. Of course there is a good chapter (12) with an eaxmple of using both BPM and BPEL. As mentioned before it has some great illustrative examples of real world scenarios. The bottom line is that I would recommend this book to people looking for a reference on SOA and integration.

Cons:
Some text seems a little dated.

Pros:
Good description of SOA and integration in general; practical ; solid introduction on the XML stuff, transactions, JCA and SCA; nice real world integration examples.

Additional reviews

If you’re interested in other reviews on this book, visit the ADF Code Corner blog by Frank Nimphius, AMIS blog by Lucas Jellema, or this SOA / BPM on Fusion Middleware blog by Niall Commiskey.

Kscope 2011 Solid Service Bus implementations


From now on counting down in days to the upcoming ODTUG Kscope 2011. ODTUG is a user group for for a wide range of technologists working with the Oracle platforms. During this conference I’ll be presenting on solid Service Bus implementations using the Oracle Service Bus, Mediator or both. The full schedule of Kscope is here.

ESB: No WebService Provider is registered at this URL

When you are using Oracle ESB 10g and encounter the message No WebService Provider is registered at this URL when testing a web service from the ESB Console try the following:

Oracle Enterprise Manager OC4J_SOA

EM OC4J_SOA

Verify the ESB web service providers are generated correctly in ESB runtime (known as the esb-rt):

  • Login into Enterprise Manager
  • Go to the Application Server Control
  • Click OC4J_SOA oc4j instance
  • Click Web Services tab
  • Verify the deployed web services are listed here, and test that services.

In case there are no web service from the ESB runtime (esb-rt) the service providers were not generated correctly, restart ESB-RT:

  • Click OC4J_SOA oc4j instance
  • Click the Applications tab
  • Expand Middleware Services, ESB and esb-dt
  • Check the esb-rt and click the restart button

Now check the j2ee/_instance_name_/log/*/oc4j/log.xml to see if the ESB application has started correctly. Verify that ESB-RT application is initialized correctly by searching for ‘ESB bootstrap: Runtime initialized‘ message in the log file. If you also restarted the ESB-DT, verify that ESB DT application is initialized correctly by searching for ‘ESB bootstrap: Repository initialized‘ message in the log file. And finally check that the ESB web service providers were generated correct (see above).

Using a Service Bus to connect the Supply Chain

So here is my presentation from the SOA Symposium 2010. Hope you enjoy it:

SOA Cloud 2011 Brazil

SOA Cloud Symposium 2011

The SOA Cloud Symposium 2011 logo has just been released. In 2011 the SOA Symposium will be held on April 27 and 28 in Brasilia the capital of Brasil.

Monitoring AQ

Although there were already several posts on querying AQ, like “in the queue” and “enqueue“, the next one is great when monitoring AQ. We are using this to monitor AQ, especially for Oracle ESB (hence the commented addition):

1
2
3
4
5
6
7
8
9
SELECT aq.name
,      aq.queue_table
,      aq.queue_type
,      v$aq.* 
FROM   v$aq
,      all_queues aq
WHERE  aq.qid = v$aq.qid
-- use if looking for ESB AQ: and    aq.queue_table = 'ESB_JAVA_DEFERRED';
;

The query will result in something like:

Monitoring AQ query results

It contains the following metrics:

  • Waiting – Number of waiting message;
  • Ready – Number of messages with status ready;
  • Expired – Number of expired messages;
  • Total Wait – Total waiting time per queue;
  • Average Wait – Average waiting time per queue.

A describtion of the AQ and Streams views can be found here.

Oracle ESB using AQ on AIX – performance boost

One of the projects I’m involved in, uses Oracle ESB (from SOA Suite 10.1.3.3 MLR 18) running on AIX 5.3. Instead of using the default in-memory JMS the ESB is configured to use Oracle Advanced Queueing (AQ). Although there were tuning efforts before, the performance wasn’t up to par. AQ seemed to be the bottleneck, especially the dequeueing part. During the day we saw the number of messages queue up. Generally these numbers would decrease after business hours. This should have been a big problem if the number of messages wasn’t going to be tripled or quadrupled in the next month(s).

Upgrade the JVM

Previous attempts in cooperation with Oracle Consulting hadn’t done the trick, yet. To get a fresh view and some out-of-the-box thinking Marc joined the team. He performed a scan of the systems settings. Researching Oracle’s knowledge system resulted (searching on AIX & ESB & DB Adapter) in the clue to upgrade the JVM (6848406 – SLOW PERFORMANCE ON AIX ESB USING FILE AND DB ADAPTER). Although we were skeptic at first, because why tried a similar path before, we decided to give it a shot. The previous attempt resulted in errors in our code without enhancing the performance of the system.
With the AIX system a Java 5 is delivered in /usr/java5, to be more precise:

1
2
3
4
5
6
7
8
$:/jdk/bin>./java -version
java version "1.5.0"
Java(TM) 2 Runtime Environment, Standard Edition (build pap32dev-20080315 (SR7))
IBM J9 VM (build 2.3, J2RE 1.5.0 IBM J9 2.3 AIX ppc-32 j9vmap3223-20080315 (JIT enabled)
J9VM - 20080314_17962_bHdSMr
JIT  - 20080130_0718ifx2_r8
GC   - 200802_08)
JCL  - 20080314

To change this stop the SOA Suite and:

1
2
3
cd $ORACLE_HOME
mv jdk jdk.orginal           # rename jdk directory
ln –s /usr/java5 jdk         # Creates symbolic link

and the start the SOA Suite. With this newer Java version we had a significant performance improvement.

Remaining issue

If the change is performed as described above Enterprise Manager doesn’t allow you to browse the oc4j_soa anymore. This will result in the following error message:

Unable to make a connection to OC4J instance oc4j_soa on Application Server someserver.local. A common cause for this failure is an authentication error. The administrator password for each OC4J instance in the Cluster must be the same as the administrator password for the OC4J instance on which Application Server Control is running.

This can be solved by applying the patch for base bug 5261515. The patch adds some security jars and properties that are not in the AIX Java5 version.
Since this patch introduces issues in our code, we are working around the Em problems for now, while trying to resolve the issue.

Oracle ESB performance enhancements

Because the growth in the number of messages that one of our customers is receiving on it’s Oracle ESB, they were looking into how to enhance it’s performance. Oracle came up with the following clues:

  • Upgrade SOA Suite to a higher patch level
  • Reduce logging (including instance tracking in ESB)
  • Increase the number of listeners in ESB on Queues
  • Turn off payload validation

In order to see what these could do for them we did some loadtesting. The upgrade to a higher patch level of SOA Suite was needed at least to get the ESB working correctly with the configured number of listeners. However because of some issues running Oracle Universal Installer there wasn’t any time left to verify that the patch resulted in increased performance.
For functional and application management reasons turning of logging was not considered an option at this stage.

Increase the number of listeners in ESB on Queues

set ESB listeners in console

set ESB listeners in console

For each ESB System the number of listeners can be configured. These listeners are used to read from JMS topics. If there is only one listener (which is the default) message are read from the JMS Topic one at a time. The first screenshot shows where to set the number of listereners using the ESB Console.

Average response time

Average response time


To determine the optimum number of listeners for the specific ESB, OS, hardware combination we did some loadtesting. The results for the average response time are shown in the next screenshot.

As expected the respons time increases at a certain point when the number of listeners is raised.

With 90% and max response time

With 90% and max response time

“Time out” messages are never created by averages. While this first graph looks great and is very convincing, we are even more interested in the 90% and maximum respons times. A chart for these statistics has been included as well. Analyzing this graph one can conclude that this has an even stronger support to set the number of listeners to an appropriate level.

From this experiments it can be concluded that the response time can significantly be reduced by setting the number of listeners of an ESB system to the appropriate level. The maximum respons time has can be decreased by over 50%. This has a significant impact on the availability of the services exposed using the ESB.

Turn off payload validation

Turn off payload validation

Turn off payload validation


Another idea to enhance the performance is to turn off payload validation. Needless to say that this also has functional aspect. In most cases the validation that is performed here has some kind of business (related goal), like data quality. To switch it off, has impact, and the validation has te be migrated to some place else in the software.
Response time

Response time


The performance results can be dramatic. In a positive way! As is shown in the figure on the right, the respons time can be reduced to a fifth of the original. This goes for both the average and the maximum respons times.

If you can change your service so payload validation at this point is no longer required, this is a great step in enhancing the Oracle ESB’s performance.

What is in the (Oracle Advanced) queue? – Monitoring AQ

As a quick follow up on the previous post… If you are using a JMS queue with Oracle´s Advanced Queueing or Streams (as they have been extended to) the previously described tool can´t help you out. You need something else for monitoring AQ.

For this case a blog by a former colleague helps us out. This approach to monitoring AQ boils down to…

Find the queue you want to monitor AQ.

1
2
3
4
5
SELECT owner
,      queue_table
,      TYPE
,      user_comment
FROM   all_queue_tables

Perform a query.

1
2
3
4
5
6
SELECT msgid
,      enq_time
,      enq_uid
,      qt.user_data.text_vc
FROM   my_queue_table qt  --alter the table name
WHERE  q_name = 'MY_QUEUE' --alter the queue name