Tag Archives: ESB

Book review: Do more with SOA integration

Book cover: Do more with SOA IntegrationRecently I read Do more with SOA integration that was published December 2011. This book is a mash-up of eight earlier published works from Packt, including Service Oriented Architecture: An Integration Blueprint, Oracle SOA Suite Developer’s Guide, WS-BPEL 2.0 for SOA Composite Applications with Oracle SOA Suite 11g, and SOA governance. More details on this title:

Target audience according to the publisher:

If you are a SOA architect or consultant who wants to extend your knowledge of SOA integration with the help of a wide variety of Packt books, particularly covering Oracle tools and products, then “Do more with SOA Integration: Best of Packt” is for you. You should have a good grasp of Service Oriented Architecture, but not necessarily of integration principles. Knowledge of vendor-specific tools would be an advantage but is not essential.

My thoughts

My assumption is that most people won’t read the around 700 pages of this book cover to cover. In my view it is a good reference book to get a solid introduction to SOA and integration in general.

To deepen you knowledge on real world scenario’s there a good examples eg given in the chapters on Extending enterprise application integration and Service oriented ERP integration. The first gives an example of of BPEL orchestrating various web service exposed on ERP systems (SAP, Siebel) using EAI (TIBCO, webMethods). This sample includes an example of centralized error handling. The latter shows an integration of PeolpleSoft CRM 8.9 and Oracle Applications 11g using BPEL 10g. The ideas and mechanismes of the integration will also hold in the 11g version.

Chapter 14 on SOA Integration a Scenario in detail, offers another example on how to use Oracle SOA technology (10g again) to integrate legacy systems into a more modern application landscape. It does a thorough job.

The chapter on Base Technologies has parts that are based on the Trivadis Integration Architecture Blueprint. Beside that it offers a good introduction on transactions, JCA, SCA and SDO. Their fundamentals are well explained without getting too technical. So should you be looking for coding examples on these topics, there are other great sources.

When reading about XML for integration I noticed that it answers questions we get from our customers on a regular basis like: How to design XSDs – XML Schema Definitions. Questions on when to use a type or an Element, chose targetNamespace or XMLSchema as the default namespace, the number of namespaces to use. These are all well adressed in the book.

Where on the other hand a complete view on the following statement could fill at least a whitepaper:

Adopt and develop design techniques, naming conventions, and other best practices similar to those used in object-oriented modelling to address the issues of reuse, modularization, and extensibility. Some of the common design techniques are discussed in the later sections.

The chapter on loose coupling offers an example of how to achieve this using the Oracle Service Bus. It is hard to overrate the importance of loose coupling since a lot of both the technical and the business advantage rely on whether or not this loose coupling is achieved.

Bottomline

As a reference this is a good starting point to learn about SOA and integration in general. It could be more consistent on some details and with the great BPEL and BPM tooling these days I wouldn’t implement processes in an ESB. Of course there is a good chapter (12) with an eaxmple of using both BPM and BPEL. As mentioned before it has some great illustrative examples of real world scenarios. The bottom line is that I would recommend this book to people looking for a reference on SOA and integration.

Cons:
Some text seems a little dated.

Pros:
Good description of SOA and integration in general; practical ; solid introduction on the XML stuff, transactions, JCA and SCA; nice real world integration examples.

Additional reviews

If you’re interested in other reviews on this book, visit the ADF Code Corner blog by Frank Nimphius, AMIS blog by Lucas Jellema, or this SOA / BPM on Fusion Middleware blog by Niall Commiskey.

Solving ORABPEL-11075 Unable to create control directory

Recently I engaged with a customer who experiences (among other problems) the following. Since there is hardly any info on Oracle Support or blogs I thought I’d share some info here:

WSIF JCA Execute of operation 'SomeService' failed due to: Unable to create control directory.
Unable to create control directory: "/mnt/queues/controldir/bpel/fileftp/controlFiles/epajYGFxPFPErSM8HcN8HQ==/outbound".
; nested exception is:
ORABPEL-11075
Unable to create control directory.
Unable to create control directory: "/mnt/queues/controldir/bpel/fileftp/controlFiles/epajYGFxPFPErSM8HcN8HQ==/outbound".
Please make sure that the directory name is valid and that there are adequate permissions to create the directory.

Solution

As the last part of the error message suggests, check for file system stuff:

  1. Is the share and/or mount your file / ftp adapter is pointing to still available?
  2. Can the share be reached from your SOA Server?
  3. Does the directory name exist?
  4. Are the required permissions available?

Correct inaccuracies.

Common Issues and Workarounds for File/FTP Adapter

There is an extensive document on solving issues with file and FTP adapters on Oracle support. The documentation for file and FTP adapters can be found:

  1. 11g JCA File / FTP adapter
  2. 10g File / FTP Adapter

Choosing your Oracle Application Integration Infrastructure

Today I presented at ODTUG Kaleidoscope. The presentation is aimed at supporting architects and especially developers to choose the right integration infrastructure for a job.

Kscope 2011 Solid Service Bus implementations


From now on counting down in days to the upcoming ODTUG Kscope 2011. ODTUG is a user group for for a wide range of technologists working with the Oracle platforms. During this conference I’ll be presenting on solid Service Bus implementations using the Oracle Service Bus, Mediator or both. The full schedule of Kscope is here.

ESB: No WebService Provider is registered at this URL

When you are using Oracle ESB 10g and encounter the message No WebService Provider is registered at this URL when testing a web service from the ESB Console try the following:

Oracle Enterprise Manager OC4J_SOA

EM OC4J_SOA

Verify the ESB web service providers are generated correctly in ESB runtime (known as the esb-rt):

  • Login into Enterprise Manager
  • Go to the Application Server Control
  • Click OC4J_SOA oc4j instance
  • Click Web Services tab
  • Verify the deployed web services are listed here, and test that services.

In case there are no web service from the ESB runtime (esb-rt) the service providers were not generated correctly, restart ESB-RT:

  • Click OC4J_SOA oc4j instance
  • Click the Applications tab
  • Expand Middleware Services, ESB and esb-dt
  • Check the esb-rt and click the restart button

Now check the j2ee/_instance_name_/log/*/oc4j/log.xml to see if the ESB application has started correctly. Verify that ESB-RT application is initialized correctly by searching for ‘ESB bootstrap: Runtime initialized‘ message in the log file. If you also restarted the ESB-DT, verify that ESB DT application is initialized correctly by searching for ‘ESB bootstrap: Repository initialized‘ message in the log file. And finally check that the ESB web service providers were generated correct (see above).

Purging BPEL and ESB instances

The question on the ability to purge instances for both Oracle BPEL and Oracle ESB keeps recurring so I decided to come up with a small overview.

Oracle BPEL

  • FMW version 11.1.1.4 and up, there is a new version of the purge_soainfra_oracle.sql. Ensure that you run the purge setup scripts from the 11.1.1.4 RCU or 11.1.1.5 RCU location, respectively, as this contains the latest purge details. Details are in the SOA Admin Guide
  • FMW version 11.1.1.1 and up, there is a purge_soainfra_oracle.sql script in the RCU_HOME/rcu/integration/soainfra/sql/purge directory. In the release before 11.1.1.4 you need to apply patch 10297850 to obtain the updated purge scripts. More details are in the FMW Admin Guide on Managing Database Growth. Mark Kelderman posted on how to improve the performance of this script by adding a specific index.
  • From BPEL in SOA Suite version 10.1.3.3.1 and up, there is a purge_instances_oracle.sql script in the SOA_HOME/bpel/system/database/scripts directory.
  • For BPEL in SOA Suite version 10.1.3.1 you need patch 7162289. After applying this patch purge_instances_oracle.sql will be located at SOA_HOME/bpel/system/database/scripts.
  • In case you’re still running BPEL 10.1.2.0.2 check Oracle support document 398578.1 and document 341102.1

Oracle ESB

  • For Mediator in FMW 11g refer to the scripts and links mentioned in the BPEL paragraph.
  • To purge ESB instances in 10.1.3.4 you need to apply at least MLR#4 to have purge scripts available. There are 3 scripts available:
  • $ORACLE_HOME/integration/esb/sql/other/purge_by_date.sql
    $ORACLE_HOME/integration/esb/sql/other/purge_by_id.sql
    $ORACLE_HOME/integration/esb/sql/other/purge_by_instance_id.sql
    

Update 2011-06 SOA Suite 11gR1 PS3

Recently the SOA Suite Engineering Team published a presentation on Purging strategies in Oracle SOA Suite 11gR1 PS3. Besides the information presented in this post, the presentation goes into targeted deletion from the EM and looks into looped and parallel purges. It also mentions the use of database partitioning. That can be very useful when you have long running instances and a large installation (the presentation gives more than 500GB total OR more than 10GB daily depending on environment specifics).

Update 2011-07 Purging the Extreme Edition

Mark Kelderman posted a blog on Purging SOA Suite 11g Extreme Edition. He give a SQL script that deletes data from Mediator, BPEL, BPM, Workflow and Composite, and after that reclaims disk space.

Update Purge Oracle Service Bus

Marc Kelderman posted on his blog on Purging OSB Report Data. Besides the default way to delete the data using the console. He also gives a SQL script to purge directly in the database.

SOA 11g database growth strategies

In January of 2012 the Oracle Fusion Middleware whitepaper on database growth strategies was published. The advice presented should facilitate better dialog between SOA and Database administrators when planning database and host requirements. An important aspect of the whitepaper is purging of the SOA database.

Related documentation:

Updated purging strategies on Oracle Support

purging in OFM 11gAs part of the Oracle Fusion Middleware (FMW) SOA 11g Infrastructure Database: Installation, Maintenance and Administration Guide [ID 1384379.1] A purging guide has been published to offer strategies and procedures for managing database growth in a production SOA 11g implementation.
Purging is not supported for MS SQL Server out of the box or through a patch in any version of 11g. The provided scripts can, however, be used as a solid base from which to create usable purge scripts for SQL Server.

InfoQ videos: Using a Service Bus to Connect the Supply Chain

The session I presented at the SOA Symposium 2010 is now available online via InfoQ. You can find it as a Service Bus case study. It is introduced as:

a case study of using a service bus in a supply channel connecting a wholesale supplier with hundreds of retailers, the overall context and challenges faced – including the integration of POS software coming from different software providers-, the solution chosen and its implementation, how it worked out and the lessons learned along the way.

Watch the video, download the mp3 or sheets.

SOA Symposium 2010 videos available via InfoQ

This year, in partnership with InfoQ.com (the largest community site for technical architects), 1/4th of the SOA Symposium sessions was filmed and will be published on InfoQ. From early November InfoQ has started publishing these videos. At the time of writing the following videos are available:

Cloudy SOA

This session on Cloudy SOA by Mark Little covers:

an introduction to cloud computing pointing to the fact that the middleware needs of the cloud are similar to SOA’s, showing some of the benefits of running SOA along with the cloud, asking if cloud computing and SOA should evolve together and giving some future directions to consider.

BPM Top Seven Architectural Topics in 2010

Hanjo Normann’s session presenting the BPM Top Seven Architectural Topics in 2010 covers:

how to design a BPM/SOA solution including: modeling human interaction, improving BPM models, orchestrating composed services, central task management, new approaches for business-IT alignment, solutions for non-deterministic processes, and choreography.

Resurrecting SOA

Anne Thomas Manes in her Resurrecting SOA session goes into details on why she:

believes organizations need SOA more than before, but using a redefined SOA based on the SOA Manifesto, focusing on models, methodologies and patterns, not on technology, intended to produce the desired business and technical goals.

For a complete overview of SOA Symposium sessions on InfoQ check their SOA Symposium page.

Tuning AQ for Oracle ESB

If you are using AQ within Oracle ESB there might be a point in time you want to tune AQ performance. In this post you’ll find the results of Metalink research, and our experience on a production system.

Queue compatibility

When creating the ORAESB schema (in version 10.1.3.3) using the script $ORACLE_HOME/integration/esb/sql/oracle/create_esb_topics.sql, the queues are created with 8.1 compatibility. This is solved in 10.1.3.4. To alter this find the statement

1
2
3
4
5
6
dbms_aqadm.create_queue_table
    ( Queue_table => qtablename
    , Queue_payload_type => 'SYS.AQ$_JMS_TEXT_MESSAGE'
    , multiple_consumers => TRUE
    , compatible => '8.1'
    );

and change this to:

1
2
3
4
5
6
dbms_aqadm.create_queue_table
    ( Queue_table => qtablename
    , Queue_payload_type => 'SYS.AQ$_JMS_TEXT_MESSAGE'
    , multiple_consumers => TRUE
    , compatible => '10.2'
    );

if you already created the queues, use this statement:

1
2
3
4
dbms_aqadm.migrate_queue_table
    ( queue_table => 'ESB_JAVA_DEFERRED'
    , compatible => '10.2'
    );

If you’re not sure check the compatibility with this query:

1
2
3
4
5
6
SELECT queue_table
,      compatible
,      recipients
FROM   dba_QUEUE_tables
WHERE  owner = 'ORAESB'
;

Streams pool size

Verify the current stream_pool_size using the following query:

1
2
3
4
5
6
7
SELECT component
,      current_size/1024/1024 "CURRENT_SIZE"
,      min_size/1024/1024 "MIN_SIZE"
,      user_specified_size/1024/1024 "USER_SPECIFIED_SIZE"
,      last_oper_type "TYPE" 
FROM   v$sga_dynamic_components
;

look for the streams pool. There are several Metalink notes on this setting (including 316889.1, 102926.1 and 335516.1). The latter has a general recommandetion per RDBMS version:

  • 11g: set STREAMS_POOL_SIZE to be greater or equal to 100 MB;
  • 10gR2: set SGA_TARGET > 0 and STREAMS_POOL_SIZE=0 to enable autotuning of the Streams pool;
  • 10gR1: use the STREAMS_POOL_SIZE init.ora parameter to configure the Streams memory allocation;

And of course you could use V$STREAMS_POOL_ADVICE to get advice for your specific situation.

Upgrade the JDK

A described here upgrading the JDK can also give a performance boost.

Oracle ESB performance enhancements

Because the growth in the number of messages that one of our customers is receiving on it’s Oracle ESB, they were looking into how to enhance it’s performance. Oracle came up with the following clues:

  • Upgrade SOA Suite to a higher patch level
  • Reduce logging (including instance tracking in ESB)
  • Increase the number of listeners in ESB on Queues
  • Turn off payload validation

In order to see what these could do for them we did some loadtesting. The upgrade to a higher patch level of SOA Suite was needed at least to get the ESB working correctly with the configured number of listeners. However because of some issues running Oracle Universal Installer there wasn’t any time left to verify that the patch resulted in increased performance.
For functional and application management reasons turning of logging was not considered an option at this stage.

Increase the number of listeners in ESB on Queues

set ESB listeners in console

set ESB listeners in console

For each ESB System the number of listeners can be configured. These listeners are used to read from JMS topics. If there is only one listener (which is the default) message are read from the JMS Topic one at a time. The first screenshot shows where to set the number of listereners using the ESB Console.

Average response time

Average response time


To determine the optimum number of listeners for the specific ESB, OS, hardware combination we did some loadtesting. The results for the average response time are shown in the next screenshot.

As expected the respons time increases at a certain point when the number of listeners is raised.

With 90% and max response time

With 90% and max response time

“Time out” messages are never created by averages. While this first graph looks great and is very convincing, we are even more interested in the 90% and maximum respons times. A chart for these statistics has been included as well. Analyzing this graph one can conclude that this has an even stronger support to set the number of listeners to an appropriate level.

From this experiments it can be concluded that the response time can significantly be reduced by setting the number of listeners of an ESB system to the appropriate level. The maximum respons time has can be decreased by over 50%. This has a significant impact on the availability of the services exposed using the ESB.

Turn off payload validation

Turn off payload validation

Turn off payload validation


Another idea to enhance the performance is to turn off payload validation. Needless to say that this also has functional aspect. In most cases the validation that is performed here has some kind of business (related goal), like data quality. To switch it off, has impact, and the validation has te be migrated to some place else in the software.
Response time

Response time


The performance results can be dramatic. In a positive way! As is shown in the figure on the right, the respons time can be reduced to a fifth of the original. This goes for both the average and the maximum respons times.

If you can change your service so payload validation at this point is no longer required, this is a great step in enhancing the Oracle ESB’s performance.