One of the projects I’m involved in, uses Oracle ESB (from SOA Suite 10.1.3.3 MLR 18) running on AIX 5.3. Instead of using the default in-memory JMS the ESB is configured to use Oracle Advanced Queueing (AQ). Although there were tuning efforts before, the performance wasn’t up to par. AQ seemed to be the bottleneck, especially the dequeueing part. During the day we saw the number of messages queue up. Generally these numbers would decrease after business hours. This should have been a big problem if the number of messages wasn’t going to be tripled or quadrupled in the next month(s).
Upgrade the JVM
Previous attempts in cooperation with Oracle Consulting hadn’t done the trick, yet. To get a fresh view and some out-of-the-box thinking Marc joined the team. He performed a scan of the systems settings. Researching Oracle’s knowledge system resulted (searching on AIX & ESB & DB Adapter) in the clue to upgrade the JVM (6848406 – SLOW PERFORMANCE ON AIX ESB USING FILE AND DB ADAPTER). Although we were skeptic at first, because why tried a similar path before, we decided to give it a shot. The previous attempt resulted in errors in our code without enhancing the performance of the system.
With the AIX system a Java 5 is delivered in /usr/java5, to be more precise:
1 2 3 4 5 6 7 8
$:/jdk/bin>./java -version java version "1.5.0" Java(TM) 2 Runtime Environment, Standard Edition (build pap32dev-20080315 (SR7)) IBM J9 VM (build 2.3, J2RE 1.5.0 IBM J9 2.3 AIX ppc-32 j9vmap3223-20080315 (JIT enabled) J9VM - 20080314_17962_bHdSMr JIT - 20080130_0718ifx2_r8 GC - 200802_08) JCL - 20080314
To change this stop the SOA Suite and:
1 2 3
cd $ORACLE_HOME mv jdk jdk.orginal # rename jdk directory ln –s /usr/java5 jdk # Creates symbolic link
and the start the SOA Suite. With this newer Java version we had a significant performance improvement.
If the change is performed as described above Enterprise Manager doesn’t allow you to browse the oc4j_soa anymore. This will result in the following error message:
Unable to make a connection to OC4J instance oc4j_soa on Application Server someserver.local. A common cause for this failure is an authentication error. The administrator password for each OC4J instance in the Cluster must be the same as the administrator password for the OC4J instance on which Application Server Control is running.
This can be solved by applying the patch for base bug 5261515. The patch adds some security jars and properties that are not in the AIX Java5 version.
Since this patch introduces issues in our code, we are working around the Em problems for now, while trying to resolve the issue.