deltalounge

Although there were already several posts on querying AQ, like “in the queue” and “enqueue“, the next one is great when monitoring queues. We are using this to monitor AQ, especially for Oracle ESB (hence the commented addition):

1
2
3
4
5
6
7
8
9

SELECT aq.name
,      aq.queue_table
,      aq.queue_type
,      v$aq.* 
FROM   v$aq
,      all_queues aq
WHERE  aq.qid = v$aq.qid
-- use if looking for ESB AQ: and    aq.queue_table = 'ESB_JAVA_DEFERRED';
;

The query will result in something like:

It contains the following metrics:

• Waiting – Number of waiting message;
• Ready – Number of messages with status ready;
• Expired – Number of expired messages;
• Total Wait – Total waiting time per queue;
• Average Wait – Average waiting time per queue.

A describtion of the AQ and Streams views can be found here.

One of the projects I’m involved in, uses Oracle ESB (from SOA Suite 10.1.3.3 MLR 18) running on AIX 5.3. Instead of using the default in-memory JMS the ESB is configured to use Oracle Advanced Queueing (AQ). Although there were tuning efforts before, the performance wasn’t up to par. AQ seemed to be the bottleneck, especially the dequeueing part. During the day we saw the number of messages queue up. Generally these numbers would decrease after business hours. This should have been a big problem if the number of messages wasn’t going to be tripled or quadrupled in the next month(s).

Upgrade the JVM

Previous attempts in cooperation with Oracle Consulting hadn’t done the trick, yet. To get a fresh view and some out-of-the-box thinking Marc joined the team. He performed a scan of the systems settings. Researching Oracle’s knowledge system resulted (searching on AIX & ESB & DB Adapter) in the clue to upgrade the JVM (6848406 – SLOW PERFORMANCE ON AIX ESB USING FILE AND DB ADAPTER). Although we were skeptic at first, because why tried a similar path before, we decided to give it a shot. The previous attempt resulted in errors in our code without enhancing the performance of the system.
With the AIX system a Java 5 is delivered in /usr/java5, to be more precise:

1
2
3
4
5
6
7
8

$:/jdk/bin>./java -version
java version "1.5.0"
Java(TM) 2 Runtime Environment, Standard Edition (build pap32dev-20080315 (SR7))
IBM J9 VM (build 2.3, J2RE 1.5.0 IBM J9 2.3 AIX ppc-32 j9vmap3223-20080315 (JIT enabled)
J9VM - 20080314_17962_bHdSMr
JIT  - 20080130_0718ifx2_r8
GC   - 200802_08)
JCL  - 20080314

To change this stop the SOA Suite and:

1
2
3

cd $ORACLE_HOME
mv jdk jdk.orginal           # rename jdk directory
ln –s /usr/java5 jdk         # Creates symbolic link

and the start the SOA Suite. With this newer Java version we had a significant performance improvement.

Remaining issue

If the change is performed as described above Enterprise Manager doesn’t allow you to browse the oc4j_soa anymore. This will result in the following error message:

Unable to make a connection to OC4J instance oc4j_soa on Application Server someserver.local. A common cause for this failure is an authentication error. The administrator password for each OC4J instance in the Cluster must be the same as the administrator password for the OC4J instance on which Application Server Control is running.

This can be solved by applying the patch for base bug 5261515. The patch adds some security jars and properties that are not in the AIX Java5 version.
Since this patch introduces issues in our code, we are working around the Em problems for now, while trying to resolve the issue.

Because the growth in the number of messages that one of our customers is receiving on it’s Oracle ESB, they were looking into how to enhance it’s performance. Oracle came up with the following clues:

• Upgrade SOA Suite to a higher patch level
• Reduce logging (including instance tracking in ESB)
• Increase the number of listeners in ESB on Queues
• Turn off payload validation

In order to see what these could do for them we did some loadtesting. The upgrade to a higher patch level of SOA Suite was needed at least to get the ESB working correctly with the configured number of listeners. However because of some issues running Oracle Universal Installer there wasn’t any time left to verify that the patch resulted in increased performance.
For functional and application management reasons turning of logging was not considered an option at this stage.

Increase the number of listeners in ESB on Queues

set ESB listeners in console

Average response time

As expected the respons time increases at a certain point when the number of listeners is raised.

With 90% and max response time

From this experiments it can be concluded that the response time can significantly be reduced by setting the number of listeners of an ESB system to the appropriate level. The maximum respons time has can be decreased by over 50%. This has a significant impact on the availability of the services exposed using the ESB.

Turn off payload validation

Turn off payload validation

Response time

If you can change your service so payload validation at this point is no longer required, this is a great step in enhancing the Oracle ESB’s performance.

As a quick follow up on the previous post… If you are using a JMS queue with Oracle´s Advanced Queueing or Streams (as they have been extended to) the previously described tool can´t help you out.

For this case a blog by a former colleague helps us out. This approach boils down to…

Find the queue you want to monitor.

1
2
3
4
5

SELECT owner
,      queue_table
,      type
,      user_comment
FROM   all_queue_tables

Perform a query.

1
2
3
4
5
6

SELECT msgid
,      enq_time
,      enq_uid
,      qt.user_data.text_vc
FROM   my_queue_table qt  --alter the table name
WHERE  q_name = 'MY_QUEUE' --alter the queue name