We observed that under the heavy load, the SendTransactedMsg scheduler job hangs waiting for the MQ connection, this waiting has in turn consumed all the free WCS DB2 connections in the pool. This is because the job's DB2 connection is still open while it's waiting for the MQ connection.

In the log file, we get error messages like this:

[1/20/11 3:56:30:049 MST] 00000ed9 CommerceSrvr E com.ibm.commerce.messaging.outboundservice.SendTransactedMsgCmdImpl assembleMsg() CMN8216E: An error com.ibm.ejs.persitence.EnumeratorException
original exception:
COM.ibm.db2.jdbc.DB2Exception: [IBM][CLI Driver][DB2/AIX64] SQL0911N The current transaction has been rolled back because of a deadlock or timeout. Reason code "68". SQLSTATE=40001 occurred while sending the message.


So we increased the DB2 connection pool size and the MQ pool size. It had no effect, we still got this error sometimes under the heavy load.

After some investigation and consulting with IBM WCS support team. We finally found out why.

We use SendTransactedMsg job in Websphere Commerce Server to send out messages to MQ for email notification processing. During the test, we found out the job sent out a few duplicate messages to MQ once a while. We were puzzled by it for quite a while. After some digging, I found out it's due to one of the job settings.

I've installed Websphere message broker (WMB) 7 on my machine. The process was to install Websphere MQ 7 first. The WMB installer offers to install MQ for you. But that step failed on my machine. I had to install it separately.

When we use DB2 with Websphere Commerce Developer, we sometimes have to face the situation that DB2 password is expired or has to be changed. Because DB2 uses native operating system user accounts, and these accounts have to obey company security regulations.

When this happens, Websphere Commerce Developer breaks down because all of its EJB enterprise beans still reference the old password.

Here's how I fix it.

Installing and configuring a Websphere Commerce Developer environment is not an easy task. It can take from a couple of days to a week to get it fully set up and functioning for project needs. Because of this, nowadays people usually configure it only once in a virtual machine and give the whole VM to new team members.

However, running the Websphere Commerce Developer in a virtual machine can be very very sloowww! Think about this, the Rational Application Developer takes 2GB of memory, the Websphere Commerce Test Server takes another 2GB. Then DB2 takes another 500MB. The VM itself can only have 2GB virtual memory.

I was unhappy about this. I hated watching the hard disk indicator light flashing all the time and the non-responsive mouse driving me nuts. However, to me, changing/upgrading the host machine is not an option at the moment. But I can try to clone the WCS development environment out of the VM into my host machine. To save time, I decided to do a file copy instead of installing the whole thing from ground zero. It worked perfectly! Now I have a much much faster development environment to work on.

Here's what I did.