HA Test Suite

To test your JMS applications in regard to High Availability can be really painful if you have to stop and start HA instances manually or by your own scripts. The best would be if the HA instances are stopped and started in intervals and to monitor your JMS applications only to see how they behave under HA conditions.

This is exactly what the HA Test Suite will do for you. It consists of launchers for each HA instance that start and stop the instance in intervals, configured via a simple XML configuration file. The HA instances are launched as a separate process and hard killed (like kill -9) after a configurable runtime. The other HA instance will take over the processing. So you will have a continuous failover scenario while your JMS application is running. Monitor tools (text and GUI-based) are also included to view System.out, System.err of the HA instance as well as launcher control commands.

Both launchers are connected via a TCP connection (called command connection). One launcher acts as master, the other one as slave. Slave connects to master. The master controls the slave and tells when to stop and start its HA instance. The master ensures that one instance is always running. Start/stop of HA instances goes round-robin, starting with the master's instance. Both launchers offer a monitor port to connect with the HA Monitor. This HA Monitor is available as a pure text monitor and a Swing GUI monitor.

Note #1: Ensure that you don't use a configuration where one instance has the "preferred-active" flag enabled. All control about failover has to be at the HA test suite. So if you run the HA Test Suite with your configuration, disable the preferred-active flag.

Note #2: The HA Test Suite is configured for the Replicated File Store. The launcher scripts will delete the whole store content of the HA instances before the launcher is executed. This ensures that you have always a clean store for your tests. If you use Shared File Store or Shared JDBC Store, you should clean the store yourself before executing the launcher scripts.

Setup and Start of the Master Launcher

It doesn't matter which launcher is the master. Just choose one of the HA instances and go to directory hatest/config/unix or /hatest/config/windows (depends on your platform). Edit master.xml. Specify the absolute directory of the SwiftMQ distribution of the HA instance and optionally change startdelay, stopinterval (all in milliseconds), masterport and monitorport. Default for stopinterval is 120000 ms (2 minutes) and 30000 ms (30 seconds) for the startdelay. That means, one of the HA instances (round-robin) is killed after 2 minutes of runtime and restarted after 30 seconds.

Note: Both instances must have enough time to turn into ACTIVE/STANDBY state before one instance is killed from the launcher due to the stopinterval. The default stopinterval of 2 minutes can be used if the store contains up to 50'000 messages. If your store is larger, increase the stopinterval accordingly.

Go to directory hatest/scripts.

On Unix/Linux perform a

./launcher ../../config/unix/master.xml instance1

On Windows perform a

launcher ../../config/windows/master.xml instance1

That's it. The master launcher is running.

Setup and Start of the Slave Launcher

Go to directory hatest/config/unix or /hatest/config/windows (depends on your platform) of the 2nd HA instance. Edit slave.xml. Specify the absolute directory of the SwiftMQ distribution of the HA instance and optionally change masterport and monitorport according to the ports you did specify in master.xml.

Go to directory hatest/scripts.

On Unix/Linux perform a

./launcher ../../config/unix/slave.xml instance2

On Windows perform a

launcher ../../config/windows/slave.xml instance2

That's it. The slave launcher is running and should be connected to the master launcher.

Using the HA Monitor Text Console

Go to directory hatest/scripts.

Call the text console. The following example has a launcher on host sun1 with monitor port 7500 and the 2nd launcher on host sun14 on monitor port 7501:

./textconsole sun1 7500 sun14 7501

Output example:

*** Connected: sun1:7500
*** Connected: sun14:7501
02.01.2006-17:10:01 Instance-1 CTL: Starting instance ...
02.01.2006-17:10:02 Instance-1 OUT: Booting SwiftMQ 6.0.0 Production ...
02.01.2006-17:10:02 Instance-1 OUT: ... startup: Trace Swiftlet
02.01.2006-17:10:03 Instance-1 OUT: ... startup: Log Swiftlet
02.01.2006-17:10:03 Instance-1 OUT: ... startup: Threadpool Swiftlet
02.01.2006-17:10:03 Instance-1 OUT: ... startup: Timer Swiftlet
02.01.2006-17:10:03 Instance-1 OUT: ... startup: Network Swiftlet
02.01.2006-17:10:03 Instance-1 OUT: ... startup: High Availability Controller Swiftlet
02.01.2006-17:06:42 Instance-2 OUT: +++ High Availability State: INITIALIZE/STANDALONE
02.01.2006-17:10:03 Instance-1 OUT: SwiftMQ 6.0.0 Production is ready.
02.01.2006-17:10:03 Instance-1 OUT: +++ Waiting for Negotiation (timeout: 120000 ms) ...
02.01.2006-17:10:03 Instance-1 OUT: +++ High Availability State: UNKNOWN/ACTIVE
02.01.2006-17:10:03 Instance-1 OUT: +++ High Availability State: INITIALIZE/ACTIVE
02.01.2006-17:06:42 Instance-2 OUT: +++ High Availability State: NEGOTIATE/STANDALONE
02.01.2006-17:06:42 Instance-2 OUT: +++ High Availability State: ACTIVE-SYNC-PREPARE/ACTI...
02.01.2006-17:10:03 Instance-1 OUT: +++ High Availability State: NEGOTIATE/ACTIVE
02.01.2006-17:10:03 Instance-1 OUT: ... standby: Authentication Swiftlet
02.01.2006-17:10:04 Instance-1 OUT: ... standby: Store Swiftlet (HA)
02.01.2006-17:10:04 Instance-1 OUT: ... standby: Queue Manager Swiftlet
02.01.2006-17:10:04 Instance-1 OUT: ... standby: Topic Manager Swiftlet
02.01.2006-17:10:04 Instance-1 OUT: ... standby: Management Swiftlet
02.01.2006-17:10:04 Instance-1 OUT: ... standby: XA Resource Manager Swiftlet
02.01.2006-17:10:04 Instance-1 OUT: ... standby: Routing Swiftlet (Unlimited Connections)
02.01.2006-17:10:04 Instance-1 OUT: ... standby: JNDI Swiftlet
02.01.2006-17:10:04 Instance-1 OUT: ... standby: JMS Swiftlet (XA/ASF)
02.01.2006-17:10:04 Instance-1 OUT: ... standby: Deploy Swiftlet
02.01.2006-17:10:04 Instance-1 OUT: ... standby: JMS Application Container Swiftlet
02.01.2006-17:10:04 Instance-1 OUT: ... standby: Scheduler Swiftlet
02.01.2006-17:10:04 Instance-1 OUT: +++ High Availability State: STANDBY-SYNC-PREPARE/STA...
02.01.2006-17:06:43 Instance-2 OUT: +++ High Availability State: ACTIVE-SYNC/ACTIVE-SYNC
02.01.2006-17:10:04 Instance-1 OUT: +++ High Availability State: STANDBY-SYNC/STANDBY-SYNC
02.01.2006-17:10:06 Instance-1 OUT: +++ High Availability State: STANDBY/STANDBY
02.01.2006-17:06:45 Instance-2 OUT: +++ High Availability State: ACTIVE/ACTIVE

Note: The timestamps shown above are the local time of the resp. hosts (not synchronized in this example).

Testing your Applications

The HA Test Suite is now running and takes care of the start/stop of the HA instances. You can disconnect and reconnect with the HA Monitors at any time (remember: only 1 at a time). The launchers will save the last 100 messages and will send them to the monitors after connecting.

You are now ready to test your JMS applications.