Queue Manager Swiftlet

Introduction

The Queue Manager Swiftlet manages all queues of a SwiftMQ router. Queues are the base of point-to-point as well as publish/subscribe messaging. Concerning publish/subscribe, each subscriber is backed by a temporary queue and a durable subscriber by a regular queue.

A queue is a very light-weighted object from the perspective of the queue itself (not from the content, of course). A SwiftMQ router can handle thousands of it. For each non-temporary queue, a persistent and a swap store is attached from the Store Swiftlet during the queue creation (a temporary queue has only a swap store attached). These stores are only used if needed. They don't consume resources such as disk space. A queue has a message cache where messages are stored for faster read access. If the cache is full, the messages are swapped to the swap store in the case that they are non-persistent. Persistent messages are always stored in the persistent store.

Access to a queue is handled in a first-in-first-out message order. The order is guaranteed within the same message priority. Messages are read and written covered by queue transactions. There is no way to access messages outside of queue transactions, so the access is always consistent. A queue is also capable to drive transactional access via the 2-Phase-Commit-Protocol. In that case, a prepare phase is added before a commit/rollback. 2PC transactions are managed by the transaction manager of the JMS XA/ASF Swiftlet as well as the XA Resource Manager Swiftlet to recover so-called in-doubt transactions (open prepared transactions).

Queue Types

SwiftMQ has different types of queues. In general, the distinction is between regular queues and system queues. Regular queues are point-to-point queues that are configured via the Queue Manager Swiftlet configuration and are available via JNDI. System queues are everything else and are not available through JNDI. Due to a $ sign in their queue name, these queues are not accessible from JMS clients and therefore invisible to them (except temporary queues).

While regular queues are directly configurable concerning their attributes (cache size etc), system queues are under the control of so-called "Queue Controllers". See the "Queue Controller" documentation.

The following table lists the different system queue types:

Consumer Modes

A consumer mode can be declared for each regular queue. It defines the behavior of the queue concerning multiple consumers.

Shared (default)

A queue with shared consumer mode distributes messages to all connected consumers.

Exclusive

A queue with exclusive consumer mode allows only a single consumer on this queue. Further consumers will get an exception when they try to connect.

Active/Standby

A queue with activestandby consumer mode allows only a single active consumer on this queue. This active consumer gets all messages. Further consumers will be parked as standby consumers. If the active consumer disconnects, one of the standby consumers is randomly selected to be the next active consumer.

Note that the active consumer may change after a restart of a SwiftMQ Router or a failover of a SwiftMQ HA Router.

Clustered Queues

Introduction

To increase scalability to an unlimited number of producers and consumers, physical queues of the same or different SwiftMQ Router can be connected to form a single virtual queue which is called a "clustered queue". A clustered queue is basically the same as a "distributed queue". It is just a different architectural view (bottom-up vs top-down).

The solid foundation of clustered queues is SwiftMQ's unique Federated Router Network. It can be used to connect any number of SwiftMQ Routers to form any network topology and to address any destination (queue or topic) network-wide.

The following pictures show a fully interconnected 4-node SwiftMQ Federated Router Network. There is one clustered queue cqueue defined which is globally available and has several linked physical queues (c1-n) on the different nodes. There are 2 producers on 2 different nodes producing messages and multiple consumers on each node consuming these messages. Message dispatch is automatic. If a consumer disconnects and has messages in its local queue, they will be automatically re-dispatched to other consumers.

A clustered queue provides the following features:

• It has knowledge about all local and remote consumers of this clustered queue.

• Messages produced to the clustered queue are dispatched to linked physical queues according to a dispatch policy.

• Messages produced in a transaction are dispatched as a single unit (support of message groups).

• It provides dynamic message re-dispatch to physical queues with consumers.

• Consumers created for a clustered queue are load balanced by attaching them round-robin to linked local physical queues.

Message Distribution

Message Dispatch

The Queue Manager Swiftlet sends and receives status messages about clustered queues and their consumers via topic swiftmq.cluster, hence, it knows about consumers of a particular clustered queue and where they are located, means to which physical queue on which router they are attached.

When a message is sent to a clustered queue, it decides on the base of its internal dispatch policy which linked physical queue the message will finally receive. The dispatch policy is round-robin over those queues with consumers. If there is no consumer at all, messages are dispatched evenly over all linked physical queues.

For example, assume there is a clustered queue cqueue with linked physical queues c1 and c2 on router1 and c1, c2, c3 on router2. If there is a consumer on c2@router1 and c1@router2 and a producer sends 100 messages, c2@router1 will get 50 and c1@router2 will get 50. If there is no consumer at all and 100 messages are produced, all 5 linked physical queues will get 20 messages.

Message Order

Because messages are distributed to multiple consumers on demand, message order is not respected, except there is only 1 consumer at a time for a particular clustered queue. In that case, message order is always the order in which the messages were produced.

Message Redispatch

Messages of linked physical queues without consumers are automatically re-dispatched to linked physical queues with consumers. This ensures that all messages of a clustered queue are consumed, no matter where the consumers are located.

Message Groups

Introduction

A message producer can mark messages which belong together and thus build message groups. The JMS specification reserves the standard message property JMSXGroupID for this purpose. All messages with the same value are considered as the same message group and will be dispatched to the same consumer.

For example, assume a producer sends 7 messages and sets the property JMSXGroupID to the values A, B, A, C, B, A, C. If there were 3 consumers on the queue, one consumer gets all messages with A, another consumer gets all with B and the 3rd gets all with C. If there were only 2 consumers on the queue, one would get all A, the other all B and one of them additionally all C.

Message Groups are being replicated to the STANDBY instance in a SwiftMQ HA Router configuration. In order to keep the assigned messages groups after a failover, a client must always connect to the same linked physical queue name.

Configuration

Message Group Enabled enables/disables message groups for this clustered queue. If this property is disabled, round-robin message distribution takes place so you have the same behavior as in previous releases.

Message Group Expiration sets a timeout in milliseconds after which a message group expires after this value has been sent the last time. For example, if the value A has been sent 30 seconds ago and the expiration is 30000 ms (30 secs), the value (group) A has expired and will be removed by the cleanup (see next section). Expiration is useful to save router memory by removing message groups that are not being used anymore.

Message Group Expiration Cleanup Interval is the interval in which expired message groups are removed.

Message Group Property is the name of the message property used for message grouping. The default is JMSXGroupID. This can be any property of any type.

Message Group Lifetime

A message group lives as long as its consumer lives or it expired. If a consumer disconnects, the remaining messages in the consumer's physical queue are re-dispatched (if enabled) and a new consumer is selected who will "own" this group.

JMS Usage

A producer marks messages as message groups by setting the message group property configured for the clustered queue:

Message message = session.createTextMessage("bla");
message.setStringProperty("JMSXGroupID", "G1");
...
producer.send(message);

Limitations

Message groups work best with Local Clustering (a clustered queue defined and used at a single router only) or in a Federated Router Network if producers on different routers produce different message groups. In case different producers on different routers produce the same message group (e.g. value XYZ), each router will select its own consumer for this group, and thus the message group is split over multiple consumers. The reason is that message group assignments are not exchanged between the clustered queues on each router.

JMS Usage

JNDI Lookup

All queues, clustered and regular ones, are automatically registered in JNDI. Starting with SwiftMQ 7.0.0, a JNDI alias is automatically created as well. Say, there is a clustered queue cqueue defined on router1, a JNDI lookup can be done with its fully qualified queue name:

Queue queue = (Queue)ctx.lookup("cqueue@router1");

But also with the JNDI alias:

Queue queue = (Queue)ctx.lookup("cqueue");

It is always recommended to use the JNDI alias for the lookup to keep the application code clean of router dependencies.

Message Consumer

To receive messages from a clustered queue, the consumer can be created either on the clustered queue or on one of its linked physical queues. With a clustered queue cqueue and 2 physical queues c1 and c2, the consumer can be created this way:

// Create consumer on the clustered queue
Queue queue = (Queue)ctx.lookup("cqueue");
QueueReceiver receiver = session.createReceiver(queue);

Or this way:

// Create consumer on linked physical queue c1@router1
Queue queue = (Queue)ctx.lookup("c1");
QueueReceiver receiver = session.createReceiver(queue);

If the consumer is created on the clustered queue (cqueue), it will be round-robin attached to the next linked physical queue. Let SwiftMQ choose the linked physical queue has some advantages. See "Local Clustering" in the section below.

Message Selectors

Due to performance reasons, consumers with message selectors are not respected by the dispatch policy of clustered queues. It is therefore not recommended to use a message selector in conjunction with clustered queues.

Multiple Queue Consumer vs Local Clustering

Multiple consumers on a single physical queue are sometimes used to increase throughput. From the JMS application point of view, this approach might be reasonable so one might get more throughput. However, from the SwiftMQ Router point of view, this is sub-optimal because all consumers are competing on messages on this single queue which adds synchronization overhead.

With clustered queues, there is a new way of "Local Clustering". On the local router, we just define a clustered queue and as many linked physical queues as we have concurrent queue consumers. The producers send to the clustered queue and the consumers are created on the clustered queue as well. The latter leads to a round-robin assignment of linked physical queues to consumers so that every concurrent consumer has its own physical input queue. Messages are distributed during send and not during receive so there is no competition at the receiver queue which eliminates synchronization.

So whenever there is a need to create multiple queue consumers, it is recommended to do it via Local Clustering. There is no change in the JMS application, except that the name of a clustered queue is used instead of a physical queue.

Basic Configuration

Define physical Queues

A physical queue acts as an input queue for a single consumer. If there are multiple local consumers for a clustered queue, a physical queue should be created for each of them:

So if there are 3 local consumers for this clustered queue, 3 physical queues need to be defined (in this example k1 to k3):

Define the clustered Queue

Next is to define the clustered queue. In case the clustered queue is used on multiple connected routers, the name must be the same. We chose cqueue1:

Bind physical Queues to the clustered Queue

The last step is to bind the physical queues k1, k2, k3 to the clustered queue cqueue1:

It is possible to enable/disable message re-dispatch for each binding. It is enabled by default. Only if message groups are used which must be consumed by the same consumer, it must be disabled for ALL bindings.

That's all to define a clustered queue. It must be repeated on all routers which are using this clustered queue.

Advanced Configuration

Cluster Exchange Topic

Defines the name of the topic on which cluster metrics are exchanged from all routers. Default is swiftmq.cluster. It should not be changed.

Cluster Metric Interval

This is the interval in which the Queue Manager Swiftlet sends local metrics about consumers of the locally defined clustered queues to the Cluster Exchange Topic. It is also the interval in which queues are checked if message re-dispatch is required and, if yes, message re-dispatch will be started. Default is 10000 ms which is sufficient.

Cluster Threadpool 'queue.cluster'

This pool is used for cluster metric exchange. The default for max-threads is 1 and must not be changed.

Cluster Threadpool 'queue.redispatcher'

This pool is used for message re-dispatch from queues without consumers to queues with consumers. The default for max-threads is 1 and should not be changed.

Deliver Expired Messages

Delivers expired messages to consumers. Should be disabled.

Log Duplicate Messages

Logs messages that are considered duplicates.

Log Expired Messages

Logs messages that are expired.

Composite Queues

Sometimes it is required by sending messages to one queue to forward them also to other queues and/or topics, for example for archiving or monitoring purposes. Or to automatically route messages to different queues based on static selectors. This can be done with composite queues.

Composite queues are virtual queues with mappings (called "binding") to collections of other queues and/or topics. Messages sent to a composite queue are copied to all bindings of this composite queue.

Composite queues are for producers only. It is not possible to create a consumer on them (but on any binding, of course).

Configuration

The name of the composite queue is the queue name. This is automatically registered in JNDI as with any other queue. A composite queue entry can have queue and topic bindings. Both are optional. Each message sent to the composite queue is copied to all defined queue and topic bindings, except there is a message selector defined which does not match.

Both queue and topic bindings can be marked as a "default delivery". This is kind of an "else". If a message was not delivered to any other queue or topic binding, the message is delivered in a second step to all queue or topic bindings which are marked as "default delivery". Message selectors for those bindings have no effect.

A queue binding points to another queue. This can be a physical queue, another composite queue, a clustered queue, a remote queue. The following attributes can be defined per queue binding:

A topic binding points to a topic. The following attributes can be defined per topic binding:

The destination of message copies for topic bindings is always set to the binding topic.

Example: Archive

This example shows how messages sent to queue orders are automatically archived in queue orders-archive:

<composite-queues>
    <composite-queue name="orders-composite">
        <queue-bindings>
            <queue-binding name="orders"/>
            <queue-binding name="orders-archive"/>
        </queue-bindings>
        <topic-bindings/>
    </composite-queue>
</composite-queues>

To automatically redirect JMS clients from orders to orders-composite, just change the resp. JNDI alias:

<alias name="orders" map-to="orders-composite@router1"/>

and let the client reconnect.

Example: Archive and Monitor

This example adds a topic binding to the composite queue which can be used to monitor the message flow on the composite queue:

<composite-queues>
    <composite-queue name="orders-composite">
        <queue-bindings>
            <queue-binding name="orders"/>
            <queue-binding name="orders-archive"/>
        </queue-bindings>
        <topic-bindings>
            <topic-binding name="monitor.queue.orders"/>
        </topic-bindings>
    </composite-queue>
</composite-queues>

Example: Selector based Message Routing

This example checks all messages sent to composite queue orderinput and routes them to different remote queues, based on selectors. The destination in the message will be set to the new destination queue name.

<composite-queues>
    <composite-queue name="orderinput">
        <queue-bindings>
            <queue-binding name="orders@frankfurt" change-destination="true" message-selector="region = 'Europe'"/>
            <queue-binding name="orders@newyork" change-destination="true" message-selector="region = 'America'"/>
            <queue-binding name="orders@sidney" change-destination="true" message-selector="region = 'Australia'"/>
            <queue-binding name="orders@tokio" change-destination="true" message-selector="region = 'Asia'"/>
        </queue-bindings>
        <topic-bindings/>
    </composite-queue>
</composite-queues>

Queue Controller

The entities to control system queues are the queue controllers. In principle, a queue controller contains attributes such as cache size, persistence mode, etc. It also contains an attribute called "predicate" which is a SQL-Like predicate. Whenever a system queue is created, the predicate of all queue controllers is applied to the queue name in the lexical order in which the controllers are defined, so the name of the controller decides the order here. If the predicate matches, the controller's attributes are used to create the system queue.

Furthermore, the queue is attached to the controller and subsequent changes to the controller's attributes are immediately applied to all attached system queues. Thus, a queue controller controls those system queues that are created from the controller. Since queue controllers can be defined with any predicate and in any order, it gives maximum flexibility and control over system queues.

Queue controllers are located in the folder System Queue Controllers (name in the configuration file is queue-controllers):

These controllers are predefined. Changes require a restart of the router. The following table lists the predefined controllers and their meanings:

The current settings for each queue can be viewed in the Usage folder of the Queue Manager Swiftlet:

Let's assume that you want to change the cleanup of expired messages for temporary queues. Go to the System Queue Controller, and select the controller with the name 01. As you see from the Predicate, it is the controller for temp queues. Change the Cleanup Interval attribute to 360000:

Now, look at the Usage table. All temporary queues have now a cleanup interval of 360000:

Message Expiration

Delivery of expired Messages

Whether a client will receive expired messages or not can be configured via the Queue Manager's attribute deliver-expired-messages. Default is false.

Default Cleanup Process

Messages which are expired (means which JMS expiration time is > 0 and < the current time of the router's host) are purged from the queue by so-called cleanup processes. These processes are being started in intervals.

The cleanup interval is specified via the attribute cleanup-interval. This attribute is part of the configuration for regular user-defined queues. For system queues (routing, temporary queues, durable subscriber queues, dead-letter queues) it is part of the configuration of the particular system queue controller.

A cleanup process will log every expired message in the router's warning log file by default. Because this can lead to huge log files, the logging can be switched off via the attribute log-expired-messages.

Using Cleanup Jobs

An alternate way to cleanup expired messages is to use the predefined Queue Manager jobs Queue Cleanup or Queue Cleanup DLQ. This saves router resources because it is often not necessary to check many queues whether they contain expired messages if they are empty or will never contain those messages.

To use these jobs, the cleanup-interval attribute of the queue must be set to 0 or -1 which disables the default cleanup process on these queues. Then schedule a Queue Cleanup or Queue Cleanup DLQ from the Scheduler Swiftlet.

The Queue Cleanup job just purges expired messages while the "Queue Cleanup DLQ" job moves the expired messages to the router's dead-letter queue routerdlq from which the expired messages can be consumed from a JMS application. The queue routerdlq is predefined from the Queue Manager Swiftlet and configured to deliver expired messages to JMS clients regardless of the setting of attribute deliver-expired-messages.

Multi-Queue Transactions

A SwiftMQ Router performs all operations on queues as single-queue transactions which are atomic. Operations on multiple queues within one local (non-XA) transaction, e.g. a receive, a send, and a JMS session commit, are called multi-queue transactions which offer some challenges concerning deadlock prevention, visibility of changes, atomicity, and scalability.

SwiftMQ 9.2.2 introduces a new Queue Manager attribute multi-queue-transaction-global-lock which influences how multi-queue transactions are performed.

Without Global Lock (default)

This is the default mode where multi-queue-transaction-global-lock is set to false. Here a multi-queue transaction is performed as a number of single-queue transactions which are committed independently and executed in parallel with other multi- and single-queue transactions.

One log record is written to the transaction log (file-based Store Swiftlet) resp. one JDBC commit is performed per single-queue transaction.

Other single-queue transactions may see messages before the corresponding multi-queue transaction is fully committed but this gap is very small. If the router crashes during commit, incomplete multi-queue transactions are resent and duplicate messages are discarded so full atomicity is provided if the clients transparently reconnect after a crash.

The JMS Swiftlet, JMS XA/ASF Swiftlet, and AMQP Swiftlet implement this model.

With Global Lock

If multi-queue-transaction-global-lock is set to true, multi-queue transactions are executed sequentially by use of a global lock that prevents deadlocks with concurrent multi-queue transactions. Queues are also locked against other concurrent single-queue transactions.

A single log record is written to the transaction log (file-based Store Swiftlet) resp. a single JDBC commit is performed per multi-queue transaction.

This mode provides the full atomicity of multi-queue transactions. Other transactions do not see changes until a multiple-queue transaction has been fully committed.

The JMS Swiftlet and JMS XA/ASF Swiftlet implement this model.

Jobs

The Queue Manager registers jobs in the job group Queue Manager at the Scheduler Swiftlet.

These jobs can be scheduled via the Scheduler Swiftlet to run at specific times or in intervals, based on calendars and so on.

Queue Cleanup

The Queue Cleanup job removes expired messages from queues. The queues are determined from a single parameter that contains a SQL-Like predicate. For example, a predicate of tmp$% performs a cleanup on all temporary queues, rt$% on all routing queues, test% on all queues starting with test like testqueue and so on. The Queue Cleanup job provides more flexibility than using the interval-based cleanup processes. Keep in mind that you should set the cleanup intervals (default or per queue) to 0 if you use the Queue Cleanup job.

Queue Cleanup DLQ

This job works like the Queue Cleanup job except that it moves expired messages to the router's dead-letter queue routerdlq for further processing.

Queue Mover

The Queue Mover job moves messages from a source queue to a target queue. The target queue may reside on a remote router. An optional message selector can be specified to select messages on base of JMS properties out of the source queue. This job is excellent for tasks like end of day processing to move all unprocessed messages to a queue on another router for further processing.

Queue Purger

The Queue Purger job removes messages from a single queue, optionally based on a message selector.

Multi-Queue Purger

The Multi-Queue Purger job removes messages from multiple queues, optionally based on a message selector. Use this job carefully!

Queue Reset

The Queue Reset job resets the consumed/produced message counter on the matching queues to zero. This can be used e.g. to reset the counters every day at midnight to get daily counts.

Configuration

The configuration of the Queue Manager Swiftlet is defined within the element

      <swiftlet name="sys$queuemanager" .../>

of the router's configuration file.

Attributes of Element "swiftlet"

Definition

Values

Element List "queue-controllers", Parent Element: "swiftlet"

Controllers for System Queues. This element list contains zero or more "queue-controller" elements with this template definition:

Definition

Values

Element List "clustered-queues", Parent Element: "swiftlet"

Clustered Queues. This element list contains zero or more "clustered-queue" elements with this template definition:

Definition

Values

Element List "queue-bindings", Parent Element: "clustered-queue"

Queue Bindings. This element list contains zero or more "queue-binding" elements with this template definition:

Definition

Values

Element List "composite-queues", Parent Element: "swiftlet"

Composite Queues. This element list contains zero or more "composite-queue" elements with this template definition:

Definition

Element List "queue-bindings", Parent Element: "composite-queue"

Queue Bindings. This element list contains zero or more "queue-binding" elements with this template definition:

Definition

Values

Element List "topic-bindings", Parent Element: "composite-queue"

Topic Bindings. This element list contains zero or more "topic-binding" elements with this template definition:

Definition

Values

Element List "queues", Parent Element: "swiftlet"

Queue Definitions. This element list contains zero or more "queue" elements with this template definition:

Definition

Values

Element List "usage", Parent Element: "swiftlet"

Active Queues. This element list contains zero or more "usage" elements with this template definition:

Definition

Values

Element List "receiver", Parent Element: "usage"

Active Queue Receiver. This element list contains zero or more "receiver" elements with this template definition:

Definition

Values

Element List "sender", Parent Element: "usage"

Active Queue Sender. This element list contains zero or more "sender" elements with this template definition:

Definition

Values

Element List "browser", Parent Element: "usage"

Active Queue Browser. This element list contains zero or more "browser" elements with this template definition:

Definition

Values