Transferring very large Messages

Note

SwiftMQ UR includes Filetransfer over JMS and the File Cache Swiftlet. Both enable users to transfer files of arbitrary size over existing JMS connections in a fast and reliable way. It should be used rather instead of transferring large JMS messages.

Introduction

SwiftMQ can handle the transfer of very large messages (10 MB and more) out of the box with its default configuration, however, the default configuration is not optimized for it. To achieve maximum throughput with very large messages, some configuration changes are necessary.

JMS Swiftlet Configuration

Creating a dedicated JMS Listener for the Producers

A dedicated JMS listener should be created for the producers because the network buffer setting differs from consumers.

Example:

    <listener name="bigmsg-producer" port="7001" router-input-buffer-size="11534336">
      ...
    </listener>

This creates a JMS listener on port 7001 and establishes a network input buffer on the router side of 11 MB (10 MB message body plus some overhead).

Creating a Connection Factory for the Producers

Create a connection factory below the producer JMS listener which is used by the producers:

Example:

  <connection-factories>
    <connection-factory name="bigmsg-producer@router1"
                        client-output-buffer-size="11534336"
                        smqp-producer-reply-interval="1"/>
  </connection-factories>

This creates a connection factory that sets the client network output buffer to 11 MB and the SMQP producer reply interval to 1 message. The latter ensures that the client acts on flow control on every message when using non-persistent messages. This attribute isn't required for persistent messages.

Creating a dedicated JMS Listener for the Consumers

A dedicated JMS listener should be created for the consumers because the network buffer setting differs from the producers.

Example:

    <listener name="bigmsg-consumer" port="7002" router-output-buffer-size="11534336">
      ...
    </listener>

This creates a JMS listener on port 7002 and creates a network output buffer on the router side of 11 MB (10 MB message body plus some overhead).

Creating a Connection Factory for the Consumers

Create a connection factory below the consumer JMS listener which is used by the consumers:

Example:

  <connection-factories>
    <connection-factory name="bigmsg-consumer@router1"
                        client-input-buffer-size="11534336"
                        smqp-consumer-cache-size="1"/>
  </connection-factories>

This creates a connection factory that sets the client network input buffer to 11 MB and the SMQP consumer cache size to 1 message. The latter ensures that the client receives messages one by one.

Note

Attribute smqp-consumer-cache-size-kb specifies the size of a consumer cache in KB. This attribute has been introduced to avoid out-of-memory errors when transferring large messages. The default value is 2048 (2 MB). So whenever smqp-consumer-cache-size or smqp-consumer-cache-size-kb is reached, the router waits until the client has consumed the cache. However, the router delivers at least one message in case the message size is greater than the smqp-consumer-cache-size-kb.

Complete JMS Swiftlet Configuration

The complete configuration of the JMS Swiftlet is, therefore:

Example:

  <swiftlet name="sys$jms">
    <listeners>
      <listener name="bigmsg-producer" port="7001" router-input-buffer-size="11534336">
        <connection-factories>
          <connection-factory name="bigmsg-producer@router1"
                              client-output-buffer-size="11534336"
                              smqp-producer-reply-interval="1"/>
        </connection-factories>
        <host-access-list/>
      </listener>
      <listener name="bigmsg-consumer" port="7002" router-output-buffer-size="11534336">
        <connection-factories>
          <connection-factory name="bigmsg-consumer@router1"
                              client-input-buffer-size="11534336"
                              smqp-consumer-cache-size="1"/>
        </connection-factories>
        <host-access-list/>
      </listener>
    </listeners>
  </swiftlet>

Producers perform a JNDI lookup on bigmsg-producer@router1 and consumer on bigmsg-consumer@router1, respectively.

Queue Manager Configuration

When transferring large messages, flow control is an important instrument to avoid overloading the router with large messages. Depending on which message model is used, point-to-point or pub/sub, different attributes have to be adjusted. Further, the queue cache size has to be adjusted to limit the number of messages held in memory.

Point-to-Point

Adjusting the flow control behavior is done via the attribute flowcontrol-start-queuesize. The cache size is defined via the attribute cache-size. Both attributes are defined per queue:

Example:

  <queue name="testqueue" cache-size="5" flowcontrol-start-queuesize="4"/>

Sets the cache size of queue testqueue to 5 messages and generates flow control delays on that queue if the queue size is at least 4.

Pub/Sub and Routing

When using pub/sub or routing, the default attributes for flowcontrol-start-queuesize and cache-size are used. These are defined in the swiftlet element itself:

Example:

  <swiftlet name="sys$queuemanager" cache-size="5" flowcontrol-start-queuesize="4">
    ...
  </swiftlet>

These values are used for all pub/sub, temporary and system (routing) queues.

Topic Manager Configuration

When using pub/sub, the pub/sub flow control has to be enabled. Because this is the default setting, you don't have to change anything.

Store Swiftlet Configuration

When using persistent messages, the Store Swiftlet's cache and checkpoint sizes have to be adjusted.

Cache Size

The cache size determines how many pages are held in memory. Say, you use a single queue with a queue cache size of 5 messages and each message is 11 MB, you'd need 55 MB of cache. Divide it by 2 KB per page and you get 28160 pages. Plus some overhead for index pages, you'd need to have a cache of 30000 pages:

Example:

  <cache min-size="30000" max-size="30000"/>

Transaction Log Checkpoint Size

The checkpoint size of the transaction log determines the interval in which checkpoints are made. A checkpoint synchronizes the transaction log with the database (page.db) and flushes all dirty cache pages. This takes time with a large cache so you should set the checkpoint size as high as possible. The following example sets it to 500 MB.

Example:

  <transaction-log checkpoint-size="524288000" path="../../store/router1/log"/>

Threadpool Swiftlet Configuration

Extend the number of threads for the jms.connection and jms.session pool to ensure that every client has always a free thread. The best way is to set the max-threads attribute to the number of clients you have:

Example:

  <pool name="jms.connection" kernel-pool="true" max-threads="5" min-threads="1">
    <threads>
      <thread name="sys$jms.connection.service"/>
    </threads>
  </pool>
  <pool name="jms.session" kernel-pool="true" max-threads="5" min-threads="1">
    <threads>
      <thread name="sys$jms.session.service"/>
    </threads>
  </pool>

Routing Swiftlet Configuration

When transferring large messages over a routing connection, the attributes inbound-transaction-size and inbound-window-size of the resp. routing listener/connector needs to be set to 1 to ensure that only a single message is put into an XA transaction and that the previous XA transaction has been fully committed before the next XA transaction will be sent.

Sender Router

A sender router sends large messages to a receiver router, thus it requires an appropriate network output buffer. Following 2 examples of a routing listener and a routing connector of a sender router. One of them needs to be defined, dependent on which router connects and which router listens.

A routing listener:

Example:

    <listener name="plainsocket"
              port="5100"
              router-output-buffer-size="11534336">
      <host-access-list/>
    </listener>

A routing connector:

Example:

    <connector name="router1"
               hostname="localhost"
               port="5100"
               router-output-buffer-size="11534336"/>

Receiver Router

A receiver router receives large messages from a sender router, thus it requires an appropriate network input buffer plus a value of 1 for the attributes inbound-transaction-size and inbound-window-size.

A routing listener:

Example:

    <listener name="plainsocket" port="5100"
              router-input-buffer-size="11534336"
              inbound-transaction-size="1"
              inbound-window-size="1">
      <host-access-list/>
    </listener>

A routing connector:

Example:

    <connector name="router1"
               hostname="localhost"
               port="5100"
               inbound-transaction-size="1"
               inbound-window-size="1">
               router-input-buffer-size="11534336"/>

Sender/Receiver Router

A sender/receiver router sends and receives large messages in both directions, thus it requires an appropriate network input and output buffer plus a value of 1 for the attributes inbound-transaction-size and inbound-window-size.

A routing listener:

Example:

    <listener name="plainsocket" port="5100"
              router-input-buffer-size="11534336"
              router-output-buffer-size="11534336"
              inbound-transaction-size="1"
              inbound-window-size="1">
      <host-access-list/>
    </listener>

A routing connector:

Example:

    <connector name="router1"
               hostname="localhost"
               port="5100"
               inbound-transaction-size="1"
               inbound-window-size="1">
               router-input-buffer-size="11534336"
               router-output-buffer-size="11534336"/>

Flow Control

During routing, messages are produced to queues during inbound processing. On each XA commit, a flow control delay is returned. An attribute inbound-flow-control-enabled can be set to true or false. Setting to true, the flow control is respected and further XA commits are delayed, depending on the flow control delays. These delays are cascaded through the whole router network back to the producer client if each hop on the route has inbound-flow-control-enabled set to true. If it is set to false (default), flow control delays are not respected and XA commits are not delayed. So in the worst case, a router is flooded with messages.

For the transfer of large messages you should enable flow control at the receiver router:

Example:

    <swiftlet name="sys$routing" inbound-flow-control-enabled="true">
      ...
    </swiftlet>