Routing Swiftlet

The Routing Swiftlet provides unlimited connectivity to other SwiftMQ routers with the transparent routing of point-to-point as well as publish/subscribe-based messages to remote destinations.

Federated Router Network

Each SwiftMQ router is a full-fledged JMS server with its own resources (queues, topics, durable subscribers, connection factories, and so on). The Routing Swiftlet is able to connect single SwiftMQ routers together. Due to dynamic routing which doesn't require any configuration (it's automatically), all resources of those connected routers become globally available. For example, a message can be sent to testqueue@router7 from a JMS client connected to router1, a message published to topic testtopic from a JMS client connected to router7 will be sent to a subscriber on testtopic which is connected to router88, and a JMS client is able to look up a connection factory, registered on router10, from a JMS client connected to router11.

For this construct of connected routers with globally available but locally managed resources, we have created the term FEDERATED ROUTER NETWORK. Each router is independent but they work together - federated. SwiftMQ includes federation support since its release 1.0 and it's very mature. In fact, there is no other JMS system in the world that implements it better than SwiftMQ, because it's SwiftMQ's core design.

There is no restriction on the number of routers you can connect nor in the form of the topology you like to build. You don't have to create a hub/spoke-like topology with the main router in the middle. You can also build a ring. No matter whether a router is directly or intermediate connected (via other routers) - it is reachable. If a connection gets lost and there is another route to the destination, the message travels that way. If no more route is available, the message stays at the last router and will be forwarded once a route becomes available again.

The following example illustrates a possible router network topology:

The Routing Swiftlet routes the messages destination-based, by finding the shortest way. To balance a load or to increase the message throughput, round-robin scheduling can be selected. In this case, the messages are distributed to all routes of a destination evenly:

For maximum reliability and guaranteed delivery, 2-phase-commit (XA) is used to transfer messages in configurable transaction sizes (bulk) from one router to the other, fast. Recovery of in-doubt XA transactions (prepared but not committed) is automatically done during the connect phase by our transaction manager.

Routing Connections

A routing connection can be established if one router provides a listener and the other a connector. There is no limit on the number of listeners and connectors. You can define any number of listeners on different ports and any number of connectors.

Listeners

A listener listens on a specific port and can be bound to a particular interface on multi homes machines. It defines a socket factory (default PlainSocketFactory) to use for sockets, created from this listener. Further attributes are a password (which is exchanged with challenge/response) and a host access list which is checked against the hostname of incoming socket connections. One can also use TLS by specifying the JSSESocketFactory instead of the PlainSocketFactory.

The following snippet defines a listener on port 4100 with the default PlainSocketFactory and another one listening on port 443 with the JSSESocketFactory and a password:

Example:

<listeners>
    <listener name="plainsocket" port="4100">
        <host-access-list/>
    </listener>
    <listener name="tlsocket" port="443"
              socketfactory-class="com.swiftmq.net.JSSESocketFactory"
              password="secret">
        <host-access-list/>
    </listener>
</listeners>

Connectors

A connector is responsible to establish a routing connection. It points to a listener on a remote router and must be of the same configuration concerning socket factory and password. A connector can be enabled and disabled. It is enabled by default. A disabled connector is inactive.

If a connection cannot be established by a connector or is lost, it retries in intervals, specified in attribute retry-time, to connect. The default retry-time is 60000 (1 minute). Retry is only performed if the connector is enabled.

The following snippet defines a connector, pointing to a routing listener on host chb.acme.com on port 17884. It uses the JSSESocketFactory and password let_me_in. The retry-time is 2 minutes:

Example:

<connectors>
    <connector name="acme" 
               hostname="chb.acme.com"
               port="17884"
               socketfactory-class="com.swiftmq.net.JSSESocketFactory"
               password="let_me_in"
               retry-time="120000"/>
</connectors>

Scheduling a Connector Job via the Scheduler Swiftlet

The Routing Swiftlet registers a job Routing Connector in the job group Routing at the Scheduler Swiftlet.

This job takes the name of the routing connector as a parameter. It enables the connector at the job start and disables it at the job stop. Therefore, a connector should be configured as being disabled if it should be used in a job:

Example:

<connectors>
    <connector name="acme"
               enabled="false"
               hostname="chb.acme.com"
               port="17884"
               socketfactory-class="com.swiftmq.net.JSSESocketFactory"
               password="let_me_in"
               retry-time="120000"/>
</connectors>

The Routing Connector should have a specified maximum runtime in its schedule, otherwise, it will not be disabled. If the connector will be enabled during the job start and the connection cannot be established, the retry behavior will be executed as described in the section above. Retry ends at the job stop.

Adjusting Network Buffers

SwiftMQ manages its own network buffers on both sides of the wire. The length of the data sent to the network is determined by the underlying SMQP protocol handlers. Only complete requests are sent. Before a request is sent, it is written to a network buffer. The receiving side receives the data into a network buffer as well. These buffers are adjusted automatically. However, an adjustment may take time when transferring large messages, so it might be necessary to configure buffer/extend sizes that meet the requirements more specifically.

Network buffers are initially created with their initial size which is 128 KB by default. If the protocol handler detects that the buffer is full before it can flush it, it will extend the buffer and continues writing until the request is finished or the buffer fills up again which then leads to another extend. The default extend size is 64 KB.

The initial buffer size and the extend size can be specified for:

• Router input/Listener

• Router output/Listener

• Router input/Connector

• Router output/Connector

So it's possible to adjust network buffers for both sides by creating the appropriate routing listeners and connectors.

Reconnect

If a routing connection is established and one side disconnects, the other side (the other router) might not detect it. That's not a problem of SwiftMQ but of the TCP protocol. To detect these invalid connections, SwiftMQ uses a keepalive mechanism on routing connections which works as follows:

If a connector of one router connects to a listener of another router, both sides pass different stages (connect, authentication, recovery, etc) until they reach the final delivery stage in which the actual message exchange takes place. The keepalive mechanism is started at the beginning of the delivery stage. Previous stages are protected by timers which ensure that such a stage will not be stuck by waiting for a reply which will never happen due to a disconnect during that stage. If the timers are fired and the stage is in waiting mode, the connection will be disconnected.

The time after which such an "early" stage becomes invalid can be specified via attribute stage-valid-timeout. Default is 15 seconds. Another attribute reject-disconnect-delay specifies the time after which a rejected connection (e.g. router name already connected, authentication failed) is closed. Both attributes are dependent on each other. If a connector connects to a listener, it initializes a timer with the value of stage-valid-timeout. This ensures that the stages become invalid if no reply arrives. Say, the router name is already connected so the listener rejects the connection and sends a negative reply. The listener initializes a timer with the value of reject-disconnect-delay which leads to a physical close when the timer is fired.

The keepalive mechanism consists of keepalive messages which are sent from both sides of the connection in the interval specified in the attribute keepalive-interval of the routing listener. This value is passed over to the connector during the connect stage. Each side has a counter which is initialized with 5. If a keepalive message is received, the counter is incremented. If a keepalive message is sent, the counter is decremented. If the counter reaches 0, the connection is probably invalid and is closed physically.

A routing connection is valid before the keepalive counter reaches 0. However, it might be already disconnected at one end and that router tries to reconnect which, in turn, will be rejected, because a given router can only be connected once. The other router is able to reconnect after the routing connection has been closed due to the keepalive mechanism. Therefore, the maximum time a router has to wait until it is able to reconnect is 5 * <keepaliveinterval>. The attribute keepalive-interval of the routing listener can be set to a lower value to shorten this time.

Message Routing

Dynamic Routing

Routing between routers of a network requires information about the routes. These routes are exchanged dynamically between each router of the network during the connect phase. That means each router is able to send messages to each other since it knows every route there. If a router connects to another router, it has the direct route to this remote router and gets all routes from here to other routers. Therefore, a message can be routed to a router that is not directly connected but reachable by other intermediate routers.

The Routing Swiftlet stores a table with all dynamic routes called the "routing table", which is displayed in the Usage node of the Routing Swiftlet:

The routing table contains an entry for each known destination router. Each of these entries contains a table of dynamic routes, received by this router. A dynamic route consists of a list of intermediate routers (hops) that a message has to travel until it reaches its final destination.

Per default, always the shortest way is chosen by the Routing Swiftlet during messages transfer, that is the route with the minimum hop count. This is the default routing mode. Therefore the message is routable if at least 1 route to its destination is known because if a route fails, the next best route is chosen and the message travels another, probably longer way. Another routing mode is "round-robin" which has to be enabled separately. Within this model, messages are distributed over all known routes to their destination evenly. Round-robin is reasonable to spread a message load over multiple routing connections.

Static Routes

With the dynamic routing architecture, a remote queue can be addressed if a route is available. Otherwise, the queue is unknown. To enable clients to address remote queues before a dynamic route is announced, static routes can be created. These are router names. Static routes are not announced to other routers. Their only sense is to enable clients to address unknown routers.

How to avoid Route Flooding

The advantage of dynamic routing is its plug-and-play nature. No administrators interaction is required. A message is routed automatically to the destination inclusive failover. However, the disadvantage is the number of routes that need to be exchanged. Every router knows every route to every other router. So, the more routers the more routes are exchanged. Updated routes are exchanged whenever a routing connection somewhere in a router network connects or disconnects. To reduce this traffic, route filters can be defined.

The following example shows a typical star-like router network topology with 2 interconnected main routers, hq1 and hq2. Each main router has 3 interconnected satellite routers. There is an additional connection between sub3 and sub4:

If a message should be send from sub6 to a queue at sub1, it will by default use the shortest way (minimum hop count): sub6 -> hq2 -> hq1 -> sub1. However, there are many other routes as well, for example: sub6 -> hq2 -> sub5 -> sub4 -> sub3 -> hq1 -> sub2 -> sub1.

Many of these routes don't make sense and will never be used. It may also be required to restrict access between routers in a network. There are 2 ways to accomplish this: to define a hop limit at a router and/or to define route filters.

Hop Limit

There is an attribute called route-announce-hop-limit to limit the routes which are announced from a router to the value specified via this attribute. The router checks each route if the number of hops exceeds the attribute value and only forwards it if the hop count is less. The default value of route-announce-hop-limit is 3. A value of -1 disabled the hop limit.

In the above example, if we define for each satellite router sub1 to sub6 the route-announce-hop-limit=1 then we would have direct routes (via the direct connection) between them and their corresponding main router hq1 resp. hq2. However, since the main routers don't have a hop limit, there was an alternate route for each satellite via its main router. For example, sub6 could reach sub5 directly but also via hq2. sub6 could also reach sub1 via hq2 and hq1 because the main router doesn't have a hop limit. However, all nonsense routes like sub6 -> hq2 -> hq1 -> sub2 -> sub3 -> sub4 -> sub5 are eliminated by the hop limit.

If we define route-announce-hop-limit=1 for the main routers hq1 and hq2 as well then each satellite could only communicate with its direct neighbor satellite and its main router. Both main routers could communicate as well but not with the satellite of each other. sub6 could reach sub5 but not sub4 and not sub1. The only exception is that sub3 and sub4 can reach each other because they have a direct connection.

Please check the routing table (see picture on top of this page) of your routers while you define your hop limits. Routing connections have to be re-established to force the re-announcement of routes.

Route Filters

Route filters provide a way for semantic filtering of routes. They can be hop or destination-based and of the include or exclude type. In this context, a hop defines a router node to which other routers are connected and which announces routes to the outside. A destination is a single router. Possible filters are:

• Announce only routes to a single destination outside (include by destination)

• Announce only routes to a hop and all hop-connected routers (= a subnet) to outside (include by hop)

• Don't announce routes to a single destination outside (exclude by destination)

• Don't announce routes to a hop and all hop-connected routers (= a subnet) to outside (exclude by hop)

In the above example, let's define a filter at sub4 for router sub3 of type include by destination with the value sub4. This must be read "to router sub3 only announce routes to destination sub4". Since the only route which has a destination sub4 is the direct route this will filter all routes except the direct route. If we define a similar route at sub3 (for sub4, type include by destination, value sub3) then both routers can reach each other via their direct connection but not any other router via this direct connection.

If we would further add a filter at hq2 of type exclude by hop with value hq1 then sub4 to sub6won't be able to reach hq1, sub1 to sub3, except that sub3 and sub4 could communicate directly. So exclude by hop excludes a complete subnet, identified by a hop. All other routes are announced. If one would like to exclude all routes except those from a particular hop then include by hop must be used.

Please check the routing table (see picture on top of this page) of your routers while you define your route filters. Routing connections have to be re-established to force the re-announcement of routes.

Routing Protocol

Up to SwiftMQ release 9.4.1 messages on routing connections are exchanged with the 2-phase-commit protocol (XA). From SwiftMQ release 9.4.2 onwards there is a choice between XA and Non-XA message exchange. Both have their advantages and disadvantages as described in the next sections.

A transaction has a specific size of messages, configurable via the attribute inbound-transaction-size on both the listener and the connector. Therefore, it is possible to specify different transaction sizes for both directions. "inbound" means that this is the size a router accepts inbound. It is passed to the sending side during the connect phase.

The Routing Swiftlet tries to load up that size messages from the corresponding routing queue and sends it as a single transaction to the other router. The default transaction size is 20 messages. Determining the proper size is dependent on the message size and on the connection speed. Small messages and high-speed connections should have a higher transaction size, e.g. 500 messages, whereas large messages on a low-speed connection should have a lower transaction size, e.g. 1. The default size of 20 fits usual environments.

The transfer of new transactions takes place while previous transactions are in transit in both directions. There is another attribute inbound-window-size for both, listeners and connectors, to specify the maximum of open transactions. The sender side of the routing connection sends new transactions until the number of open transactions reaches this window size. Thereafter it waits until it has a window again (which means, one or more transactions are committed) and continues sending. The higher the window size, the more transactions can be handled asynchronously and the higher the throughput. However, the more open transactions, the longer takes recovery during the startup/reboot of the router. Therefore, the default value is 10 which is useful.

XA Protocol

The exchange of messages between routers takes place in form of XA transactions. An XA transaction is a transaction under the control of SwiftMQ's transaction manager, utilizing the 2-phase-commit protocol to move it from one router to the other.

If a router is shut down, it might have many open XA transactions, dependent on the number of active routing connections and the configured window sizes. Like JMS XA transactions, these are handled by the XA Resource Manager Swiftlet. It checks the XA log during startup and eventually displays a warning message like this:

Example:

+++ WARNING! 10 prepared transactions found!
+++          Routing XA transactions are automatically recovered.
+++          You may also use Explorer/CLI for commit or rollback.

These open (prepared) XA transactions are automatically recovered once the corresponding router connects again. If that will not be the case, e.g. due to changes in the router network topology, the SwiftMQ Explorer or CLI can be used to commit or roll back these open XA transactions manually. See the XA Resource Manager Swiftlet documentation for details.

The advantage of using XA transactions is that exactly-once delivery (no duplicates, no loss) is ensured by the XA protocol. Messages of prepared but not committed XA transactions are locked until they are committed or rolled back.

The disadvantage is the latency overhead. A single XA transaction requires 3 protocol messages exchanged between the source and destination router until the messages are available at the destination router. The overall throughput can be increased by increasing transaction concurrency via the attribute inbound-window-size attribute but this does not reduce the latency of a single transaction.

Non-XA Protocol (since 9.4.2)

Non-XA can be requested by a listener and/or connector. It will be used if both sides request it and if both sides are at release 9.4.2 or later.

A transaction is immediately committed at the receiving router and the sending router is asynchronously informed. If a routing connection is disconnected, the router dies or is shut down, and open transactions are rolled back. When the connection is re-established, these messages are marked as redelivered and duplicate message detection at the destination queues will check these messages on arrival if they are duplicates and may discard them.

The advantage of the non-XA transactions is the reduction of latency. A single transaction requires only a single protocol message.

The disadvantage is that duplicates may occur under certain scenarios:

• Duplicate message detection relies on the JMS message-id. It does not work if the JMS message id was disabled which is possible to configure.

• If a destination queue gets input from other sources while a routing connection is down, the JMS message-ids of the routing connection may be rolled out of the duplicate log in the meantime because it has a limited size.

• If there are multiple routes over multiple hops to the destination, the rolled-back messages will travel these routes. Duplicate detection works only for the next hop so duplicates may not be detected.

Therefore, a scenario that will not generate possible duplicates is

• All messages have a JMS message-id.

• Destination queues do not get other input than from the routing connection.

• There is no further route with more than 1 hop to the destination router.

Flow Control

Flow control is an important instrument to avoid overloading a router with messages. SwiftMQ generates flow control delays on every queue during the commit of a producer. The flow control delay is a computed estimate of how long it takes to consume queue messages with the current production and consumption rate. If the flow control is respected, an optimal throughput flow will be established which holds the messages of a queue just below the maximum cache size. Therefore, the throughput is maximum at the current consumption rate.

During routing, messages are produced to queues during inbound processing. On each commit, a flow control delay is returned. An attribute inbound-flow-control-enabled can be set to true or false. Setting to true, the flow control is respected and further commits are delayed, depending on the flow control delays. These delays are cascaded through the whole router network back to the producer client if each hop on the route has inbound-flow-control-enabled set to true. If it is set to false (default), flow control delays are not respected and commits are not delayed. So in the worst case, a router is flooded with messages.

Thread Pools

The Routing Swiftlet uses the following thread pools:

• routing.connection.mgr

• routing.exchanger

• routing.scheduler

• routing.service

• routing.throttle

routing.connection.mgr

This thread pool is used by the routing connection manager. This is a single instance, performing several tasks during the connect and disconnect phase of a routing connection. The max threads setting has to be 1.

routing.exchanger

Used from the route exchanger to send, receive, and process routes from other routers. There is only a single route exchanger instance per SwiftMQ router so only 1 thread is required on demand. Therefore, the max threads setting has to be 1.

routing.scheduler

Used from routing schedulers. These are asynchronous running components, responsible to package messages into XA transactions and move them to a connection service stage, determined via dynamic routes. A scheduler exists for every routable destination router. It runs on demand and acquires threads from the routing.scheduler pool. The usage of this pool is low. Min/max of each 3 is quite enough for most use cases.

routing.service

This pool is used from the connection service stages. Dependent on the state of a connection and the protocol handling, a connection service has a specific stage running. The most frequented stage is the delivery stage which is activated after the whole protocol handshaking is done and the connection is activated for delivery. The delivery stage sends and received messages and does the whole XA handling, except XA commits if inbound flow control is enabled (see next point). The usage of this pool is high. The default setting for min/max of each 3 is enough to handle about 10 concurrent routing connections under high load.

routing.throttle

This pool is only used if inbound flow control is set to true. In that case, XA commits are dispatched to a throttle handler which throttles XA commits depending on the given flow control delays. That means threads are blocked during the delay. Because each delivery stage has an associated throttle handler and all handlers are served from this pool, a delay on one connection may block other non-delayed connections if the pool wents out of threads. Increasing the max threads should help here.

Configuration

The configuration of the Routing Swiftlet (Unlimited Connections) is defined within the element

      <swiftlet name="sys$routing" .../>

of the router's configuration file.

Attributes of Element "swiftlet"

Definition

Values

Element List "static-routes", Parent Element: "swiftlet"

Static Route Definitions. This element list contains zero or more "static-route" elements with this template definition:

Definition

Element List "filters", Parent Element: "swiftlet"

Route Filter Definitions. This element list contains zero or more "filter" elements with this template definition:

Definition

Values

Element List "routers", Parent Element: "filter"

Routers to filter. This element list contains zero or more "router" elements with this template definition:

Definition

Element List "listeners", Parent Element: "swiftlet"

Listener Definitions. This element list contains zero or more "listener" elements with this template definition:

Definition

Values

Element List "host-access-list", Parent Element: "listener"

Host Access List. This element list contains zero or more "host-access-entry" elements with this template definition:

Definition

Element List "connectors", Parent Element: "swiftlet"

Connector Definitions. This element list contains zero or more "connector" elements with this template definition:

Definition

Values

Element "usage", Parent Element: "swiftlet"

Usage.

Element List "connections", Parent Element: "usage"

Active Routing Connections. This element list contains zero or more "usage" elements with this template definition:

Definition

Values

Element List "routing-table", Parent Element: "usage"

Routing Table. This element list contains zero or more "destination" elements with this template definition:

Definition

Element List "dynamic-routes", Parent Element: "destination"

Dynamic Routes. This element list contains zero or more "dynamic-route" elements with this template definition:

Definition