Topic Manager Swiftlet

The JMS publish/subscribe message model is based on topics that are managed by the Topic Manager Swiftlet in SwiftMQ.

Publish/Subscribe

Topics are location independent and available on all routers in a Federated Router Network due to dynamic subscription exchange. That is, publishers can publish messages on any router, and subscribers can connect and consume from the same topic on any other router.

SwiftMQ provides topic hierarchies that enable subscribers to filter messages directly out of the topic without the need to declare expensive message selectors.

Publish/Subscribe flow control creates constant message streams with maximum throughput between publishers and subscribers. Slow subscriber conditions can be declared to handle slow subscribers, delete malfunctioning durable subscribers, and so on.

Topic Hierarchies

SwiftMQ provides the possibility to define topic hierarchies in the form <roottopic>.<subtopic>.<subsubtopic> etc.

Example:

    iit.sales.EU
    iit.sales.US
    iit.projects.swiftmq.users
    news.weather.forecasts
    sports.formula1.MichaelSchumacher

Publishing and subscribing rules to nodes in a hierarchy:

• Selecting a specific node in a topic hierarchy publishes messages to the addressed node and all concerning sub-nodes. Example: publishing to node iit.sales send the messages to iit.sales, iit.sales.US, iit.sales.EU.

• Subscribing to a specific node in a topic hierarchy means receiving all messages addressed to this node and all parent and sub-nodes. Example: subscribing to iit.sales.US one receives all messages published in iit, iit.sales, and iit.sales.US; subscribing to iit.sales one receives all messages published in iit, iit.sales, iit.sales.US, iit.sales.EU, but you don't receive the messages published in the iit.projects hierarchy.

• A subscriber receives a message only once, regardless of the number of topics his subscription matches.

SQL-Like Predicate Topic Addressing (Indirect Subscriber Selection)

As an extension to the normal topic node- and subnode-addressing scheme, SwiftMQ provides SQL-Like predicate topic addressing. So every subnode could be a SQL-Like predicate. A SQL-Like predicate could contain wildcards that match a single character (the underscore '_'), or any characters (the percent sign '%'). The escape character is '\'.

Root nodes cannot be addressed as a predicate; i. e. it is not possible to specify i%.%.swiftmq.

Predicate topics are normal topic destinations and are not available via JNDI lookups. To create a predicate topic, one has to use the createTopic() method from a TopicSession.

Example:

    Topic allSwiftMQ = topicSession.createTopic("iit.%.US");
    msg.setJMSDestination(allSwiftMQ);
    publisher.publish(msg);

Access to predicate topics must be granted by the Authentication Swiftlet just like normal topics.

What means "Indirect Subscriber Selection"? The topic name inside the message is a SQL-Like predicate. It is used to select those topic names which are defined at the Topic Manager Swiftlet configuration. This is the first step. The next step is to use those matched topics and match them against subscribers. Subscribers are also using SQL-Like predicates. Therefore, in this mode, a topic hierarchy needs to be declared in the Topic Manager configuration in order to work. This is the price for the flexibility and sometimes a disadvantage. For this reason, we introduced "Direct Subscriber Selection" (see next section).

Direct Subscriber Selection

This feature does not interpret the topic name in the message as SQL-Like predicate but as a string. Therefore it is possible to select the subscribers directly by matching the subscriber's SQL-Like predicates against the topic name of the message. You don't need to declare the topics in the Topic Manager Swiftlet configuration.

Because it is a very rare case that someone uses a predicate as a topic name when sending a message, e.g. send to iit.%.US, the direct subscriber selection is the default mode but can be disabled:

Durable Subscriptions

Durable subscriptions are implemented as queues. The name of a these queues are formed by this pattern:

      <clientid>$<durablename>

A durable subscription is usually managed by a JMS client. It creates and deletes it by use of the JMS API. However, SwiftMQ provides also ways to create a durable subscription administratively by use of SwiftMQ Explorer or CLI.

Managing Durable Subscriber Queues

Durable subscriber queues are system queues that are automatically created and their configuration is not stored in the router's configuration file. Instead, they are controlled by a "Queue Controller" that matches the name of the durable subscriber queue.

It is possible to declare dedicated queue controllers for specific sets of durable subscribers, e.g. that with a particular client id, and configure these sets differently.

Multiple Consumers on a Durable Subscription

The JMS 1.1 specification defines a durable subscription as exclusive, so it cannot have more than a single consumer. This is enforced by requiring that all active JMS connections have a unique JMS client id. Later versions of the JMS specification have relaxed this requirement and allow multiple consumers on a durable subscription to handle higher loads.

To allow multiple consumers on a single durable subscription, one needs to allow connections with the same client id by setting the specific attribute in the JMS Swiftlet configuration:

Multiple consumers then behave as defined in the attribute consumer-mode of the respective queue controller.

For example, let's configure 2 different sets of durable subscriptions, one defined by client id failsafe, and the other set covers simply all others. The failsave set should allow having "standby" consumers that jump in when the active consumer fails. The other set should be in a shared mode where messages should be distributed to all connected consumers.

The configuration of the queue controllers would be:

      <queue-controllers>
        <queue-controller name="01" persistence-mode="non_persistent" predicate="tmp$%"/>
        <queue-controller name="02" predicate="sys$%"/>
        <queue-controller name="03" predicate="swiftmq%"/>
        <queue-controller name="04" predicate="rt$%"/>
        <queue-controller name="05" predicate="unroutable"/>
        <queue-controller name="06" consumer-mode="activestandby" predicate="failsave$%"/>
        <queue-controller name="07" predicate="%$%"/>
        <queue-controller name="08" predicate="routerdlq"/>
        <queue-controller name="09" predicate="%"/>
      </queue-controllers>

In the above example queue controller 06 matches for the failsave durable subscription and queue controller 07 for all others. The latter uses the default consumer-mode of shared. Note the lexical order of the name attribute which is important because there is a top-down matching sequence and the first match on the queue name is used.

Publish/Subscribe Flow Control

SwiftMQ provides 2 different ways to control the message flow:

• delay-based flow control

• slow subscriber conditions

Delay-Based Flow Control

SwiftMQ provides publish/subscribe flow control. As for point-to-point queues, the flow control is based on flow control delays. During the publish/commit operation, all flow control delays of the selected subscriber queues are determined and the maximum delay is delivered back. So, the flow control delay is that of the slowest subscriber.

Slow Subscriber Conditions

In some use-case scenarios, it is very important to maintain a constant publishing rate. For example, a trading feed in a financial application requires that feed data (stock prices) are published without delays. If there are slow subscribers on that feed topic, either a backlog of messages will increase for those subscribers or, when using pub/sub flow control, the publisher will be throttled down from SwiftMQ to the rate of the slowest subscriber.

To avoid these situations, slow subscriber conditions can be created. Such a condition identifies a slow subscriber during the publish transaction and if the condition matches avoid sending further messages to those matching subscribers until the condition doesn't match with future publish transactions. Thus, a constant publishing rate is created.

SwiftMQ also allows the removal of non-durable subscriber connections and removes durable subscriber connections including deleting durable subscriptions. See the sections below.

A slow subscriber condition is defined per 'root topic', which is the first node of a topic hierarchy. Say, all your feeds are summarized under the hierarchy 'feed', e.g. 'feed.bloomberg', 'feed.reuters', a condition can be defined on the root topic 'feed'. The condition is valid for all sub-topics in the hierarchy. Further condition attributes are the persistence-mode (all, persistent, non_persistent), the subscription-type (all, local, remote), and the maximum number of messages in the subscriber queue during the time of the publish operation.

Example:

      <swiftlet name="sys$topicmanager">
        <slow-subscriber-conditions>
          <topic name="feed" max-messages="100" persistence-mode="non_persistent" subscription-type="local"/>
          <topic name="events" max-messages="300" persistence-mode="all" subscription-type="remote"/>
        </slow-subscriber-conditions>
        <topics>
          <topic name="feed"/>
          <topic name="feed.bloomberg"/>
          <topic name="feed.reuters"/>
          <topic name="events"/>
        </topics>
      </swiftlet>

The example defines 2 conditions. The first is for the topic hierarchy 'feed' and matches for nonpersistent messages and local subscribers if the subscriber queue has reached 100 messages. The second condition is for topic hierarchy 'events', matches for all messages but only for remote subscriptions and permits max. 300 messages in the subscriber queue.

Note that for remote subscriptions the subscriber queue is the routing queue of the resp. router and thus the message count of that queue is taken.

Disconnecting Non-Durable Subscribers

If attribute disconnect-non-durable-subscriber is set to true (default is false), all non-durable subscriber connections for which the condition matches will be removed. This option is useful if there are malfunctioning JMS clients which have topic subscriptions.

Disconnecting Durable Subscribers and Delete Durable Subscriptions

If attribute disconnect-delete-durable-subscriber is set to true (default is false), all durable subscriber connections for which the condition matches will be removed and their durable subscriptions will be deleted. That is, all messages in their durable subscriber queue will be lost! So be careful with this option. This option is useful if there are malfunctioning JMS clients which have durable topic subscriptions or if there are "dead" durable subscriptions on a topic (a JMS client created a durable subscription but doesn't use it anymore so the durable subscriber queue grows and grows).

Publish/Subscribe in a Federated Router Network

Dynamic Exchange of remote Subscriptions

All Topic Manager Swiftlets in a router network are working federated. They exchange subscription messages on a "some/none" basis. That is, they send a subscription message to all other Topic Manager Swiftlets if they have at least 1 subscription for a particular topic and they send an unsubscription message around if the count reaches zero. The subscription exchange is made on the base of root topics. A root topic is the first node in a topic hierarchy, e.g. the root topic of the topic 'iit.sales.eu' is 'iit'. For a local Topic Manager Swiftlet, these remote subscriptions are just normal subscriptions. The only difference is that the subscriber queue is a remote queue tpc$<roottopic> at the particular remote router.

Remote subscriptions are listed in the Topic Manager's Usage section of node "Remote Router Subscriptions":

Each router node consists of 1 or more root topics for which the particular router has at least 1 subscription.

Usually, these dynamic remote subscriptions are only informational. However, there is one case where an administrator has to delete these remote subscriptions: to put a node of a federated router network out of service.

If a node should be removed from a federated router network, this can be done by shutting down its JMS clients which in turn removes all their subscriptions and forces the router node to send an unsubscription notification to all other routers when the last local JMS client unsubscribes from this particular topic. The last action is to shut down the router.

But if the router node did not have an active routing connection to one or more of its neighbors, they will never receive the unsubscription notification and will continuously send pub/sub messages for the router's topic subscriptions. In this case, an administrator has to delete the resp. remote router subscription entry:

This must be done on all router nodes of the federated router network.

Static Remote Subscriptions

In a federated router network, subscriptions are exchanged dynamically between routers. If a router starts up, it has initially no knowledge about remote subscriptions. Once another router connects to this router, subscriptions are exchanged and messages are forwarded between the routers on the base of these subscriptions. To enable message forwarding for times when a router has not yet remote subscriptions (it has just been started and no other router has connected yet), static remote subscriptions can be defined.

Note that, per default, static remote subscriptions are only valid before the remote router connects the first time. Thereafter the subscriptions are turning dynamically. So in the case the remote router unsubscribes from a topic, no further messages are forwarded, even if there is a static remote subscription for it. Starting with release 6.1.0 this behavior can be changed by enabling the attribute keep-on-unsubscribe of the static remote subscription. In that case, dynamic unsubscription requests for that topic of that router are not respected and messages are still forwarded. This is very important in the case the remote router performs an orderly shutdown (e.g. for maintenance) and has durable subscribers because a router unsubscribes from all topics at remote routers during the shutdown. If keep-on-unsubscribe is true, messages are still forwarded into the routing queue and will be delivered once the remote router restarts. So no messages are lost in the meantime. If keep-on-unsubscribe is set to false (default), no messages are delivered while the remote router is down and messages are lost.

Static remote subscriptions are defined on a per-router-base. It requires that a static route is defined to the particular remote router.

Example:

      <swiftlet name="sys$routing">
        <connectors/>
        <filters/>
        <listeners>
          <listener name="plainsocket" port="4100">
            <host-access-list/>
          </listener>
        </listeners>
        <static-routes>
          <static-route name="router2"/>
          <static-route name="router3"/>
          <static-route name="router4"/>
        </static-routes>
      </swiftlet>

Thereafter, static remote subscriptions can be defined under the Topic Manager Swiftlet configuration:

Example:

      <swiftlet name="sys$topicmanager" flowcontrol-enabled="false">
        <static-remote-router-subscriptions>
          <static-remote-router-subscription name="router2">
            <static-topic-subscriptions>
              <static-topic-subscription name="testtopic"/>
            </static-topic-subscriptions>
          </static-remote-router-subscription>
          <static-remote-router-subscription name="router3">
            <static-topic-subscriptions>
              <static-topic-subscription name="testtopic"/>
            </static-topic-subscriptions>
          </static-remote-router-subscription>
          <static-remote-router-subscription name="router4">
            <static-topic-subscriptions>
              <static-topic-subscription name="iit" keep-on-unsubscribe="true"/>
              <static-topic-subscription name="testtopic"/>
            </static-topic-subscriptions>
          </static-remote-router-subscription>
        </static-remote-router-subscriptions>
        <topics>
          <topic name="testtopic"/>
          <topic name="iit"/>
          <topic name="iit.sales"/>
          <topic name="iit.sales.eu"/>
          <topic name="iit.sales.us"/>
          <topic name="iit.sales.asia"/>
        </topics>
      </swiftlet>

Note that the 'name' attribute of an element 'static-remote-router-subscription' must contain the remote router name and the 'name' attribute of an element 'static-topic-subscription' must contain the root topic (the first node of a topic hierarchy; if you don't have a topic hierarchy then it's just the topic name).

The above configuration ensures that messages are forwarded to the topic hierarchy beginning with 'testtopic' to routers 'router2', 'router3', and 'router4'. Messages to the topic hierarchy beginning with 'iit' are also forwarded to 'router4'. Forwarding of messages for this topic continues even if the remote router has no subscriptions for it.

Topic Announce Filters

Since 12.5.3

If you have a large router network and many different topics with subscriptions, all nodes will announce these subscriptions to all other nodes. This may put a high load on your router network until all of these subscriptions have been announced.

To reduce these announcements, you should use hierarchical topics. This will limit the announcements to root topics.

To reduce this further, you can explicitly define filters to allow or disallow certain topic patterns to be announced to particular nodes or groups of nodes:

  <swiftlet name="sys$topicmanager"> 
    <announce-filters> 
      <router-name-filter name="01" routername="primary" filter-type="include"> 
        <topic-filters> 
          <topic-name-filter name="global%"/> 
        </topic-filters> 
      </router-name-filter>  
      <router-name-filter name="02" routername="util1" filter-type="include"> 
        <topic-filters> 
          <topic-name-filter name="global%"/>  
          <topic-name-filter name="request%"/> 
        </topic-filters> 
      </router-name-filter>  
      <router-name-filter name="03" routername="app1%" filter-type="include"> 
        <topic-filters> 
          <topic-name-filter name="global%"/>  
          <topic-name-filter name="request%"/> 
        </topic-filters> 
      </router-name-filter>  
      <router-name-filter name="04" routername="%" filter-type="exclude"> 
        <topic-filters> 
          <topic-name-filter name="request%"/>  
          <topic-name-filter name="local%"/> 
        </topic-filters> 
      </router-name-filter> 
    </announce-filters>  
    <slow-subscriber-conditions/>  
    <static-remote-router-subscriptions/>  
    <topics> 
      <topic name="global1"/>  
      <topic name="global2"/>  
      <topic name="request1"/>  
      <topic name="request2"/>  
      <topic name="local1"/>  
      <topic name="local2"/> 
    </topics> 
  </swiftlet>  

Announce filters contain filter per router that contain topic filters. The name of a router-name-filter should be numeric to guarantee the order of application as entity lists are always sorted by their names.

The routername and the name of a topic-name-filter are SQL-LIKE predicates with a backslash escape character. router-name-filter is matched in the order of the name field.

A filter-type="include" means only the topics matching the topic filters are announced. All other topics are excluded.

A filter-type="exclude" means all topics are announced, except those matching the topic filters.

To illustrate the above example we use a tree-like router network. The node at the top is primary. It has several sub-nodes called util1, util2, and so on. Each util router has several sub-nodes called app11, app12, and so on.

Topics used in this network are non-hierarchical (flat topic names):

• Topic names starting with global are used within the whole network.

• Topic names starting with request are used within each util network only.

• Topic names starting with local are only used within an app node.

The example above defines announce filters used on app11 node that connects to its parent util1. It announces global% to primary, global% and request% to its parent util1, and all topics without request% and local% to any other node.

This ensures that:

• global% can be served by any node in the network,

• request% can only be served by a util node and its app nodes,

• local% can only be served from the local app node.

Jobs

These jobs can be scheduled via the Scheduler Swiftlet to run at specific times or in intervals, based on calendars and so on.

Delete Durable

The Delete Durable job deletes durable subscribers (incl. all messages) which are selected by 2 SQL-Like predicates, client id and durable name, specified as job parameters. For example, a client id predicate of sales and a durable name predicate of % will delete all durable subscribers of client id sales whereas a client id predicate of s\_\_es% and a durable name predicate of dur% will delete those from saukestaand dur2003. If you set both parameters to %, it will match all durable subscribers and therefore all durable subscribers are deleted.

Configuration

The configuration of the Topic Manager Swiftlet is defined within the element

      <swiftlet name="sys$topicmanager" .../>

of the router's configuration file.

Attributes of Element "swiftlet"

Definition

Values

Element List "topics", Parent Element: "swiftlet"

Topic Definitions. This element list contains zero or more "topic" elements with this template definition:

Definition

Element List "slow-subscriber-conditions", Parent Element: "swiftlet"

Slow Subscriber Conditions. This element list contains zero or more "topic" elements with this template definition:

Definition

Values

Element List "static-remote-router-subscriptions", Parent Element: "swiftlet"

Static Remote Router Subscriptions. This element list contains zero or more "static-remote-router-subscription" elements with this template definition:

Definition

Element List "static-topic-subscriptions", Parent Element: "static-remote-router-subscription"

Static Topic Subscriptions. This element list contains zero or more "static-topic-subscription" elements with this template definition:

Definition

Values

Element "usage", Parent Element: "swiftlet"

Active Topics.

Element List "durables", Parent Element: "usage"

Active Durable Topic Subscriber. This element list contains zero or more "durables" elements with this template definition:

Definition

Values

Element List "subscriber", Parent Element: "usage"

Active Topic Subscriber. This element list contains zero or more "subscriber" elements with this template definition:

Definition

Values

Element List "subscriber-remote", Parent Element: "usage"

Subscriptions from remote Routers. This element list contains zero or more "router" elements with this template definition:

Definition

Element List "topics", Parent Element: "router"

Subscriptions from this Router. This element list contains zero or more "topic" elements with this template definition:

Definition