FileCache Swiftlet

Introduction

Sending very large Messages with JMS?

The Java Message Service standard is not designed to handle the transfer of very large messages as it requires storing messages completely in memory. So whenever very large messages are being sent, one pays with high memory consumption. It is impossible to send messages of gigabyte size. This is where protocols like FTP were designed for.

What Users want

JMS users don't want to use another protocol connection along with their existing JMS connection. The handling is difficult so client applications become more complex, JMS-like reliability is not ensured, firewall issues may arise, and so on. JMS users just want to send any kind of data of any size over their existing JMS infrastructure.

What the File Cache Swiftlet provides

Send and receive Files of any Size over existing JMS Connection, multithreaded

The File Cache Swiftlet provides a complete and sophisticated solution to transfer files of arbitrary size between JMS clients and a SwiftMQ Universal Router. JMS clients send and receive files with an easy-to-use Filetransfer client API that uses an existing JMS connection and performs the actual file transfer over a dedicated JMS session which allows transferring files in different threads.

Separate Store, asynchronous/non-persistent Transfer, Digest

The storage of files takes place outside of the persistent message store of a router in a separate directory so it doesn't affect persistent message performance. The transfer itself has performed asynchronously over pairs of temporary queues by splitting a file into pieces and sending them as non-persistent JMS messages. This guarantees very high performance. The integrity of a file is ensured by a digest.

Direct File-to-File Transfer, Links, Filetransfer API

Files are transferred directly from/to a File object on disk and a file cache at the respective Router. Once a file has been stored, a unique link is returned which identifies the file at the file cache. This link can then be used to retrieve the file from the file cache through the Filetransfer API. The same file can be retrieved by one (like point-to-point) or multiple receivers (like publish/subscribe).

Expiration, Download Limit, Password Protection, Private Files, Message Selectors

It is possible to specify an expiration time or a maximum number of downloads per file. They can also be protected by a password or marked as private. JMS properties such as filename, size, etc are automatically set from the Filetransfer API during transfer. Arbitrary custom JMS properties can be added. Both sets can be used in JMS selectors in a file cache query to retrieve files from a cache like the following example which selects all zip files of a particular customerid range that are bigger than 1 MB:

     JMS_SWIFTMQ_FT_FILENAME like '%.zip'
       and customerid between 10000 and 11500
         and JMS_SWIFTMQ_FT_SIZE > 1024 * 1024

Multiple File Caches, Default Behavior, Copy/Move between File Caches, Authentication

An unlimited number of different files caches can be created for different purposes. Each file cache can have different attributes concerning default expiration, maximum sizes, etc. Files can be copied or moved between file caches and access to a file cache can be protected through the Authentication Swiftlet.

Managing File Caches

Overview

A file cache stores files and provides access to it. It has 2 components: the directory where the files are stored and a file registry where the information about the files is held.

The registry is realized as a regular queue where a persistent message for each file is stored which contains all the declared message properties from the file sender. This makes it possible to provide the powerful Filetransfer.query(selector) functionality by simply applying the JMS selector on the file registry queue.

Queues used by a File Cache

The following queues are used from a file cache. They will be automatically created if they do not exist.

The actual file transfer is done over 2 dedicated temp queues which are created per Filetransfer session.

Access Authentication

Access to a file cache can be controlled by granting selective access to the request queue of a file cache. A further way is to protect single files with a password and/or to declare them as private files.

See the Filetransfer Client for detailed information.

Creating a File Cache

File Cache with SwiftMQ Universal Router (non HA)

To create a new file cache, go to File Cache Swiftlet / Caches:

Then click Create new Entity:

The attributes have the following meanings:

File Cache with SwiftMQ HA Router

To operate a file cache with SwiftMQ HA Router it is required to use a shared directory accessible from both HA instances.

Therefore the HA version of the File Cache Swiftlet asks for two directory names.

Both directories must point to the same physical storage and must be specified as the local directory name of the respective HA instance. The directories are either relative to the working directory scripts/platform or can be absolute.

Deleting a File Cache

To delete an existing file cache, open File Cache Swiftlet / Caches, select the file cache, and select Delete Entity:

Note that it deletes the file cache configuration only. It does not delete the files of the cache nor the directory or the file registry. This needs to be performed separately.

Monitoring a File Cache

All file caches are listed under the Usage section of the File Cache Swiftlet. To show a summary of all file caches, select Active Caches:

The table shows a summary of all file caches:

Listing Files

To list files of the cache, click on the file cache entry in the Active Caches table and select List Cache Files:

Which shows the files contained in the file cache:

Selecting an entry will show you the details:

The following message properties are automatically set during send:

In addition to the Filetransfer properties listed above, all custom properties can be used in a message selector, too. For example, if a custom property orderid was set as a custom property and one wants to get all PDF files for a particular order id, a query might look like this:

      orderid = 12345 and JMS_SWIFTMQ_FT_FILENAME like '%.pdf'

To select all zip files of a particular customerId range that are bigger than 1 MB, use this selector:

      JMS_SWIFTMQ_FT_FILENAME like '%.zip'
        and customerid between 10000 and 11500
          and JMS_SWIFTMQ_FT_SIZE > 1024 * 1024

Monitoring Uploads

The currently active uploads to a file cache can be monitored by selecting the Active Uploads node of an active file cache:

Monitoring Downloads

The currently active downloads from a file cache can be monitored by selecting the Active Downloads node of an active file cache:

Schedulable File Cache Jobs

Overview

The File Cache Swiftlet registers jobs in the job group File Cache at the Scheduler Swiftlet:

These jobs can be scheduled via the Scheduler Swiftlet to run at specific times or in intervals, based on calendars and so on.

File Cache Cleanup

The File Cache Cleanup job removes expired files from file caches. The file caches are determined from a single parameter that contains a SQL-Like predicate. For example, a predicate of order% performs a cleanup on all file caches whose names start with order. The File Cache Cleanup job provides more flexibility than using the interval-based cleanup per file cache. Keep in mind that you should set the cleanup intervals to 0 if you use the File Cache Cleanup job.

File Copier

The File Copier job copies files from a source cache, which must reside on the router where the job runs, to a target cache, which can be the local or a remote router. Transferred files at the target cache are identified by a new link.

File Purger

The File Purger job removes messages from a single file cache, optionally based on a message selector.

Configuration

The configuration of the FileCache Swiftlet is defined within the element

      <swiftlet name="sys$filecache" .../>

of the router's configuration file.

Attributes of Element "swiftlet"

Definition

Values

Element List "caches", Parent Element: "swiftlet"

Caches. This element list contains zero or more "cache" elements with this template definition:

Definition

Values

Element List "usage", Parent Element: "swiftlet"

Active Caches. This element list contains zero or more "usage" elements with this template definition:

Definition

Values

Element List "uploads", Parent Element: "usage"

Active Uploads. This element list contains zero or more "upload" elements with this template definition:

Definition

Values

Element List "downloads", Parent Element: "usage"

Active Downloads. This element list contains zero or more "download" elements with this template definition:

Definition

Values