Skip to main content
Skip table of contents

FileCache Swiftlet

Introduction

Sending very large Messages with JMS?

The Java Message Service standard is not designed to handle the transfer of very large messages as it requires storing messages completely in memory. So whenever very large messages are being sent, one pays with high memory consumption. It is impossible to send messages of gigabyte size. This is where protocols like FTP were designed for.

What Users want

JMS users don't want to use another protocol connection along with their existing JMS connection. The handling is difficult so client applications become more complex, JMS-like reliability is not ensured, firewall issues may arise, and so on. JMS users just want to send any kind of data of any size over their existing JMS infrastructure.

What the File Cache Swiftlet provides

Send and receive Files of any Size over existing JMS Connection, multithreaded

The File Cache Swiftlet provides a complete and sophisticated solution to transfer files of arbitrary size between JMS clients and a SwiftMQ Universal Router. JMS clients send and receive files with an easy-to-use Filetransfer client API that uses an existing JMS connection and performs the actual file transfer over a dedicated JMS session which allows transferring files in different threads.

Separate Store, asynchronous/non-persistent Transfer, Digest

The storage of files takes place outside of the persistent message store of a router in a separate directory so it doesn't affect persistent message performance. The transfer itself has performed asynchronously over pairs of temporary queues by splitting a file into pieces and sending them as non-persistent JMS messages. This guarantees very high performance. The integrity of a file is ensured by a digest.

Direct File-to-File Transfer, Links, Filetransfer API

Files are transferred directly from/to a File object on disk and a file cache at the respective Router. Once a file has been stored, a unique link is returned which identifies the file at the file cache. This link can then be used to retrieve the file from the file cache through the Filetransfer API. The same file can be retrieved by one (like point-to-point) or multiple receivers (like publish/subscribe).

Expiration, Download Limit, Password Protection, Private Files, Message Selectors

It is possible to specify an expiration time or a maximum number of downloads per file. They can also be protected by a password or marked as private. JMS properties such as filename, size, etc are automatically set from the Filetransfer API during transfer. Arbitrary custom JMS properties can be added. Both sets can be used in JMS selectors in a file cache query to retrieve files from a cache like the following example which selects all zip files of a particular customerid range that are bigger than 1 MB:

SQL
     JMS_SWIFTMQ_FT_FILENAME like '%.zip'
       and customerid between 10000 and 11500
         and JMS_SWIFTMQ_FT_SIZE > 1024 * 1024

Multiple File Caches, Default Behavior, Copy/Move between File Caches, Authentication

An unlimited number of different files caches can be created for different purposes. Each file cache can have different attributes concerning default expiration, maximum sizes, etc. Files can be copied or moved between file caches and access to a file cache can be protected through the Authentication Swiftlet.

Managing File Caches

Overview

A file cache stores files and provides access to it. It has 2 components: the directory where the files are stored and a file registry where the information about the files is held.

The registry is realized as a regular queue where a persistent message for each file is stored which contains all the declared message properties from the file sender. This makes it possible to provide the powerful Filetransfer.query(selector) functionality by simply applying the JMS selector on the file registry queue.

Queues used by a File Cache

The following queues are used from a file cache. They will be automatically created if they do not exist.

Queue Name

Purpose

sys$fcreg-<cachename>

The file registry. One persistent message is stored here for each file.

swiftmqfcreq-<cachename>

The request queue used by Filetransfer clients to establish filetransfer sessions.

The actual file transfer is done over 2 dedicated temp queues which are created per Filetransfer session.

Access Authentication

Access to a file cache can be controlled by granting selective access to the request queue of a file cache. A further way is to protect single files with a password and/or to declare them as private files.

See the Filetransfer Client for detailed information.

Creating a File Cache

File Cache with SwiftMQ Universal Router (non HA)

To create a new file cache, go to File Cache Swiftlet / Caches:

Then click Create new Entity:

The attributes have the following meanings:

Attribute

Default

Description

Name

The name of the file cache.

Chunk Size (KB)

10

The chunk size in KB used for the file transfer. Files are transferred in chunks of this size. Do not change the default of 10 KB except you know why.

Cleanup Interval (ms)

120000

The interval in which this file cache deletes expired files.

Default File Expiration (ms)

-1

This will be applied to all files stored without an expiration time. Default is -1 (no expiration).

Directory

The directory in which the files will be stored. This directory is either relative to the working directory scripts/platform or can be absolute.

Enabled

false

Enables/disables this file cache.

Inactivity Timeout (ms)

60000

Timeout after which a Filetransfer session is closed due to inactivity.

Maximum Cache Size (MB)

-1 (unlimited)

The limit in MB this file cache can store.

Maximum File Size (MB)

-1 (unlimited)

The limit in MB of files stored in this file cache.

Maximum Number of File

-1 (unlimited)

The limit in number of files stored in this file cache.

File Cache with SwiftMQ HA Router

To operate a file cache with SwiftMQ HA Router it is required to use a shared directory accessible from both HA instances.

Therefore the HA version of the File Cache Swiftlet asks for two directory names.

Attribute

Default

Description

Directory

The directory in which the files will be stored of the currently ACTIVE HA instance.

Directory 2nd HA Instance

The directory in which the files will be stored of the currently STANDBY HA instance.

Both directories must point to the same physical storage and must be specified as the local directory name of the respective HA instance. The directories are either relative to the working directory scripts/platform or can be absolute.

Deleting a File Cache

To delete an existing file cache, open File Cache Swiftlet / Caches, select the file cache, and select Delete Entity:

Note that it deletes the file cache configuration only. It does not delete the files of the cache nor the directory or the file registry. This needs to be performed separately.

Monitoring a File Cache

All file caches are listed under the Usage section of the File Cache Swiftlet. To show a summary of all file caches, select Active Caches:

The table shows a summary of all file caches:

Listing Files

To list files of the cache, click on the file cache entry in the Active Caches table and select List Cache Files:

Which shows the files contained in the file cache:

Selecting an entry will show you the details:

The following message properties are automatically set during send:

Property

Type

Value

JMS_SWIFTMQ_FT_USERNAME

String

The JMS user who sent the file.

JMS_SWIFTMQ_FT_FILENAME

String

The original filename

JMS_SWIFTMQ_FT_SIZE

long

File size in bytes

JMS_SWIFTMQ_FT_EXPIRATION

long

The absolute expiration time in ms (since 01/01/1970)

JMS_SWIFTMQ_FT_DIGESTTYPE

String

The digest type, e.g. "SHA1".

JMS_SWIFTMQ_FT_STORETIME

long

The absolute store time in ms (since 01/01/1970)

JMS_SWIFTMQ_FT_DELAFcolumnL

boolean

The number of downloads after which the file will be deleted.

JMS_SWIFTMQ_FT_DOWNLOADCOUNT

String

The current number of downloads.

In addition to the Filetransfer properties listed above, all custom properties can be used in a message selector, too. For example, if a custom property orderid was set as a custom property and one wants to get all PDF files for a particular order id, a query might look like this:

SQL
      orderid = 12345 and JMS_SWIFTMQ_FT_FILENAME like '%.pdf'

To select all zip files of a particular customerId range that are bigger than 1 MB, use this selector:

SQL
      JMS_SWIFTMQ_FT_FILENAME like '%.zip'
        and customerid between 10000 and 11500
          and JMS_SWIFTMQ_FT_SIZE > 1024 * 1024

Monitoring Uploads

The currently active uploads to a file cache can be monitored by selecting the Active Uploads node of an active file cache:

Monitoring Downloads

The currently active downloads from a file cache can be monitored by selecting the Active Downloads node of an active file cache:

Schedulable File Cache Jobs

Overview

The File Cache Swiftlet registers jobs in the job group File Cache at the Scheduler Swiftlet:

These jobs can be scheduled via the Scheduler Swiftlet to run at specific times or in intervals, based on calendars and so on.

File Cache Cleanup

The File Cache Cleanup job removes expired files from file caches. The file caches are determined from a single parameter that contains a SQL-Like predicate. For example, a predicate of order% performs a cleanup on all file caches whose names start with order. The File Cache Cleanup job provides more flexibility than using the interval-based cleanup per file cache. Keep in mind that you should set the cleanup intervals to 0 if you use the File Cache Cleanup job.

Parameter

Mandatory

Description

Cache Name Predicate

Yes

SQL-Like predicate to select file cache names

File Copier

The File Copier job copies files from a source cache, which must reside on the router where the job runs, to a target cache, which can be the local or a remote router. Transferred files at the target cache are identified by a new link.

Parameter

Mandatory

Description

Source Cache

Yes

Name of the source file cache

Target Router

Yes

Name of the target router

Target Cache

Yes

Name of the target file cache

Message Selector

No

Optional SQL92 message selector to select files

Delete After Transfer

No

Delete file after transfer (= move), default is false

File Purger

The File Purger job removes messages from a single file cache, optionally based on a message selector.

Parameter

Mandatory

Description

Cache Name

Yes

File cache name

Message Selector

No

Optional SQL92 message selector to select files

Configuration

The configuration of the FileCache Swiftlet is defined within the element

XML
      <swiftlet name="sys$filecache" .../>

of the router's configuration file.

Attributes of Element "swiftlet"

Definition

Attribute

Type

Mandatory

Description

collect-interval

java.lang.Long

No

Collect Interval (ms) for the Usage Section

Values

Attribute

Values

collect-interval

Default: 1000

Element List "caches", Parent Element: "swiftlet"

Caches. This element list contains zero or more "cache" elements with this template definition:

Definition

Attribute

Type

Mandatory

Description

name

java.lang.String

Yes

Name of this Cache

enabled

java.lang.Boolean

No

Enables/Disables this Cache

directory

java.lang.String

Yes

Local Directory to store Files

max-cache-size-mb

java.lang.Integer

No

Maximum Size of all Files in this Cache

chunk-size-kb

java.lang.Integer

No

Size of the Chunks for the File Transfer

max-file-size-mb

java.lang.Integer

No

Maximum Size of a single File in this Cache

max-number-files

java.lang.Integer

No

Maximum Number Files in this Cache

default-expiration

java.lang.Long

No

Default Exiration of Files in this Cache

inactivity-timeout

java.lang.Long

No

Inactivity Timeout after which a File Transfer is stopped

cleanup-interval

java.lang.Long

No

Interval to Cleanup expired Files of this Cache

Values

Attribute

Values

enabled

Default: false

directory

max-cache-size-mb

Default: -1

chunk-size-kb

Min: 1
Default: 10

max-file-size-mb

Default: -1

max-number-files

Default: -1

default-expiration

Default: -1

inactivity-timeout

Default: 60000

cleanup-interval

Default: 120000

Element List "usage", Parent Element: "swiftlet"

Active Caches. This element list contains zero or more "usage" elements with this template definition:

Definition

Attribute

Type

Mandatory

Description

name

java.lang.String

Yes

Name of this Active Cache

a01-cache-size-files

java.lang.Integer

No

Current Number Files in this Cache

a02-cache-size-mb

java.lang.Integer

No

Current Size of all Files in this Cache

a03-active-uploads

java.lang.Integer

No

Active Uploads

a04-active-downloads

java.lang.Integer

No

Active Downloads

Values

Attribute

Values

a01-cache-size-files

Default: 0

a02-cache-size-mb

Default: 0

a03-active-uploads

Default: 0

a04-active-downloads

Default: 0

Element List "uploads", Parent Element: "usage"

Active Uploads. This element list contains zero or more "upload" elements with this template definition:

Definition

Attribute

Type

Mandatory

Description

name

java.lang.String

Yes

Name of this Active Upload

a00-file-name

java.lang.String

No

File Name

a01-file-size-mb

java.lang.String

No

File Size (MB)

a02-uploaded-size-mb

java.lang.String

No

Uploaded Size (MB)

a03-upload-mb-sec

java.lang.String

No

Upload MB/sec

a04-uploaded-pct

java.lang.Integer

No

Upload Percent

Values

Attribute

Values

a00-file-name

a01-file-size-mb

Default: 0

a02-uploaded-size-mb

Default: 0

a03-upload-mb-sec

Default: 0

a04-uploaded-pct

Default: 0

Element List "downloads", Parent Element: "usage"

Active Downloads. This element list contains zero or more "download" elements with this template definition:

Definition

Attribute

Type

Mandatory

Description

name

java.lang.String

Yes

Name of this Active Download

a00-file-name

java.lang.String

No

File Name

a01-file-size-mb

java.lang.String

No

File Size (MB)

a02-downloaded-size-mb

java.lang.String

No

Downloaded Size (MB)

a03-download-mb-sec

java.lang.String

No

Download MB/sec

a04-downloaded-pct

java.lang.Integer

No

Download Percent

Values

Attribute

Values

a00-file-name

a01-file-size-mb

Default: 0

a02-downloaded-size-mb

Default: 0

a03-download-mb-sec

Default: 0

a04-downloaded-pct

Default: 0

JavaScript errors detected

Please note, these errors can depend on your browser setup.

If this problem persists, please contact our support.