Store Swiftlet

The Store Swiftlet provides a transactional file-based message store where the following data is stored:

• Persistent messages within a transactional store

• Non-persistent messages within a swap store

• Durable subscriber definitions within a durable subscriber store

Tuning

Disk access has the most influence on the SwiftMQ performance. Normally, the default settings of the Store Swiftlet are sufficient and are optimized for the average standard operation. Concerning advanced applications, various tuning possibilities are discussed in this chapter.

Changing the Page Size

You can dramatically increase the performance of the persistent store by using a different page size that matches the size of your stored messages. We have created a How-To document that explains it in detail. Have a look!

Database Settings

The initial size of the file page.db may be configured via the attribute initial-db-page-size. The default setting is 25600 pages (50 MB). It is not guaranteed, but most of operating systems create a continuous space if the file size is given and if procurable. Such a coherent disk partition reduces the number of positionings during disk access. If your page.db is permanently greater (thus you store simultaneously many messages), you should increment the initial-page-size to allocate a maximum coherent area. Please notice, this attribute only has an effect if the file does not exist. Therefore you should first delete page.db and then start the router with the modified initial-db-page-size. Further, please take notice that all pages are read during the router startup to locate free pages. Thus, a larger size implicates a longer startup time.

Cache Size

The cache may be configured by the two attributes min-size and max-size. Both values are numbers of pages. The cache manager makes sure that the cache size retains during that interval. When reaching themax-size, automatic flushing of all dirty pages takes place unless these are in use (pin). The flush is continued as long as the min-size is reached resp. no dirty unpinned pages exist. The default settings are 1024 pages concerning the min-size and 2048 pages concerning the max-size. This is optimal for a write/read-transaction mix, i. e. messages are stored but they are reread at the best time. In that case, no disk access takes place with the exception of the transaction log. All transactions are served out of the cache as deallocated pages also remain within the cache. If you have many write-transactions and an extended message dwell, you should increase respectively the max-size to avoid the automatical flush.

Checkpoint Size

The attribute checkpoint-size specifies the size of the transaction log (transaction.log). After reaching, a checkpoint is performed. This means a pause of all transactions, a complete flush of the Cache, sync of the disk, and the enabling of all transactions. The checkpoint-size default setting is 10 MB. This proves optimal. A checkpoint then requires a few milliseconds. A minor size implicates more checkpoints and, as a result, more disk syncs. A larger size accordingly requires a bigger cache so that the duration of a checkpoint increases as more is written. Please consider that all transactions are stopped during a checkpoint.

Transaction Log Disk Sync and Group Commit

A disk sync of the transaction log may be forced over the attribute force-sync. The default setting is false. In the case of true a disk sync is executed with every interval of the log manager. During one interval, the log manager may write several log records into the transaction log (group commit) as it works asynchronously. However, this is only the case when having high parallelism of transactions. It doesn't increase the throughput of single transactions but only the overall throughput (scalability).

The default mode of false is less reliable and you may lose data if the computer crashes. However, the throughput is many times higher than with disk sync. Forcing disk syncs is reliable, but the performance is then bound to the speed of the disk, which varies.

Backup and Restore

Backup

A backup is done by intercepting a checkpoint. That is, a checkpoint is initiated and the backup is performed before transactions are restarted to get a consistent state of the store. During a backup operation, a save set is generated which includes all necessary files of the store. A save set has a timestamp and is generated below the backup directory. The number of generated save sets to keep is configurable and is maintained by the backup processor. A backup can be initiated manually via SwiftMQ Explorer / CLI or automatically through a backup job via the Scheduler Swiftlet.

Backup Configuration

The backup configuration takes place via the element backup of the Store Swiftlet. It contains 2 attributes. One is path and points per default to the directory backup below the router's store directory. This is the root path for backups. All save sets are created below that directory. The other attribute is keep-generations and specifies how many save sets should be kept in the backup directory. The default is 3 generations. That is, if a new save set is created, the backup processor checks the number of save sets in the backup directory and deletes the oldest save set until the number of save sets matches the number, specified in attribute keep-generations. If the value is changed, it will be automatically applied to the number of save sets. For example, if keep-generations is 4 and the number of save sets is also 4, and you change keep-generations to 2, the backup processor will automatically delete 2 save sets.

The following example uses an alternative backup path and specifies to keep 2 generations:

Example:

    <backup path="/usr/local/backup/swiftmq" keep-generations="2"/>

Performing a Backup via CLI

CLI provides a command backup, available within context /sys$store/backup. So you have to change to this context first. Thereafter, initiate the backup command:

Example:

    router1> cc /sys$store/backup
    router1/sys$store/backup> backup

    Backup initiated. Please watch Folder 'Generated Backup Save Sets'.

    router1/sys$store/backup>_

The backup runs asynchronously. Once it is finished, you'll find it below the /sys$store/usage context:

Example:

    router1/sys$store/backup> cc /sys$store/usage
    router1/sys$store/usage> lc

    Entity List: Generated Backup Save Sets
    Description: Generated Backup Save Sets

    Entities in this List:
    ----------------------
    saveset_20030217125000008
    saveset_20030220124842148
    saveset_20030220125000023

    router1/sys$store/usage>_

Performing a Backup via SwiftMQ Explorer App

Go to the folder Backup of the Store Swiftlet, select it, and do a right mouse click. A popup menu Perform Backup Now appears:

Click it and you'll see this message:

The backup operation is now performed asynchronously (during the next checkpoint). Once it is finished, you'll see a new entry in the folder Generated Backup Save Sets:

Schedule a Backup Job via the Scheduler Swiftlet

The Store Swiftlet registers a job Backup in the job group Store at the Scheduler Swiftlet:

This job can be used to schedule a backup at a specific time of the day. The following example schedules the Backup job every day at 00:00, 10:00, and 18:00:

Restore

To restore the content of a save set to the router's store, you have to stop the router. It is not possible to perform a restore while the router is running.

Thereafter you have to delete the following files from the router's store:

• page.db

• transaction.log

• all files ending with *.durable

Now go to the backup root path (specified in the path directory) and select the save set you to want to restore. Verify whether a file named .completed is contained in the save set. This file is created as the last operation during a backup after all files are synced with the disk. If it is not contained, then the backup is incomplete, you cannot use it and you have to switch to the previous generation (just check the name of the save set which contains a timestamp).

From the save set, copy:

• page.db

• all files ending with *.durable

to the resp. router's store directories. Don't wonder why the file transaction.log isn't contained in the save set. It is just that the backup is performed right after a checkpoint when the transaction.log is empty and thus there is no need to save it.

Start the router.

Shrinking the Database

Monitoring the Data Store

The usage of the data store page.db can be monitored. To this, the Usagesection of the Store Swiftlet contains a Files entity list with one entry page.db. This entry consists of attributes file-size, free-pages, and used-pages:

By clicking on the node, the entry is displayed as a table and you can add charts on the right side:

Shrinking the Data Store

When pages are released on the data store page.db, they are kept in a free page list and will be reused. The file is not automatically shrunk for performance reasons. This could lead to a large data store with many free pages. The Store Swiftlet allows performing a shrink during operation. The shrink command is available in context sys$store/database. It can be invoked from CLI as well as from the SwiftMQ Explorer App.

To invoke it from CLI, switch to context sys$store/database and perform the command shrink:

Example:

    router1> cc sys$store
    router1/sys$store> cc database
    router1/sys$store/database> shrink

    Shrink initiated.

    router1/sys$store/database> _

To router initiates a checkpoint and runs the shrink thereafter.

You can also shrink it from SwiftMQ Explorer App if you click on the Database node:

Schedule a Shrink Job via the Scheduler Swiftlet

The Store Swiftlet registers a job Shrink in the job group Store at the Scheduler Swiftlet:

This job can be used to schedule a shrink at a specific time of the day. The following example schedules the Shrink job at 00:00 on each day:

Analyzing the Store

Analyzing the Checkpoint Duration

To analyze how long a checkpoint takes, set this system property to true:

    -Dswiftmq.store.checkpoint.verbose=true

The Store Swiftlet writes a line on each checkpoint to System.out containing the time in milliseconds the checkpoint took.

Analyzing the Store Content

To analyze the content of the transactional store, set this system property to true:

    -Dswiftmq.store.analyze=true

On startup, the Store Swiftlet creates one file per queue with the extension .analyze and one file called rootindex.analyze within the working directory. These files contain a dump of the queue index resp. the root index. The queue messages aren't dumped.

Configuration

The configuration of the Store Swiftlet is defined within the element

      <swiftlet name="sys$store" .../>

of the router's configuration file.

Element "backup", Parent Element: "swiftlet"

Backup Settings.

Definition

Values

Element "database", Parent Element: "swiftlet"

Database Settings.

Definition

Values

Element "cache", Parent Element: "swiftlet"

Cache Settings.

Definition

Values

Element "transaction-log", Parent Element: "swiftlet"

Transaction Log Settings.

Definition

Values

Element "swap", Parent Element: "swiftlet"

Swap Settings.

Definition

Values

Element "durable-subscriber", Parent Element: "swiftlet"

Durable Subscriber Settings.

Definition

Values

Element "usage", Parent Element: "swiftlet"

Usage.

Element List "backup", Parent Element: "usage"

Generated Backup Save Sets. This element list contains zero or more "backup" elements with this template definition:

Definition

Element List "files", Parent Element: "backup"

Files. This element list contains zero or more "file" elements with this template definition:

Definition

Values

Element List "files", Parent Element: "usage"

Files. This element list contains zero or more "file" elements with this template definition:

Definition

Values

Element List "scan", Parent Element: "usage"

Files. This element list contains zero or more "scan" elements with this template definition:

Definition

Values