Chapter 6. Persistence

6.1.1. Estimating Resources

A persistence library allows MRG Messaging to store messages and queue configuration, ready to be reloaded in the event of machine or network failure. When the persistent store module is loaded, it allows messages and other persistent state information to be recovered when a broker is restarted.

When a transaction occurs, a journal called the Transaction Prepared List (or TPL) is initialized. This journal stores the state of the transaction in the broker as it occurs. If the broker then crashes or is stopped for any reason, the journal can restore the information to the broker when it is restarted. In this way, any in-progress transactions that were either prepared or committed when the broker failed will be recovered.

For storage of the broker itself, and for exchange and binding information, the Berkeley Database is used to store and recover information.

When adjusting parameters for persistence, there are three settings specific to the TPL. This allows alternate settings for those messages that are recovered.

In order for messages to be stored the persistence store must be loaded. The --store-dir command specifies the directory used for for the persistence store and any configuration information. The default directory is /var/lib/qpidd. See Table 6.1, “Persistence Options” for options on how to change this behavior.

Important

If the persistence module is not loaded, messages and the broker state will not be stored to disk, even if the queue and messages sent to it are marked persistent.

Important

Only one running broker can access a data directory at a time. If another broker attempts to access the data directory it will fail with an error stating: Exception: Data directory is locked by another process.

In addition to loading the persistence store, queues and messages also need to be identified as durable. This can be done in the client application or by using the qpid-config command line tool. See the MRG Messaging Tutorial for more information about creating client applications.

It is important to set the store journal size to match the anticipated persistent message queue size. The store uses a fixed-size circular file buffer, so it is possible for the store to run out of space to queue messages if consumers are slow or messages are very large.

Each queue that is marked persistent will cause the broker to create an instance of the store together with its files. The broker will stop accepting persistent messages when approximately 80% of the capacity is reached. Consuming of messages (dequeueing), however will be allowed to continue. Once the dequeued messages have cleared sufficient space, message queuing will continue as normal. As a rule of thumb, the journal capacity should be about double the size expected to be stored on the disk at any one moment in time.

Warning

A totally full condition on the journal (in which there is no more write space in the circular buffer) is a fatal condition. It is possible to read the messages in a full journal, but not to dequeue them, as dequeueing requires the ability to write dequeue records. To prevent this, message queuing is disabled at 80% capacity.

Persistence Options
`--store-dir DIRECTORY`	Specifies the directory used for storage of persistence configuration information. The default is `/var/lib/qpidd`.
`--num-jfiles NUMBER`	Set the number of files for each instance of the persistence journal. The default is 8.
`--jfile-size-pgs NUMBER`	Set the size of each journal file in multiples of 64KB. The default is 24.
`--wcache-page-size NUMBER`	The size (in KB) of the pages in the write page cache. Allowable values must be powers of 2 (1, 2, 4, ... 128). Lower values will decrease latency but also decrease throughput. The default is 32.
`--tpl-num-jfiles NUMBER`	Set the number of files for each instance of the TPL journal. The default is 8.
`--tpl-jfile-size-pgs NUMBER`	Set the size of each TPL journal file in multiples of 64KB. The default is 24.
`--tpl-wcache-page-size NUMBER`	The size (in KB) of the pages in the TPL write page cache. Allowable values must be powers of 2 (1, 2, 4, ... 128). Lower values will decrease latency but also decrease throughput. The default is 32.

Table 6.1. Persistence Options

6.1. Persistent Queues

When a broker creates persistent queues, the store module is used to create, write and maintain the queue journal files on disk for as long as the queue exists. However, opening a number of persistent queues at the same time can cause system resource limitations. This section discusses some considerations to avoid reaching default limits or exhausting system resources.

6.1.1. Estimating Resources

Estimate the number of simultaneous persistent queues

Determine the number of simultaneous persistent queues that will be required. This number is the primary value that determines resource consumption, so it will be helpful to estimate typical, best and worst case scenarios.

Estimate the required number of file handles

Once the likely range of simultaneous persistent queues is known, it is possible to estimate the number of file handles that will be consumed. This calculation is effected by the broker store settings.

The broker store parameter --num-jfiles (or the equivalent setting in the broker configuration file) sets the default number of files used for all persistent queues that will be created on that broker – and thus determines the rate at which file handles will be consumed. This parameter defaults to 8 files.

Note

The MRG Management Console may be used to override these defaults and create individual queues that have file geometry parameters (including the number of files) that differ from this setting.

The store uses a single journal instance for storing transaction boundaries. This journal is called the Transaction Prepared List (TPL). The TPL is initialized when the first transaction occurs on any persistent queue, and has its own geometry parameters, separate from the other journal instances. The --tpl-num-jfiles parameter (or the equivalent setting in the broker configuration file) has little effect on overall file handle consumption and can be ignored in the calculation.

The broker opens one write file handle for each file. It will also open one for the read pipeline on initialization. Other file handles are opened temporarily during recovery, but are closed again. Once normal operations begin, the read pipeline will eventually be invalidated by overwrite, and the read handle closed and released. However, if flow-to-disk is initiated on a queue the read pipeline may be reinitialized, and a read file handle will be opened for that queue until it is invalidated by overwrite once again.

Note

When using default settings, expect to consume nine file handles per queue with no transactions.

This example uses the formulas given above to calculate the required number of file handles.

Using the default settings, 6,144 simultaneous persistent queues requires 6,144 * 9 = 55,296 file handles.

Example 6.1. Calculating the number of file handles

Estimate the required number of AIO event handles

The store reserves one AIO (Asynchronous Input/Output) event handle for each page of the journal memory cache for both read and write pipelines. Since the overall size of each of these caches is fixed at 1MB, the size of the pages will directly affect the number of pages and hence the number of AIO event handles being reserved on queue creation. The broker parameter --wcache-page-size sets the journal write cache page size. The total number of pages is obtained by dividing the total cache size by the page size. The journal read cache page size is not adjustable because it is used internally only to keep the cache full for read operations.

The write cache page size affects message storage latency for persistent queues and messages. Smaller page sizes means that messages are written to disk more quickly, although this occurrs at the expense of throughput and a greater number of pages and consumed AIO event handles. Conversely, larger page sizes improve overall persistent message throughput and lowers the page count, at the expense of message latency.

The kernel manages this resource on a system-wide basis. If any other processes use the AIO system, then their AIO event handle usage must be added to any estimates.

Note

When using default settings, expect to consume 64 AIO event handles per queue with no transactions.

The number of pages in the cache is obtained by dividing the page size by the fixed total cache size.

This example uses the formula given above to calculate the required number of AIO event handles.

Using the default settings, 6144 simultaneous queues requires 6,144 * 64 = 393,216 AIO handles.

Example 6.2. Calculating the number of AIO event handles

Estimate the required memory

There are two main memory allocations resulting from persistent queue declarations.

Journal cache: each persistent queue is allocated 1 MB for a journal write cache and 1 MB for a journal read cache. The journal cache sizes cannot be adjusted, however the number of pages used for the journal write cache can be adjusted.

Kernel: the kernel reserves space for expected events. This amount is then rounded up to the memory page size (4KB). An overhead of 312 bytes for every AIO context is then added to the total.

Note

When using default settings, expect to consume about 2MB per queue. The kernel effects are neglible.

This example uses the formulas given above to estimate the required memory.

Using the default settings, 5,000 simultaneous persistent queues will require about 10GB of cache memory. If the average queue depth is 1,000 messages and the average message size is 1KB, the broker will need 5,000 * 1,000 * 1 = 5,000,000KB (5GB) to keep the messages in the queues.

Example 6.3. Estimating the required memory

Changing the resource limits

Change the file handle limit by switching to the root user and opening the /etc/security/limits.conf in your preferred text editor. Add the following line for the qpidd user to set both the hard and soft limits:
```
qpidd - nofile 32768
```
Change the AIO handle limit by opening the /etc/sysctl.conf and adding the following line:
```
fs.aio-max-nr = 262144
```
Check that the system that will run the broker has sufficient memory to support the required number of queues.
After making the changes, perform a reboot of the system to start using the new settings. Check the appropriate files to ensure the changes have persisted before starting the broker.

Common errors

There are two errors that can occur as a result of resource problems.

jexception 0x0400 fcntl::clean_file() threw JERR_FCNTL_OPENWR: Unable to open file for write. 
(open() failed: errno=24 (Too many open files))

This error occurs if the broker runs out of available file handles. Under typical default conditions, the broker store will consume all available handles at around 110 queues.

jexception 0x0103 pmgr::initialize() threw JERR__AIO: AIO error. 
(io_queue_init() failed: errno=11 (Resource temporarily unavailable))

This error occurs if the broker runs out of available AIO event handles.