Chapter 7. Clustering and federation

7.1. Messaging Clusters
7.2. Federation

7.1. Messaging Clusters

A Messaging Cluster is a group of brokers that act as a single broker. Changes on any broker are replicated to all other brokers in the same Messaging Cluster, so if one broker fails, its clients can fail-over to another broker without loss of state. The brokers in a Messaging Cluster may run on the same host or on different hosts.

Messaging Clusters are implemented using using OpenAIS, which provides a reliable multicast protocol, tools, and infrastructure for implementing replicated services. You must install and configure OpenAIS to use MRG broker groups.

Messaging Clusters can be used together with Red Hat Clustering Services (RHCS) by starting brokers with the --cluster-cman option

Message Clusters in MRG Messaging uses an Active/Active model. In this model, all possible brokers are active at all times. Producers and consumers can be connected to any broker in the cluster. Additionally, any broker can be killed and restarted, and the cluster will retain its operational state. This model provides scalability and enhanced load-balancing.

Clustering requirements

The user that runs the broker must be a member of the group root and must have their primary group set to ais. A user called 'qpidd' with the correct membership is created during installation.

Starting the AIS service

Clustering requires the Application Interface Specification (AIS) to be running. AIS requires the openais package.

Ensure the openais service is not running:

# service openais stop
Stopping OpenAIS daemon (aisexec):              [  OK  ]

Create a file at /etc/ais/openais.conf using the following information:

totem {
  version: 2
  secauth: off
  threads: 0
  interface {
    ringnumber: 0
    bindnetaddr: 192.168.1.0    # Modify for your network
    mcastaddr: 226.94.1.1
    mcastport: 5405
  }
}

logging {
  to_syslog: yes
}

amf {
  mode: disabled
}

The bindnetaddr entry should be set to the subnet you will use for cluster multicast. Use the same subnet for for all hosts in the cluster. You can find the subnet with /sbin/ifconfig. For example:
```
# /sbin/ifconfig eth0
  eth0      Link encap:Ethernet  HWaddr 00:1E:37:88:72:8A  
            inet addr:192.168.1.103  Bcast:192.168.1.255  Mask:255.255.255.0
```
In this case the local address is 192.168.1.103 and the subnet mask is 255.255.255.0. The subnet is the bitwise AND of the mask and the local address, 192.168.1.255.

Restart the openais service:

# service openais start
Starting OpenAIS daemon (aisexec):              [  OK  ]

Setting up the redundant ring

OpenAIS supports a redundant ring protocol (RRP), which uses two physically separate networks for cluster communication, so that the cluster can continue to operate if one network fails.

Choose the replication mode for your environment. RRP has 3 modes:
- active
  Active replication can offer slightly lower latency in faulty network environments, however it can reduce throughput
- passive
  Passive replication can nearly double the speed from transmit to delivery, but also carries the potential for the protocol to become bound to a single CPU
- none
  Disables redundant ring.
To enable RRP make the following changes to openais.conf:
1. In the totem section, add rrp_mode=active or rrp_mode=passive
2. Add a second interface section with a different bindnetaddr for your second network.
For more information about configuring openAIS see the openais.conf(5) man page.
Note
Make sure you run brokers that are members of a cluster as a user with primary group "ais".

Once the cluster is running, you can view the active state from qpid-tool:

Object of type org.apache.qpid.cluster:cluster: (last sample time: 15:07:23)
Type      Element       110
=============================================================================
property  brokerRef     102
property  clusterName   test_cluster
property  clusterID     1a911c67-6fcd-4ffa-8fc8-0dc74ec14ec0
property  publishedURL  amqp:tcp:10.16.18.96:5672
property  clusterSize   2
property  status        ACTIVE
property  members       amqp:tcp:10.16.18.96:5672 amqp:tcp:10.16.18.96:5673

You can also see this information from the MRG Management Console. See Section 2.1, “MRG Management Console” for more information.

Options for clustering
`--cluster-name NAME`	Name of the Messaging Cluster to join. A Messaging Cluster consists of all brokers started with the same cluster-name and openais configuration.
`--cluster-url URL`	An AMQP URL containing the local address advertised to clients for fail-over connections. This is different for each host. By default, all local addresses are advertized. You only need to set this if Your host has more than one active network interface. You want to restrict client fail-over to a specific interface or interfaces. The URL is of the form `amqp:tcp:<host>:<port>[,tcp:<host>:<port> ...]` For example: `amqp:tcp:192.168.1.103:5672,tcp:192.168.1.105:5672`
`--cluster-cman`	CMAN protects against the "split-brain" condition, in which a network failure splits the cluster into two sub-clusters that cannot communicate with each other. When "split-brain" occurs, each of the sub-clusters can access shared resources without knowledge of the other sub-cluster, resulting in corrupted cluster integrity. To avoid "split-brain", CMAN uses the notion of a "quorum". If more than half the cluster nodes are active, the cluster has quorum and can act. If half (or fewer) nodes are active, the cluster does not have quorum, and all cluster activity is stopped. There are other ways to define the quorum for particular use cases (e.g. a cluster of only 2 members), see the CMAN documentation for more detail. When enabled, the MRG broker will wait until it belongs to a quorate cluster before accepting client connections. It continually monitors the quorum status and shuts down immediately if the node it runs on loses touch with the quorum.