Highly Available and Redundant
Cluster continuous replication (CCR) is one of the high availability features of Microsoft Exchange Server 2007 which includes the asynchronous log shipping and replay technology built into Exchange 2007. This technology removes the requirement for shared disk and duplicate hardware while providing for a highly available system to the databases themselves. This removes the single point of failure associated with Shared Disk that is present in a single copy cluster when a quorum disk, transaction log spindle and database spindle are present and shared between multiple nodes.
- CCR is designed to provide high availability for Exchange 2007 Mailbox servers by providing a solution that addresses the following requirements:
- No Single point of failure
- Data availability and resilience to failure
- Service availability and resilience to failure
- Backup data center support, or "Geo Clustering"
- Rapid recovery with current data
- Low cost
- Wide range of support of disk technologies
- A Simplified solution
Core Technology
CCR combines the architecture of existing Single Copy Clusters but addresses the single point of failure of using shared disk associated with one copy of the database. The core technology consist of the following features:
- Failover and virtualization features provided by Microsoft failover clusters allowing for redundant nodes.
- Transaction log replication and replay features in Exchange 2007 for replication of the database data to an alternate location.
- Message queue feature of the Hub Transport server called the transport dumpster to prevent mail loss.
As one can see in the previous diagram CCR uses two computers (referred to as nodes) joined in a single Microsoft failover cluster, in this case Active Node 1 and Passive Node 2. The nodes in the failover cluster host a single clustered mailbox server called an Exchange Virtual Server. The concept of an Exchange Virtual Server existed in previous versions of Exchange Clustering. The node running a clustered mailbox server in control of the Exchange Virtual Server is an active node, and a node that is not running a clustered mailbox server or the Exchange Virtual Server, but is part of the cluster, is a passive node.
Majority Node Site Quorum
The failover cluster is built using the Windows Cluster service, and a new type of cluster quorum. CCR uses the file share witness on a third computer to avoid an occurrence of network partition within the cluster, also known as split brain syndrome. Split brain syndrome occurs when all networks designated to carry internal cluster communications fail, and nodes cannot receive heartbeat signals from each other. Split brain syndrome is prevented by always requiring a majority of the two nodes and the file share to be available and interacting for the clustered mailbox server to be operational. When a majority of the computers are communicating, the cluster is said to have a quorum. The file share for the file share witness can be hosted on any computer running Windows Server 2003 SP1 or Windows Server 2003 R2. It is recommended that a Hub Transport Server in the Active Directory server site containing the cluster nodes be used to home the File Share Witness. For details on the new Quorum model see KB921181 hotfix.
There are two new features introduced though the use of this hotfix
- The file share witness feature is an improvement to the current MNS quorum model. This feature lets you use a file share that is external to the cluster as an additional vote to determine the status of the cluster in a two-node MNS quorum cluster deployment.
- The configurable cluster heartbeats feature enables you to configure cluster heartbeat parameters. This may help avoid unnecessary cluster failovers. These failovers occur because of a temporary network problem that may cause packets to be dropped or delayed. The configurable cluster heartbeats feature is designed for environments where cluster nodes are geographically dispersed, as in the case of a multiple data center CCR deployment.
CCR and the File Share Witness
CCR uses the file share witness on a third computer as a "voter node" or referee to determine which node should own the cluster. When a CCR exists in a geographically dispersed fashion a syndrome can develop in which neither location knows who is the owner of the cluster. Such a scenario is known as "split brain". The "split brain" syndrome occurs when the networks responsible for the internal cluster communications fail, and the active and passive nodes cannot receive a "heartbeat" or communicate with one another. The split brain syndrome is prevented in requiring that the "voter node" or "File Share Witness" always be available in combination with one node in order for a quorum to exist. Because the Hub Transport server is required for a new concept called a "transport dumpster" it is recommended that the file share witness also be a "Hub Transport Server". A "Hub Transport Server" is one of the five roles for Exchange 2007. The Hub Transport server role processes all messages that are sent inside the Exchange 2007 organization before the messages are delivered to a recipient’s Inbox or are routed to users outside the organization. There are no exceptions to this behavior; messages are always passed through a server that runs the Hub Transport server role.
Transport Dumpster
The transport dumpster is required piece of CCR that is turned on by default when preparing a CCR implementation. The transport dumpster is a feature enabled at the Hub Transport server role specifically for CCR when a lossless failover is not guaranteed. When the transport Dumpster is enabled a queue of mail is maintained which had been previously delivered to the virtual mailbox server hosted by the cluster. When a failover is seen which is not lossless CCR will automatically request that every Hub Transport Server in the site resubmit it’s transport dumpster queue. To prevent duplicates the information stores automatically deletes them and redelivers any lost mail.
There are known scenarios in which data loss is not prevented by the use of the transport dumpster. These scenarios include the following.
- Drafts folder for any Microsoft Outlook clients in online mode
- Appointments, contact updates, property updates, tasks, and task updates.
- Outgoing mail that is in transit from the client to the Hub Transport server. There is a period of time during which the e-mail message only exists on the sender’s Mailbox server.
The transport dumpster is required for CCR, and it is turned on by default. The transport dumpster is a feature of the Hub Transport server role. The Hub Transport server maintains a queue of mail that was recently delivered to a clustered mailbox server. In the event of a failover that is not Lossless, CCR automatically requests every Hub Transport server in the site to resubmit mail from the transport dumpster queue. The information store automatically deletes the duplicates and redelivers mail that was lost.
File Share Witness
The file share witness uses a file share on the previously described "voter node" which is not one of the two nodes which are qualified to host the cluster. This "voter node" if you will is used by the Active and Passive nodes to track which node is in control of the cluster. When the two nodes cannot communicate with each other the file share witness or voter node is used to determine whether the Active node should maintain ownership or should be passed to the passive node. The file share witness or "voter node" is only used when the two nodes are unable to communicate with each other. In such a circumstance both nodes will attempt communication with the file share witness. In this circumstance if the node currently holding the role of Active and in control for the Exchange Virtual Server cannot communicate with the "voter node" or file share witness but the passive node can communicate with the file share witness, the passive node can become the active one.
The file share witness or "voter node" support provides a periodic access check of the file share witness. If the access check fails, an event is generated. The event can be collected, detected, and reported by a monitoring system. This even allows the Exchange support team to correct the issue, prior to the issue causing an outage due to a second concurrent failure.
When using the Majority Node Set (MNS) quorum with file share witness, only precisely two servers can exist in the cluster. If more than two servers exist in the cluster, the file share witness feature of the MNS quorum cannot be used.
The file share is accessed under the following conditions:
- When a cluster node comes up and only one cluster node is available.
- When a network connection problem prevents a previously reachable node from communicating with the cluster.
- When a cluster node leaves the cluster.
- Periodically for validation purposes. This validation frequency is a configurable setting.
The load on the file share is considered very light and as a result a single file share witness or "voter node" server can provide the file shares for multiple CCR pairs.
Transaction Log Replication and Replay
Transaction log replication is used to replay a copy of the transaction logs to the passive copy of the database. The replication process takes advantage of the change history produced by the Extensible Storage Engine (ESE). The change history is represented by a sequence of 1 megabyte (MB) transaction log files. The replication mechanism copies these transaction log files to the passive node as each transaction log file is generated on the active node. This replication mechanism is asynchronous to the online database. When the transaction logs are copied to the passive node, they are first checked for corruption prior to being replayed into the copy of the database that is stored on the passive node. The replay process then makes the changes to the passive database copy which had been recorded in the transaction logs of the active node. This results in the passive nodes copy being a replica of the active nodes copy. There is however delay based on the time required to copy the transaction logs to the passive node, check the transaction logs integrity and reply them to the passive nodes copy of the database.
Because the data is replicated between the nodes, the Exchange Virtual Server can operate against either node in the cluster. This capability provides increased availability because scheduled outages and failures of one node do not cause an extended outage of the Exchange Virtual Server. In addition, service outages caused by database corruption on one node will not impact availability of the other node and the Exchange Virtual Server. Assuming that the file share or "voter node" is still available and that it can communicate with the passive node, an outage of the active nodes causes the Exchange Virtual Server to move to a remaining node, and it continues to operate.
In a CCR environment, the transaction log file folder on the active node is shared using a standard Windows file share. The object GUID for the storage group is used for the share name, and a dollar sign ($) is added to the end of the share to hide it from the environment. The Microsoft Exchange Replication Service on the passive node connects to the file share on the active node and copies, or pulls, the transaction log files using the Server Message Block (SMB) protocol. The passive node then vlaidates each of the transaction log files and replays them into the copy of the database on the passive node. It should be noted that the SMB traffic for transaction log file replication is not encrypted between the two nodes. If this is needed, Internet Protocol security (IPsec) should be used to encrypt this traffic. Only transaction log file replication occurs using the SMB protocol. Re-seeding a passive copy of the databases occurs using the ESE backup API, which is encrypted, remote procedure call (RPC)-based communication.
