Understanding Shadow Redundancy

Applies to: Exchange Server 2010

High availability strategies for Exchange have focused on the availability and recoverability of data stored in mailbox databases. When you implement a highly available solution for your Mailbox servers, the e-mail messages won't be lost, and they can easily be recovered after a failure, after they arrive in a mailbox.

However, these strategies didn't extend to messages while they're in transit. If a Hub Transport server fails while processing messages and can't be recovered, data loss could occur. As the volume of messages processed by Hub Transport servers increases, potential data loss becomes an increasing concern for administrators.

Microsoft Exchange Server 2007 introduced the transport dumpster feature for the Hub Transport server role. An Exchange 2007 Hub Transport server maintains a queue of messages delivered recently to recipients whose mailboxes are on a clustered mailbox server. When a failover is experienced, the clustered mailbox server automatically requests every Hub Transport server in the Active Directory site to resubmit mail from the transport dumpster queue. This prevents mail from being lost during the time taken for the cluster to fail over. While this does provide a basic level of transport redundancy, it's only available for message delivery in a cluster continuous replication (CCR) environment and doesn't address potential message loss when messages are in transit between Hub Transport and Edge Transport servers.

Exchange Server 2010 introduces the shadow redundancy feature to provide redundancy for messages for the entire time they're in transit. The solution involves a technique similar to the transport dumpster. With shadow redundancy, the deletion of a message from the transport databases is delayed until the transport server verifies that all of the next hops for that message have completed delivery. If any of the next hops fail before reporting back successful delivery, the message is resubmitted for delivery to that next hop.

Shadow redundancy provides the following benefits:

• It eliminates the reliance on the state of any specific Hub Transport or Edge Transport server. As long as redundant message paths exist in your routing topology, any transport server becomes disposable.
• If a transport server fails, you can remove it from production without emptying its queues or losing messages.
• If you want to upgrade a Hub Transport or Edge Transport server, you can bring that server offline at any time without the risk of losing messages.
• It eliminates the need for storage hardware redundancy for transport servers.
• It consumes less bandwidth than creating duplicate copies of messages on multiple servers. The only additional network traffic generated with shadow redundancy is the exchange of discard status between transport servers. Discard status is the information each transport server maintains. It indicates when a message is ready to be discarded from the transport database.
• It provides resilience and simplifies recovery from a transport server failure.

Shadow redundancy is implemented by extending the SMTP service. The service extensions allow SMTP hosts to negotiate shadow redundancy support and exchange discard status for shadow messages.

Looking for management tasks related to managing transport servers? See Managing Transport Servers.

Contents

Shadow Redundancy Components

Shadow Redundancy Message Flow

Shadow Redundancy Manager

Message Processing After an Outage

Extended Rights Required for Shadow Redundancy

Shadow Redundancy Components

The following table provides descriptions of all the components of shadow redundancy.

Shadow redundancy components

Return to top

Shadow Redundancy Message Flow

To illustrate the mail flow with shadow redundancy enabled, consider the simple scenario where a Hub Transport server sends a message to a third-party mail server via an Edge Transport server in the perimeter network.

Message flow with shadow redundancy

In this scenario, the message flow goes through following stages:

1. The Hub Transport server delivers a message to the Edge Transport server.
   1. The Hub Transport server opens an SMTP session with the Edge Transport server.
   2. The Edge Transport server advertises shadow redundancy support.
   3. The Hub Transport server notifies the Edge Transport server to track discard status.
   4. The Hub Transport server submits the message to the Edge Transport server.
   5. The Edge Transport server acknowledges the receipt of the message and records the Hub Transport server name for sending discard information for the message.
   6. The Hub Transport server moves the message to the shadow queue for the Edge Transport server and marks the Edge Transport server as the primary server. The Hub Transport server becomes the shadow server.
2. The Edge Transport server delivers the message to the next hop.
   1. The Edge Transport server submits the message to a third-party mail server.
   2. The third-party mail server acknowledges the receipt of the message.
   3. The Edge Transport server updates the discard status for the message as delivery complete.
3. The Hub Transport server queries the Edge Transport server for discard status (success case).
   1. At the end of each SMTP session with the Edge Transport server, the Hub Transport server queries the Edge Transport server for discard status on messages previously submitted. If the Hub Transport server hasn't opened any SMTP sessions with the Edge Transport server after the initial message submission, it will open an SMTP session with the Edge Transport server just to query for the discard status after a specific amount of time.
   2. The Edge Transport server checks the local discard status and sends back the list of messages that have been delivered, and removes the discard information.
   3. The Hub Transport server deletes the list of messages from its shadow queue.
4. The Hub Transport server queries the Edge Transport server for the discard status and resubmits the message (failure case).

   1. If the Hub Transport server can't contact the Edge Transport server, the Hub Transport server resumes the primary server role and resubmits the messages in the shadow queue.

   2. Resubmitted messages are delivered to another Edge Transport server and the workflow starts from stage 1.

      Note

      If there are no alternative routes available for a shadow message (such as the second Edge Transport server shown in the preceding figure), it won't be resubmitted, but remain in the shadow queue.

For more information about message flow in various different scenarios, see Shadow Redundancy Mail Flow Scenarios.

Multiple Hop Scenario

If a message travels through multiple servers that support shadow redundancy, the shadow messages are retained on a server only until the next server in the message path confirms delivery. To illustrate how this works, consider an organization that has five Active Directory sites with Hub Transport servers installed. The sites are connected to each other as shown in the following figure. The organization has New York and London sites configured as hub sites, so the messages from Chicago or Atlanta need to go through Hub Transport servers in the New York and London sites to get to the Dublin site.

Sample topology for multiple hop scenario

Assume that a message is sent by a user in the Chicago site to a user in the Dublin site. This message will need to travel through the New York and London sites to get to Dublin. In this case, the following occurs:

1. The Hub Transport server in Chicago will send the message to the Hub Transport server in New York, and it will retain a shadow copy of the message.
2. The New York Hub Transport server will send the message to the Hub Transport server in London and queue a discard status for the Chicago hub.
3. The Chicago hub queries the New York hub for discard status and receives the discard notification for the message. At this time, it can remove the shadow message from its database. Whether the message was delivered from London to Dublin doesn't have an impact on when the Chicago server deletes the shadow message.

Interoperability

Whether shadow redundancy will be used or not is decided while establishing a new SMTP connection. If both servers in an SMTP connection support shadow redundancy, the workflow mentioned previously is used. However, there will be situations where Exchange 2010 transport servers exchange messages with mail servers that don't support shadow redundancy. These could be third-party mail servers, earlier versions of Exchange, or an Exchange 2010 organization that hasn't enabled shadow redundancy.

When an Exchange 2010 transport server that supports shadow redundancy establishes a connection with a server that doesn't support shadow redundancy, the following events take place:

1. Exchange establishes an SMTP connection to the target server.
2. The target server doesn't advertise shadow redundancy support.
3. Because the target server doesn't support redundancy, Exchange will perform the following for each message:
   1. Deliver the message to the target server.
   2. Shadow Redundancy Manager will mark that the message is delivered to the next hop.
   3. Delete the message after it's delivered to all of the next hops.

When a server that doesn't support shadow redundancy establishes a connection with an Exchange 2010 server, the following events take place:

1. The sending server establishes an SMTP connection with Exchange.
2. Exchange advertises shadow redundancy support.
3. The sending server doesn't support shadow redundancy and therefore it won't use it. It will deliver messages to the Exchange server.
4. For each message Exchange receives, it will do the following:
   1. Deliver the message to the next hop, or make a shadow copy of it.
   2. Send acknowledgement to the sending server.

Delayed Acknowledgement

The main principle behind shadow redundancy is maintaining a copy of the message on the previous hop until the server verifies that it has successfully delivered it to all the next hops. This isn't possible when an Exchange 2010 transport server is receiving a message from a mail server that doesn't support shadow redundancy. This mail server can be an Exchange server running an older version of Exchange, a standard SMTP client, or a non-Exchange mail server on the Internet. In this case, Exchange attempts to achieve shadow redundancy by delaying the acknowledgement to the mail server until it verifies that the message has been successfully delivered to all the next hops internally. This way, if the Exchange 2010 server fails, the sending mail server will assume that the message was never delivered to Exchange and will attempt delivery again.

However, the delivery of the message to the next hops may take a long time due to the complexity of your routing infrastructure, or failure of one of the next hops. In this case, to prevent the SMTP session from timing out, the Exchange 2010 transport server will send an acknowledgement to the sending mail server. In this case, the mail redundancy isn't guaranteed, but it's a best effort. For example, a message may be lost in the following scenario: An Internet mail server transmits a message to an Edge Transport server. The Edge Transport server can't communicate with the Hub Transport server due to a network problem and acknowledges the receipt of the message to the Internet mail server. The Edge Transport server then fails and can't be recovered before the network problem is resolved. At this point, the message is lost.

The delayed acknowledgement time-out value is controlled by the MaxAcknowledgementDelay attribute of each Receive connector. The default value is 30 seconds. To learn more about configuring this attribute, see Configure Shadow Redundancy.

Return to top

Shadow Redundancy Manager

Shadow Redundancy Manager is the core component of an Exchange 2010 transport server that's responsible for managing shadow redundancy.

Shadow Redundancy Manager is responsible for maintaining the following information for all the primary messages that a server is currently processing:

• The shadow server for each primary message being processed.
• The discard status to be sent to shadow servers.

Shadow Redundancy Manager is responsible for the following for all the shadow messages that a server has in its shadow queues:

• Maintaining the list of primary servers for each shadow message.
• Checking the availability of each primary server for which a shadow message is queued.
• Processing discard notifications from primary servers.
• Removing the shadow messages from the database after all expected discard notifications are received.
• Deciding when the shadow server should take ownership of shadow messages, becoming a primary server.

In addition, Shadow Redundancy Manager is also responsible for managing performance counters related to shadow redundancy.

Heartbeat

Shadow Redundancy Manager uses heartbeat to determine the availability of the servers for which shadow messages are queued. During the SMTP session between two servers that both support shadow redundancy, the server that initiates the connection queries the target server for discard status of messages previously submitted to that server. The initiating server accomplishes this by issuing an XQUERYDISCARD command. In response, the target server returns the discard notifications. This exchange between the two servers is used as the heartbeat for shadow redundancy.

There is a time-out value for the heartbeat. If no connections are established to a server for which Shadow Redundancy Manager is maintaining a shadow queue for that duration, the server will attempt to establish an SMTP connection with the primary server specifically to query the discard status and reset the timer. The time-out value is controlled by the ShadowHeartbeatTimeoutInterval parameter of the Set-TransportConfig cmdlet, and it's set to 300 seconds by default.

If the server can't establish a connection to a primary server when the time-out value is reached, it will reset the timer and try again. If the time-out value is reached three times in a row, the server will conclude that the primary server has failed and will assume ownership of the shadow messages and begin to generate discard notifications for them to send to the primary server that failed. The number of time-outs a server will wait before deciding a primary server has failed is controlled by the ShadowHeartbeatRetryCount parameter of the Set-TransportConfig cmdlet.

To learn more about configuring the shadow redundancy heartbeat, see Configure Shadow Redundancy.

Return to top

Message Processing After an Outage

Shadow redundancy minimizes message loss due to server outages. When a transport server comes back online after an outage, there are two scenarios:

• The server comes back online with a new transport database   In this scenario, the transport database is unrecoverable due to data corruption or hardware failure. In this case, because the transport server will have a new database ID, it will be recognized as a new route by the other transport servers in the organization. This also applies to the situation where a server couldn't be recovered, and a new server was provisioned as a replacement.
• The server comes back online with the same transport database   In this scenario, the particular transport server didn't fail, but was offline for an extended period of time. For example, a network card failure, or a long maintenance on the server would cause this scenario.

The following table summarizes how transport reacts to these two scenarios when shadow redundancy is enabled. For clarity, assume that the server that had an outage is named Hub01.

Message processing in recovery scenarios

Return to top

Extended Rights Required for Shadow Redundancy

Exchange 2010 introduces the following two extended rights, which are required for shadow redundancy:

• ms-Exch-SMTP-Accept-XSHADOW
• ms-Exch-SMTP-Send-XSHADOW

When an SMTP connection is established to an Exchange 2010 transport server, it will advertise shadow redundancy support if the ms-Exch-SMTP-Accept-XSHADOW extended right exists on the Receive connector being used. In addition, the authentication mechanism on the Receive connector should be either Exchange Server authentication or Externally Secured.

When an Exchange 2010 transport server establishes an SMTP connection to another server that advertises shadow redundancy support, it will issue an XSHADOW command only if the session has been granted the ms-Exch-SMTP-Send-XSHADOW extended right.

By default, these extended rights are granted to the Exchange Servers group on all internal Send connectors and Receive connectors.

Return to top