Share via

Transitioning to Exchange Web Services Notifications

  • Article

Jason Henderson announced in his Exchange Developer Roadmap blog post that we are officially deprecating, among other things, Store Event Sinks and SMTP Event Sinks. This means that Exchange developers will have to rely on Exchange Web Services (EWS) notifications and transport agents to perform many of the tasks previously handled by these event sinks. To help with the transition, I’m going to discuss how to manage the notification subscription lifecycle. Specifically, I’m going to focus on push notifications. I’m going to assume that the reader is familiar with both the Notifications documentation that is available on MSDN (Event Notifications) and the Exchange Web Services API. Also, Inside Microsoft Exchange Server 2007 Web Services is a great book that was written by some of the people who developed Exchange Web Services and has a lot of detailed information on this and other topics.

Architecture

Notification subscriptions (both push and pull) are stored in memory on the Client Access server (CAS). In the case of push notifications, when a subscription comes in, the CAS begins to poll the Mailbox server every couple of seconds for new events. If any events have occurred since the last poll, the Mailbox server responds to the CAS with notifications about the events and the associated watermarks. The CAS then filters these events according to the information that is provided in the subscription, and creates a notification response that is populated with those events. Along with the events, the Client Access server appends the watermark of each event, as well as the “previous” watermark – the watermark of the last event of the previous message (the Mailbox server maintains the watermarks and updates them as events occur). Finally, the CAS sends this message to the client.

Ordering of Events

We are frequently asked – are notifications guaranteed to arrive in the order in which the events happened? The answer is yes! When the CAS sends a notifications message to the client application, the client must reply with either an “OK” or an “Unsubscribe” response. Until the CAS receives an “OK” response, it will not send out any more NEW notifications messages (even if new events occur). It will, however, go into a retry mode, if it doesn’t receive a response within the timeout amount of time, which is set on the subscription request (this is the topic of the next section). Thus, no new notifications message will be sent out by the CAS until the previous one has been acknowledged by the server. This guarantees that the order is preserved.

Timeouts

We are also often asked how push notifications timeouts work. Suppose your client subscribed to push notifications, and then unexpectedly goes offline. The server will not be able to push out the next set of events as notifications to the client (or, more importantly, the server will not get a response to its notifications message). Rather than give up right away, the server will retry to push them out up to three times. After the first failure, the server will wait the timeout amount of time (that’s the timeout that you set in the subscription request) and try again. If that attempt fails, the server will wait twice the timeout time and try again. If that fails, the server will wait three times the timeout time and try again. Finally, if the third retry attempt fails, the server will give up and delete the subscription. There is no Web service method to check on the life of the subscription, but the CAS sends StatusEvent messages every timeout amount of time, so if you don’t get a StatusEvent message within the specified time, you can assume that the subscription was deleted.

If your client application goes offline for long enough, such that the CAS went through its retry procedure and deleted the subscription, it is still possible (in almost all cases; see the next section for clarification) for your client to receive notifications that happened since the last notifications message that the client processed. To do this, you must issue a subscribe request and pass in the watermark from the point where you want to get notifications. Most of the time, you will want to pass in the last watermark seen (thus we recommend you keep track of it).

In summary, there are really two ways to end a subscription on a CAS:

  • You can send “Unsubscribe” in the response message (as a response to a notifications message from the CAS).
  • You can become unresponsive to notifications messages.

We recommend the first method (sending an unsubscribe message) rather than the second method, which causes the Client Access server to perform unnecessary work.

Reliability

Reliability of notifications is another area of confusion. Notifications are reliable – that is, all events for a mailbox that happen on the back end can be seen by the client, unless the backend server goes down. A client can always re-subscribe with the last watermark (a thing to keep in mind here: watermarks are good for about 30 days, after which they expire) and in this way get any potential events that have happened since that last watermark. The one exception to this is mailbox moves. Remember that watermarks are unique to mailbox databases, and only the Mailbox server that the watermark came from knows about that watermark. In the case of a mailbox move to another server, after the mailbox is moved, the original server notifies the CAS that no more events for that mailbox will occur, and the CAS deletes any subscriptions for that mailbox. Prior to deleting the subscriptions, the CAS will send a failure message to the client application, thus notifying it that the subscription will be deleted. The client application may now try to re-subscribe with the latest watermark. However, because the mailbox is now on another server, the previous watermarks are no longer valid (sending a subscription that includes an invalid watermark will result in an error returned by the server). Therefore, the client will have to create a new subscription from that point on. This means that any events that happened to that mailbox between the mailbox move and the creation of the new subscription creation will not be delivered.

Comments

  • Anonymous
    February 04, 2009
    I am getting notificaton on single mailbox .I have developed client side using JAXWS. Now, I want all mailboxes notification using Administrators credientials? Also How to get all exchage mailbox user list? My mail ID : shantikumar9@gmail.com Thanks in advance.

  • Anonymous
    March 30, 2010
    Hello. Can EWS handle 10,000 PUSH notifications? (for 10,000 mail account). I mean, what is the max limit to have a functional push mechanism?

  • Anonymous
    April 30, 2010
    The comment has been removed

  • Anonymous
    November 09, 2011
    goodin - I had the same problem because I had a client opened to the mailbox in cache mode. clear out the contents of Inbox, Calendar & Contacts and close the client and test again.

  • Anonymous
    July 10, 2014
    Assuming you are pulling [notification] once a day. Is it better a) to set the timeout to a day b) to set it pretty short , let it time outand just resubscribe (on the theory the watermark sticks around for 30 days) c) to pull, unsubscribe and keep the water mark [will it still be valid for resubscription?]

  • Anonymous
    July 11, 2014
    mjb- I'm quite certain that c is the best option, but I'm getting confirmation. Why not just sync once a day instead?

  • Anonymous
    July 13, 2014
    I am syncing once a day. But it's unclear to me how to deal with the fact that a) subscription may have expired due to unluckiness of the world (can't guarantee absolutely I will pull within 24 hours). That doesn't appear to matter, I've seen word that watermarks last 30 days? b) Since I am pulling at another time, the api doesn't provide a way to provide the SUBSCRIPTION ID at the same time as the water mark. So if I do another beginSubscription with the watermark, will that magically reconnect to the old subscription, or will it create another. c) If it will create another, then obviously I might be better off with having a short timeout of the original or unsubscribing. If I unsubscribe though, wont the watermark be garbage collected?

  • Anonymous
    July 15, 2014
    a) Yes, the watermark can get overrun and fail if too many events have happened.   b) If you call SubscribeToPullNotifications again, it will create a new subscription with a new SubscriptionId. c) You can include the old watermark in the new subscription, but it's not recommended. From "Recovering from lost subscriptions "(msdn.microsoft.com/.../dn458788(v=exchg.150).aspx): When a subscription is lost, or is no longer accessible, it is best to create a new subscription and not include the old watermark in the new subscription. Resubscribing with the old watermark causes a linear scan for events, which is costly. Instead, create a new subscription and compare folder properties to look for content changes that occurred between the lost subscription and the new subscription. The extended folder properties that we recommend that you check are PR_LOCAL_COMMIT_TIME_MAX (0x670a0040) and PR_DELETED_COUNT_TOTAL (0x670b0003). You can do this by creating an extended property definition. You might also want to look at this video by Matt Stehle, he discusses notifications and lost subscriptions at the 31 minute mark: channel9.msdn.com/.../Exchange-Web-Services-Best-Practices-Part-2

  • Anonymous
    July 15, 2014
    Mimi - thanks for the answer. It doesn't appear to be relevant though a) under what circumstances? It's the duty of watermark to survive a nuclear attack minus contractual status. b) Right, exactly. So how do you keep using the old subscription. Remember. it's a new PROCESS. c) Well that appears to be the one and only correct answer. I haven't lost the subscription, microsoft has, because there is no way for me to create a PullSubscription with a given subscriptionID! Am I being clear. Let me try again.

  1. Subscribe to a pullnotification with null watermark. I get a watermark and a subscription id. I put them in a database.
  2. 2 hours later, in a completely separate thread and process, I need to resume the Pull Subscription. (because I want to catch everything changing in those 2 hours). The ONLY Api call available is another Subscribe, which will take a watermark, but not a subscribe ID. Do you see the problem?