Running .NET4 Windows Workflows in Azure
This blog reviews the current (January 2011) set of options available for hosting existing .NET4 Workflow (WF) programs in Windows Azure and also provides a roadmap to the upcoming features that will further enhance support for hosting and monitoring the Workflow programs. The code snippets included below are also available as an attachment for you to download and try it out yourself.
Workflow in Azure – Today
Workflow programs can broadly classified as durable or non-durable (aka non-persisted Workflow Instances). Durable Workflow Services are inherently long running, persist their state, and use correlation for follow-on activities. Non-durable Workflows are stateless, effectively they start and run to completion in a single burst.
Today non-durable Workflows are readily supported by Windows Azure of course with a few configuration/trivial changes. Hosting durable Workflows today is a challenge; since we do not yet have a ‘Windows Server AppFabric’ equivalent for Azure which can persist, manage and monitor the Service. In brief the big buckets of functionality required to host the durable Workflow Services are:
- Monitoring store: There is no Event Collection Service available to gather the ETW events and write them to the SQL Azure based Monitoring database. There is also no schema that ships with .NET Framework for creating the monitoring database, and the one that ships with Windows Server AppFabric is incompatible with SQL Azure – an example, the scripts that are provided with Windows Server AppFabric make use of the XML column type which is currently not supported by SQL Azure.
- Instance Store: The schemas used by the SqlWorkflowInstanceStore have incompatibilities with SQL Azure. Specifically, the schema scripts require page locks, which are not supported on SQL Azure.
- Reliability: While the SqlWorkflowInstanceStore provides a lot of the functionality for managing instance lifetimes, the lack of the AppFabric Workflow Management Service means that you need to manually implement a way to start your WorkflowServiceHosts before any messages are received (such as when you bring up a new role instance or restart a role instance), so that the contained SqlWorkflowInstanceStore can poll for workflow service instances having expired timers and subsequently resume their execution.
The above limitations make it rather difficult to run a durable Workflow Service on Azure – the upcoming release of Azure AppFabric (Composite Application) is expected to make it possible to run durable Workflow Services. In this blog, we will focus on the design approaches to get your non-durable Workflow instances running within Azure roles.
Today you can run your non-durable Workflow on Azure. What this means, is that your Workflow programs really cannot persist their state and wait for subsequent input to resume execution- they must complete following their initial launch. With Azure you can run non-durable Workflows programs in one of the three ways:
- Web Role
- Worker Roles
- Hybrid
The Web Role acts very much like IIS does on premise as an HTTP server, and is easier to configure and requires little code to integrate and is activated by an incoming request. The Worker Role acts like an on-premise Windows Service and is typically used in backend processing scenarios have multiple options to kick off the processing – which in turn add to the complexity. The hybrid approach, which bridges communication between Azure hosted and on-premise resources, has multiple advantages: it enables you to leverage existing deployment models and also enables use of durable Workflows on premise as a solution until the next release. The following sections, succinctly, provide details on these three approaches and in the ‘Conclusion’ section we will also provide you pointers on the appropriateness of each approach.
Host Workflow Services in a Web Role
The Web Role is similar to a ‘Web Application’ and can also provide a Service perspective to anything that uses the http protocol - such as a WCF service using basicHttpBinding. The Web Role is generally driven by a user interface – the user interacts with a Web Page, but a call to a hosted Service can also cause some processing to happen. Below are the steps that enable you to host a Workflow Service in a Web Role.
First step is to create a Cloud Project in Visual Studio, and add a WCF Service Web Role to it. Delete the IService1.cs, Service1.svc and Service1.svc.cs added by the template since they are not needed and will be replaced by the workflow service XAMLX.
To the Web Role project, add a WCF Workflow Service. The structure of your solution is now complete (see the screenshot below for an example), but you need to add a few configuration elements to enable it to run on Azure.
Windows Azure does not include a section handler in its machine.config for system.xaml.hosting as you have in an on-premises solution. Therefore, the first configuration change (HTTP Handler for XAMLX and XAMLX Activation) is to add the following to the top of your web.config, within the configuration element:
<configSections>
<sectionGroup name="system.xaml.hosting" type="System.Xaml.Hosting.Configuration.XamlHostingSectionGroup, System.Xaml.Hosting, Version=4.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35">
<section name="httpHandlers" type="System.Xaml.Hosting.Configuration.XamlHostingSection, System.Xaml.Hosting, Version=4.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35" />
</sectionGroup>
</configSections>
Next, you need to add XAML http handlers for WorkflowService and Activities root element types by adding the following within the configuration element of your web.config, below the configSection that we included above:
<system.xaml.hosting>
<httpHandlers>
<add xamlRootElementType="System.ServiceModel.Activities.WorkflowService, System.ServiceModel.Activities, Version=4.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35" httpHandlerType="System.ServiceModel.Activities.Activation.ServiceModelActivitiesActivationHandlerAsync, System.ServiceModel.Activation, Version=4.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35" />
<add xamlRootElementType="System.Activities.Activity, System.Activities, Version=4.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35" httpHandlerType="System.ServiceModel.Activities.Activation.ServiceModelActivitiesActivationHandlerAsync, System.ServiceModel.Activation, Version=4.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35" />
</httpHandlers>
</system.xaml.hosting>
Finally, configure the WorkflowServiceHostFactory to handle activation for your service by adding a serviceActivation element to system.serviceModel\serviceHostingEnvironment element:
<serviceHostingEnvironment multipleSiteBindingsEnabled="true" >
<serviceActivations>
<add relativeAddress="~/Service1.xamlx" service="Service1.xamlx" factory="System.ServiceModel.Activities.Activation.WorkflowServiceHostFactory"/>
</serviceActivations>
</serviceHostingEnvironment>
The last step is to deploy your Cloud Project and with that you now have your Workflow service hosted on Azure – graphic below!
Note: Sam Vanhoutte from CODit in his blog also elaborates on Hosting workflow services in Windows Azure and focuses on troubleshooting configuration by disabling custom errors-- do review.
Host Workflows in a Worker Role
The Worker Role is similar to a Windows Service and would start up ‘automatically’ and be running all the time. While the Workflow Programs could be initiated by a timer, it could use other means to activate such as a simple while (true) loop and a sleep statement. When it ‘ticks’ it performs work. This is generally the option for background or computational processing.
In this scenario you use Workflows to define Worker Role logic. Worker Roles are created by deriving from the RoleEntryPoint Class and overriding a few of its method. The method that defines the actual logic performed by a Worker Role is the Run Method. Therefore, to get your workflows executing within a Worker Role, use WorkflowApplication* or WorkflowInvoker to host an instance of your non-service Workflow (e.g., it doesn’t use Receive activities) within this Method. In either case, you only exit the Run Method when you want the Worker Role to stop executing Workflows.
The general strategy to accomplish this is to start with an Azure Project and add a Worker Role to it. To this project you Add a reference to an assembly containing your XAML Workflow types. Within the Run Method of WorkerRole.cs, you initialize one of the host types (WorkflowApplication or WorkflowInvoker), referring to an Activity type contained in the referenced assembly. Alternatively, you can initialize one of the host types by loading an Activity instance from the XAML Workflow file available on the file system.. You will also need to add references to .NET Framework assemblies (System.Activities and System.Xaml - if you wish to load XAML workflows from a file).
Host Workflow (Non-Service) in a Worker Role
For ‘non-Service’ Workflows, your Run method needs to describe a loop that examines some input data and passes it to a workflow instance for processing. The following shows how to accomplish this when the Workflow type is acquired from a referenced assembly:
public override void Run()
{
Trace.WriteLine("WFWorker entry point called", "Information");
while (true)
{
Thread.Sleep(1000);
/* ...
* ...Poll for data to hand to WF instance...
* ...
*/
//Create a dictionary to hold input data
Dictionary<;string, object> inputData = new Dictionary<string,object>();
//Instantiate a workflow instance from a type defined in a referenced assembly
System.Activities.Activity workflow = new Workflow1();
//Execute the WF passing in parameter data and capture output results
IDictionary<;string, object> outputData =
System.Activities.WorkflowInvoker.Invoke(workflow, inputData);
Trace.WriteLine("Working", "Information");
}
}
Alternatively, you could perform the above using the WorkflowApplication to host the Workflow instance. In this case, the main difference is that you need to use semaphores to control the flow of execution because the workflow instances will be run on threads separate from the one executing the Run method.
public override void Run()
{
Trace.WriteLine("WFWorker entry point called", "Information");
while (true)
{
Thread.Sleep(1000);
/* ...
* ...Poll for data to hand to WF...
* ...
*/
AutoResetEvent syncEvent = new AutoResetEvent(false);
//Create a dictionary to hold input data and declare another for output data
Dictionary<;string, object> inputData = new Dictionary<string,object>();
IDictionary<;string, object> outputData;
//Instantiate a workflow instance from a type defined in a referenced assembly
System.Activities.Activity workflow = new Workflow1();
//Run the workflow instance using WorkflowApplication as the host.
System.Activities.WorkflowApplication workflowHost = new System.Activities.WorkflowApplication(workflow, inputData);
workflowHost.Completed = (e) =>
{
outputData = e.Outputs;
syncEvent.Set();
};
workflowHost.Run();
syncEvent.WaitOne();
Trace.WriteLine("Working", "Information");
}
}
Finally, if instead of loading Workflow types from a referenced assembly, you want to load the XAML from a file available, for example, one included with WorkerRole or stored on an Azure Drive, you would simply replace the line that instantiates the Workflow in the above two examples with the following, passing in the appropriate path to the XAML file to XamlServices.Load:
System.Activities.Activity workflow = (System.Activities.Activity)
System.Xaml.XamlServices.Load(@"X:\workflows\Workflow1.xaml");
By and large, if you are simply hosting logic described in a non-durable workflow, WorkflowInvoker is the way to go. As it offers fewer lifecycle features (when compared to WorkflowApplication), it is also more light weight and may help you scale better when you need to run many workflows simultaneously.
Host Workflow Service in a Worker Role
When you need to host Workflow Service in a Worker Role, there are a few more steps to take. Mainly, these exist to address the fact that Worker Role instances run behind a load balancer. From a high-level, to host a Workflow Service means creating an instance of a WorkflowServiceHost based upon an instance of an Activity or WorkflowService defined either in a separate assembly or as a XAML file. The WorkflowService instance is created and opened in the Worker Role’s OnStart Method, and closed in the OnStop Method. It is important to note that you should always create the WorkflowServiceHost instance within the OnStart Method (as opposed to within Run as was shown for non-service Workflow hosts). This ensures that if a startup error occurs, the Worker Role instance will be restarted by Azure automatically. This also means the opening of the WorkflowServiceHost will be attempted again.
Begin by defining a global variable to hold a reference to the WorkflowServiceHost (so that you can access the instance within both the OnStart and OnStop Methods).
public class WorkerRole : RoleEntryPoint
{
System.ServiceModel.Activities.WorkflowServiceHost wfServiceHostA;
…
}
Next, within the OnStart Method, add code to initialize and open the WorkflowServiceHost, within a try block. For example:
public override bool OnStart()
{
Trace.WriteLine("Worker Role OnStart Called.");
//…
try
{
OpenWorkflowServiceHostWithAddressFilterMode();
}
catch (Exception ex)
{
Trace.TraceError(ex.Message);
throw;
}
//…
return base.OnStart();
}
Let’s take a look at the OpenWorkflowServiceHostWithAddressFilterMode method implementation, which really does the work. Starting from the top, notice how either an Activity or WorkflowService instance can be used by the WorkflowServiceHost constructor, they can even be loaded from a XAMLX file on the file-system. Then we acquire the internal instance endpoint and use it to define both the logical and physical address for adding an application service endpoint using a NetTcpBinding. When calling AddServiceEndpoint on a WorkflowServiceHost, you can specify either just the service name as a string or the namespace plus name as an XName (these values come from the Receive activity’s ServiceContractName property).
private void OpenWorkflowServiceHostWithAddressFilterMode()
{
//workflow service hosting with AddressFilterMode approach
//Loading from a XAMLX on the file system
System.ServiceModel.Activities.WorkflowService wfs =
(System.ServiceModel.Activities.WorkflowService)System.Xaml.XamlServices.Load("WorkflowService1.xamlx");
//As an alternative you can load from an Activity type in a referenced assembly:
//System.Activities.Activity wfs = new WorkflowService1();
wfServiceHostA = new System.ServiceModel.Activities.WorkflowServiceHost(wfs);
IPEndPoint ip = RoleEnvironment.CurrentRoleInstance.InstanceEndpoints["WorkflowServiceTcp"].IPEndpoint;
wfServiceHostA.AddServiceEndpoint(System.Xml.Linq.XName.Get("IService", "https://tempuri.org/"),
new NetTcpBinding(SecurityMode.None),
String.Format("net.tcp://{0}/MyWfServiceA", ip));
//You can also refer to the implemented contract without the namespace, just passing the name as a string:
//wfServiceHostA.AddServiceEndpoint("IService",
// new NetTcpBinding(SecurityMode.None),
// String.Format("net.tcp://{0}/MyWfServiceA", ip));
wfServiceHostA.ApplyServiceMetadataBehavior(String.Format("net.tcp://{0}/MyWfServiceA/mex", ip));
wfServiceHostA.ApplyServiceBehaviorAttribute();
wfServiceHostA.Open();
Trace.WriteLine(String.Format("Opened wfServiceHostA"));
}
In order to enable our service to be callable externally, we next need to add an Input Endpoint that Azure will expose at the load balancer for remote clients to use. This is done within the Worker Role configuration, on the Endpoints tab. The figure below shows how we have defined a single TCP Input Endpoint on port 5555 named WorkflowServiceTcp. It is this Input Endpoint, or IPEndpoint as it appears in code, that we use in the call to AddServiceEndpoint in the previous code snippet. At runtime, the variable ip provides the local instance physical address and port which the service must use, to which the load balancer will forward messages. The port number assigned at runtime (e.g., 20000) is almost always different from the port you specify in the Endpoints tab (e.g., 5555), and the address (e.g., 10.26.58.148) is not the address of your application in Azure (e.g., myapp.cloudapp.net), but rather the particular Worker Role instance.
It is very important to know that currently, Azure Worker Roles do not support using HTTP or HTTPS endpoints (primarily due to permissions issues that only Worker Roles face when trying to open one). Therefore, when exposing your service or metadata to external clients, your only option is to use TCP.
Returning to the implementation, before opening the service we add a few behaviors. The key concept to understand is that any workflow service hosted by an Azure Worker Role will run behind a load balancer, and this affects how requests must be addressed. This results in two challenges which the code above solves:
- How to properly expose service metadata and produce metadata which includes the load balancer’s address (and not the internal address of the service hosted within a Worker Role instance).
- How to configure the service to accept messages it receives from the load balancer, that are addressed to the load balancer.
To reduce repetitive work, we defined a helper class that contains extension methods for ApplyServiceBehaviorAttribute and ApplyServiceMetadataBehavior that apply the appropriate configuration to the WorkflowServiceHost and alleviate the aforementioned challenges.
//Defines extensions methods for ServiceHostBase (useable by ServiceHost &; WorkflowServiceHost)
public static class ServiceHostingHelper
{
public static void ApplyServiceBehaviorAttribute(this ServiceHostBase host)
{
ServiceBehaviorAttribute sba = host.Description.Behaviors.Find<;ServiceBehaviorAttribute>();
if (sba == null)
{
//For WorkflowServices, this behavior is not added by default (unlike for traditional WCF services).
host.Description.Behaviors.Add(new ServiceBehaviorAttribute() { AddressFilterMode = AddressFilterMode.Any });
Trace.WriteLine(String.Format("Added address filter mode ANY."));
}
else
{
sba.AddressFilterMode = System.ServiceModel.AddressFilterMode.Any;
Trace.WriteLine(String.Format("Configured address filter mode to ANY."));
}
}
public static void ApplyServiceMetadataBehavior(this ServiceHostBase host, string metadataUri)
{
//Must add this to expose metadata externally
UseRequestHeadersForMetadataAddressBehavior addressBehaviorFix = new UseRequestHeadersForMetadataAddressBehavior();
host.Description.Behaviors.Add(addressBehaviorFix);
Trace.WriteLine(String.Format("Added Address Behavior Fix"));
//Add TCP metadata endpoint. NOTE, as for application endpoints, HTTP endpoints are not supported in Worker Roles.
ServiceMetadataBehavior smb = host.Description.Behaviors.Find<;ServiceMetadataBehavior>();
if (smb == null)
{
smb = new ServiceMetadataBehavior();
host.Description.Behaviors.Add(smb);
Trace.WriteLine("Added ServiceMetaDataBehavior.");
}
host.AddServiceEndpoint(
ServiceMetadataBehavior.MexContractName,
MetadataExchangeBindings.CreateMexTcpBinding(),
metadataUri
);
}
}
Looking at how we enable service metadata in the ApplyServiceMetadataBehavior method, notice there are three key steps. First, we add the UseRequestHeadersForMetadataAddressBehavior. Without this behavior, you could only get metadata by communicating directly to the Worker Role instance, which is not possible for external clients (they must always communicate through the load balancer). Moreover, the WSDL returned in the metadata request would include the internal address of the service, which is not helpful to external clients either. By adding this behavior, the WSDL includes the address of the load balancer. Next, we add the ServiceMetadataBehavior and then add a service endpoint at which the metadata can be requested. Observe that when we call ApplyServiceMetadataBehavior, we specify a URI which is the service’s internal address with mex appended. The load balancer will now correctly route metadata requests to this metadata endpoint.
The rationale behind the ApplyServiceBehaviorAttribute method is similar to ApplyServiceMetadataBehavior. When we add a service endpoint by specifying only the address parameter (as we did above), the logical and physical address of the service are configured to be the same. This causes a problem when operating behind a load balancer, as messages coming from external clients via the load balancer will be addressed to the logical address of the load balancer, and when the instance receives such a message it will not accept—throwing an AddressFilterMismatch exception. This happens because the address in the message does not match the logical address at which the endpoint was configured. With traditional code-based WCF services, we could resolve this simply by decorating the service implementation class with [ServiceBehavior(AddressFilterMode=AddressFilterMode.Any)], which allows the incoming message to have any address and port. This is not possible with Workflow Services (as there is no code to decorate with an attribute), hence we have to add it in the hosting code.
If allowing an incoming address concerns you, an alternative to using AddressFilterMode is simply to specify the logical address that is to be allowed. Instead of adding the ServiceBehaviorAttribute, you simply open the service endpoint specifying both the logical (namely the port the load balancer will receives messages on) and physical address (at which your service listens). The only complication, is that your Workflow Role instance does not know which port the load balancer is listening- so you need to add this value to configuration and read it from their before adding the service endpoint. To add this to configuration, return to the Worker Role’s properties, Settings tab. Add string setting with the value of the of the port you specified on the Endpoints tab, as we show here for the WorkflowServiceEndpointListenerPort.
With that setting in place, the rest of the implementation is fairly straightforward:
private void OpenWorkflowServiceHostWithoutAddressFilterMode()
{
//workflow service hosting without AddressFilterMode
//Loading from a XAMLX on the file system
System.ServiceModel.Activities.WorkflowService wfs = (System.ServiceModel.Activities.WorkflowService)System.Xaml.XamlServices.Load("WorkflowService1.xamlx");
wfServiceHostB = new System.ServiceModel.Activities.WorkflowServiceHost(wfs);
//Pull the expected load balancer port from configuration...
int externalPort = int.Parse(RoleEnvironment.GetConfigurationSettingValue("WorkflowServiceEndpointListenerPort"));
IPEndPoint ip = RoleEnvironment.CurrentRoleInstance.InstanceEndpoints["WorkflowServiceTcp"].IPEndpoint;
//Use the external load balancer port in the logical address...
wfServiceHostB.AddServiceEndpoint(System.Xml.Linq.XName.Get("IService", "https://tempuri.org/"),
new NetTcpBinding(SecurityMode.None),
String.Format("net.tcp://{0}:{1}/MyWfServiceB", ip.Address, externalPort),
new Uri(String.Format("net.tcp://{0}/MyWfServiceB", ip)));
wfServiceHostB.ApplyServiceMetadataBehavior(String.Format("net.tcp://{0}/MyWfServiceB/mex", ip));
wfServiceHostB.Open();
Trace.WriteLine(String.Format("Opened wfServiceHostB"));
}
With that, we can return to the RoleEntryPoint definition of our Worker Role and override the Run and OnStop Methods. For Run, because the WorkflowServiceHost takes care of all the processing, we just need to have a loop that keeps Run from exiting.
public override void Run()
{
Trace.WriteLine("Run - WFWorker entry point called", "Information");
while (true)
{
Thread.Sleep(30000);
}
}
For OnStop we simply close the WorkflowServiceHost.
public override void OnStop()
{
Trace.WriteLine(String.Format("OnStop - Called"));
if (wfServiceHostA != null)
wfServiceHostA.Close();
base.OnStop();
}
With OnStart, Run and OnStop Methods defined, our Worker Role is fully capable of hosting a Workflow Service.
Hybrid Approach - Host Workflow On-Premise and Reach From the Cloud
Unlike ‘pure’ cloud solutions, hybrid solutions have a set of “on-premises” components: business processes, data stores, and services. These must be on-premises, possibly due to compliance or deployment restrictions. A hybrid solution is one which has parts of the solution deployed in the cloud while some applications remain deployed on-premises.
This is a great interim approach, leveraging on-premise Workflows hosted within on-premise Windows Server AppFabric (as illustrated in the diagram below) to various components and application that are hosted in Azure. This approach may also be applied if stateful/durable Workflows are required to satisfy scenarios. You can build a Hybrid solution and run the Workflows on-premise and use either the AppFabric Service Bus or Windows Azure Connect to reach into your on-premise Windows Server AppFabric instance.
Source: MSDN Blog Hybrid Cloud Solutions with Windows Azure AppFabric Middleware
Conclusion
How do you choose which approach to take? The decision ultimately boils down to your specific requirements, but here are some pointers that can help.
Hosting Workflow Services in a Web Role is very easy and robust. If your Workflow is using Receive Activities as part of its definition, you should be hosting in a Web Role. While you can build and host a Workflow Service within a Worker Role, you take on the responsibility of rebuilding the entire hosting infrastructure provided by IIS in the Web Role- which is a fair amount of non-value added work. That said, you will have to host in a Worker Role when you want to use a TCP endpoint, and a Web Role when you want to use an HTTP or HTTPS endpoint.
Hosting non-service Workflows that poll for their tasks is most easily accomplished within a Worker Role. While you can build another mechanism to poll and then call Workflow Services hosted in a Web Role, the Worker Role is designed to support and keep a polling application alive. Moreover, if your Workflow design does not already define a Service, then you should host it in a Worker Role-- as Web Role hosting would require you to modify the Workflow definition to add the appropriate Receive Activities.
Finally, if you have existing investments in Windows Server AppFabric as hosted Services that need to be called from Azure hosted applications, then taking a hybrid approach is a very viable option. One clear benefit, is that you retain the ability to monitor your system’s status through the IIS Dashboard. Of course this approach has to be weighed against the obvious trade-offs of added complexity and bandwidth costs.
The upcoming release of Azure AppFabric Composite Applications will enable hosting Workflow Services directly in Azure while providing feature parity to Windows Server AppFabric. Stay tuned for the exciting news and updates on this front.
Sample
The sample project attached provides a solution that shows how to host non-durable Workflows, in both Service and non-Service forms. For non-Service Workflows, it shows how to host using a WorkflowInvoker or WorkflowApplication within a Worker Role. For Services, it shows how to host both traditional WCF service alongside Workflow services, in both Web and Worker Roles.
Code for Blog Entry Running NET4 Windows Workflows in Azure.zip
Comments
- Anonymous
January 13, 2011
Great article, the advice is very timely for me. Out of interest where is the sample project attached? I can't seem to find it on the article. - Anonymous
January 13, 2011
Nikolai,Thank you for the compliment and pointing out the missing attachment. - the sample is now attached. All the best to you.Suren - Anonymous
January 20, 2011
Suren,Nice post.. congratulations.Do you have a date or at least a prevision about the next Azure AppFabric´s release?I have to know this to make de decision about making our own WF4 Host For Azure or wait for the Azure AppFabric...It would help us a lot..Best Regards - Anonymous
May 04, 2011
Now that the persistence scripts are supported with SQL Azure are there any plans to update this article?blogs.msdn.com/.../microsoft-net-framework-4-platform•SqlWorkflowInstanceStore support for SQL Azure◦The scripts that create the persistence database are modified to be compatible with Sql Azure◦Retry semantics have been added and new error codes added that are specific to SQL Azure scenarios