Hyper-V: Using Live Migration with Cluster Shared Volumes in Windows Server 2008 R2

Applies To: Windows Server 2008 R2

This guide details the steps required to perform a live migration of Hyper-V™ virtual machines from one node in a Windows Server® 2008 R2 failover cluster to another node.

Live migration overview

Live migration is a new Hyper-V feature in Windows Server 2008 R2, which requires the failover clustering feature to be added and configured on the servers running Hyper-V. Hyper-V and failover clustering can be used together to make a virtual machine that is highly available, thereby minimizing disruptions and interruptions to clients. Live migration allows you to transparently move running virtual machines from one node of the failover cluster to another node in the same cluster without a dropped network connection or perceived downtime. In addition, failover clustering requires shared storage for the cluster nodes. This can include an iSCSI or Fiber-Channel Storage Area Network (SAN). All virtual machines are stored in the shared storage area, and the running virtual machine state is managed by one of the nodes. For a detailed overview of live migration and the benefits of using it, see Windows Server 2008 R2 & Microsoft Hyper-V Server 2008 R2 - Hyper-V Live Migration Overview & Architecture.

Network recommendations for using live migration

The following recommendations will help you configure your networking environment for using live migration:

• Network adapters. For each node of the failover cluster, use more than one network adapter and configure at least one network adapter for the private virtual network. We recommend that you configure a dedicated private network with Gigabit speed for live migration traffic, and this network should be separate from the network for private communication between the cluster nodes, from the network for the virtual machine, and from the network for storage. For information about the network traffic that can occur on a network used for Cluster Shared Volumes, see “Understanding redirected I/O mode in CSV communciation” in Requirements for Using Cluster Shared Volumes in a Failover Cluster in Windows Server 2008 R2 (https://go.microsoft.com/fwlink/?LinkId=182153).

  Each node must also have a network adapter that carries Cluster Shared Volumes communication, and in the network adapter properties, Client for Microsoft Networks and File and Printer Sharing for Microsoft Networks must be enabled to support SMB. For more information on the network used for CSV communication, see Managing the network used for Cluster Shared Volumes.

  We recommend that you do not use the same network adapter for virtual machine access and management. If you are limited by the number of network adapters, you should configure a virtual local area network (VLAN) to isolate traffic. VLAN recommendations include 802.1q and 802.p.

• Hardware and system settings. It is required that the storage hardware on the failover cluster be identical and that the cluster nodes used for live migration have virtualization-capable processors by the same processor manufacturer. If this is not possible, it is recommended that the hardware and system settings are as similar as possible to minimize potential problems.

• Security policies. If possible, do not apply IPsec policies on a private network for live migration because this can significantly impact the performance of live migration.

• IP subnet configuration. Ensure that the source and destination nodes (for the live migration) in the failover cluster are connected through the same IP subnet. This is so the virtual machine can retain the same IP address after live migration. For each network in a failover cluster where Cluster Shared Volumes is enabled, all nodes must be on the same logical subnet, which means that multisite clusters that use Cluster Shared Volumes must use a VLAN.

  For storage network recommendations, you should review the guidelines provided by your storage vendor.

Node requirements for using Cluster Shared Volumes

The following recommendations will help you configure the nodes for using Cluster Shared Volumes and live migration:

• Operating system. All nodes must run Windows Server 2008 R2 or Hyper-V Server 2008 R2. All nodes must run the same operating system, except that you can combine nodes running Hyper-V Server 2008 R2 in the same cluster with nodes running the Server Core installation of Windows Server 2008 R2.

• Drive letter of system disk. On all nodes, the drive letter for the system disk must be the same.

• Authentication protocol. The NTLM protocol must be enabled on all nodes.

Processor compatibility

If you are using different processor versions on the nodes in the cluster, live migration may fail. To perform a live migration of a virtual machine to another physical computer with a different processor, you must first select the Migrate to a physical computer with a different processor version setting in Hyper-V Manager. This setting ensures that the virtual machine uses only the features of the processor that are available on all versions of a virtualization-capable processor by the same processor manufacturer. It does not provide compatibility between different processor manufacturers. This allows you to move a running virtual machine to a physical computer with different processor features without restarting the virtual machine. The setting is also useful for high availability and backup and recovery scenarios because it makes it easier to move a highly available virtual machine to another node in a cluster or restore the virtual machine to different hardware.

Snapshots and virtual machines

You should avoid taking a snapshot of a running virtual machine. If you revert back to a snapshot of a running virtual machine, the memory state is restored in addition to the disk. Before reverting a clustered virtual machine back to a snapshot, you should first shut down the virtual machine from Failover Cluster Manager, take a snapshot of the virtual machine, and restart the virtual machine.

Attempting to back up a migrating virtual machine

Because live migration is a transition state, the Hyper-V VSS writer waits for the migration to complete before continuing with backup. However, once migration is complete, the virtual machine is no longer on the cluster node on which backup occurs. At this stage, backup continues and correctly backs up the files (it can still access the files on the CSV volume), but it is only a file copy. The VSS writer does not perform the steps that it usually would for an online backup. You should be aware that the VSS writer does not return a failure error code to VSS, and therefore, does not log any errors. However, it does log two warning messages that the virtual machine is not found.

Steps for implementing live migration

Use the following steps to implement live migration:

• Install Windows Server 2008 R2 and enable the Hyper-V role on all nodes in the failover cluster. For more information, see the Hyper-V Planning and Deployment Guide.

• Install the Failover Clustering feature on all servers that you want to include in the cluster. For more information, see the Hyper-V Step-by-Step Guide: Hyper-V and Failover Clustering

• Configure as nodes in a failover cluster. For more information, see the Failover Cluster Deployment Guide.

• Validate the cluster configuration. For more information, see the Failover Cluster Step-by-Step Guide: Validating Hardware for a Failover Cluster.

• Configure Cluster Shared Volumes

• Set up a virtual machine for live migration

• Configure cluster networks for live migration

• Initiate a live migration of a virtual machine

Configure Cluster Shared Volumes

Cluster Shared Volumes are volumes in a failover cluster that multiple nodes can read from and write to at the same time. The nodes coordinate the reading and writing activity so that the disk is not corrupted. In contrast, disks (LUNs) in cluster storage that are not Cluster Shared Volumes are always owned by a single node. Cluster Shared Volumes have the same requirements as non-Cluster Shared Volumes disk resources. The storage location in the Cluster Shared Volumes is under SystemDrive/ClusterStorage. When creating the virtual machine, we recommend that you use this storage location.

It is recommended that you first validate the cluster configuration before configuring Cluster Shared Volumes. For more information about how to validate a cluster configuration, see the Failover Cluster Step-by-Step Guide: Validating Hardware for a Failover Cluster and The Microsoft Support Policy for Windows Server 2008 Failover Clusters.

• The network connection that is used by Cluster Shared Volumes is fault tolerant; therefore, if the network that is used by Cluster Shared Volumes experiences problems, network traffic will be moved to another network.
  
  
• Cluster Shared Volumes can only be enabled once per cluster.
  
  
• By enabling Cluster Shared Volumes for a failover cluster, all nodes in the cluster will be enabled to use shared volumes.
  
  

To enable Cluster Shared Volumes using Failover Cluster Manager

1. In the Failover Cluster Manager snap-in, if the cluster that you want to configure is not displayed, in the console tree, right-click Failover Cluster Manager, click Manage a Cluster, and then select or specify the cluster that you want.

2. Right-click the failover cluster, and then click Enable Cluster Shared Volumes. Or, under Configure (center pane), click Enable Cluster Shared Volumes. The Enable Cluster Shared Volumes dialog box opens. Read and accept the terms and restrictions, and click OK.

To add a disk to the Cluster Shared Volumes

1. In the Failover Cluster Manager snap-in, if the cluster that you want to configure is not displayed, in the console tree, right-click Failover Cluster Manager, click Manage a Cluster, and then select or specify the cluster that you want.

2. If the console tree is collapsed, expand the tree under the cluster that you want to add a disk to the Cluster Shared Volumes.

3. Click Cluster Shared Volumes.

4. Under Actions (on the right), click Add storage.

5. In Add Storage, select from the list of available disks, and click OK. The disk or disks you selected appear in the Results pane for Cluster Shared Volumes.

The storage location appears as SystemDrive\ClusterStorage on all nodes of the failover cluster. Under SystemDrive\ClusterStorage, a specific folder appears for each volume on the disk (or disks) that was added to the Cluster Shared Volumes. You can view the list of volumes in Failover Cluster Manager.

Managing the network used for Cluster Shared Volumes

Failover clusters include a setting to prioritize the networks used for communication between the nodes in the cluster and for the network used for CSV traffic. You can identify the network used for CSV traffic and change the settings of the network using the Windows PowerShell cmdlet, Get-ClusterNetwork.

Each network in a cluster has two settings for network prioritization – Metric and AutoMetric. The Metric setting is used to determine the priority of the network (the network with the lowest value is the most preferred for CSV). The AutoMetric setting identifies whether the Metric setting was set manually or automatically by the failover cluster. For private networks, the Metric settings are between 1000 and 10,000, and for public networks, the Metric settings start at 10,000.

To manage the network used by Cluster Shared Volumes

1. Open PowerShell. Click Start, point to All Programs, click Windows Powershell 2.0, and then click Windows Powershell 2.0.

   The Failover Clustering feature must be installed on the computer on which you are starting the PowerShell session. Or, you can use Remote Server Administration Tools for Windows® 7 to run the PowerShell session.

2. To install the Failover Clustering feature, type:

   Import-Module FailoverClusters

3. To identify the networks of a failover cluster and the properties of each network, type:

   Get-ClusterNetwork | fl*

   A list of cluster networks and their properties appears.

4. To change the metric setting to 1100 for the network named cluster network 1, type:

   ( Get-ClusterNetwork "Cluster Network 1" ).Metric = 1100

   The AutoMetric setting changes from True to False after you manually change the Metric setting. This is to prevent the failover cluster from automatically assigning a Metric setting. If you want the cluster to start automatically assigning the Metric setting again, change the AutoMetric setting back to True.

Set up a virtual machine for live migration

To set up a virtual machine for live migration, you need to do the following:

1. Create the virtual machine

2. Configure the virtual machine to use Cluster Shared Volumes

3. Reconfigure the automatic start action on the virtual machine

4. Make the virtual machine highly available

For information about how to perform these procedures, see Steps 6 and 7 in Hyper-V Step-by-Step Guide: Hyper-V and Failover Clustering.

Configure cluster networks for live migration

Cluster networks are automatically configured for live migration. You can use Failover Cluster Manager to perform this procedure.

To configure a cluster network for live migration

1. In the Failover Cluster Manager snap-in, if the cluster that you want to configure is not displayed, in the console tree, right-click Failover Cluster Manager, click Manage a Cluster, and then select or specify the cluster that you want.

2. Expand Services and applications.

3. In the console tree (on the left), select the clustered virtual machine for which you want to configure the network for live migration.

4. Right-click the virtual machine resource displayed in the center pane (not on the left), and then click Properties.

5. Click the Network for live migration tab, and select one or more cluster networks to use for live migration. Use the buttons on the right to move the cluster networks up or down to ensure that a private cluster network is the most preferred. The default preference order is as follows: networks that have no default gateway should be located first; networks that are used by cluster shared volumes and cluster traffic should be located last.

   Live migration will be attempted in the order of the networks specified in the list of cluster networks. If the connection to the destination node using the first network is not successful, the next network in the list is used until the complete list is exhausted, or there is a successful connection to the destination node using one of the networks.

• When you configure a network for live migration for a specific virtual machine, the setting is global and therefore applies to all virtual machines.
  
  
• If you have more than one cluster network listed in Network for live migration, you should change the priority order to avoid having live migration and Cluster Shared Volumes use the same network.
  
  

Initiate a live migration of a virtual machine

You can use either Failover Cluster Manager or PowerShell to initiate live migration to move a virtual machine from one node to another node in a failover cluster.

• Depending on the number of nodes in the failover cluster, you may be able to use live migration to move more than one virtual machine at a time. However, a cluster node can participate as the source or destination node in only one live migration at a time. For example, if there are 4 nodes in the failover cluster, then two live migrations can occur at the same time.
  
  
• If live migration fails, the virtual machine continues to operate on the source node with no disruption.
  
  

The amount of time it takes to move a virtual machine using live migration is dependent on the following items:

• The network connection speed and bandwidth that is available between the source cluster node and the destination cluster node.

• The load on the source cluster node and the destination cluster node.

• The amount of RAM configured for the virtual machine.

To initiate live migration using Failover Cluster Manager

1. In the Failover Cluster Manager snap-in, if the cluster that you want to configure is not displayed, in the console tree, right-click Failover Cluster Manager, click Manage a Cluster, and then select or specify the cluster that you want.

2. Expand Nodes.

3. In the console tree (on the left), select the node under which you want to move a clustered virtual machine using live migration.

4. Right-click the virtual machine resource displayed in the center pane (not on the left), and then click Live migrate virtual machine to another node.

5. Select the node that you want to move the virtual machine to. When migration is complete, the virtual machine is running on the new node.

6. To verify that the virtual machine successfully migrated, you will see the virtual machine listed under the new node (in Current Owner).

To initiate a live migration using PowerShell

1. Open PowerShell. Click Start, point to All Programs, click Windows Powershell 2.0, and then click Windows Powershell 2.0.

   The Failover Clustering feature must be installed on the computer on which you are starting the PowerShell session. Or, you can use Remote Server Administration Tools for Windows® 7 to run the PowerShell session.

2. To install the Failover Clustering feature, type:

   Import-Module FailoverClusters

3. Type:

   Get-Cluster “<Cluster Name>” | Move-ClusterVirtualMachineRole -Name “<VM group name>” -Node “<Destination node name>”

   Where:

   • <Cluster Name> is the name of the cluster that the virtual machine is included in.

   • <VM group name> is the virtual machine resource group.

   • <Destination node name> is the name of the destination node to which you would like to move the virtual machine using live migration.

Troubleshooting live migrations

This section covers some of the issues that you may encounter when you are performing a live migration. Before you review the troubleshooting items in this section, you should confirm that:

• The cluster validation tool has successfully run.

• The antivirus scan excludes files that are related to the virtual machine, such as configuration files, VHDs, and snapshots. For more information, see the Microsoft Support article Error codes when you create or start a virtual machine on a Windows Server 2008-based computer that has Hyper-V.

Basic troubleshooting considerations

The following list covers some of the basic live migration troubleshooting issues.

• Make sure that all virtual network names are the same across the entire cluster.

• If the cluster consists of both x86-based and x64-based processor computers, note that live migration between both processor types will fail.

• If you initiate a second live migration before the clean-up of the first migration is complete, the second migration may fail. You should wait for several seconds before initiating the second migration.

• If a cluster service is restarted, or a virtual machine is moved and hosted by a new RHS.exe cluster process, the network that is used for the migration needs time to initialize. It may take up to 30 seconds for the network to be ready. If you initiate a live migration during this time, it will fail with the error “no migration network available”.

• Make sure that virtual switches and virtual switch ports do not have the same name. Live migration will fail on virtual machines connected to virtual switch ports that have the same name as the virtual switch.

Flowcharts for troubleshooting a live migration

Use the following two flowcharts to troubleshoot a live migration.

Collecting configuration information

The following list outlines the configuration information that will help diagnose troubleshooting issues.

Cluster information:

• Determine whether Cluster Shared Volumes is used. If so, note how many volumes.

• Determine the CPU model of each cluster node and verify that they are compatible.

• Analyze the cluster validation report.

• Determine the number of networks and each network’s configuration. For example, 1GB or 10GB, IPv4 or IPv6, and whether IPsec is used.

Network information:

• Determine whether the network adapters are capable of TCP Offload (Chimney), and whether TCP Offload (Chimney) is enabled in the management and guest operating systems.

• Determine whether the network adapters are bound to a virtual switch that is capable of virtual machine queue (VMQ).

• Determine whether the network adapters are teamed.

Virtual machine information:

• Determine how many virtual machines have been made highly available, and whether the virtual machines are using Cluster Shared Volumes.

• Determine what operating systems the virtual machines are using and what the workload is.

Retrieving event log information

To help diagnose troubleshooting issues, you can retrieve Hyper-V and cluster event log information.

• To enable the high availability analytic log, type the following command at a command prompt:

  Wevtutil sl “<Log Name>” /e:true /q

  Where:

  <Log Name> is the name of the high availability analytic log to enable.

• To retrieve information from the log, type the following command at a command prompt:

  Wevtutil epl query.txt /sq:true <Log Name>.evtx

  Where:

  <Log Name> is the name of the high availability analytic log from which to retrieve information.

  Query.txt code:

  <QueryList>
    <Query Id="0" Path="System">
      <Select Path="System">*[System[Provider[@Name='Microsoft-Windows-Hyper-V-Hypervisor']]]</Select>
      <Select Path="Microsoft-Windows-Hyper-V-Config-Admin">*</Select>
      <Select Path="Microsoft-Windows-Hyper-V-High-Availability-Admin">*</Select>
      <Select Path="Microsoft-Windows-Hyper-V-High-Availability-Analytic">*</Select>
      <Select Path="Microsoft-Windows-Hyper-V-Hypervisor-Admin">*</Select>
      <Select Path="Microsoft-Windows-Hyper-V-Integration-Admin">*</Select>
      <Select Path="Microsoft-Windows-Hyper-V-Network-Admin">*</Select>
      <Select Path="Microsoft-Windows-Hyper-V-SynthStor-Admin">*</Select>
      <Select Path="Microsoft-Windows-Hyper-V-SynthNic-Admin">*</Select>
      <Select Path="Microsoft-Windows-Hyper-V-Image-Management-Service-Admin">*</Select>
      <Select Path="Microsoft-Windows-Hyper-V-VMMS-Admin">*</Select>
      <Select Path="Microsoft-Windows-Hyper-V-Worker-Admin">*</Select>
    </Query>
  </QueryList>
  

• To retrieve information from cluster event logs, type the following from a PowerShell session:

  Get-ClusterLog or Get-ClusterLog –Destination <logdir>

  Where:

  <Logdir> is the location of the event logs.

  To retrieve information from cluster event logs using Cluster.exe at a command prompt, type:

  Cluster.exe log /g /copy:<logdir>