Backup to remote Azure URL failed due to throttling at VM level

There are multiple levels of throttling with Azure.  You can get throttled at disk level, storage account level or VM/host level.    These types of throttling can manifest into something that may not be that obvious.

Recently we received a call from customer who was backing up databases from Azure VM to Azure blob storage.  The VM was DS13 using premium storage and backup URL is standard storage.  Periodically, he received the following error:

Backup     BackupIoRequest::ReportIoError: write failure on backup device 'https://<accountname>.blob.core.windows.net/dbbackup/backupfilename.bak'. Operating system error Backup to URL received an exception from the remote endpoint. Exception Message: The client could not finish the operation within specified timeout..

The error message seemed to indicate that something is wrong from remote storage.  So we engaged our storage team to analyze it. In this case, it turned out that the throttling occurred at VM level.   Customer was on DS13, this size of VM only allows 256MB IO throughput per https://azure.microsoft.com/en-us/documentation/articles/virtual-machines-windows-sizes/.    Below is a screenshot (“max uncached disk throughput:IOPS/MBps).  Note that throttling at VM level includes all IO incurred and combined including backup, data and log IO.

So the question is how do know if you are throttled at VM/host level?   You won’t get any errors in that nature.  But you can use two counters in the Azure Portal for the VM to estimate how much data was pushed through to estimate.  You can enable “Disk Read Guest OS” and “Disk Write Guest OS” to measure data going in and out.

Solution

So what’s the solution for this scenario?  The easiest is to upgrade VM size.  DS14 allows you to have twice as much throughput.  You can also make sure you don’t do backups all at once (spread them out)

Jack Li |Senior Escalation Engineer | Microsoft SQL Server

twitter| pssdiag |Sql Nexus