Share via


Working with Azure Storage Queues from PowerShell

 

Hello Everyone, in this blog post I would like to introduce you to my latest PowerShell module, similarly to the AzureRmStorageTables module that I developed a while ago, this module works with Storage Queues, enabling IT Pros to rely on this great service that Azure Storage provides.

A fully function example of how IT Pros could leverage tables and queues is presented in this GitHub repository where I published an Azure Automation based solution that helps customers upload custom virtual hard drives to Azure and have it distributed amongst different regions/subscriptions and generating managed images. This solution will have a separate blog post shortly.

Introduction

Azure Storage Queues is one of the four Microsoft Azure Storage abstractions available (Blobs, Tables and Azure Files are the other ones) at the time that this blog was written. It provides cloud messaging between application components. In designing applications for scale, application components are often decoupled, so that they can scale independently. Queue storage delivers asynchronous messaging for communication between application components, whether they are running in the cloud, on the desktop, on an on-premises server, or on a mobile device. Azure also provides a more full featured alternative called Azure Service Bus which is similar in concept but provides other features.

For more information about Service Bus, please refer to Service Bus messaging: flexible data delivery in the cloud.

For more information about Azure Storage Queue, please refer to Queue storage and Get started with Azure Queue storage using .NET.

If you want to compare Service Bus against Storage Queue, please refer to Storage queues and Service Bus queues - compared and contrasted.

 

Since up to today there are no official cmdlets to support placing/retrieving/deleting/modifying messages on Storage Queues from Azure PowerShell module, I decided to create this simple module to help IT Pros to leverage this service without having knowledge of .NET framework through some simple cmdlets as follows:

Cmdlet Description
Add-AzureRmStorageQueueMessage Adds a message into the queue
Clear-AzureRmStorageQueue Clears all messages in a queue
Get-AzureRmStorageQueueQueue Creates/Gets a queue object/resource
Invoke-AzureRmStorageQueueGetMessage Get a message from queue and marks it as invisible in the queue for a period of time
Invoke-AzureRmStorageQueuePeekMessage Gets a single message from the queue without removing it from the queue
Invoke-AzureRmStorageQueuePeekMessageList Gets a list of messages from the queue without removing it from the queue
Remove-AzureRmStorageQueueMessage Deletes a message from the queue
Update-AzureRmStorageQueueMessage Updates a message in the queue

 

There are a number of use cases for using it with PowerShell, specially during automation scenarios where you want to scale your automation to the next level. One example is the before mentioned solution I posted in GitHub. I would love to get more examples from the community over time in the comments section of this post.

 

Requirements

  • PowerShell 4.0 or greater
  • This module requires Azure PowerShell module installed, which can be downloaded from https://aka.ms/webpi-azps or the following minimum modules installed via PSGet module from PowerShell Gallery:
    • AzureRM.Profile
    • AzureRM.Storage
    • AzureRM.Resources
    • Azure.Storage

 

Installation/Source Code

Since this module is published on PowerShell Gallery, you can install this module directly from PowerShell 5.0 and Windows 10 / Windows Server 2016 by executing the following cmdlet in an elevated PowerShell command prompt window:

 Install-Module AzureRmStorageQueue

 

Note that if you want to use the PowerShellGet module in Windows 2012 R2 with PowerShell 4, please look at the https://www.powershellgallery.com/ initial page on how to download and install it. The process is not that straight forward to enable it on Windows 2012 R2 and because of that I placed a summary of the steps below:

  1. Download PowerShellGet source code from GitHub (https://github.com/PowerShell/PowerShellGet/releases)

  2. Right-click the ZIP flie and click “Unblock”

  3. Extract it anywhere

  4. Copy the folder <your extraction folder>\PowerShellHGet-XYZ\PowerShellGet to C:\Program Files\WindowsPowerShell\Modules

  5. Download and install PackageManagement PowerShell Modules Preview from https://www.microsoft.com/en-us/download/details.aspx?id=51451

  6. Open an elevated PowerShell command prompt and execute

     Install-PackageProvider Nuget –Force
    
  7. Close the command prompt

  8. Open a new elevated PowerShell command prompt and you’re ready to install the Azure Rm Storage Queue module

As an alternative, you can manually download the module from my GitHub repository (here) and extract to C:\Program Files\WindowsPowerShell\Modules and rename the folder to AzureRmStorageQueue. Please remember to unblock the zip file before extracting it.

 

Working with Azure Rm Storage Queue PowerShell Module

The following steps will walk you through loading the module and perform one or more example tasks of the basic operations offered in this module.

 

Before you start working with it, you need to authenticate to Azure and select the correct subscription if you have multiple subscriptions:

 Add-AzureRmAccount
 Select-AzureRmSubscription -SubscriptionName <your subscription>

 

Next, lets import AzureRmTableQueue PowerShell module and list the functions available on it:

 Import-Module AzureRmStorageQueue
 Get-Command -Module AzureRmStorageQueue -CommandType Function

 

You should see the following cmdlets

image

 

For the sake of simplicity, we need to define some variables at this point to make our examples a little bit more clean, please, make sure you have a general purpose storage account to use and that you change the values of the variables below to reflect your environment. Notice that one of the variables is called $partitionKey, in this example we are using only one partition, please refer to the documentation at the end of this blog in order to get a better understanding on benefits of partitions.

 $resourceGroup = "imageprocess-rg"
 $storageAccountName = "pmcstorage07tier0"
 $queueName = "my-test-queue"

 

In this example I’m using the storage account named pmcstorage77tier0 and as we can see below I don’t have any queues created yet, this screen shot snippet was taken from Azure Storage Explorer, that can be installed from here.

 

image

 

Getting/Creating a queue

 $queue = Get-AzureRmStorageQueueQueue -resourceGroup $resourceGroup -storageAccountName $storageAccountName -queueName $queueName

 

This is the result of the cmdlet below, it created a new queue since I didn’t had yet one named my-test-queue, notice that queue names cannot contain capital letters and spaces.

 

image

 

If we look at the https://portal.azure.com you can see that now I do have a new queue in my storage account:

 

image

 

Adding a message to the queue

In order to add messages to the queue we need to define a hash table, it can be in a separate object or directly defined in the cmdlet as you can see:

Example 1

 Add-AzureRmStorageQueueMessage -queue $queue -message @{"type"="copy";"vhdname"="newvhd.vhd";"sourceStorageAccount"="pmcstorage05";"subscription"="mysubscription"}

 

image

 

Example 2

 $message = @{"type"="copy";
              "vhdname"="newvhd01.vhd";
              "sourceStorageAccount"="pmcstorage05";
              "subscription"="mysubscription"}
  
 Add-AzureRmStorageQueueMessage -queue $queue -message $message

 

image

 

The effect behind the scenes is that this hash table will become a json string in the queue so you can easily consume it in whatever application you need.

 

image

 

Details of message 1

image

 

Peeking messages from the queue

Peek process in a queue allows you to get the message without de-queuing it, useful when you need to just inspect values inside of it, we do have two cmdles for this process, one that peeks the first available message (Invoke-AzureRmStorageQueuePeekMessage) or one to get the all messages in the queue (Invoke-AzureRmStorageQueuePeekMessageList).

 

Example 1 – Peeking a single message

 $message = Invoke-AzureRmStorageQueuePeekMessage -queue $queue

 

Result

image

 

Example 2 – Peeking all messages

 $messageList = Invoke-AzureRmStorageQueuePeekMessageList -queue $queue

 

Result

image

 

De-queuing messages from the queue

This process is a little different because this is the regular queue operation, it hides the message as soon as it is de-queued for processing for example.

 $message =  Invoke-AzureRmStorageQueueGetMessage -queue $queue

 

Result

image

 

Please notice the DequeueCount property of the message, stating that it was de-queued one time. Now if we perform a peek to get all messages we just see the second message added because it was not de-queued. You can verify because the message ID is different as well.

image

 

Clearing queue from all messages

 Clear-AzureRmStorageQueue -queue $queue

 

Result

image

 

if you don’t want to be asked for confirmation you can just add –Confirm:$false to the end of the line.

 Clear-AzureRmStorageQueue -queue $queue -Confirm:$false

 

image

 

Removing messages from the queue

In this situation we have two ways to preform this, one is by passing the CloudQueueMessage object which is obtained by using the Invoke-AzureRmStorageQueueGetMessage cmdlet or if you know the message id and the popReceipt values you can use them directly. Notice that you can only delete a message if you de-queue it first.

 

Example 1 – Passing CloudQueueMessage object as argument

 

Adding a test message to queue

 Add-AzureRmStorageQueueMessage -queue $queue -message @{"type"="copy";"vhdname"="vhd01.vhd";"sourceStorageAccount"="pmcstorage05";"subscription"="mysubscription"}

image

De-queuing de message

 $message =  Invoke-AzureRmStorageQueueGetMessage -queue $queue

image

 

Removing the message based on the CloudQueueMessage object

 Remove-AzureRmStorageQueueMessage -queue $queue -message $message

image

 

Notice that after deleting it I tried to de-queue the message but nothing came out because I deleted the message.

 

Example 2 – Passing message ID and popReceipt as argument

 

Adding another test message in the queue

 Add-AzureRmStorageQueueMessage -queue $queue -message @{"type"="copy";"vhdname"="vhd02.vhd";"sourceStorageAccount"="pmcstorage05";"subscription"="mysubscription"}

 

De-queuing message

 $message =  Invoke-AzureRmStorageQueueGetMessage -queue $queue

 

Displaying message contents

image

Those red highlighted properties are the ones we will pass in the next cmdlet to delete the message.

 Remove-AzureRmStorageQueueMessage -queue $queue -messageId "969f7550-982d-49d5-bff8-26324adcf587" -popReceipt "AgAAAAMAAAAAAAAAgi9WalsU0wE="

Of course I could use the $message object I already as $message.id and $message.popReceipt but my idea was to show how the actual values looks like.

image

As a side note, message only gets the popReceipt when you de-queue it.

 

Updating a message and placing it back to the queue

Sometimes all you need is to update the message content, this operation exemplifies how to perform this operation.

 

Adding a test message to the queue

 Add-AzureRmStorageQueueMessage -queue $queue -message @{"type"="copy";"vhdname"="vhd03.vhd";"sourceStorageAccount"="pmcstorage05";"subscription"="mysubscription"}

 

De-queuing the message

 $msg = Invoke-AzureRmStorageQueueGetMessage -queue $queue

 

image

 

In this example we are changing the VHD name property of the JSON string with a random name, but first we need convert this JSON string into a PowerShell object to make things easier.

 $msgObj = $msg.AsString | ConvertFrom-Json

 

image

Changing the property content

 $msgObj.vhdname = [string]::format("{0}.vhd",[guid]::NewGuid().guid)

image

 

Updating the message object

 $msg.SetMessageContent(($msgObj | ConvertTo-Json -Depth 100))

image

 

Updating the message in the queue

 Update-AzureRmStorageQueueMessage -queue $queue -message $msg

image

 

Making the message visible again in the queue and peeking the queue to check the changes made, notice that the popReceipt is null meaning the message is ready to get de-queued again.

 Update-AzureRmStorageQueueMessage -queue $queue -message $msg -visibilityTimeout 0
 Invoke-AzureRmStorageQueuePeekMessage -queue $queue
  

image

 

Well, that’s it for this post, I hope you enjoy this new PowerShell module and that it helps you build great PowerShell scripts using Azure Storage Queues.

 

Regards

 

Paulo

Comments

  • Anonymous
    August 16, 2017
    Great description and samples!
  • Anonymous
    November 10, 2017
    This is brilliant, but I found a limitation. If you try and run Invoke-AzureRmStorageQueuePeekMessageList when there are more then 32 messages in the queue, you will get an error."Exception calling "PeekMessages" with "1" argument(s): "The remote server returned an error: (400) Bad Request." " This is not a limitation on the module, but of Azure, since even on the web portal and Storage Manager you can only see 32 messages.Since the module already counts the total messages in the queue, it might be worth it to have some messaging about a maximum queue message count.
    • Anonymous
      November 13, 2017
      Hi DC in TX,Thanks for the feedback, I will look at this and update the module.RegardsPaulo