Rotate secrets in Azure Stack Hub

This article provides guidance for performing secret rotation, to help maintain secure communication with Azure Stack Hub infrastructure resources and services.

Overview

Azure Stack Hub uses secrets to maintain secure communication with infrastructure resources and services. To maintain the integrity of the Azure Stack Hub infrastructure, operators need the ability to rotate secrets at frequencies that are consistent with their organization's security requirements.

When secrets are nearing expiration, the following alerts are generated in the administrator portal. Completing secret rotation will resolve these alerts:

• Pending service account password expiration
• Pending internal certificate expiration
• Pending external certificate expiration

For more information on alert monitoring and remediation, see Monitor health and alerts in Azure Stack Hub.

Prerequisites

1. It's highly recommended that you're running a supported version of Azure Stack Hub and that you apply the latest available hotfix for the Azure Stack Hub version your instance is running. For example, if you're running 2008, make sure you've installed the latest hotfix available for 2008.

   Important

   For pre-1811 versions:

   • If secret rotation has already been performed, you must update to version 1811 or later before you perform secret rotation again. Secret Rotation must be executed via the Privileged Endpoint and requires Azure Stack Hub Operator credentials. If you don't know whether secret rotation has been run on your environment, update to 1811 before performing secret rotation.
   • You don't need to rotate secrets to add extension host certificates. You should follow the instructions in the article Prepare for extension host for Azure Stack Hub to add extension host certificates.

2. Notify your users of planned maintenance operations. Schedule normal maintenance windows, as much as possible, during non-business hours. Maintenance operations may affect both user workloads and portal operations.

3. Generate certificate signing requests for Azure Stack Hub.

4. Prepare Azure Stack Hub PKI certificates.

5. During rotation of secrets, operators may notice alerts open and automatically close. This behavior is expected and the alerts can be ignored. Operators can verify the validity of these alerts using the Test-AzureStack PowerShell cmdlet. For operators, using System Center Operations Manager to monitor Azure Stack Hub systems, placing a system in maintenance mode will prevent these alerts from reaching their ITSM systems. However, alerts will continue to come if the Azure Stack Hub system becomes unreachable.

Rotate internal secrets

Internal secrets include certificates, passwords, secure strings, and keys used by the Azure Stack Hub infrastructure, without intervention of the Azure Stack Hub Operator. Internal secret rotation is only required if you suspect one has been compromised, or you've received an expiration alert.

Complete the following steps to rotate internal secrets:

1. Run the following PowerShell script. Notice for internal secret rotation, the "Run Secret Rotation" section uses only the -Internal parameter to the Start-SecretRotation cmdlet:

   # Create a PEP Session
   winrm s winrm/config/client '@{TrustedHosts= "<IP_address_of_ERCS>"}'
   $PEPCreds = Get-Credential
   $PEPSession = New-PSSession -ComputerName <IP_address_of_ERCS_Machine> -Credential $PEPCreds -ConfigurationName "PrivilegedEndpoint" -SessionOption (New-PSSessionOption -Culture en-US -UICulture en-US)
   
   # Run Secret Rotation
   Invoke-Command -Session $PEPSession -ScriptBlock {
       Start-SecretRotation -Internal
   }
   Remove-PSSession -Session $PEPSession
   

   Note

   Pre-1811 versions don't require the -Internal flag.

2. After successful completion, your console will display a ActionPlanInstanceID ... CurrentStatus: Completed message, followed by Action plan finished with status: 'Completed'.

   Note

   If secret rotation fails, follow the instructions in the error message and rerun Start-SecretRotation with the -Internal and -ReRun parameters.

   Start-SecretRotation -Internal -ReRun
   

   Contact support if you experience repeated secret rotation failures.

Rotate Azure Stack Hub root certificate

The Azure Stack Hub root certificate is provisioned during deployment with an expiration of five years. Starting with 2108, internal secret rotation also rotates the root certificate. The standard secret expiration alert identifies the expiry of the root certificate and generates alerts at both 90 (warning) and 30 (critical) days.

To rotate the root certificate, you must update your system to 2108 and perform internal secret rotation.

The following code snippet uses the Privileged Endpoint to list the expiration date of the root certificate:

$pep = New-PSSession -ComputerName <ip address> -ConfigurationName PrivilegedEndpoint -Credential $cred -SessionOption (New-PSSessionOption -Culture en-US -UICulture en-US) 
 
$stampInfo = Invoke-Command -Session $pep -ScriptBlock { Get-AzureStackStampInformation }

$rootCert = $stampInfo.RootCACertificates| Sort-Object -Property NotAfter | Select-Object -Last 1
"The Azure Stack Hub Root Certificate expires on {0}" -f $rootCert.NotAfter.ToString("D") | Write-Host -ForegroundColor Cyan

Update the BMC credential

The baseboard management controller monitors the physical state of your servers. Refer to your original equipment manufacturer (OEM) hardware vendor for instructions to update the user account name and password of the BMC.

2. Open a privileged endpoint in Azure Stack Hub sessions. For instructions, see Using the privileged endpoint in Azure Stack Hub.

3. After opening a privileged endpoint session, run one of the PowerShell scripts below, which use Invoke-Command to run Set-BmcCredential. If you use the optional -BypassBMCUpdate parameter with Set-BMCCredential, credentials in the BMC aren't updated. Only the Azure Stack Hub internal datastore is updated. Pass your privileged endpoint session variable as a parameter.

   Here's an example PowerShell script that will prompt for user name and password:

   # Interactive Version
   $PEPIp = "<Privileged Endpoint IP or Name>" # You can also use the machine name instead of IP here.
   $PEPCreds = Get-Credential "<Domain>\CloudAdmin" -Message "PEP Credentials"
   $NewBmcPwd = Read-Host -Prompt "Enter New BMC password" -AsSecureString
   $NewBmcUser = Read-Host -Prompt "Enter New BMC user name"
   
   $PEPSession = New-PSSession -ComputerName $PEPIp -Credential $PEPCreds -ConfigurationName "PrivilegedEndpoint" -SessionOption (New-PSSessionOption -Culture en-US -UICulture en-US)
   
   Invoke-Command -Session $PEPSession -ScriptBlock {
       # Parameter BmcPassword is mandatory, while the BmcUser parameter is optional.
       Set-BmcCredential -BmcPassword $using:NewBmcPwd -BmcUser $using:NewBmcUser
   }
   Remove-PSSession -Session $PEPSession
   

   You can also encode the user name and password in variables, which may be less secure:

   # Static Version
   $PEPIp = "<Privileged Endpoint IP or Name>" # You can also use the machine name instead of IP here.
   $PEPUser = "<Privileged Endpoint user for example Domain\CloudAdmin>"
   $PEPPwd = ConvertTo-SecureString '<Privileged Endpoint Password>' -AsPlainText -Force
   $PEPCreds = New-Object System.Management.Automation.PSCredential ($PEPUser, $PEPPwd)
   $NewBmcPwd = ConvertTo-SecureString '<New BMC Password>' -AsPlainText -Force
   $NewBmcUser = "<New BMC User name>"
   
   $PEPSession = New-PSSession -ComputerName $PEPIp -Credential $PEPCreds -ConfigurationName "PrivilegedEndpoint" -SessionOption (New-PSSessionOption -Culture en-US -UICulture en-US)
   
   Invoke-Command -Session $PEPSession -ScriptBlock {
       # Parameter BmcPassword is mandatory, while the BmcUser parameter is optional.
       Set-BmcCredential -BmcPassword $using:NewBmcPwd -BmcUser $using:NewBmcUser
   }
   Remove-PSSession -Session $PEPSession
   

Reference: Start-SecretRotation cmdlet

Start-SecretRotation cmdlet rotates the infrastructure secrets of an Azure Stack Hub system. This cmdlet can only be executed against the Azure Stack Hub privileged endpoint, by using an Invoke-Command script block passing the PEP session in the -Session parameter. By default, it rotates only the certificates of all external network infrastructure endpoints.

Syntax

For external secret rotation

Start-SecretRotation [-PfxFilesPath <string>] [-PathAccessCredential <PSCredential>] [-CertificatePassword <SecureString>]  

For internal secret rotation

Start-SecretRotation [-Internal]  

For external secret rotation rerun

Start-SecretRotation [-ReRun]

For internal secret rotation rerun

Start-SecretRotation [-ReRun] [-Internal]

Examples

Rotate only internal infrastructure secrets

This command must be run via your Azure Stack Hub environment's privileged endpoint.

PS C:\> Start-SecretRotation -Internal

This command rotates all of the infrastructure secrets exposed to the Azure Stack Hub internal network.

Rotate only external infrastructure secrets

# Create a PEP Session
winrm s winrm/config/client '@{TrustedHosts= "<IP_address_of_ERCS>"}'
$PEPCreds = Get-Credential
$PEPSession = New-PSSession -ComputerName <IP_address_of_ERCS> -Credential $PEPCreds -ConfigurationName "PrivilegedEndpoint" -SessionOption (New-PSSessionOption -Culture en-US -UICulture en-US)

# Create Credentials for the fileshare
$CertPassword = ConvertTo-SecureString '<CertPasswordHere>' -AsPlainText -Force
$CertShareCreds = Get-Credential
$CertSharePath = "<NetworkPathOfCertShare>"
# Run Secret Rotation
Invoke-Command -Session $PEPsession -ScriptBlock {
    param($CertSharePath, $CertPassword, $CertShareCreds )
    Start-SecretRotation -PfxFilesPath $CertSharePath -PathAccessCredential $CertShareCreds -CertificatePassword $CertPassword
} -ArgumentList ($CertSharePath, $CertPassword, $CertShareCreds)
Remove-PSSession -Session $PEPSession

This command rotates the TLS certificates used for Azure Stack Hub's external network infrastructure endpoints.

# Create a PEP Session
winrm s winrm/config/client '@{TrustedHosts= "<IP_address_of_ERCS>"}'
$PEPCreds = Get-Credential
$PEPSession = New-PSSession -ComputerName <IP_address_of_ERCS> -Credential $PEPCreds -ConfigurationName "PrivilegedEndpoint" -SessionOption (New-PSSessionOption -Culture en-US -UICulture en-US)
# Create Credentials for the fileshare
$CertPassword = ConvertTo-SecureString '<CertPasswordHere>' -AsPlainText -Force
$CertShareCreds = Get-Credential
$CertSharePath = "<NetworkPathOfCertShare>"
# Run Secret Rotation
Invoke-Command -Session $PEPSession -ScriptBlock {
    Start-SecretRotation -PfxFilesPath $using:CertSharePath -PathAccessCredential $using:CertShareCreds -CertificatePassword $using:CertPassword
}
Remove-PSSession -Session $PEPSession

Next steps

Learn more about Azure Stack Hub security