Ensuring ARM template idempotency for Azure Kubernetes Service

Hannu Piki 0 Reputation points
2024-10-09T08:26:13.33+00:00

Is it possible to make ARM template deployment for "Microsoft.ContainerService/managedClusters" to be entirely idempotent when azureKeyvaultSecretsProvider and/or azurepolicy addonProfiles are being enabled?

If I run template once, it works perfectly well. If I run that second time, I need to wait 10 minutes to get response even there is zero need to change anything. I know when azureKeyvaultSecretsProvider or azurepolicy are enabled, it will create automatically managed identity and assigns it for the VMSS. Can't be prevented. Because of that it isn't possible to assign into addonProfile identity block, just read its output. That would be fine, if it wouldn't change cluster into updating mode when redeploying the same template and no further changes can be made during that.

Few screenshots to illustrate how Activity Log is showing when deployment starts and its outcome (hidden changes are just cluster/nodepool state changes). It does remove identity from keyvault secrets provider and then exactly the same comes back.

image (11)

Currently running deployments for this from Crossplane with ResourceGroupTemplateDeployment (resources.azure.upbound.io/v1beta1) resource, which is using TF SDK and AzureRM's azurerm_resource_group_template_deployment resource in the background. But exactly the same behavior can be experienced with raw ARM template only.

Any suggestions how to resolve this would be highly appreciated.

Actual ARM template with masked GUIDs:

{
    "$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#",
    "contentVersion": "1.0.0.0",
    "outputs": {
        "clusterFqdn": {
            "type": "string",
            "value": "[reference(resourceId('Microsoft.ContainerService/managedClusters', 'aks-hpixpspoke01-aks')).fqdn]"
        },
        "id": {
            "type": "string",
            "value": "[resourceId('Microsoft.ContainerService/managedClusters', 'aks-hpixpspoke01-aks')]"
        },
        "oidcIssuerUrl": {
            "type": "string",
            "value": "[reference(resourceId('Microsoft.ContainerService/managedClusters', 'aks-hpixpspoke01-aks')).oidcIssuerProfile.issuerUrl]"
        }
    },
    "parameters": {},
    "resources": [
        {
            "apiVersion": "2024-06-02-preview",
            "identity": {
                "UserAssignedIdentities": {
                    "/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/rg-hpixpspoke01-aks/providers/Microsoft.ManagedIdentity/userAssignedIdentities/id-aks-hpixpspoke01-aks-ctrlplane": {}
                },
                "type": "UserAssigned"
            },
            "location": "westeurope",
            "name": "aks-hpixpspoke01-aks",
            "properties": {
                "aadProfile": {
                    "adminGroupObjectIDs": [
                        "00000000-0000-0000-0000-000000000000"
                    ],
                    "enableAzureRbac": true,
                    "managed": true,
                    "tenantID": "00000000-0000-0000-0000-000000000000"
                },
                "addonProfiles": {
                    "azureKeyvaultSecretsProvider": {
                        "config": {
                            "enableSecretRotation": "true",
                            "rotationPollInterval": "2m"
                        },
                        "enabled": true
                    },
                    "azurepolicy": {
                        "enabled": false
                    }
                },
                "agentPoolProfiles": [
                    {
                        "availabilityZones": [
                            "1",
                            "2",
                            "3"
                        ],
                        "count": 1,
                        "enableAutoScaling": true,
                        "enableFIPS": false,
                        "enableNodePublicIP": false,
                        "kubeletDiskType": "OS",
                        "maxCount": 2,
                        "maxPods": 110,
                        "minCount": 1,
                        "mode": "System",
                        "name": "nodepool1",
                        "osDiskSizeGB": 0,
                        "osDiskType": "Managed",
                        "osSKU": "AzureLinux",
                        "osType": "Linux",
                        "type": "VirtualMachineScaleSets",
                        "vmSize": "Standard_D2as_v4",
                        "vnetSubnetID": "[resourceId('rg-hpixpspoke01-default', 'Microsoft.Network/virtualNetworks/subnets', 'vnet-hpixpspoke01', 'aks-system')]"
                    }
                ],
                "autoScalerProfile": {
                    "balance-similar-node-groups": "false",
                    "daemonset-eviction-for-occupied-nodes": true,
                    "max-node-provision-time": "15m",
                    "scale-down-delay-after-add": "10m",
                    "skip-nodes-with-local-storage": "false"
                },
                "autoUpgradeProfile": {
                    "nodeOSUpgradeChannel": "NodeImage",
                    "upgradeChannel": "patch"
                },
                "disableLocalAccounts": true,
                "dnsPrefix": "hpixpspoke01-aks",
                "enableRBAC": true,
                "identityProfile": {
                    "kubeletidentity": {
                        "clientId": "00000000-0000-0000-0000-000000000000",
                        "objectId": "00000000-0000-0000-0000-000000000000",
                        "resourceId": "/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/rg-hpixpspoke01-aks/providers/Microsoft.ManagedIdentity/userAssignedIdentities/id-aks-hpixpspoke01-aks-kubelet"
                    }
                },
                "kubernetesVersion": "1.30",
                "networkProfile": {
                    "dnsServiceIP": "10.222.0.10",
                    "ipFamilies": [
                        "IPv4"
                    ],
                    "kubeProxyConfig": {
                        "enabled": false,
                        "ipvsConfig": {
                            "scheduler": "LeastConnection",
                            "tcpFinTimeoutSeconds": 120,
                            "tcpTimeoutSeconds": 900,
                            "udpTimeoutSeconds": 300
                        },
                        "mode": "IPVS"
                    },
                    "loadBalancerSku": "standard",
                    "networkPlugin": "none",
                    "networkPolicy": "none",
                    "outboundType": "loadBalancer",
                    "podCidr": "10.223.0.0/16",
                    "podCidrs": [
                        "10.223.0.0/16"
                    ],
                    "serviceCidr": "10.222.0.0/24",
                    "serviceCidrs": [
                        "10.222.0.0/24"
                    ]
                },
                "oidcIssuerProfile": {
                    "enabled": true
                },
                "publicNetworkAccess": "Enabled",
                "securityProfile": {
                    "workloadIdentity": {
                        "enabled": true
                    }
                },
                "servicePrincipalProfile": {
                    "clientId": "msi"
                },
                "storageProfile": {
                    "diskCSIDriver": {
                        "enabled": true
                    },
                    "fileCSIDriver": {
                        "enabled": true
                    },
                    "snapshotController": {
                        "enabled": true
                    }
                },
                "supportPlan": "KubernetesOfficial",
                "upgradeSettings": {
                    "overrideSettings": {}
                }
            },
            "sku": {
                "name": "Base",
                "tier": "Standard"
            },
            "type": "Microsoft.ContainerService/managedClusters"
        }
    ],
    "variables": {}
}

Azure Kubernetes Service (AKS)
Azure Kubernetes Service (AKS)
An Azure service that provides serverless Kubernetes, an integrated continuous integration and continuous delivery experience, and enterprise-grade security and governance.
2,127 questions
{count} votes

Your answer

Answers can be marked as Accepted Answers by the question author, which helps users to know the answer solved the author's problem.