Share via

AKS cannot pull image from ACR with kubelet identity - 401 Unauthorized

QuyHoang 5 Reputation points
2024-10-18T03:39:56.6466667+00:00

After using service principal for AKS for a year, I switched to user-assigned managed identity. Everything is working well except pulling image from ACR, here is the error:

Failed to pull image "myacr.azurecr.io/fluent/fluent-bit:3.1.6":[rpc error: code = Unknown desc = failed to pull and unpack image "myacr.azurecr.io/fluent/fluent-bit:3.1.6": failed to resove reference "myacr.azurecr.io/fluent/fluent-bit:3.1.6": failed to authorize: failed to fetch oauth token: unexpected status from GET request to https://myacr.azurecr.io/oauth2/token?scope=repository%3Afluent%2Ffluent-bit%3Apull&service=myacr.azurecr.io: 401 Unauthorized, rpc error: code = Unknown desc = failed to pull and unpack image "myacr.azurecr.io/fluent/fluent-bit:3.1.6": failed to resolve reference "myacr.azurecr.io/fluent/fluent-bit:3.1.6": failed to authorize: failed to fetch anonymous token: unexpected status from GET request to https://myacr.azurecr.io/oauth2/token?scope=repository%3Afluent%2Ffluent-bit%3Apull&service=myacr.azurecr.io: 401 Unauthorized]

My resources meet all the requirements:

  • AKS already have kubelet identity assigned.
  • Kubelet identity was assigned AcrPull role with scope of exactly ACR.
  • Kubelet identity was associated with VMSS of AKS node pools.
  • There is no connection issue between ACR and AKS (It worked if I use secret in AKS)
  • Service principal client id on AKS is msi.

I tried to follow these documents:

However, it didn't work. I even tried to scale new node on node pool.

It seems like this is a bug, anyone have solution for this?

Azure Container Registry
Azure Container Registry
An Azure service that provides a registry of Docker and Open Container Initiative images.
461 questions
Azure Kubernetes Service (AKS)
Azure Kubernetes Service (AKS)
An Azure service that provides serverless Kubernetes, an integrated continuous integration and continuous delivery experience, and enterprise-grade security and governance.
2,127 questions

1 answer

Sort by: Most helpful
Most helpful Newest Oldest
  1. Mahesh Goud Juvvadi 1,065 Reputation points Microsoft Vendor
    2024-10-18T16:08:35.6833333+00:00

    Hi QuyHoang,

    Thank you for reaching out to the Microsoft Q&A platform.

    Encountering a 401 Unauthorized error when pulling images from Azure Container Registry (ACR) using a user-assigned managed identity in Azure Kubernetes Service (AKS).

    Here are a few steps to troubleshoot and resolve this issue:

    verify if the kubelet identity is correctly assigned, run the following command:

    az aks show --resource-group <nameofResourcegrp> --name <cluster-name> --query "identityProfile.kubeletidentity"

    After confirming the kubelet identity, check if the required role is assigned:

    az role assignment list --assignee <your-managed-identity-client-id> --scope /subscriptions/<your-subscription-id>/resourceGroups/<your-resource-group>/providers/Microsoft.ContainerRegistry/registries/<your-acr-name>

    Create Role Assignment (if missing) if the role is not assigned, you can create it with:

    az role assignment create --assignee <your-managed-identity-client-id> --role AcrPull --scope /subscriptions/<your-subscription-id>/resourceGroups/<your-resource-group>/providers/Microsoft.ContainerRegistry/registries/<your-acr-name>

    Update AKS Cluster Ensure the AKS cluster is updated to use the correct managed identity:

    az aks update --resource-group <your-resource-group> --name <your-cluster-name> --assign-identity <your-managed-identity-resource-id>

    Verify Kubelet Identity for Node Pools to check if the kubelet identity is applied to all node pools, use:

    az vmss identity show --resource-group <your-resource-group> --name <your-node-pool-name>

    Set Up Diagnostic Settings for ACR Optionally, you can create diagnostic settings for your Azure Container Registry (ACR):

    az acr diagnostic-settings create --registry-name <your-acr-name> --logs Read --metrics Read --storage-account <your-storage-account>

    If an answer has been helpful, please consider accept the answer and "Upvote" to help increase visibility of this question for other members of the Microsoft Q&A community. 

    User's image

    Thank you.

    0 comments

Your answer

Answers can be marked as Accepted Answers by the question author, which helps users to know the answer solved the author's problem.