Share via

ImagePullBackOff kubelet Failed to pull image .westeurope.data.azurecr.io : failed to resolve reference : failed to do request : tls: failed to verify certificate: x509: certificate is valid for *.azurecr.io, not .westeurope.data.azurecr.io

Thomas FOUREST (admintenant.tfo) 0 Reputation points
2024-10-01T08:34:41.7966667+00:00

Hi,

I 've got an issue on my new AKS cluster to pull image from my ACR :

  • it seems to be great created:
  • networking is good (can login/pull/push to the acr from a podman pod in the cluster),
  • managedidentity with push/pull is role well created (az aks check-acr ... : "Your cluster can pull images from acrblablabla.azurecr.io")
    • but : when I create a pod , node can get the image from the acr
Name:         mypodubi8
Namespace:    default
Priority:     0
Node:         aks-.../x.x.x.x
Start Time:   Thu, 26 Sep 2024 14:47:15 +0000
Labels:       <none>
Annotations:  cni.projectcalico.org/containerID: a662e7ce540900006173f76ad65a7e16c6701fa86bfaf1a05c1d116dae4875e4
              cni.projectcalico.org/podIP: 172.25.5.24/32
              cni.projectcalico.org/podIPs: 172.25.5.24/32
Status:       Pending
IP:           172.25.5.24
IPs:
  IP:  172.25.5.24
Containers:
  mypodubi8:
    Container ID:  
    Image:         acrblablabla.westeurope.data.azurecr.io/valid/ubi8:latest
    Image ID:           Port:          <none>
    Host Port:     <none>
    Args:
      sleep
      1000000
    State:          Waiting
      Reason:       ErrImagePull
    Ready:          False
    Restart Count:  0
    Environment:
      HTTP_PROXY:   http://proxy-cloud.xxx.cloud:80/
      http_proxy:   http://proxy-cloud.xxx.cloud:80/
      HTTPS_PROXY:  http://proxy-cloud.xxx.cloud:80/ 
      https_proxy:  http://proxy-cloud.xxx.cloud:80/
      NO_PROXY:     blablabla,acrblablabla.westeurope.data.azurecr.io,acrblablabla.azurecr.io 
      no_proxy:     blablabla,acrblablabla.westeurope.data.azurecr.io,acrblablabla.azurecr.io 
      Mounts:       /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-lkfkj (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  kube-api-access-lkfkj:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason     Age                From               Message
  ----     ------     ----               ----               -------
  Normal   Scheduled  43s                default-scheduler  Successfully assigned default/mypodubi8 to aks-xxx
  Normal   Pulling    27s (x2 over 43s)  kubelet            Pulling image "acrblablabla.westeurope.data.azurecr.io/valid/ubi8:latest" 
  Warning  Failed     27s (x2 over 42s)  kubelet            Failed to pull image "acrblablabla.westeurope.data.azurecr.io/valid/ubi8:latest": failed to pull and unpack image "acrblablabla.westeurope.data.azurecr.io/valid/ubi8:latest": failed to resolve reference "acrblablabla.westeurope.data.azurecr.io/valid/ubi8:latest": failed to do request: Head "https://acrblablabla.westeurope.data.azurecr.io/v2/v
alid/ubi8/manifests/latest": tls: failed to verify certificate: x509: certificate is valid for *.azurecr.io, not acrblablabla.westeurop e.data.azurecr.io 
  Warning  Failed     27s (x2 over 42s)  kubelet            Error: ErrImagePull 
  Normal   BackOff    12s (x3 over 42s)  kubelet            Back-off pulling image "acrblablabla.westeurope.data.azurecr.io/valid/ubi8: latest" 
  Warning  Failed     12s (x3 over 42s)  kubelet            Error: ImagePullBackOff

Please, help is needed,
Thomas

Azure Container Registry
Azure Container Registry
An Azure service that provides a registry of Docker and Open Container Initiative images.
460 questions
Azure Kubernetes Service (AKS)
Azure Kubernetes Service (AKS)
An Azure service that provides serverless Kubernetes, an integrated continuous integration and continuous delivery experience, and enterprise-grade security and governance.
2,128 questions
0 comments

3 answers

Sort by: Most helpful
Most helpful Newest Oldest
  1. Mounika Reddy Anumandla 505 Reputation points Microsoft Vendor
    2024-10-01T10:45:43.9733333+00:00

    Hi Thomas FOUREST (admintenant.tfo),
    Welcome to the Microsoft Q&A Platform! Thank you for asking your question here.

    An ImagePullBackOff is the result of repeat ErrImagePull errors, meaning the kubelet tried to pull a container image several times and failed. This indicates a persistent problem that needs to be addressed.
    The issue you're experiencing here is due to a certificate verification error when trying to pull an image from your Azure Container Registry (ACR). The error message indicates that the certificate is valid for *.azurecr.io, but not for acrblablabla.westeurope.data.azurecr.io.

    I can provide you the below document which matches with your scenario:-)
    https://github.com/Azure/acr/issues/425

    If you have any further queries, do let us know.

    If the answer is helpful, please click "Accept Answer" and "Upvote it."

  2. Thomas FOUREST (admintenant.tfo) 0 Reputation points
    2024-10-16T11:17:53.99+00:00

    Hello,

    First of all, Thanks for the support !

    We solve the issue with new entries in our private DNS zone for a best resolution of the private endpoint that link the ACR PaaS in our subnet.
    So the certificat error will be the good symptom to find the root cause.

    Best regards,

    Thomas.

    0 comments
  3. Mounika Reddy Anumandla 505 Reputation points Microsoft Vendor
    2024-10-16T15:15:52.2433333+00:00

    Hi Thomas FOUREST (admintenant.tfo),

    Hope you are doing good!
    Glad to know that you were able to resolve your issue and thank you for posting your solution so that others experiencing the same thing can easily reference this! Since the Microsoft Q&A community has a policy that "The question author cannot accept their own answer. They can only accept answers by others ", I'll repost your solution in case you'd like to "Accept " the answer.

    Issue: The AKS cluster is unable to pull the image "acrblablabla.westeurope.data.azurecr.io/valid/ubi8:latest" from your ACR. The error is due to a TLS certificate verification failure. The certificate is valid for "*.azurecr.io", but not for "acrblablabla.westeurope.data.azurecr.io".

    Solution: Created new entries in their private DNS zone, which helps to properly resolve the private endpoint linking the Azure Container Registry (ACR) PaaS in their subnet. The certificate error is a good indication of the root cause of the problem.

    If you have any other questions or are still running into more issues, please let me know. Thank you again for your time and patience throughout this issue.

    Please remember to "Accept Answer" if any answer/reply helped, so that others in the community facing similar issues can easily find the solution.

    User's image

    Thank you!

Your answer

Answers can be marked as Accepted Answers by the question author, which helps users to know the answer solved the author's problem.