Unable to connect another tenant SMB file share from my Aks cluster

Hello Hemabhushan,

Welcome to Microsoft Q&A Forum. Thanks for posting your query here!

I understand that your Unable to connect another tenant SMB file share from Aks cluster.

Further to my investigation i see that the (Error Message: mount error (13): Permission denied) indicates that the service principal used by your AKS cluster lacks the necessary permissions to access the file share in the other tenant.

Firstly, we request you to Verify File Share Permissions of Tenant A Authorization. In Tenant A (where the file share resides), ensure the service principal used by your AKS cluster in Tenant B has the appropriate permissions (e.g., Storage File Contributor or equivalent) to access the file share and also ensure Azure Active Directory (AAD) Configuration, double-check that AAD is configured correctly for cross-tenant access. This might involve granting specific permissions to the service principal in Tenant A's AAD.

We recommend you adhere to the steps outlined below before attempting to connect the tenant to the SMB file share from the AKS cluster.

1. If your AKS cluster pod uses a managed identity for accessing resources, confirm that it has the required Azure RBAC permissions to access the file share in Tenant A.
2. When mounting the file share, ensure you're providing the correct credentials for the service principal or managed identity with access to the file share. You might need to use a secret or config map to store the credentials securely within the pod.
3. Consider using additional options while mounting the file share, such as:
   • vers=3.0 to specify the SMB version (may be required for older SMB servers)
   • cifs_username and cifs_password to explicitly provide credentials (if not using a managed identity)
4. Verify that there are no firewall rules within the security groups associated with the storage account or virtual network that might be blocking access from your AKS cluster's subnet.

The above steps should be able to identify the root cause of the permission error and successfully mount the SMB file share from another tenant in your AKS cluster. If you're still encountering the same issue after trying these steps, please helps us with the screenshots to investigate further.

Additional information: Documentation for mounting SMB file shares in AKS pods: https://medium.com/@intruder2021/how-to-mount-folder-from-azure-files-to-a-deployment-in-aks-6de90d11aced

Please let us know you have any further quires. We will be glad to assist you closely.

Please do consider to “up-vote” wherever the information provided helps you, this can be beneficial to other community members.

Hi Teja,

We have configured private endpoint connections for Tenant A's AKS and Tenant B's storage account, and we also provided the necessary permissions for the storage account. However, we are encountering an error while using a Linux script to mount the file share on the pod in Tenant A.

Interestingly, the same script runs without issues on one of the virtual machines, which has its IP address allowed in the storage account's networking settings. In that case, the mounting process works perfectly.

Hello Hemabhushan,

Thank you for the quick response.

I see that you're encountering an error while using a Linux script to mount the file share on the pod in Tenant A.
We request you to kindly check the NSG Rules ensure that the Network Security Groups (NSGs) associated with the subnets of both the AKS cluster in Tenant A and the storage account in Tenant B allow the necessary traffic. Specifically, check for rules that might be blocking SMB traffic on the private endpoint and also Verify that the AKS pods in Tenant A can resolve the private endpoint's FQDN. Check the DNS resolution within the AKS cluster's pods.

Regarding the Linux Script and Mounting Options If using service principal credentials, ensure they are correctly stored and accessed within the Linux script. Consider using secrets or environment variables for secure storage. You can check the mount command by reviewing the YAML file used to create the Kubernetes manifest for the pod. Please find the below mounting commands.

1. Review the exact mount command used in the script
2. Ensure the correct syntax and options are used
3. Verify that the vers option (e.g., vers=3.0) is specified if required by the SMB server
4. Consider using options like cifs_username and cifs_password if necessary.

please check the firewall settings on the storage account are not blocking the traffic from your AKS cluster. You can check the firewall settings by opening the Azure portal, navigating to the storage account that contains the file share, and selecting "Firewalls and virtual networks" and Ensure that the permissions on the file share are set up correctly to allow access from the AKS cluster. You can check the permissions by opening the Azure portal, navigating to the storage account that contains the file share, and selecting "Shared access signature".

Please let us know you have any further quires. We will be glad to assist you closely.

I'm able to connect the storage from cluster but unable to mount the storage, Allowed the Aks cluster load balancer frontend IPs also.Still i didn't find it where is it blocking.

one more thing ,when i try to follow the same steps on same tenant everything working fine.

Hello Hemabhushan,

Thank you for the kind response!

If the provided steps worked in the same tenant, it might be issue with the cross tenant with improper settings under Tenant A.

We request you to kindly Check if the AKS cluster is configured to use the private DNS zone for resolving the storage account's FQDN and verify DNS resolution within AKS pods using nslookup. If using a managed identity, confirm it has the required permissions. Post which use the private endpoint's FQDN in the mount command (e.g., storage-account-name.privatelink.file.core.windows.net) and ensure correct credentials are provided in the script (e.g., using secrets or environment variables).

1. Enable debug logging for the CIFS client on AKS nodes.
2. Examine system logs for error messages.
3. Use telnet or nc from an AKS pod to test connectivity to the storage account on port 445.

Prerequisites

• Cross-Tenant Security: Prioritize security when accessing resources across tenants.
• Testing: Thoroughly test the mount operation after making changes.

Please refer to the link for reference: https://zcusa.951200.xyz/en-us/troubleshoot/azure/azure-kubernetes/storage/fail-to-mount-azure-file-share#mounterror13

Additional information:

1. Identify the Node: Run the following command to identify the node hosting the faulty pod: kubectl get pod -n <namespace> -o wide.
2. Check VNET and Subnet: Go to the AKS cluster in the Azure portal, select Properties > Infrastructure resource group, access the virtual machine scale set (VMSS) associated with the node, and check the Virtual network/subnet to identify the VNET and subnet.
3. Allow VNET and Subnet: Access the storage account in the Azure portal, select Networking, and ensure that the VNET and subnet of the AKS cluster are allowed under Firewalls and virtual networks. If not, add them.
4. Check Network Security Group (NSG): Ensure that the NSG is not blocking traffic between the AKS cluster and the storage account. Use the following commands to check connectivity:

Let me know if you have any concerns we are here at your service.