Share via

Any Limitation on number of mounts of a storage container on different servers using blobfuse2 ?

Himanshu Singh 30 Reputation points
2024-10-04T22:53:13.2066667+00:00

I have 12 on-prem servers at different locations and they all run k3s on it. I have mounted the same Azure storage account container on all these servers using blobfuse2 (Documentation referred). Please note that all 12 servers use the same container SAS token to mount the same container.
All these servers act as individual clusters (makes it 12 clusters). Each cluster has 3 pods responsible for writing data on azure blob container. All three pods write data in their unique directory.

Problem:
The above setup was working fine on 10 servers/clusters which crashed after 2 weeks of live run with the below error. As of today, the above setup is running only on 5-7 servers at a time.

Logs from csi-blob-node pod and container named blob :

I1002 15:47:26.575930 13323 nodeserver.go:127] NodePublishVolume: volume mystorage_mycontainer1 mounting /var/lib/kubelet/plugins/kubernetes.io/csi/blob.csi.azure.com/d15383f56406a2c08a1896bfb52c576a0500c32c4238d8ba1fcf556b4f0d4b07/globalmount at /var/lib/kubelet/pods/2774de23-e83b-4c38-b163-69f51a2bdc8b/volumes/kubernetes.io~csi/pv-blob/mount with mountOptions: [bind]

I1002 15:47:26.575943 13323 mount_linux.go:218] Mounting cmd (mount) with arguments ( -o bind /var/lib/kubelet/plugins/kubernetes.io/csi/blob.csi.azure.com/d15383f56406a2c08a1896bfb52c576a0500c32c4238d8ba1fcf556b4f0d4b07/globalmount /var/lib/kubelet/pods/2774de23-e83b-4c38-b163-69f51a2bdc8b/volumes/kubernetes.io~csi/pv-blob/mount)

E1002 15:47:26.577082 13323 mount_linux.go:230] Mount failed: exit status 32

Mounting command: mount

Mounting arguments: -o bind /var/lib/kubelet/plugins/kubernetes.io/csi/blob.csi.azure.com/d15383f56406a2c08a1896bfb52c576a0500c32c4238d8ba1fcf556b4f0d4b07/globalmount /var/lib/kubelet/pods/2774de23-e83b-4c38-b163-69f51a2bdc8b/volumes/kubernetes.io~csi/pv-blob/mount

Output: mount: /var/lib/kubelet/pods/2774de23-e83b-4c38-b163-69f51a2bdc8b/volumes/kubernetes.io~csi/pv-blob/mount: special device /var/lib/kubelet/plugins/kubernetes.io/csi/blob.csi.azure.com/d15383f56406a2c08a1896bfb52c576a0500c32c4238d8ba1fcf556b4f0d4b07/globalmount does not exist. DRIVER INFORMATION:

Build Date: "2023-10-19T11:51:15Z"

Compiler: gc

Driver Name: blob.csi.azure.com

Driver Version: v1.23.1

Git Commit: d6e08f4b0bcddbe42db9169c1bf9461d25b91093

Go Version: go1.21.1

Platform: linux/amd64

Is there a limit on number of read-write mounts for a single container using blobfuse2 ? Can you please help me in identifying the root cause of the problem?

Any help would be much appreciated.

Azure Storage Accounts
Azure Storage Accounts
Globally unique resources that provide access to data management services and serve as the parent namespace for the services.
3,192 questions
Azure Blob Storage
Azure Blob Storage
An Azure service that stores unstructured data in the cloud as blobs.
2,884 questions
Accepted answer
  1. Vinodh247 21,966 Reputation points
    2024-10-05T09:26:02.5266667+00:00

    Hi Himanshu Singh,

    Thanks for reaching out to Microsoft Q&A.

    The error you're experiencing is most likely due to the expiration of the SAS token used for mounting the Azure Blob Storage container with blobfuse2. Since the issue appeared after about two weeks, it aligns with common default expiry times for SAS tokens. By renewing the SAS token and ensuring it has a suitable validity period and permissions, you should be able to resolve the mounting issues.

    Is There a Limit on the Number of Read-Write Mounts for a Single Container Using Blobfuse2?

    No, Azure Blob Storage does not impose a hard limit on the number of read-write mounts or connections to a single container using blobfuse2.

    1. The error indicates that the globalmount directory does not exist at the time of the bind mount.
    2. In Kubernetes CSI drivers, a volume is first mounted to a global mount path. Subsequent bind mounts are made from this global path to individual pod volume paths.
    3. If the initial mount fails, the global mount path will not exist, causing the bind mount to fail.

    Possible Reasons for the Global Mount Path Not Existing:

    • Expired SAS Token: If the SAS token used for authentication has expired, blobfuse2 cannot mount the container, resulting in the absence of the global mount path.
    • Blobfuse2 Process Failure: The blobfuse2 process may have crashed or terminated unexpectedly on the affected nodes.
    • Network Issues: Connectivity problems between the servers and Azure Blob Storage could prevent successful mounting.
    • Resource Limits on the Node: Although unlikely, if the node has hit a limit on the number of file descriptors or mounts, new mounts might fail.

    Why the Issue Occurred After 2 Weeks:

    • SAS Token Expiry: By default, SAS tokens can have a predefined expiry time. If all tokens were generated with a 2-week validity period, they would all start failing around the same time.
    • Credential Rotation: If credentials were rotated or revoked, this could disrupt existing mounts.

    Explanation:

    Azure Blob Storage Scalability: Azure Blob Storage is designed to handle a large number of concurrent connections and operations. It supports thousands of simultaneous clients without performance degradation.

    Blobfuse2 Limitations: Blobfuse2 itself does not impose a limit on the number of mounts or clients. It relies on FUSE and the underlying operating system, which also do not have strict limits on the number of mounts.

    Potential Throttling: While there is no hard limit, Azure may throttle requests if the account exceeds certain thresholds (e.g., ingress/egress bandwidth, transactions per second). However, this would typically result in throttling errors, not mount failures.

    Possible workarounds:

    Check Expiry: Ensure that the SAS token used has not expired. If it has, generate a new SAS token with an appropriate expiry time or consider using a storage account key for long-term access.

    Token Scope: Make sure the SAS token has the necessary permissions (read, write, list) for the container.

    Please 'Upvote'(Thumbs-up) and 'Accept' as an answer if the reply was helpful. This will benefit other community members who face the same issue.

    1 person found this answer helpful.

0 additional answers

Sort by: Most helpful
Most helpful Newest Oldest

Your answer

Answers can be marked as Accepted Answers by the question author, which helps users to know the answer solved the author's problem.