Share via

AzCopy between FileShare to Blob (with private endpoint)

Alex 375 Reputation points
2024-02-14T07:55:58.03+00:00

Hello,

I am facing an issue as below, would appreciate if any insights/guidance provided for it.

My setup:

  1. A File Share in SA1 - no private endpoints, selected networks allowed, trusted Azure services exception is enabled.
  2. A Blob in SA2 - private endpoint configured in Vnet1/Subnet1, no public access allowed.
  3. I run AzCopy command as below from a VM in Vnet2/Subnet1,
    azcopy sync <FS_url_with_sas> <Blob_url_with_sas> --recursive
  4. I get below error when I run it,
    INFO: Authentication Failed, it is either not correct, or expired, or does not have the correct permission PUT <blob_url>
    RESPONSE 403
    ERROR CODE: CannotVerifyCopySource

NOTE:

  1. I can access both FS and Blob SA endpoints from the AzCopy VM - able to do all operations on FS and Blob separately, but when I trigger the AzCopy command, I get above error.
  2. The same AzCopy works if I configure private endpoint for the FS in Vnet1/Subnet1 where blob endpoint is configured.

Qs:

  1. Does Blob considers the FS as the source or the AzCopy VM as the source?
  2. If FS is the source and if it directly connects to Blob,
    1. Which IP of the blob does FS use? would it be the public Traffic manager IP of the Blob endpoint or the private endpoint IP which I configured for the blob?
    2. How can I make the FS to connect to Blob using its private endpoint?

Thank you so much in advance.

Azure Files
Azure Files
An Azure service that offers file shares in the cloud.
1,295 questions
Azure Storage Accounts
Azure Storage Accounts
Globally unique resources that provide access to data management services and serve as the parent namespace for the services.
3,192 questions
Azure Blob Storage
Azure Blob Storage
An Azure service that stores unstructured data in the cloud as blobs.
2,883 questions
0 comments
Accepted answer
  1. Sam Cogan 10,742 Reputation points MVP
    2024-02-14T11:06:49.1366667+00:00

    When you use AzCopy to move files between two storage accounts, the request runs from the storage account, not from your machine. hence why it works when you configure a private endpoint for the FS in your vNet. To make this copy operation work, you either need the private endpoint so that the blob and file storage account are connected to the same vnet, or allow public access for the duration of the copy. The outbound public IP of the copy operation will not be one configured on the storage account, it could be any IP on the MS backend and you won't have control of this.

    1 person found this answer helpful.

1 additional answer

Sort by: Most helpful
Most helpful Newest Oldest
  1. Nehruji R 7,811 Reputation points Microsoft Vendor
    2024-02-14T11:40:31.27+00:00

    Hello Alex,

    Welcome to Microsoft Q&A Forum, thank you for posting your query here!

    Adding to above response, the error message you’re encountering during the AzCopy operation indicates an issue with authentication or permissions.

    Please ensure that the SAS (Shared Access Signature) tokens you’re using for both the file share and blob are valid and not expired. Double-check that the SAS tokens have the correct permissions (e.g., read, write, list) required for the operation, verify that the account key or connection string provided to AzCopy is accurate.

    Similar SO thread - https://stackoverflow.com/questions/73709699/azure-copy-blobs-across-storage-accounts-fails-with-errorcodecannotverifycopyso,https://stackoverflow.com/questions/70246046/azure-devops-azcopy-authentication-failed-it-is-either-not-correct-or-expired for reference.

    The CannotVerifyCopySource Error occurs when AzCopy cannot validate the source during the copy operation, to fix this issue, consider generating a new SAS token for the source blob with at least read permission and use that SAS URL as the copy source and update your command to use the new SAS token accordingly.

    refer troubleshooting for more information - https://zcusa.951200.xyz/en-us/troubleshoot/azure/azure-storage/storage-use-azcopy-troubleshoot

    Answering to your Qs:

    1.     Does Blob consider the FS as the source or the AzCopy VM as the source?

    ->AzCopy considers the Source as FS in SA1 and destination as Blob in SA2.

    refer - https://zcusa.951200.xyz/en-us/azure/storage/common/storage-use-azcopy-files

     

    2.If FS is the source and if it directly connects to Blob,

    a. Which IP of the blob does FS use? would it be the public Traffic manager IP of the Blob endpoint or the private endpoint IP which I configured for the blob?

    As the private endpoint is configured for the Blob in Vnet1/Subnet1, the FS will use the private IP address of the Blob’s private endpoint to connect and accordingly the Blob will respond using its private IP address within the same virtual network.

    b. How can I make the FS to connect to Blob using its private endpoint?

    Configure a private endpoint for the FS in Vnet1/Subnet1 similar to did for Blob and update the AzCopy command to use the private endpoint URL for the FS.

       

    refer- https://zcusa.951200.xyz/en-us/azure/storage/files/storage-files-networking-endpoints?tabs=azure-portal , https://zcusa.951200.xyz/en-us/azure/storage/common/storage-private-endpoints

    Hope this answer helps! Please let us know if you have any further queries. I’m happy to assist you further.

    Please do not forget to "Accept the answer” and “up-vote” wherever the information provided helps you, this can be beneficial to other community members.

Your answer

Answers can be marked as Accepted Answers by the question author, which helps users to know the answer solved the author's problem.