Azure ML | Support for ADLS Gen2 Datastore with SAS Token Authentication

Question

Hello,

I’m currently working on an application where we’re connecting various data sources—such as file shares and ADLS Gen2—to Azure Machine Learning. While we can create file-share datastores with SAS token authentication, I noticed that this option isn’t available for ADLS Gen2 datastores in Azure ML. Interestingly, this type of SAS-based authentication seems to be supported in Azure Databricks. Could you clarify if there’s a particular reason for this limitation in Azure ML? Enabling SAS token support for ADLS Gen2 would simplify our integration significantly.

Our use case involves utilizing ADLS Gen2 for blob storage, but we don’t have direct access to the ADLS resource via service principals. Instead, the data lake is primarily accessed through an API, and we are only allowed to generate SAS tokens. Would it be possible to create custom datastores in Azure ML that communicate with the data lake via the API?

Additionally, is it feasible to configure datastores using non-root-level SAS tokens? For instance, if a SAS token provides access only to a specific directory within a blob (e.g., root/some/path/), can we use it to create a datastore scoped to that directory?

Thank you for your assistance!

Answer

Hello Pål Hannus,

Welcome to the Microsoft Q&A and thank you for posting your questions here.

I understand that you are in need to create a data store that support Azure ML | Support for ADLS Gen2 Datastore with SAS Token Authentication.

This might be complex than expected, but I will give you best solution and links to read further for more steps and instructions, also utilize the additional resources by the right side of this page.

Therefore,

Currently, Azure Machine Learning does not support SAS token authentication for ADLS Gen2 datastores. - https://knowledge.informatica.com/s/article/FAQ-What-is-Shared-Access-Signature-Token-in-ADLS-Gen2 This limitation is likely due to security and management considerations, as SAS tokens can be less secure compared to other authentication methods like service principals or managed identities. Azure Databricks, on the other hand, does support SAS tokens for ADLS Gen2, which might be due to different security models and use cases. - https://github.com/easonlai/sas_access_to_adls_databricks
While Azure ML does not natively support SAS tokens for ADLS Gen2, you can potentially create a custom solution. One approach could be to use Azure Functions or Azure Logic Apps to act as an intermediary, generating SAS tokens and accessing the data via the API. This would involve setting up a service that handles the authentication and data access, then connecting this service to your Azure ML workspace.
Yes, it is feasible to configure datastores using non-root-level SAS tokens. If your SAS token provides access to a specific directory within a blob, you can scope your datastore to that directory. When creating the datastore, you would specify the path within the blob storage that the SAS token has access to. This allows you to limit the scope of access to just the necessary data. - https://zcusa.951200.xyz/en-us/azure/machine-learning/how-to-access-data-interactive?view=azureml-api-2

This links below cover all detailed steps on setting up a custom solution or any other specific aspect of your integration:

I hope this is helpful! Do not hesitate to let me know if you have any other questions.

Please don't forget to close up the thread here by upvoting and accept it as an answer if it is helpful.

Share via

Azure ML | Support for ADLS Gen2 Datastore with SAS Token Authentication

1 answer

Your answer