AdlsClient Class

Reference

Definition

Namespace:: Microsoft.Azure.DataLake.Store

Assembly:: Microsoft.Azure.DataLake.Store.dll

Package:: Microsoft.Azure.DataLake.Store v1.1.24

Package:: Microsoft.Azure.DataLake.Store v1.2.5-alpha

Important

Some information relates to prerelease product that may be substantially modified before it’s released. Microsoft makes no warranties, express or implied, with respect to the information provided here.

Client of Azure data lake store. It contains the public APIs to perform operations of REST API which are easier to call and more usable than Core APIs. Core APIs provide more freedom but ADLSClient provide more commonly used forms. It encapsulates the Authorization token and token refresh. Contains factory methods that takes a ServiceClientCredential or a string auth token and returns instance of this class. For every operation it provides a async and sync version. Every sync method is a wait on async method with exception of Create and Concurrent append. All APIs are thread safe with some exceptions in CreateFile and GetAppendStream. CreateFile and GetAppendStream cannot be called for the same path from different threads because writing is done with a lease so there will lease conflicts If an application wants to perform multi-threaded operations using this SDK it is highly recomended to set ServicePointManager.DefaultConnectionLimit to the number of threads application wants the sdk to use before creating any instance of AdlsClient. By default ServicePointManager.DefaultConnectionLimit is set to 2. Adls use NLog for logging. adls.dotnet.* is the name of the logger to obtain all logs.

public class AdlsClient

type AdlsClient = class

Public Class AdlsClient

Inheritance: Object
AdlsClient

Derived: Microsoft.Azure.DataLake.Store.MockAdlsFileSystem.MockAdlsClient

Constructors

AdlsClient()

Protected constructor for moq tests

Fields

ClientLogger

Logger to log information (debug/error/trace) regarding client

Properties

AccountFQDN	Azure data lake store account name including full domain name
ClientId	Client object ID
DefaultNumThreads	Default number of threads used by tools like uploader/downloader, recursive acl change and other multi-threaded tools using the SDK. Can be used to set ServicePointManager.DefaultConnectionLimit if you want the SDK to decide number of threads it uses for multi-threaded tools.

Methods

AddUserAgentSuffix(String)	Adds the user agent suffix
BulkDownload(String, String, Int32, IfExists, Boolean, IProgress<TransferStatus>, Boolean, Boolean, CancellationToken)	Download directory or file from remote server to local. Transfers the contents under source directory under the destination directory. Transfers the source file and saves it as the destination path. This method does not throw any exception for any entry's transfer failure. Refer the return value TransferStatus to get the status/exception of each entry's transfer. By default logs the transfer progress in system's temp path, so that user can recover using `resume` if upload has crashed. This progress logging can be disabled using `disableTransferLogging`. It is highly recomended to set ServicePointManager.DefaultConnectionLimit to the number of threads application wants the sdk to use before creating any instance of AdlsClient. By default ServicePointManager.DefaultConnectionLimit is set to 2.
BulkDownload(String, String, Int32, IfExists, IProgress<TransferStatus>, Boolean, Boolean, CancellationToken)	Download directory or file from remote server to local. Transfers the contents under source directory under the destination directory. Transfers the source file and saves it as the destination path. This method does not throw any exception for any entry's transfer failure. Refer the return value TransferStatus to get the status/exception of each entry's transfer. It is highly recomended to set ServicePointManager.DefaultConnectionLimit to the number of threads application wants the sdk to use before creating any instance of AdlsClient. By default ServicePointManager.DefaultConnectionLimit is set to 2.
BulkUpload(String, String, Int32, IfExists, Boolean, IProgress<TransferStatus>, Boolean, Boolean, Boolean, CancellationToken)	Upload directory or file from local to remote. Transfers the contents under source directory under the destination directory. Transfers the source file and saves it as the destination path. This method does not throw any exception for any entry's transfer failure. Refer the return value TransferStatus to get the status/exception of each entry's transfer. By default logs the transfer progress in system's temp path, so that user can recover using `resume` if upload has crashed. This progress logging can be disabled using `disableTransferLogging`. It is highly recomended to set ServicePointManager.DefaultConnectionLimit to the number of threads application wants the sdk to use before creating any instance of AdlsClient. By default ServicePointManager.DefaultConnectionLimit is set to 2. By default files are uploaded at new line boundaries. However if files does not have newline within 4MB chunks, the transfer will fail. In that case it is required to pass true to `isBinary` to avoid uploads at newline boundaries.
BulkUpload(String, String, Int32, IfExists, IProgress<TransferStatus>, Boolean, Boolean, Boolean, CancellationToken)	Upload directory or file from local to remote. Transfers the contents under source directory under the destination directory. Transfers the source file and saves it as the destination path. This method does not throw any exception for any entry's transfer failure. Refer the return value TransferStatus to get the status/exception of each entry's transfer. It is highly recomended to set ServicePointManager.DefaultConnectionLimit to the number of threads application wants the sdk to use before creating any instance of AdlsClient. By default ServicePointManager.DefaultConnectionLimit is set to 2. By default files are uploaded at new line boundaries. However if files does not have newline within 4MB chunks, the transfer will fail. In that case it is required to pass true to `isBinary` to avoid uploads at newline boundaries.
ChangeAcl(String, List<AclEntry>, RequestedAclType, Int32, IProgress<AclProcessorStats>, CancellationToken)	Change Acl (Modify, set and remove) recursively on a directory tree It is highly recomended to set ServicePointManager.DefaultConnectionLimit to the number of threads application wants the sdk to use before creating any instance of AdlsClient. By default ServicePointManager.DefaultConnectionLimit is set to 2.
ChangeAcl(String, List<AclEntry>, RequestedAclType, Int32)	Change Acl (Modify, set and remove) recursively on a directory tree It is highly recomended to set ServicePointManager.DefaultConnectionLimit to the number of threads application wants the sdk to use before creating any instance of AdlsClient. By default ServicePointManager.DefaultConnectionLimit is set to 2.
CheckAccess(String, String, CancellationToken)	Checks if the user/group has specified access of the given path
CheckAccessAsync(String, String, CancellationToken)	Asynchronously checks if the user/group has specified access of the given path
CheckExists(String)	Checks whether file or directory exists
ConcatenateFiles(String, List<String>, Boolean, CancellationToken)	Synchronous API to concatenate source files to a destination file
ConcatenateFilesAsync(String, List<String>, Boolean, CancellationToken)	Asynchronous API to concatenate source files to a destination file
ConcurrentAppend(String, Boolean, Byte[], Int32, Int32)	Synchronous API to perform concurrent append at server. The offset at which append will occur is determined by server. It is highly recomended to call this api with data size less than equals 4MB. Backend gurantees 4MB atomic appends.
ConcurrentAppendAsync(String, Boolean, Byte[], Int32, Int32, CancellationToken)	Asynchronous API to perform concurrent append at server. The offset at which append will occur is determined by server. Asynchronous operation. It is highly recomended to call this api with data size less than equals 4MB. Backend gurantees 4MB atomic appends.
CreateClient(String, ServiceClientCredentials, DelegatingHandler[])	Factory method that creates an instance of AdlsClient using ServiceClientCredential. If an application wants to perform multi-threaded operations using this SDK it is highly recomended to set ServicePointManager.DefaultConnectionLimit to the number of threads application wants the sdk to use before creating any instance of AdlsClient. By default ServicePointManager.DefaultConnectionLimit is set to 2.
CreateClient(String, ServiceClientCredentials, Int32, DelegatingHandler[])	Factory method that creates an instance of AdlsClient using ServiceClientCredential. If an application wants to perform multi-threaded operations using this SDK it is highly recomended to set ServicePointManager.DefaultConnectionLimit to the number of threads application wants the sdk to use before creating any instance of AdlsClient. By default ServicePointManager.DefaultConnectionLimit is set to 2.
CreateClient(String, ServiceClientCredentials, Int32)	Factory method that creates an instance of AdlsClient using ServiceClientCredential. If an application wants to perform multi-threaded operations using this SDK it is highly recomended to set ServicePointManager.DefaultConnectionLimit to the number of threads application wants the sdk to use before creating any instance of AdlsClient. By default ServicePointManager.DefaultConnectionLimit is set to 2.
CreateClient(String, ServiceClientCredentials)	Factory method that creates an instance of AdlsClient using ServiceClientCredential. If an application wants to perform multi-threaded operations using this SDK it is highly recomended to set ServicePointManager.DefaultConnectionLimit to the number of threads application wants the sdk to use before creating any instance of AdlsClient. By default ServicePointManager.DefaultConnectionLimit is set to 2.
CreateClient(String, String, Int32)	Factory method that creates an instance AdlsClient using the token key. If an application wants to perform multi-threaded operations using this SDK it is highly recomended to set ServicePointManager.DefaultConnectionLimit to the number of threads application wants the sdk to use before creating any instance of AdlsClient. By default ServicePointManager.DefaultConnectionLimit is set to 2.
CreateClient(String, String)	Factory method that creates an instance AdlsClient using the token key. If an application wants to perform multi-threaded operations using this SDK it is highly recomended to set ServicePointManager.DefaultConnectionLimit to the number of threads application wants the sdk to use before creating any instance of AdlsClient. By default ServicePointManager.DefaultConnectionLimit is set to 2.
CreateDelegatingHandler(HttpClientHandler, DelegatingHandler[])	Based on Microsoft.Rest.ServiceCLient immplementation. USed for mock framework's delegation handler
CreateDirectory(String, String, CancellationToken)	Synchronous API to create a directory
CreateDirectoryAsync(String, String, CancellationToken)	Asynchronous API to create a directory.
CreateFile(String, IfExists, String, Boolean)	Synchronous API that creates a file and returns the stream to write data to that file in ADLS. The file is opened with exclusive access - any attempt to open the same file for append will fail while this stream is open. Threading: The returned stream is not thread-safe.
CreateFileAsync(String, IfExists, String, Boolean, CancellationToken)	Asynchronous API that creates a file and returns the stream to write data to that file in ADLS. The file is opened with exclusive access - any attempt to open the same file for append will fail while this stream is open. Threading: The returned stream is not thread-safe.
Delete(String, CancellationToken)	Synchronous api to delete a file or directory. For directory it will only delete if it is empty.
DeleteAsync(String, CancellationToken)	Asynchronous api to delete a file or directory. For directory it will only delete if it is empty.
DeleteRecursive(String, CancellationToken)	Synchronous api to delete a file or directory recursively. If it is a non-empty directory then it deletes the sub-directories or files.
DeleteRecursiveAsync(String, CancellationToken)	Asynchronous api to delete a file or directory recursively
EnumerateDeletedItems(String, String, Int32, IProgress<EnumerateDeletedItemsProgress>, CancellationToken)	Search trash under a account with hint and a starting point. This is a long running operation, and user is updated with progress periodically. Caution: Undeleting files is a best effort operation. There are no guarantees that a file can be restored once it is deleted. The use of this API is enabled via whitelisting. If your ADL account is not whitelisted, then using this api will throw Not immplemented exception. For further information and assistance please contact Microsoft support.
EnumerateDeletedItemsAsync(String, String, Int32, IProgress<EnumerateDeletedItemsProgress>, CancellationToken)	Asynchronously gets the trash entries Caution: Undeleting files is a best effort operation. There are no guarantees that a file can be restored once it is deleted. The use of this API is enabled via whitelisting. If your ADL account is not whitelisted, then using this api will throw Not immplemented exception. For further information and assistance please contact Microsoft support.
EnumerateDirectory(String, UserGroupRepresentation, CancellationToken)	Returns a enumerable that enumerates the sub-directories or files contained in a directory. By default listAfter and listBefore is empty and we enumerate all the directory entries.
GetAclStatus(String, UserGroupRepresentation, CancellationToken)	Gets the ACL entry list, owner ID, group ID, octal permission and sticky bit (only for a directory) of the file/directory
GetAclStatusAsync(String, UserGroupRepresentation, CancellationToken)	Asynchronously gets the ACL entry list, owner ID, group ID, octal permission and sticky bit (only for a directory) of the file/directory
GetAppendStream(String, CancellationToken)	Synchronous API that returns the stream to write data to a file in ADLS. The file is opened with exclusive access - any attempt to open the same file for append will fail while this stream is open. Threading: The returned stream is not thread-safe.
GetAppendStreamAsync(String, CancellationToken)	Asynchronous API that returns the stream to write data to a file in ADLS. The file is opened with exclusive access - any attempt to open the same file for append will fail while this stream is open. Threading: The returned stream is not thread-safe.
GetContentSummary(String, Int32, CancellationToken)	Gets content summary of a file or directory. It is highly recomended to set ServicePointManager.DefaultConnectionLimit to the number of threads application wants the sdk to use before creating any instance of AdlsClient. By default ServicePointManager.DefaultConnectionLimit is set to 2.
GetDirectoryEntry(String, UserGroupRepresentation, CancellationToken)	Synchronously gets meta data like full path, type (file or directory), group, user, permission, length,last Access Time,last Modified Time, expiry time, acl Bit, replication Factor
GetDirectoryEntryAsync(String, UserGroupRepresentation, CancellationToken)	Asynchronously gets meta data like full path, type (file or directory), group, user, permission, length,last Access Time,last Modified Time, expiry time, acl Bit, replication Factor
GetExceptionFromResponse(OperationResponse, String)	Returns a ADLS Exception based on response from the server
GetFileProperties(String, Boolean, String, Boolean, Boolean, Int32, Boolean, Boolean, Int64, CancellationToken)	Recursively dumps file property of alldirectories or/and files under the given path to a local or adl file. File property can be disk usage or Acl or both. It is highly recomended to set ServicePointManager.DefaultConnectionLimit to the number of threads application wants the sdk to use before creating any instance of AdlsClient. By default ServicePointManager.DefaultConnectionLimit is set to 2.
GetHttpClientForCert()	This method needs to be overriden by inheriting class to provide a ideal solution of reusing httclients. Inheriting class has to dispose of the previous httpclient and then get a new httpclient
GetReadStream(String, CancellationToken)	Synchronous API that returns the stream to read data from file in ADLS
GetReadStream(String, Int32, CancellationToken)	Synchronous API that returns the stream to read data from file in ADLS
GetReadStreamAsync(String, CancellationToken)	Asynchronous API that returns the stream to read data from file in ADLS
GetReadStreamAsync(String, Int32, CancellationToken)	Asynchronous API that returns the stream to read data from file in ADLS
ModifyAclEntries(String, List<AclEntry>, CancellationToken)	Modifies acl entries of a file or directory with given ACL list. It merges the exisitng ACL list with given list.
ModifyAclEntriesAsync(String, List<AclEntry>, CancellationToken)	Asynchronously modifies acl entries of a file or directory with given ACL list. It merges the exisitng ACL list with given list.
RemoveAclEntries(String, List<AclEntry>, CancellationToken)	Removes specified Acl Entries for a file or directory.
RemoveAclEntriesAsync(String, List<AclEntry>, CancellationToken)	Asynchronously removes specified Acl Entries for a file or directory.
RemoveAllAcls(String, CancellationToken)	Removes all Acl Entries for a file or directory.
RemoveAllAclsAsync(String, CancellationToken)	Asynchronously removes all Acl Entries for a file or directory.
RemoveDefaultAcls(String, CancellationToken)	Removes all Acl Entries of AclScope default for a file or directory.
RemoveDefaultAclsAsync(String, CancellationToken)	Asynchronously removes all Acl Entries of AclScope default for a file or directory.
Rename(String, String, Boolean, CancellationToken)	Synchronous API to rename a file or directory. For renaming directory: If the destination exists then it puts the source directory one level under the destination.
RenameAsync(String, String, Boolean, CancellationToken)	Asynchronous API to rename a file or directory. For renaming directory: If the destination exists then it puts the source directory one level under the destination.
RestoreDeletedItems(String, String, String, String, CancellationToken)	Synchronously Restores trash entry Caution: Undeleting files is a best effort operation. There are no guarantees that a file can be restored once it is deleted. The use of this API is enabled via whitelisting. If your ADL account is not whitelisted, then using this api will throw Not immplemented exception. For further information and assistance please contact Microsoft support.
RestoreDeletedItemsAsync(String, String, String, String, CancellationToken)	Asynchronously Restores trash entry Caution: Undeleting files is a best effort operation. There are no guarantees that a file can be restored once it is deleted. The use of this API is enabled via whitelisting. If your ADL account is not whitelisted, then using this api will throw Not immplemented exception. For further information and assistance please contact Microsoft support.
SetAcl(String, List<AclEntry>, CancellationToken)	Sets Acl Entries for a file or directory. It wipes out the existing Acl entries for the path.
SetAclAsync(String, List<AclEntry>, CancellationToken)	Asynchronously sets Acl Entries for a file or directory. It wipes out the existing Acl entries for the path.
SetConditionalCreateWithOverwrite(Boolean)	Sets whether to perform conditional create with overwrite. Helps resolving atomic issues with retries
SetExpiryTime(String, ExpiryOption, Int64, CancellationToken)	Synchronously sets the expiry time
SetExpiryTimeAsync(String, ExpiryOption, Int64, CancellationToken)	Asynchronously sets the expiry time
SetOwner(String, String, String, CancellationToken)	Sets the owner or/and group of the path
SetOwnerAsync(String, String, String, CancellationToken)	Asynchronously sets the owner or/and group of the path
SetPermission(String, String, CancellationToken)	Sets the permission of the specified path
SetPermissionAsync(String, String, CancellationToken)	Asynchronously sets the permission of the specified path
SetPerRequestTimeout(TimeSpan)	Sets the per request timeout. Highly recommended to set it after creating client. Not recommended to set it while requests are underway from a different thread.
SetToken(String)	Sets the auth token.

Applies to

Share via