Kusto.Ingest ingestion status reporting

Applies to: ✅ Microsoft Fabric ✅ Azure Data Explorer

This article explains how to use IKustoQueuedIngestClient features to track the status of an ingestion request.

Description classes

These description classes contain important details about the source data to be ingested, and should be used in the ingestion operation.

• SourceDescription
• DataReaderDescription
• StreamDescription
• FileDescription
• BlobDescription

The classes are all derived from the abstract class SourceDescription, and they're used to instantiate a unique identifier for each data source. Each identifier will then be used for status tracking and will show up in all reports, traces, and exceptions related to the relevant operation.

Class SourceDescription

Large datasets will be split into 1GB chunks, and each part will be ingested separately. The same SourceId will then apply to all ingest operations originated from the same dataset.

public abstract class SourceDescription
{
    public Guid? SourceId { get; set; }
}

Class DataReaderDescription

public class DataReaderDescription : SourceDescription
{
    public  IDataReader DataReader { get; set; }
}

Class StreamDescription

public class StreamDescription : SourceDescription
{
    public Stream Stream { get; set; }
}

Class FileDescription

public class FileDescription : SourceDescription
{
    public string FilePath { get; set; }
}

Class BlobDescription

public class BlobDescription : SourceDescription
{
    public string BlobUri { get; set; }
    // Setting the Blob size here is important, as this saves the client the trouble of probing the blob for size
    public long? BlobSize { get; set; }
}

Ingestion result representation

Interface IKustoIngestionResult

This interface captures the result of a single queued ingestion operation and can be retrieved by SourceId.

public interface IKustoIngestionResult
{
    // Retrieves the detailed ingestion status of the ingestion source with the given sourceId.
    IngestionStatus GetIngestionStatusBySourceId(Guid sourceId);

    // Retrieves the detailed ingestion status of all data ingestion operations into Kusto associated with this IKustoIngestionResult instance.
    IEnumerable<IngestionStatus> GetIngestionStatusCollection();
}

Class IngestionStatus

IngestionStatus contains a complete status of a single ingestion operation.

public class IngestionStatus
{
    // The ingestion status returns from the service. Status remains 'Pending' during the ingestion process and
    // is updated by the service once the ingestion completes. When <see cref="IngestionReportMethod"/> is set to 'Queue' the ingestion status
    // will always be 'Queued' and the caller needs to query the report queues for ingestion status, as configured. To query statuses that were
    // reported to queue, see: <see href="https://zcusa.951200.xyz/azure/kusto/api/netfx/kusto-ingest-client-status#ingestion-status-in-azure-queue"/>.
    // When <see cref="IngestionReportMethod"/> is set to 'Table', call <see cref="IKustoIngestionResult.GetIngestionStatusBySourceId"/> or
    // <see cref="IKustoIngestionResult.GetIngestionStatusCollection"/> to retrieve the most recent ingestion status.
    public Status Status { get; set; }
    // A unique identifier representing the ingested source. Can be supplied during the ingestion execution.
    public Guid IngestionSourceId { get; set; }
    // The URI of the blob, potentially including the secret needed to access the blob.
    // This can be a filesystem URI (on-premises deployments only), or an Azure Blob Storage URI (including a SAS key or a semicolon followed by the account key)
    public string IngestionSourcePath { get; set; }
    // The name of the database holding the target table.
    public string Database { get; set; }
    // The name of the target table into which the data will be ingested.
    public string Table { get; set; }
    // The last updated time of the ingestion status.
    public DateTime UpdatedOn { get; set; }
    // The ingestion's operation Id.
    public Guid OperationId { get; set; }
    // The ingestion operation activity Id.
    public Guid ActivityId { get; set; }
    // In case of a failure - indicates the failure's error code.
    public IngestionErrorCode ErrorCode { get; set; }
    // In case of a failure - indicates the failure's status.
    public FailureStatus FailureStatus { get; set; }
    // In case of a failure - indicates the failure's details.
    public string Details { get; set; }
    // In case of a failure - indicates whether or not the failures originate from an Update Policy.
    public bool OriginatesFromUpdatePolicy { get; set; }
}

Status enumeration

Tracking Ingestion Status (KustoQueuedIngestClient)

IKustoQueuedIngestClient is a 'fire-and-forget' client. The ingestion operation on the client side ends by posting a message to an Azure queue. After the posting, the client job is done. For the client user's convenience, KustoQueuedIngestClient provides a mechanism for tracking the individual ingestion status. This mechanism isn't intended for mass usage on high-throughput ingestion pipelines. This mechanism is for precision ingestion when the rate is relatively low and the tracking requirements are strict.

The following properties (set on KustoQueuedIngestionProperties) control the level and transport for ingestion success or failure notifications.

IngestionReportLevel enumeration

public enum IngestionReportLevel
{
    FailuresOnly = 0,
    None,
    FailuresAndSuccesses
}

IngestionReportMethod enumeration

public enum IngestionReportMethod
{
    Queue = 0,
    Table,
    QueueAndTable
}

To track the status of your ingestion, provide the following to the IKustoQueuedIngestClient that you do the ingest operation with:

1. Set IngestionReportLevelproperty to the required level of report. Either FailuresOnly (which is the default value) or FailuresAndSuccesses. When set to None, nothing will be reported at the end of the ingestion.
2. Specify the IngestionReportMethod - Queue, Table, or QueueAndTable.

A usage example can be found on the Kusto.Ingest Examples page.

Ingestion status in the Azure table

The IKustoIngestionResult interface that is returned from each ingest operation contains functions that can be used to query the status of the ingestion. Pay special attention to the Status property of the returned IngestionStatus objects:

• Pending indicates that the source has been queued for ingestion and is yet to be updated. Use the function again to query the status of the source
• Succeeded indicates that the source has been ingested successfully
• Failed indicates that the source failed to be ingested

Ingestion status in Azure queue

The IKustoIngestionResult methods are only relevant for checking a status in an Azure table. To query statuses that were reported to an Azure queue, use the following IKustoQueuedIngestClient methods.

Ingestion failures retrieved from the Azure queue

The ingestion failures are represented by the IngestionFailure object that contains useful information about the failure.