Data sources overview
To configure Dynamics 365 Customer Insights - Data, first bring in source data for processing. Customer Insights - Data provides several types of data connectors to connect to and ingest data from a broad set of sources.
This article describes options that can affect the performance of data ingestion.
Delta format
The Delta data format is the native processing format that Customer Insights – Data uses. If you can provide source data in the Delta format, there are many advantages.
- Better efficiency with Customer Insights - Data: Customer Insights - Data uses Delta format internally.
- Faster data ingestion: Delta format provides superior compression for faster data transfers.
- Faster unification: When the Delta time travel feature is used, Customer Insights – Data unifies only data that changed. It doesn't reprocess the whole set of data. For incremental unification, all data inputs to unification must be in the Delta format.
- Fewer data corruption issues: Delta format reduces the corrupted partitions and common data corruption issues that are caused by older formats, such as comma-separated value (CSV) format.
- More reliable data management design: Delta format doesn't require manual updates to manifests, schemas, and partition files.
- Higher data validity: Delta format provides atomicity, consistency, isolation, durability (ACID) transactions, and isolation levels in Spark.
You can use Delta format with the following connectors:
Delta is a term introduced with Delta Lake, the foundation for storing data and tables in the Databricks Lakehouse Platform. Delta Lake is an open-source storage layer that brings ACID (atomicity, consistency, isolation, and durability) transactions to big data workloads. For more information, see the Delta Lake Documentation Page.
Data source attachment or import
When you're deciding how to ingest your data, a key consideration is whether the data connector attaches to the data or makes copies of it. When you use Customer Insights - Data, we recommend that you use a connector that attaches to the data. In this way, the data is directly accessed when it's time to process it. If you use a connector that copies the data, delays can occur when the data is updated.
The following data connectors attach to your data:
- Azure Data Lake Storage Delta tables
- Azure Data Lake Storage Common Data Model tables
- Microsoft Dataverse
- Azure Synapse Analytics (preview)
If you can't use a connector that attaches to your data, copy the data by using one of the Power Query connectors. Power Query provides a useful way to transform the data.
Data profiling
When data is ingested, Customer Insights – Data performs some basic data profiling. For example, it profiles the frequency of repeated values in a column. You can use profile data to understand your data and address issues. For example, if you're matching on the FullName column, data profiling can help you detect that the default value, Enter your name, appears in 10,000 rows. Therefore, if you match on this value, 10,000 rows that should not be matched are matched. When you add your Azure data lake, Delta tables, or Azure Synapse data sources, you can enable data profiling for more columns.
After you ingest the data, you can view the results of data profiling.
Go to Data > Tables, and select a table. Then, in the row for a field (for example,
DateOfBirth), in the Summary column, select the Summary icon.
Review the details for any errors or missing values.
Data sources page
The Data sources page lists the data sources in two sections:
- Managed by me: Microsoft Power Platform dataflows that you created and manage. Other users can only view these dataflows. They can't edit, refresh, or delete them.
- Managed by others: Microsoft Power Platform dataflows that other admins created. You can only view these dataflows. This section shows the owner of each dataflow, so that you can contact them if you need any assistance.
Note
Other users can view and use all tables. Although each data source is owned by the user who created it, every user of Customer Insights - Data can use all the tables that result from data ingestion.
