Common Data Model metadata: Introducing manifest

In a data lake, a Common Data Model folder is a collection, spread over sub-folders or accounts, of the data files and schema description files that constitute a set of related entities that have been organized together for some purpose, such as to back an application or perform analysis. This collection of related data is sometimes called a solution. A Common Data Model manifest object and the document that contains one (*.manifest.cdm.json) is an organizing document that acts as an entry point directory that points to the items in the Common Data Model folder. The manifest object describes the list of entities in the solution, giving a detailed schema description for each entity, a collection of data partition files for each entity, a list of the known relationships between entities, and potentially other sub-manifest objects that are nested solutions.

Those familiar with the model.json file and the format that describes a Common Data Model folder will recognize the manifest as a similar but expanded concept. In fact, the Common Data Model object model offers backward compatibility with the model.json format.

You can also use a manifest to organize purely abstract entity definitions as a way of sharing standard schemas. In this form (a Common Data Model domain), the manifest lists the entities with schema documents, known relationships, and sub-manifests, but doesn't give the data partition file locations for the entities.

A folder may contain multiple manifest documents, each presenting an alternate view or organization of the entities. You can use these views to hide, simplify, or focus the entities in a solution for different personas, or to show different business perspectives. By convention, if there are multiple manifest documents in a folder, the document named default.manifest.cdm.json should be used when there is no specific manifest name.

A manifest document has this structure:

• Version information

• Imports

• Manifest definition

  • Entity list

    • Local entity declaration

      • Data partition list

      • Data partition pattern list

    • Referenced entity declaration

  • Relationship list

  • Sub-manifest list

Imports

Use imports to specify the schema documents that need to be imported first.

Because a manifest uses Common Data Model traits to provide the details about data partitions, attributes, and so on, most of the time, a single import of cdm:/foundations.cdm.json is sufficient to get the standard definitions for these traits.

Shared concepts inside the manifest

The manifest object and the Common Data Model objects it contains, share some properties and methods.

Entities

The list of entities in a manifest represents a joint context or purpose, such as the entities used for an application or a data analysis project. Some of these entities are owned by the manifest—that is, the manifest knows the location of the schema definition files and all of the data partition files. The owner of the manifest document is responsible for managing that set of data. It's also possible for a manifest to simply borrow an entity that a different manifest owns. The application that owns the main manifest can write and maintain the data for the entities it owns but it can only act as a reader for the borrowed entities. An entity listing can be one of two types: a local entity declaration or a referenced entity declaration.

Local entity declaration

For an entity that the manifest owns, the entity declaration describes the location of the schema document where details about the entity metadata, the set of data partition files that contains the data records for the entity, and the file location hints that can be used to discover new data partitions, can all be found. The declaration also has the shared features described earlier in the shared concepts inside the manifest section.

Data partition

A data partition object describes and points to an entity's data file. Common Data Model traits on this object give detailed information about the file format and the options for parsing them. By default, the data should be interpreted using the schema description that the associated local entity declaration indicates. However, as schemas can change over time and older files may have different content than newer files, an override of the default entity schema can be used for a specific partition file.

Data partitions may be generated at runtime by the object model using a data partition pattern. A property indicates if the partition description was generated in-memory or if the description was explicitly created and stored with the folder.

Data partition specific traits

A data partition's trait collection is used to hold data partition specific settings and behaviors.

Data partition patterns

An entity declaration can specify one or more data partition pattern objects, one for each search pattern that can be used to discover and describe partition files. The use of a pattern (using the Common Data Model object model) may result in dynamically discovered data partition objects in the collection.

Here are a few examples that show the interplay of RootLocation, RegularExpression, and Parameters.

All .csv files in a folder

Make a partition for each ‘Customer’ CSV data file that's located in the Customer/dataFiles/folder (relative to the folder in which the manifest document exists). Assume there are no sub-directories.

Customer/dataFiles/august.csv

Customer/dataFiles/last-year.csv

Customer/dataFiles/last-year.csv.backup (not matched)

All .csv files in structured folders with information in the file name

Extract the year, month, and cohort number from the file paths and make them arguments.

dataFiles/2016/June/cohort001.csv

dataFiles/2016/June/cohort002.csv

dataFiles/2017/April/cohort001.csv

dataFiles/2017/May/cohort001.csv

All files in any sub-folder

Extract the file extension.

dataFiles/2016/June/cohort001.csv

dataFiles/2016/June/backups/cohort002.csv

dataFiles/2017/April/cohort001.txt

dataFiles/2017/May/cohort001.csv.save

Order of processing and conflicts

Since one entity can have many partition patterns that use different root locations and different regular expressions, it's possible for the multiple partition patterns to find the same files. It's also possible for a "found" partition to conflict with one of the non-inferred (explicitly listed) partitions for the entity. When a conflict happens, the following rules determine which argument values and specialized schema apply to a new inferred partition:

1. Inferred partitions never overwrite existing, non-inferred partitions on the same entity.

2. The last pattern processed takes precedence.

Referenced entity declaration

The reference to an entity owned by a different manifest is simply a name and a location.

Sub-manifests

When a set of related manifests form a logical hierarchy, the hierarchy can be described using sub-manifests. A manifest object can indicate a list of sub-manifests. A sub-manifest may be an independent solution but often the entities in a sub-manifest are "aware of", "extensions of", or "built upon" the entities in the manifest(s) that are the "parent" of the sub-manifest.

A manifest has a collection of manifest declarations called SubManifests.

Entity relationships

A manifest may contain a list of the known entity-to-entity relationships. When you store relational data in a lake, you store the different entities in separate sets of data partitions. these relationships are expressed in the data through foreign key references in one entity’s partitions. These foreign key references align with the primary key values from another entity.

These entity-to-entity relationship descriptions can be extracted from an external metadata repository and stored in the manifest through automation, or simply listed in the document through other means. Alternatively, the Common Data Model object model offers functionality to analyze the logical metadata of the entities in the corpus and populate the discovered relationships for a manifest.

Currently, only single-attribute relationships are supported. All relationships are modeled from the "many side" to the "one side", that is, from the referencing entity to the one referenced record in another entity. A “many-to-many” relationship is modeled from the intersection entity, which correlates two “many-to-one” relationships.

File status check and modified times

The manifest object and the objects it contains, collect and report information about the modification times for the files being referenced. The following properties are shared:

The meanings of these times and the scope of the FileStatusCheckAsync() method depend upon the specific object.

Example manifest document

This sample manifest document demonstrates each of the above-mentioned capabilities for the manifest object:

{
    "manifestName": "MySolution",
    "jsonSchemaSemanticVersion": "0.9.0",
    "imports": [{
        "corpusPath": "cdm:/foundations.cdm.json"
    }],
    "lastFileStatusCheckTime": "2019-10-01T02:09:08Z",
    "lastFileModifiedTime": "2019-10-01T02:09:08Z",
    "lastChildFileModifiedTime": "2019-10-01T02:09:08Z",
    "entities": [{
        "entityName": "Person",
        "entitySchema": "PersonData/Person.cdm.json/Person",
        "dataPartitions": [{
            "location": "PersonData/all-people.csv",
            "exhibitsTraits": [{
                "traitReference": "is.partition.format.CSV",
                "arguments": [{
                    "name": "columnHeaders",
                    "value": "true"
                }, {
                    "name": "delimiter",
                    "value": ","
                }]
            }]
        }]
    }, {
        "entityName": "EmailMessage",
        "entityPath": "EmailMessageData/EmailMessage.cdm.json/EmailMessage",
        "dataPartitions": [{
            "location": "EmailMessageData/Sep/emails.csv",
            "exhibitsTraits": [],
            "arguments": [{
                "month": [
                    "Sep"
                ]
            }],
            "lastFileStatusCheckTime": "2019-10-01T02:09:08Z",
            "lastFileModifiedTime": "2019-09-01T02:19:06Z"
        }, {
            "location": "EmailMessageData/Oct/emails.csv",
            "exhibitsTraits": [],
            "arguments": [{
                "month": [
                    "Oct"
                ]
            }],
            "lastFileStatusCheckTime": "2019-10-01T02:09:08Z",
            "lastFileModifiedTime": "2019-10-01T02:09:08Z"
        }],
        "dataPartitionPatterns": [{
            "name": "EmailByMonth",
            "rootLocation": "EmailMessageData/",
            // This pattern is what describes the partitions found above 
            "regularExpression": "(\\\\w{3})/emails.csv)",
            "parameters": [
                "month"
            ]
        }],
        "lastFileStatusCheckTime": "2019-10-01T02:09:08Z",
        "lastFileModifiedTime": "2019-09-27T02:09:08.149Z",
        "lastChildFileModifiedTime": "2019-09-27T02:09:08.149Z"
    }, {
        // This is a referenced entity declaration. 
        // We are just pointing at another manifest and entity.
        "entityName": "FrequentPairs",
        "explanation": "Borrow the analysis of frequent email pairs from that project",
        "entityPath": "networkProject/NetworkAnalysis.manifest.cdm.json/FrequentPairs"
    }],
    "relationships": [
        // A relationship between the entities.
        {
            "fromEntity": "EmailMessageData/EmailMessage.cdm.json/EmailMessage",
            "fromEntityAttribute": "ownerId",
            "toEntity": "PersonData/Person.cdm.json/Person",
            "toEntityAttribute": "personId"
        }
    ],
    "submanifests": [
        // This sub-manifest is independent but described here so we know it's related
        {
            "name": "NetworkAnalysis",
            "definition": "networkProject/NetworkAnalysis.manifest.cdm.json"
        }
    ]
}