Troubleshoot Manufacturing data solutions (preview)

This troubleshooting guide is designed to help you identify and resolve common issues that can arise during and after the deployment of Manufacturing data solutions service.

Debug failed deployment

You can check if your deployment succeeded or not by looking at status of the resource. If the status of your Manufacturing data solutions resource is Failed, go to Managed Resource group for that resource using the following naming convention MDS-{your-deployment-name}-MRG-{UniqueID}. You can also refer to Resource JSON details to fetch the details of Managed Resource group.

Go to the specified resource group, select Deployments tab from the left-hand menu, and review any deployment errors that have occurred.

If you receive a BadRequest error code, it indicates that the deployment values you sent don't match the expected values by Resource Manager. Check the inner message for more details.

Deployment can fail due to various reasons as listed in the following table:

Debug custom entity CRUD

To check status of create Entity API, perform the following steps:

1. Call the POST API. The response includes an Operation-Location header containing a URL. For example, the code is trying to register device entity.

    curl -X POST https://{{serviceUrl}}/mds/service/entities \
    -H "Authorization: {BEARER_TOKEN}" \
    -H "Content-Type: application/json" \
    -d '{
        "name": "Device",
        "columns": [
            {
                "name": "id",
                "description": "A unique id",
                "type": "String",
                "mandatory": true,
                "semanticRelevantFlag": true,
                "isProperNoun": false,
                "groupBy": true,
                "primaryKey": true
            },
            {
                "name": "description",
                "description": "Additional information about the equipment",
                "type": "String",
                "mandatory": false,
                "semanticRelevantFlag": true,
                "isProperNoun": false,
                "groupBy": false,
                "primaryKey": false
            },
            {
                "name": "hierarchyScope",
                "description": "Identifies where the exchanged information",
                "type": "String",
                "mandatory": false,
                "semanticRelevantFlag": true,
                "isProperNoun": false,
                "groupBy": false,
                "primaryKey": false
            },
            {
                "name": "equipmentLevel",
                "description": "An identification of the level in the role-based equipment hierarchy.",
                "type": "Enum",
                "mandatory": false,
                "semanticRelevantFlag": true,
                "isProperNoun": false,
                "groupBy": false,
                "primaryKey": false,
                "enumValues": [
                    "enterprise",
                    "site",
                    "area",
                    "workCenter",
                    "workUnit",
                    "processCell",
                    "unit",
                    "productionLine",
                    "productionUnit",
                    "workCell",
                    "storageZone",
                    "Storage Unit"
                ]
            },
            {
                "name": "operationalLocation",
                "description": "Identifies the operational location of the equipment.",
                "type": "String",
                "mandatory": false,
                "semanticRelevantFlag": true,
                "isProperNoun": false,
                "groupBy": false,
                "primaryKey": false
            },
            {
                "name": "operationalLocationType",
                "description": "Indicates whether the operational",
                "type": "Enum",
                "mandatory": false,
                "semanticRelevantFlag": true,
                "isProperNoun": false,
                "groupBy": false,
                "primaryKey": false,
                "enumValues": [
                    "Description",
                    "Operational Location"
                ]
            },
            {
                "name": "assetsystemrefid",
                "description": "Asset/ERP System of Record Identifier for Equipment (AssetID)",
                "type": "Alphanumeric",
                "mandatory": false,
                "semanticRelevantFlag": false,
                "isProperNoun": false,
                "isProperNoun": false,
                "groupBy": false,
                "primaryKey": false
            },
            {
                "name": "messystemrefid",
                "description": "Manufacturing system Ref ID",
                "type": "Alphanumeric",
                "mandatory": false,
                "semanticRelevantFlag": false,
                "isProperNoun": false,
                "groupBy": false,
                "primaryKey": false
            },
            {
                "name": "temperature",
                "description": "temperature of device",
                "type": "Double",
                "mandatory": false,
                "semanticRelevantFlag": true,
                "isProperNoun": false,
                "groupBy": false,
                "primaryKey": false
            },
            {
                "name": "pressure",
                "description": "pressure of device",
                "type": "Double",
                "mandatory": false,
                "semanticRelevantFlag": true,
                "isProperNoun": false,
                "groupBy": false,
                "primaryKey": false
            }
        ],
        "tags": {
            "ingestionFormat": "Batch",
            "ingestionRate": "Hourly",
            "storage": "Hot"
        },
        "dtdlSchemaUrl": "https%3A%2F%2Fl2storage.blob.core.windows.net%2Fcustomentity%2FDevice.json",
        "semanticRelevantFlag": true
    }'
   

   It populates header's Operation-Location with the URL to check status of API.

2. Decode the URL provided in the Operation-Location header in the previous call to check the status via GET request.

   curl -X GET "https://{serviceUrl}/mds/service/entities/status/RegisterEntity/efd66e795001492fb67def8a94336384" \
   -H "Authorization: Bearer {BEARER_TOKEN}"
   

   Check

   Action

   API Response

   Ensure you receive a 202 response from the API and check the status using the Operation-Location header. Perform a GET request on the URL provided in the header and verify that the status in the response payload indicates success. "createdDateTime": "2024-05-27T03:22:50.1367676Z", "resourceLocationUrl": "https://{serviceUrl}/mds/service/entities/Device", "status": "Succeeded" Path: https://{serviceUrl}/mds/service/entities

To debug the error, check the response generated by the API. For more instructions, follow Entity registration Updation.

Troubleshoot via logs

1. Go to Application Insights Resource in the Managed Resource Group.

   

2. Select Logs under Monitoring navigation section on left hand side.

   

3. Run the KQL in Application Insights provided in Table:

   For cases where CRUD API returned failed status

   Check	Action
   API Response	Ensure you receive a 202 response from the API and check the status using the Operation-Location header. Perform a GET request on the URL provided in the header and if the status is Failed	Path: https://{serviceUrl}/mds/service/entities
   Application Insights Check	Use the following query to check whether it's Redis issue, run the following query in Application Insights: traces \| sort by timestamp \| where message contains "Error while creating entry in Redis". If it's a Redis issue, check the actual exception and check Redis resource is up and running.	Expected Result - Should return a single entry if an entity was registered within the specified time
   Application Insights Check	Use the following query to check whether it's Cosmos DB issue, run the following query in Application Insights: `traces | sort by timestamp | where message contains "Error occurred while updating the status in Cosmos". If it's a Cosmos DB issue, check the inner exception and verify the Cosmos DB is up and running.	Expected Result - Should return a single entry if an entity was registered within the specified time

   For cases where CRUD API returned faulted status -

   Check	Action
   API Response	Ensure you receive a 202 response from the API and check the status using the "Operation-Location" header. Perform a GET request on the URL provided in the header and if the status is Faulted Path: {serviceUrl}/mds/service/entities
   Application Insights Check	This check occurs when another job is already processing this entity traces | sort by timestamp | where message contains " Another job is already processing this entity"

Commonly encountered errors

Troubleshoot data mapping

Initial checks

For assistance in verifying that the mapping is done correctly, contact the Microsoft Support Team at mdssupport@microsoft.com. Additionally, ensure you understand the mapping by reviewing the examples provided on the mapping page.

Troubleshoot data ingestion

Initial checks

1. Match the actual Twin and relationship count to expected one to use the following API:

   curl -X GET "https://{serviceUrl}/mds/service/query/ingestionStatus?operator=BETWEEN&endDate=2024-05-20&startDate=2024-05-01" \
       -H "Authorization: Bearer {BEARER_TOKEN}"
   

   Check the following link for more detailed information:

   Ingestion Status API

2. Check that you get a healthy response from Manufacturing data solutions Health check API.

   curl -H "Authorization: Bearer {BEARER_TOKEN}" https://{{serviceUrl}}/mds/service/health
   

   Output:

   {
       "message": "Manufacturing data solutions is ready to serve the requests.",
       "setupStatus": "Succeeded",
       "setupInfo": {
           "registerEntityStatus": "Succeeded"
       },
       "errorMessage": [],
       "id": "f26dcdeb-6453-4521-b28e-610171db3997",
       "version": 1
   }
   

Debug ingestion validation errors

Batch ingestion

When files are uploaded to the lakehouse, they undergo validation checks.

For batch ingestion, the first 200 validation failures per entity are recorded. These errors are stored in a subfolder named ValidationErrors within the specified lake path provided by the customer. A CSV file is then created in this folder, where the errors are recorded following the format provided in the sample CSV file.

During reingestion, a new file is created with a new timestamp, and any validation errors encountered are recorded in this new file.

Cleanup of files in ValidationErrors isn't being performed. On reingestion, new files get created with latest timestamp. It's user's responsibility to clean up older files if not needed.

Sample CSV file

"File Name","Entity Name","File Type" , "Error Message", "Record"
"Operations Request_20240501134049.csv", "Operations Request", "CSV","DataTypeMisMatch: Cell Value(s) do(es) not adhere to the agreed upon data contract",""ulidoperationsrequest3762901per","YrbidoperationsRequest3762901", "descriptionoperationsRequest732483", "Production", "hierarchyscopeoperationsRequest577992", "2024-02-13T06:00:04.12", "2019-06-13T00:00:00","priorityoperationsRequest646820", "aborted""

Stream ingestion

When streaming data is sent to Manufacturing data solutions, errors are captured in a single file per day at the validation layer.

On performing multiple reingestions, errors are appended in the same file.

The limit on file size is set to 10 MB; if a file reaches this maximum size, no more errors are written to the file per day.

Sample file

"TimeStamp", "Entity Name", "DataFormat" , "ValidationErrorDataCategory" ,"Error Message", "Properties","Relationships"
"20240501130605", "Operations Request", "JSON", "Property", "Invalid property value as there is mismatch with its type ", "{"id":"0128341001001","description":"EXTRUDE","hierarchyScope":"BLOWNLINE06","startTime":"2023-09-24T02:43","endTime":"2023-09-24T19:23:48","requestState":"completed"}", "[]"
"20240501130623", "Equipment Property", "JSON", "Property", "Mandatory Property Cannot be null or empty", "{"id":"","description":"Address of the hierarchy scope object","value":"Contoso Furniture  HQ  Hyderabad  India","valueUnitOfMeasure":"NA","externalreferenceid":"externalreferenceid7ET2VF39EH","externalreferencedictionary":"externalreferencedictionaryVfMZahPNcL","externalreferenceuri":"https://www.bing.com/"}", "[]"
"20240501130624", "Operations Request", "JSON", "Property", "Invalid property value as there is mismatch with its type ", "{"id":"0128341001001","description":"EXTRUDE","hierarchyScope":"BLOWNLINE06","startTime":"2023-09-24T02:43","endTime":"2023-09-24T19:23:48","requestState":"completed"}", "[]"
"20240501130625", "", "JSON", "Relationship", "Provide a valid entity name in the relationship", "{}", "[{"targetEntityName":"","keys":{"id":"FHnAddress1","value":"NA","valueUnitOfMeasure":"NA"},"relationshipType":null}]"
"20240501130627", "", "JSON", "Relationship", "Provide a valid entity name in the relationship", "{}", "[{"targetEntityName":"","keys":{"id":"FHnAddress1","value":"NA","valueUnitOfMeasure":"NA"},"relationshipType":null}]"
"20240501130628", "Equipment Property", "JSON", "Property", "Mandatory Property Cannot be null or empty", "{"id":"","description":"Ownership Type of the estabslishment","value":"Private Limited Company","valueUnitOfMeasure":"NA","externalreferenceid":"externalreferenceid6U5CFU5OVT","externalreferencedictionary":"externalreferencedictionarylxzpGxVYEN","externalreferenceuri":"https://www.bing.com/"}", "[]"
"20240501130630", "", "JSON", "Relationship", "Provide a valid entity name in the relationship", "{}", "[{"targetEntityName":"","keys":{"id":"EZWProducts Handled1","value":"NA","valueUnitOfMeasure":"NA"},"relationshipType":null}]"
"20240501134049", "Operations Request", "JSON", "Property", "Invalid property value as there is mismatch with its type ", "{"id":"0128341001001","description":"EXTRUDE","hierarchyScope":"BLOWNLINE06","startTime":"2023-09-24T02:43","endTime":"2023-09-24T19:23:48","requestState":"completed"}", "[]"

OPC UA stream ingestion

When a streaming OPC UA data or metadata is sent to Manufacturing Data Solutions, errors are captured in a single file per day at the validation layer.

On performing multiple reingestions, errors are appended in the same file. The ValidationErrorDataCategory field indicates whether the issue is with metadata or data.

A limit of 10 MB is set on file size; if a file reaches this maximum size, no more errors are written to the file per day.

"TimeStamp", "DataFormat" , "ValidationErrorDataCategory" ,"Error Message", "DataMessage"
"20240513112048", "OPCUA", "OpcuaData", "[MDS_Ingestion_Response_Code : MDS_IG_OPCUA_MappingDataFieldsNotMatching] : Ingestion failed as Field key not matching with field in metadata with namespaceUri", "{"DataSetWriterId":200,"Payload":{"new key":{"Value":"Contoso Furniture  HQ  Hyderabad  Indianew"}},"Timestamp":"2023-10-10T09:27:39.2743175Z"}"
"20240508115927", "OPCUA", "OpcuaMetadata", "[MDS_Ingestion_Response_Code : MDS_IG_OPCUAMetadataPublisherIdMissing]: Validation failed as publisherId is missing", "{"MessageType":"ua-metadata","PublisherId":"","MessageId":"600","DataSetWriterId":200,"MetaData":{"Name":"urn:assembly.seattle;nsu=http://opcfoundation.org/UA/Station/;i=403","Fields":[{"Name":"Address Hyderabad","FieldFlags":0,"BuiltInType":6,"DataType":null,"ValueRank":-1,"MaxStringLength":0,"DataSetFieldId":"b6738567-05eb-4b48-8cd4-f38c9b120129"}],"ConfigurationVersion":null}}"
"20240508120004", "OPCUA", "OpcuaMetadata", "[MDS_Ingestion_Response_Code : MDS_IG_OPCUAMetadataPublisherIdMissing]: Validation failed as publisherId is missing", "{"MessageType":"ua-metadata","PublisherId":"","MessageId":"600","DataSetWriterId":200,"MetaData":{"Name":"urn:assembly.seattle;nsu=http://opcfoundation.org/UA/Station/;i=403","Fields":[{"Name":"Address Hyderabad","FieldFlags":0,"BuiltInType":6,"DataType":null,"ValueRank":-1,"MaxStringLength":0,"DataSetFieldId":"b6738567-05eb-4b48-8cd4-f38c9b120129"}],"ConfigurationVersion":null}}"
"20240508120329", "OPCUA", "OpcuaMetadata", "[MDS_Ingestion_Response_Code : MDS_IG_OPCUAMetadataPayloadDeserializationFailed] : OPCUAMetadataIngestionFailed as JsonDeserialization to MetadataMessagePayload payload failed", "{"MessageType":"ua-metadata","PublisherId":"publisher.seattle","MessageId":"602","MetaData":{"Name":"urn:assembly.seattle;nsu=http://opcfoundation.org/UA/Station/;i=405","Fields":[{"Name":"Address Peeniya","FieldFlags":0,"BuiltInType":6,"DataType":null,"ValueRank":-1,"MaxStringLength":0,"DataSetFieldId":"b6738567-05eb-4b48-8cd4-f38c9b120129"}],"ConfigurationVersion":null}}"
"20240508121318", "OPCUA", "OpcuaMetadata", "[MDS_Ingestion_Response_Code : MDS_IG_OPCUAMetadataPayloadDeserializationFailed] : OPCUAMetadataIngestionFailed as JsonDeserialization to MetadataMessagePayload payload failed", "{"MessageType":"ua-metadata","PublisherId":"publisher.seattle","MessageId":"602","MetaData":{"Name":"urn:assembly.seattle;nsu=http://opcfoundation.org/UA/Station/;i=405","Fields":[{"Name":"Address Peeniya","FieldFlags":0,"BuiltInType":6,"DataType":null,"ValueRank":-1,"MaxStringLength":0,"DataSetFieldId":"b6738567-05eb-4b48-8cd4-f38c9b120129"}],"ConfigurationVersion":null}}"

Debug other ingestion errors

Troubleshoot via logs

Make sure that you at a minimum have Log Analytics Reader role to the deployed Log Analytics Workspace instance.

1. Go to Application Insights Resource in the Managed Resource Group.

1.Select "Logs" under "Monitoring" section on left hand side.

1.Run the following KQL in Application Insights to get more information.

traces
| where message contains "ingestion failed"

Example of Batch ingestion csv row failure in Application Insights.

For Twins

[05:46:50 ERR] Ingestion failed for row: [rbsawmill1,KupSawMill1,Production Line for wood sawing operations,Large Wood Saw Medium Wood Saw,workUnit,ProductionLine,Description,assetsystemrefidYKUQX8MDDD,messystemrefidBA3XM6UOAL,dkjsfhdks] at 04/29/2024 05:46:50 for entity Equipment with filename : sampleLakehouse.Lakehouse/Files/dpfolder/Equipment_202404291105102034.csv due to There is a mismatch in total number of columns defined.

For Relationships

[12:09:41 ERR] Ingestion failed for row: [Person,dhyardhandler11Test,,bbhandlinglocationyard3511,hasValuesOf] at 04/30/2024 12:09:41 with filename : sampleLakehouse.Lakehouse/Files/dpfolder/Person_Mapping_202403190947077573.csv due to Source Type and Target Type cannot be empty for the record.

Commonly encountered errors

Common errors related to OPC UA

Start fresh

If for some reason you need to start Manufacturing data solutions with a clean slate, you can delete all the ingested data via the Cleanup API. This step makes sure you can reingest the data and don't get bothered with one or more previous ingested datasets.

Debug response of health API

Various Issues encountered related to health API.

Unable to access health API

After the deployment is completed, you can call the health API to check if the system is ready state. There can be Authentication or Authorization issue while trying to invoke the health API. Ensure that you created the App registration as per the prerequisites and has necessary roles assigned.

To invoke any of the Manufacturing data solutions APIs, you must fetch the access token for the App ID specified during deployment and use it as the Bearer token.

Failed status in health API response

Sometimes the Manufacturing data solutions health API can return the response with status as Failed. The response also contains the actual error message. Depending on the error message, you need to get in touch with the Microsoft support team.

Troubleshoot control plane issues

Initial checks

Check metrics emitted by Azure resources using Azure Monitor

Diagnostic settings are enabled by default to capture extra metrics for our Infrastructure resources. These metrics are enabled for following resources and send to Log Analytics workspace which gets procured as part of deployment. These settings are enabled for all Manufacturing data solutions SKUs.

1. Azure Data explorer
2. Event-hub
3. Function App
4. Azure Kubernetes Service
5. Cosmos DB
6. Azure OpenAI
7. App Service Plan
8. Azure Redis Cache

How to view metrics in Azure Monitor

Follow the guidance on Azure Monitor in MS Learn.

Debugging control plane issues

Common encountered scenarios

How to change network settings

• Ensure you have at least Contributor role on the Resource. Go to Networking tab for the required resource and add your IP address.

How to grant database access

Prerequisite: Your IP address is added in networking settings.

1. Ensure you have corresponding role required to access the Resource. For example, if you were to access storage account, ensure you have Storage Blob Data Reader or Storage Blob Data Contributor on the Storage account. Similarly, for Cosmos Database.

2. For Azure Data Explorer (ADX), you would need Database Reader permissions to query the data. For detailed instructions, refer to Manage Cluster Permissions.

Troubleshoot copilot

Initial checks

Try to fetch information from a table by running Copilot Query API.

POST https://{serviceUrl}/mds/copilot/v3/query?api-version=2024-06-30-preview

{
    "ask": "string", // natural language question
}

Debug Copilot issues

Troubleshoot via logs

Make sure that you at a minimum have Log Analytics Reader role to the deployed Log Analytics Workspaceinstance.

1. Run the following KQL in Application Insights with updated start and end time to get more information.

    let startTime = datetime('2024-07-04T00:00:00');
    let endTime = datetime('2024-07-05T00:00:00');
    let reqs = requests
        | where timestamp between (startTime .. endTime)
        | where cloud_RoleInstance startswith "aks-dmmcopilot-"
        | where url has 'copilot/v3/query'
        | project operation_Id, timestamp, resultCode, duration;
    let exc = exceptions
        | where timestamp between (startTime .. endTime)
        | where cloud_RoleInstance startswith "aks-dmmcopilot-"
        | distinct operation_Id, outerMessage
        | summarize exception_list = make_list(outerMessage, 10) by operation_Id;
    let copilot_traces = traces
        | where timestamp between (startTime .. endTime)
        | where cloud_RoleInstance startswith "aks-dmmcopilot-";
    let instructions = copilot_traces
        | where message has 'The instruction id used in prompt is '
        | extend instructions = parse_json(customDimensions)['Instructionid']
        | project operation_Id, instructions;
    let alias = copilot_traces
        | where message has 'AliasDictionary -'
        | extend alias = tostring(parse_json(customDimensions)['aliasDictionary'])
        | where isnotempty(alias)
        | summarize aliases = take_any(alias) by operation_Id
        | project operation_Id, aliases;
    let retries = copilot_traces
        | where message has 'Retry attempt {autoCorrectCount} out of'
        | summarize arg_max(timestamp,customDimensions) by operation_Id
        | project retries = toint(parse_json(customDimensions)['autoCorrectCount']), operation_Id;
    let total_vector_search_ms = copilot_traces
        | where message has 'Time taken to do similarity search on collection'
        | summarize total_vector_search_ms = sum(todouble((parse_json(customDimensions)['timeTaken']))) by operation_Id
        | project  total_vector_search_ms, operation_Id;
    let query = copilot_traces
        | where message startswith 'User input -'
        | extend query = tostring(parse_json(customDimensions)['input'])
        | extend intent = iff(message contains 'invalid', 'invalid', 'valid')
        | distinct operation_Id, query, intent;
    let tokens = copilot_traces
        | where message has 'Prompt tokens:' and message has 'Total tokens:'
        | extend prompt_tokens = toint(parse_json(customDimensions)['PromptTokens'])
        | extend completion_tokens = toint(parse_json(customDimensions)['CompletionTokens'])
        | summarize make_list(prompt_tokens), make_list(completion_tokens) by operation_Id;
    let total_tokens = copilot_traces
        | where message has 'Prompt tokens:' and message has 'Total tokens:'
        | extend total_tokens = toint(parse_json(customDimensions)['TotalTokens'])
        | summarize total_token = sum(total_tokens) by operation_Id;
    let kql = copilot_traces
        | where message has 'Generated Graph KQL Query -'
        | extend KQLquery = parse_json(customDimensions)['kqlQuery']
        | summarize kqls = make_list(KQLquery) by operation_Id
        | project operation_Id, kqls;
    let kql_sanitized = copilot_traces
        | where message has 'Sanitized Graph KQL Query -'
        | extend KQLquery = parse_json(customDimensions)['kqlQuery']
        | summarize s_kqls = make_list(KQLquery) by operation_Id
        | project operation_Id, s_kqls;
    reqs
    | extend operation_id_req = operation_Id
    | join kind=leftouter exc on $left.operation_id_req == $right.operation_Id
    | join kind=leftouter query on $left.operation_id_req == $right.operation_Id
    | join kind=leftouter instructions on $left.operation_id_req == $right.operation_Id
    | join kind=leftouter alias on $left.operation_id_req == $right.operation_Id
    | join kind=leftouter tokens on $left.operation_id_req == $right.operation_Id
    | join kind=leftouter total_tokens on $left.operation_id_req == $right.operation_Id
    | join kind=leftouter kql on $left.operation_id_req == $right.operation_Id
    | join kind=leftouter kql_sanitized on $left.operation_id_req == $right.operation_Id
    | join kind=leftouter retries on $left.operation_id_req == $right.operation_Id
    | join kind=leftouter total_vector_search_ms on $left.operation_id_req == $right.operation_Id
    | project
        timestamp,
        operation_Id,
        resultCode,
        query,
        duration,
        intent,
        exception_list,
        instructions,
        aliases,
        list_prompt_tokens,
        list_completion_tokens,
        total_token,
        kqls,
        s_kqls,
        retries,
        total_vector_search_ms
    | order by timestamp desc

1. Go to Application Insights Resource in the Managed Resource Group.

   

2. Select "Logs" under Monitoring navigation section on left hand side.

   

Commonly encountered issue

Alias dictionary

Example Query

Instruction

Operation

Query

Feedback

Create a support e-mail ticket

Select Support in the left hand pane in Solution Center. Select Microsoft Cloud for Manufacturing (Preview) and then Data Solutions and Factory Operations Agent(Preview) under Solution Area. Select the support email template to send a support request to the Manufacturing data solutions team.

You can also create a support ticket from the announcements in Solution Center in the Microsoft Cloud for Manufacturing Home page. Select Send Email to send a support request to the Manufacturing data solutions team.