Unable to get local issuer certificate on launching iotedge demon

Hello Azure team,

We are attepting to put Microsoft Azure Edge v1.2 on an embedded device which does not have any standard package managers like apt, snaps etc.We are using the "Quick start approach with symmetric keys". We copied the relevant binaries

on to the edge device and made all the necessary configuration. Generated the certificates using the tooling scripts and provided the paths in config.toml

We are apble to use openssl s_connect and verify the toolchain. But, azure IoT Edge runtime is not able to communicate with the server.

Below is the logs from "iotedge check --verbose" and "iot runtime". We faced a similar issue when the edged is running. Can you please help us fix the issue?

We currently cannot upgrade to the latest version because of dependency on libc. We first want to test with this version and then consider upgrading.

iotedge check --verbose logs as below:

Configuration checks (aziot-identity-service) --------------------------------------------- ΓêÜ keyd configuration is well-formed - OK ΓêÜ certd configuration is well-formed - OK ΓêÜ tpmd configuration is well-formed - OK ΓêÜ identityd configuration is well-formed - OK ΓêÜ daemon configurations up-to-date with config.toml - OK ΓêÜ identityd config toml file specifies a valid hostname - OK ├ù aziot-identity-service package is up-to-date - Error     could not query https://aka.ms/latest-aziot-identity-service for latest available version         caused by: error trying to connect: error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:ssl/statem/statem_clnt.c:1915:: unable to get local issuer certificate         caused by: error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:ssl/statem/statem_clnt.c:1915:: unable to get local issuer certificate         caused by: error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:ssl/statem/statem_clnt.c:1915:         caused by: error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:ssl/statem/statem_clnt.c:1915: ΓêÜ host time is close to reference time - OK ΓêÜ preloaded certificates are valid - OK ΓêÜ keyd is running - OK ΓêÜ certd is running - OK ΓêÜ identityd is running - OK ΓêÜ read all preloaded certificates from the Certificates Service - OK ΓêÜ read all preloaded key pairs from the Keys Service - OK ΓêÜ ensure all preloaded certificates match preloaded private keys with the same ID - OK

Connectivity checks (aziot-identity-service) -------------------------------------------- ├ù host can connect to and perform TLS handshake with iothub AMQP port - Error     Could not connect to ArchuHub.azure-devices.net : could not complete TLS handshake         caused by: error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:ssl/statem/statem_clnt.c:1915:: unable to get local issuer certificate ├ù host can connect to and perform TLS handshake with iothub HTTPS / WebSockets port - Error     Could not connect to ArchuHub.azure-devices.net : could not complete TLS handshake         caused by: error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:ssl/statem/statem_clnt.c:1915:: unable to get local issuer certificate ├ù host can connect to and perform TLS handshake with iothub MQTT port - Error     Could not connect to ArchuHub.azure-devices.net : could not complete TLS handshake         caused by: error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:ssl/statem/statem_clnt.c:1915:: unable to get local issuer certificate

Configuration checks -------------------- ΓêÜ aziot-edged configuration is well-formed - OK ΓêÜ configuration up-to-date with config.toml - OK ΓêÜ container engine is installed and functional - OK ├ù configuration has correct URIs for daemon mgmt endpoint - Error     One or more errors occurred. (Connection refused /var/run/iotedge/mgmt.sock)         caused by: docker returned exit code: 1, stderr = One or more errors occurred. (Connection refused /var/run/iotedge/mgmt.sock) ├ù aziot-edge package is up-to-date - Error     Could not spawn aziot-edged process         caused by: No such file or directory (os error 2) ΓêÜ container time is close to host time - OK ΓêÜ DNS server - OK ΓÇ╝ production readiness: container engine - Warning     Device is not using a production-supported container engine (moby-engine).     Please see https://aka.ms/iotedge-prod-checklist-moby for details. ΓêÜ production readiness: logs policy - OK ├ù production readiness: Edge Agent's storage directory is persisted on the host filesystem - Error     Could not check current state of edgeAgent container         caused by: docker returned exit code: 1, stderr = Error: No such object: edgeAgent ├ù production readiness: Edge Hub's storage directory is persisted on the host filesystem - Error     Could not check current state of edgeHub container         caused by: docker returned exit code: 1, stderr = Error: No such object: edgeHub ΓêÜ Agent image is valid and can be pulled from upstream - OK

Connectivity checks ------------------- ΓêÜ container on the default network can connect to upstream  AMQP port - OK ΓêÜ container on the default network can connect to upstream HTTPS / WebSockets port - OK ΓêÜ container on the default network can connect to upstream MQTT port - OK ΓêÜ container on the IoT Edge module network can connect to upstream AMQP port - OK ΓêÜ container on the IoT Edge module network can connect to upstream HTTPS / WebSockets port - OK ΓêÜ container on the IoT Edge module network can connect to upstream MQTT port - OK 27 check(s) succeeded. 1 check(s) raised warnings. 8 check(s) raised errors.

IoT Runtime logs as below:

<6>2024-10-28T10:49:05Z [INFO] - Starting Azure IoT Edge Module Runtime

<6>2024-10-28T10:49:05Z [INFO] - Version - 1.2.0

<6>2024-10-28T10:49:05Z [INFO] - Initializing the module runtime...

<6>2024-10-28T10:49:05Z [INFO] - Initializing module runtime...

<6>2024-10-28T10:49:05Z [INFO] - Using runtime network id azure-iot-edge

<6>2024-10-28T10:49:05Z [INFO] - Successfully initialized module runtime

<6>2024-10-28T10:49:05Z [INFO] - Finished initializing the module runtime.

<6>2024-10-28T10:49:05Z [INFO] - Obtaining edge device provisioning data...

<6>2024-10-28T10:49:05Z [INFO] - <-- POST /identities/device?api-version=2020-09-01 {"content-type": "application/json", "host": "2f72756e2f617a696f742f6964656e74697479642e736f636b:0", "content-length": "16"}

<6>2024-10-28T10:49:05Z [INFO] - <-- GET /key/device-id?api-version=2020-09-01 {"host": "keyd.sock"}

<6>2024-10-28T10:49:05Z [INFO] - --> 200 {"content-type": "application/json"}

<6>2024-10-28T10:49:05Z [INFO] - --> 200 {"content-type": "application/json"}

<6>2024-10-28T10:49:05Z [INFO] - Finished provisioning edge device.

<6>2024-10-28T10:49:05Z [INFO] - Stopping all modules...

<6>2024-10-28T10:49:05Z [INFO] - Finished stopping modules.

<6>2024-10-28T10:49:05Z [INFO] - Detecting if device information has changed...

<6>2024-10-28T10:49:05Z [INFO] - Starting management API...

<6>2024-10-28T10:49:05Z [INFO] - Starting workload API...

<6>2024-10-28T10:49:05Z [INFO] - Starting watchdog with 60 second frequency...

<6>2024-10-28T10:49:05Z [INFO] - Listening on unix:///var/run/iotedge/mgmt.sock with 1 thread for management API.

<6>2024-10-28T10:49:05Z [INFO] - Checking edge runtime status

<6>2024-10-28T10:49:05Z [INFO] - Creating and starting edge runtime module edgeAgent

<6>2024-10-28T10:49:05Z [INFO] - <-- GET /certificates/aziot-edged-ca?api-version=2020-09-01 {"host": "2f72756e2f617a696f742f63657274642e736f636b:0"}

<6>2024-10-28T10:49:05Z [INFO] - --> 200 {"content-type": "application/json"}

<6>2024-10-28T10:49:05Z [INFO] - Listening on unix:///var/run/iotedge/workload.sock with 1 thread for workload API.

<6>2024-10-28T10:49:05Z [INFO] - <-- PUT /identities/modules/$edgeAgent?api-version=2020-09-01&type=aziot {"content-type": "application/json", "host": "2f72756e2f617a696f742f6964656e74697479642e736f636b:0", "content-length": "40"}

<6>2024-10-28T10:49:05Z [INFO] - <-- GET /key/device-id?api-version=2020-09-01 {"host": "keyd.sock"}

<6>2024-10-28T10:49:05Z [INFO] - --> 200 {"content-type": "application/json"}

<6>2024-10-28T10:49:05Z [INFO] - <-- POST /sign?api-version=2020-09-01 {"content-type": "application/json", "host": "keyd.sock", "content-length": "362"}

<6>2024-10-28T10:49:05Z [INFO] - --> 200 {"content-type": "application/json"}

<6>2024-10-28T10:49:05Z [INFO] - !!! Hub client error

<6>2024-10-28T10:49:05Z [INFO] - !!! caused by: error trying to connect: error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:ssl/statem/statem_clnt.c:1915:: unable to get local issuer certificate

<6>2024-10-28T10:49:05Z [INFO] - !!! caused by: error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:ssl/statem/statem_clnt.c:1915:: unable to get local issuer certificate

<6>2024-10-28T10:49:05Z [INFO] - !!! caused by: error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:ssl/statem/statem_clnt.c:1915:

<6>2024-10-28T10:49:05Z [INFO] - !!! caused by: error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:ssl/statem/statem_clnt.c:1915:

<6>2024-10-28T10:49:05Z [INFO] - --> 404 {"content-type": "application/json"}

<4>2024-10-28T10:49:05Z [WARN] - Error in watchdog when checking for edge runtime status:

<4>2024-10-28T10:49:05Z [WARN] - A module runtime error occurred.

<4>2024-10-28T10:49:05Z [WARN] -        caused by: HTTP response error: [404 Not Found] {"message":"Hub client error\ncaused by: error trying to connect: error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:ssl/statem/statem_clnt.c:1915:: unable to get local issuer certificate\ncaused by: error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:ssl/statem/statem_clnt.c:1915:: unable to get local issuer certificate\ncaused by: error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:ssl/statem/statem_clnt.c:1915:\ncaused by: error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:ssl/statem/statem_clnt.c:1915:"}

Version of different components are as below:

Linux:3.18

libc:2.26

Crun:1.17

Containerd: 1.7

Moby version:25.0

Azure Edge :1.2