Problems to connect iotedge behind a zscaler proxy

[ma3gool-gf]

Hello all,

I'm trying to connect an IoT Edge device to Azure IoT Central, but after more than a week of troubleshooting, I haven't been successful.

What I’ve done so far:
I added the Zscaler root certificate to the trusted store on the device.

I configured the proxy for Moby, aziot-edged, and aziot-identityd as documented here:
https://learn.microsoft.com/en-gb/azure/iot-edge/how-to-configure-proxy-support?view=iotedge-2018-06

Current Status:
The device is successfully provisioned via DPS and gets assigned to the correct IoT Hub, but it fails to connect to that hub.

Here are the relevant log excerpts:

Jun 02 11:14:11 pschshsv00010 aziot-identityd[33469]: 2025-06-02T11:14:11Z [INFO] - DPS registration complete.
Jun 02 11:14:11 pschshsv00010 aziot-identityd[33469]: 2025-06-02T11:14:11Z [INFO] - Successfully provisioned with DPS.
Jun 02 11:14:11 pschshsv00010 aziot-identityd[33469]: 2025-06-02T11:14:11Z [INFO] - Provisioning complete.
Jun 02 11:14:11 pschshsv00010 aziot-identityd[33469]: 2025-06-02T11:14:11Z [INFO] - Identity reconciliation started. Reason: Startup
Jun 02 11:14:11 pschshsv00010 aziot-identityd[33469]: 2025-06-02T11:14:11Z [INFO] - Could not reconcile Identities with current device data. Reprovisioning.
Jun 02 11:14:11 pschshsv00010 aziot-keyd[22542]: 2025-06-02T11:14:11Z [INFO] - <-- GET /keypair/device-id?api-version=2021-05-01 {"host": "keyd.sock"}
Jun 02 11:14:11 pschshsv00010 aziot-keyd[22542]: 2025-06-02T11:14:11Z [INFO] - --> 200 {"content-type": "application/json"}
Jun 02 11:14:11 pschshsv00010 aziot-certd[22554]: 2025-06-02T11:14:11Z [INFO] - <-- GET /certificates/device-id?api-version=2020-09-01 {"host": "certd.sock"}
Jun 02 11:14:11 pschshsv00010 aziot-certd[22554]: 2025-06-02T11:14:11Z [INFO] - --> 200 {"content-type": "application/json"}
Jun 02 11:14:11 pschshsv00010 aziot-keyd[22542]: 2025-06-02T11:14:11Z [INFO] - <-- POST /parameters/algorithm?api-version=2021-05-01 {"content-length": "248", "content-type": "application/json"}
Jun 02 11:14:11 pschshsv00010 aziot-keyd[22542]: 2025-06-02T11:14:11Z [INFO] - --> 200 {"content-type": "application/json"}
Jun 02 11:14:11 pschshsv00010 aziot-keyd[22542]: 2025-06-02T11:14:11Z [INFO] - <-- POST /parameters/rsa-modulus?api-version=2021-05-01 {"content-length": "248", "content-type": "application/json"}
Jun 02 11:14:11 pschshsv00010 aziot-keyd[22542]: 2025-06-02T11:14:11Z [INFO] - --> 200 {"content-type": "application/json"}
Jun 02 11:14:11 pschshsv00010 aziot-keyd[22542]: 2025-06-02T11:14:11Z [INFO] - <-- POST /parameters/rsa-exponent?api-version=2021-05-01 {"content-length": "248", "content-type": "application/json"}
Jun 02 11:14:11 pschshsv00010 aziot-keyd[22542]: 2025-06-02T11:14:11Z [INFO] - --> 200 {"content-type": "application/json"}
Jun 02 11:14:11 pschshsv00010 aziot-keyd[22542]: 2025-06-02T11:14:11Z [INFO] - <-- POST /parameters/algorithm?api-version=2021-05-01 {"content-length": "248", "content-type": "application/json"}
Jun 02 11:14:11 pschshsv00010 aziot-keyd[22542]: 2025-06-02T11:14:11Z [INFO] - --> 200 {"content-type": "application/json"}
Jun 02 11:14:11 pschshsv00010 aziot-keyd[22542]: 2025-06-02T11:14:11Z [INFO] - <-- POST /parameters/rsa-modulus?api-version=2021-05-01 {"content-length": "248", "content-type": "application/json"}
Jun 02 11:14:11 pschshsv00010 aziot-keyd[22542]: 2025-06-02T11:14:11Z [INFO] - --> 200 {"content-type": "application/json"}
Jun 02 11:14:11 pschshsv00010 aziot-keyd[22542]: 2025-06-02T11:14:11Z [INFO] - <-- POST /parameters/rsa-exponent?api-version=2021-05-01 {"content-length": "248", "content-type": "application/json"}
Jun 02 11:14:11 pschshsv00010 aziot-keyd[22542]: 2025-06-02T11:14:11Z [INFO] - --> 200 {"content-type": "application/json"}
Jun 02 11:14:11 pschshsv00010 aziot-identityd[33469]: 2025-06-02T11:14:11Z [INFO] - Sending DPS registration request.
Jun 02 11:14:11 pschshsv00010 aziot-keyd[22542]: 2025-06-02T11:14:11Z [INFO] - <-- POST /encrypt?api-version=2021-05-01 {"content-length": "636", "content-type": "application/json"}
Jun 02 11:14:11 pschshsv00010 aziot-keyd[22542]: 2025-06-02T11:14:11Z [INFO] - --> 200 {"content-type": "application/json"}
Jun 02 11:14:16 pschshsv00010 aziot-identityd[33469]: 2025-06-02T11:14:16Z [INFO] - Checking DPS registration status.
Jun 02 11:14:16 pschshsv00010 aziot-keyd[22542]: 2025-06-02T11:14:16Z [INFO] - <-- POST /encrypt?api-version=2021-05-01 {"content-length": "636", "content-type": "application/json"}
Jun 02 11:14:16 pschshsv00010 aziot-keyd[22542]: 2025-06-02T11:14:16Z [INFO] - --> 200 {"content-type": "application/json"}
Jun 02 11:14:17 pschshsv00010 aziot-identityd[33469]: 2025-06-02T11:14:17Z [INFO] - Assigned to IoT hub: iotc-897b8b9e-98b6-4cd1-9d00-04b11436d80e.azure-devices.net
Jun 02 11:14:17 pschshsv00010 aziot-identityd[33469]: 2025-06-02T11:14:17Z [INFO] - DPS registration complete.
Jun 02 11:14:17 pschshsv00010 aziot-identityd[33469]: 2025-06-02T11:14:17Z [INFO] - Successfully provisioned with DPS.
Jun 02 11:14:17 pschshsv00010 aziot-identityd[33469]: 2025-06-02T11:14:17Z [ERR!] - Failed to provision with IoT Hub, and no valid device backup was found: Hub client error
Jun 02 11:14:17 pschshsv00010 aziot-identityd[33469]: 2025-06-02T11:14:17Z [ERR!] - service encountered an error
Jun 02 11:14:17 pschshsv00010 aziot-identityd[33469]: 2025-06-02T11:14:17Z [ERR!] - caused by: Hub client error
Jun 02 11:14:17 pschshsv00010 aziot-identityd[33469]: 2025-06-02T11:14:17Z [ERR!] - caused by: {"errorCode":401002,"message":"Unauthorized access","trackingId":"0A19F43D01A742DDB063BCBE712DDB3D-G2:-TimeStamp:2025-06-02T11:14:17.459763985Z","timestampUtc":"2025-06-02T11:14:17.459763985Z","info":null}
Jun 02 11:14:17 pschshsv00010 aziot-identityd[33469]: 2025-06-02T11:14:17Z [ERR!] - 0:
Jun 02 11:14:17 pschshsv00010 aziot-identityd[33469]: 1:
Jun 02 11:14:17 pschshsv00010 aziot-identityd[33469]: 2:
Jun 02 11:14:17 pschshsv00010 aziot-identityd[33469]: 3:
Jun 02 11:14:17 pschshsv00010 aziot-identityd[33469]: 4:
Jun 02 11:14:17 pschshsv00010 aziot-identityd[33469]: 5:
Jun 02 11:14:17 pschshsv00010 aziot-identityd[33469]: 6:
Jun 02 11:14:17 pschshsv00010 aziot-identityd[33469]: 7:
Jun 02 11:14:17 pschshsv00010 aziot-identityd[33469]: 8:
Jun 02 11:14:17 pschshsv00010 aziot-identityd[33469]: 9:
Jun 02 11:14:17 pschshsv00010 aziot-identityd[33469]: 10: __libc_start_main
Jun 02 11:14:17 pschshsv00010 aziot-identityd[33469]: 11:
Jun 02 11:14:17 pschshsv00010 systemd[1]: aziot-identityd.service: Main process exited, code=exited, status=1/FAILURE
Jun 02 11:14:17 pschshsv00010 systemd[1]: aziot-identityd.service: Failed with result 'exit-code'.
Jun 02 11:14:22 pschshsv00010 systemd[1]: aziot-identityd.service: Scheduled restart job, restart counter is at 91.
Jun 02 11:14:22 pschshsv00010 systemd[1]: Stopped aziot-identityd.service - Azure IoT Identity Service.
Jun 02 11:14:22 pschshsv00010 aziot-edged[22524]: 2025-06-02T11:14:22Z [WARN] - Failed to send HTTP request (attempt 1 of 2): connection error: Connection reset by peer (os error 104)

It looks like authentication against the assigned IoT Hub is failing with a 401 Unauthorized error, even though DPS provisioning completes successfully.

Output from iotedge check:
Configuration checks (aziot-identity-service)

√ keyd configuration is well-formed - OK
√ certd configuration is well-formed - OK
√ tpmd configuration is well-formed - OK
√ identityd configuration is well-formed - OK
√ daemon configurations up-to-date with config.toml - OK
√ identityd config toml file specifies a valid hostname - OK
√ aziot-identity-service package is up-to-date - OK
√ host time is close to reference time - OK
√ production readiness: identity certificates expiry - OK
√ preloaded certificates are valid - OK
√ keyd is running - OK
√ certd is running - OK
√ identityd is running - OK
√ read all preloaded certificates from the Certificates Service - OK
√ read all preloaded key pairs from the Keys Service - OK
√ check all EST server URLs utilize HTTPS - OK
√ ensure all preloaded certificates match preloaded private keys with the same ID - OK

Connectivity checks (aziot-identity-service)

‼ host can connect to and perform TLS handshake with iothub AMQP port - Warning
Could not retrieve iothub_hostname from provisioning file.
Please specify the backing IoT Hub name using --iothub-hostname switch if you have that information.
Since no hostname is provided, all hub connectivity tests will be skipped.
‼ host can connect to and perform TLS handshake with iothub HTTPS / WebSockets port - Warning
Could not retrieve iothub_hostname from provisioning file.
Please specify the backing IoT Hub name using --iothub-hostname switch if you have that information.
Since no hostname is provided, all hub connectivity tests will be skipped.
‼ host can connect to and perform TLS handshake with iothub MQTT port - Warning
Could not retrieve iothub_hostname from provisioning file.
Please specify the backing IoT Hub name using --iothub-hostname switch if you have that information.
Since no hostname is provided, all hub connectivity tests will be skipped.
√ host can connect to and perform TLS handshake with DPS endpoint - OK

Configuration checks

√ aziot-edged configuration is well-formed - OK
√ configuration up-to-date with config.toml - OK
√ container engine is installed and functional - OK
× configuration has correct URIs for daemon mgmt endpoint - Error
SocketError - SocketErrorCode (TimedOut) : Operation timed out
One or more errors occurred. (Got bad response: )
√ aziot-edge package is up-to-date - OK
√ container time is close to host time - OK
‼ DNS server - Warning
Container engine is not configured with DNS server setting, which may impact connectivity to IoT Hub.
Please see https://aka.ms/iotedge-prod-checklist-dns for best practices.
You can ignore this warning if you are setting DNS server per module in the Edge deployment.
‼ production readiness: logs policy - Warning
Container engine is not configured to rotate module logs which may cause it run out of disk space.
Please see https://aka.ms/iotedge-prod-checklist-logs for best practices.
You can ignore this warning if you are setting log policy per module in the Edge deployment.
× production readiness: Edge Agent's storage directory is persisted on the host filesystem - Error
Could not check current state of edgeAgent container
× production readiness: Edge Hub's storage directory is persisted on the host filesystem - Error
Could not check current state of edgeHub container
‼ proxy settings are consistent in aziot-edged, aziot-identityd, moby daemon and config.toml - Warning
The proxy setting for IoT Edge Agent "http://...:80", IoT Edge Daemon "http://...:80", IoT Identity Daemon "http://...:80", and Moby "" may need to be identical.

Connectivity checks

23 check(s) succeeded.
6 check(s) raised warnings. Re-run with --verbose for more details.
3 check(s) raised errors. Re-run with --verbose for more details.
7 check(s) were skipped due to errors from other checks. Re-run with --verbose for more details.

Questions:
Has anyone encountered a similar issue when using Zscaler or a corporate proxy?

Is there any additional step required to allow aziot-edged to authenticate with the IoT Hub after DPS succeeds?

Could this be related to a missing trusted root cert in the Docker/Moby container layer?

Any help or suggestions would be greatly appreciated!

Thanks in advance,

[nyanzebra]

@ma3gool-gf just to make sure the problem is understood. You don't have any issues with connecting device to upstream without proxy and using DPS? This only happens with ZScaler proxy in place?