IoTEdge fails to remove Docker Containers after an update on an Child EdgeDevice with 25+ modules deployed

[silvestropomestetrapak]

Expected Behavior

In a Nested Edge scenario, a Child EdgeDevice has a fairly good amount of modules deployed (more than 25).
We pushed a new deployment manifest where some modules are missing, and all the others have a different CreateOptions.
I would expect:

• the missing modules to be removed by the edgeAgent
• the remaining modules to be recreated with the different CreateOptions

Current Behavior

• the missing modules are removed, but aziot-identityd logs:

Oct 24 13:41:15 x3439-07861-ipc1.rsu.local aziot-identityd[864599]: 2025-10-24T13:41:15Z [INFO] - <-- DELETE /identities/modules/tetrapak_ara_servicetool_frontend?api-version=2020-09-01&type=aziot {"host": "identityd.sock"}
Oct 24 13:41:15 x3439-07861-ipc1.rsu.local aziot-identityd[864599]: 2025-10-24T13:41:15Z [INFO] - !!! Hub client error
Oct 24 13:41:15 x3439-07861-ipc1.rsu.local aziot-identityd[864599]: 2025-10-24T13:41:15Z [INFO] - !!! caused by: unexpected HTTP status code: 412 Precondition Failed
Oct 24 13:41:15 x3439-07861-ipc1.rsu.local aziot-identityd[864599]: 2025-10-24T13:41:15Z [INFO] - --> 404 {"content-type": "application/json"}

• the remaining modules are not recreated, and the edgeAgent logs the following error:

<6> 2025-10-24 13:41:16.579 +00:00 [INF] - Unable to process module tetrapak_mdl_datastorage_api add or update as the module identity could not be obtained
<6> 2025-10-24 13:41:16.579 +00:00 [INF] - Unable to process module tetrapak_mdl_breakfast add or update as the module identity could notbe obtained
<6> 2025-10-24 13:41:16.579 +00:00 [INF] - Unable to process module tetrapak_mdl_plmsrecordpublisher_localstorage add or update as the module identity could not be obtained
<6> 2025-10-24 13:41:16.579 +00:00 [INF] - Unable to process module tetrapak_mdl_plantmodelpublisher add or update as the module identitycould not be obtained
<6> 2025-10-24 13:41:16.579 +00:00 [INF] - Unable to process module tetrapak_mdl_plmsphases add or update as the module identity could not be obtained
<6> 2025-10-24 13:41:16.579 +00:00 [INF] - Unable to process module tetrapak_mdl_timeseriespublisher add or update as the module identity could not be obtained

I also noticed that by deleting manually all the containers, the edgeAgent has been able to recreate them correctly with the latest CreateOptions.

Steps to Reproduce

Provide a detailed set of steps to reproduce the bug.

1. Setup a parent device and a child device
2. in the Child device, go back and forth between these two versions of deployment manifest (attached as example, I've purged them from secrets, password and certificates)
   manifest_6.0.36.json
   manifest_6.0.53.json

Context (Environment)

Output of iotedge check

Click here

{"additional_info":{"aziot_edged_version":null,"docker_version":"28.3.1-1","now":"2025-10-30T12:35:12.445837384Z","os":{"arch":"x86_64","bitness":64,"id":"ol","version_id":"9.6"},"system_info":{"disks":[],"total_ram":"15.17 GiB","total_swap":"0 B","used_ram":"2.67 GiB","used_swap":"0 B"}},"checks":{"aziot-edge-version":{"result":{"result":"ignored"},"additional_info":{"actual_version":null,"expected_version":null}},"aziot-edged-config-well-formed":{"result":{"result":"ok"},"additional_info":{}},"aziot-version":{"result":{"result":"ignored"},"additional_info":{"actual_version":null,"expected_version":null}},"certd-config-well-formed":{"result":{"result":"ok"},"additional_info":{}},"certd-running":{"result":{"result":"ok"},"additional_info":{}},"certs-match-private-keys":{"result":{"result":"ok"},"additional_info":{}},"certs-preloaded":{"result":{"result":"ok"},"additional_info":{}},"certs-read":{"result":{"result":"ok"},"additional_info":{}},"check-agent-image":{"result":{"result":"ok"},"additional_info":{}},"config-up-to-date":{"result":{"result":"ok"},"additional_info":{}},"configs-up-to-date":{"result":{"result":"ok"},"additional_info":{}},"connect-management-uri":{"result":{"result":"error","details":["Unable to find image 'itdeiot-de-08.rsu.local/azureiotedge-diagnostics:1.5.21' locally\ndocker: Error response from daemon: manifest for itdeiot-de-08.rsu.local/azureiotedge-diagnostics:1.5.21 not found: manifest unknown: manifest unknown\n\nRun 'docker run --help' for more information\n","docker returned exit status: 125, stderr = Unable to find image 'itdeiot-de-08.rsu.local/azureiotedge-diagnostics:1.5.21' locally\ndocker: Error response from daemon: manifest for itdeiot-de-08.rsu.local/azureiotedge-diagnostics:1.5.21 not found: manifest unknown: manifest unknown\n\nRun 'docker run --help' for more information\n"]},"additional_info":{"connect_management_uri":"unix:///var/run/iotedge/mgmt.sock","listen_management_uri":"fd://aziot-edged.mgmt.socket"}},"container-connect-upstream-amqp":{"result":{"result":"error","details":["Container on the azure-iot-edge network could not connect to itdeiot-de-08.rsu.local:5671","docker returned exit status: 125, stderr = Unable to find image 'itdeiot-de-08.rsu.local/azureiotedge-diagnostics:1.5.21' locally\ndocker: Error response from daemon: manifest for itdeiot-de-08.rsu.local/azureiotedge-diagnostics:1.5.21 not found: manifest unknown: manifest unknown\n\nRun 'docker run --help' for more information\n"]},"additional_info":{"diagnostics_image_name":"/azureiotedge-diagnostics:1.5.21","network_name":"azure-iot-edge","proxy":null,"upstream_hostname":"itdeiot-de-08.rsu.local","upstream_port":"Amqp"}},"container-connect-upstream-https":{"result":{"result":"error","details":["Container on the azure-iot-edge network could not connect to itdeiot-de-08.rsu.local:443","docker returned exit status: 125, stderr = Unable to find image 'itdeiot-de-08.rsu.local/azureiotedge-diagnostics:1.5.21' locally\ndocker: Error response from daemon: manifest for itdeiot-de-08.rsu.local/azureiotedge-diagnostics:1.5.21 not found: manifest unknown: manifest unknown\n\nRun 'docker run --help' for more information\n"]},"additional_info":{"diagnostics_image_name":"/azureiotedge-diagnostics:1.5.21","network_name":"azure-iot-edge","proxy":null,"upstream_hostname":"itdeiot-de-08.rsu.local","upstream_port":"Https"}},"container-connect-upstream-mqtt":{"result":{"result":"error","details":["Container on the azure-iot-edge network could not connect to itdeiot-de-08.rsu.local:8883","docker returned exit status: 125, stderr = Unable to find image 'itdeiot-de-08.rsu.local/azureiotedge-diagnostics:1.5.21' locally\ndocker: Error response from daemon: manifest for itdeiot-de-08.rsu.local/azureiotedge-diagnostics:1.5.21 not found: manifest unknown: manifest unknown\n\nRun 'docker run --help' for more information\n"]},"additional_info":{"diagnostics_image_name":"/azureiotedge-diagnostics:1.5.21","network_name":"azure-iot-edge","proxy":null,"upstream_hostname":"itdeiot-de-08.rsu.local","upstream_port":"Mqtt"}},"container-default-connect-upstream-amqp":{"result":{"result":"error","details":["Container on the default network could not connect to itdeiot-de-08.rsu.local:5671","docker returned exit status: 125, stderr = Unable to find image 'itdeiot-de-08.rsu.local/azureiotedge-diagnostics:1.5.21' locally\ndocker: Error response from daemon: manifest for itdeiot-de-08.rsu.local/azureiotedge-diagnostics:1.5.21 not found: manifest unknown: manifest unknown\n\nRun 'docker run --help' for more information\n"]},"additional_info":{"diagnostics_image_name":"/azureiotedge-diagnostics:1.5.21","network_name":"azure-iot-edge","proxy":null,"upstream_hostname":"itdeiot-de-08.rsu.local","upstream_port":"Amqp"}},"container-default-connect-upstream-https":{"result":{"result":"error","details":["Container on the default network could not connect to itdeiot-de-08.rsu.local:443","docker returned exit status: 125, stderr = Unable to find image 'itdeiot-de-08.rsu.local/azureiotedge-diagnostics:1.5.21' locally\ndocker: Error response from daemon: manifest for itdeiot-de-08.rsu.local/azureiotedge-diagnostics:1.5.21 not found: manifest unknown: manifest unknown\n\nRun 'docker run --help' for more information\n"]},"additional_info":{"diagnostics_image_name":"/azureiotedge-diagnostics:1.5.21","network_name":"azure-iot-edge","proxy":null,"upstream_hostname":"itdeiot-de-08.rsu.local","upstream_port":"Https"}},"container-default-connect-upstream-mqtt":{"result":{"result":"error","details":["Container on the default network could not connect to itdeiot-de-08.rsu.local:8883","docker returned exit status: 125, stderr = Unable to find image 'itdeiot-de-08.rsu.local/azureiotedge-diagnostics:1.5.21' locally\ndocker: Error response from daemon: manifest for itdeiot-de-08.rsu.local/azureiotedge-diagnostics:1.5.21 not found: manifest unknown: manifest unknown\n\nRun 'docker run --help' for more information\n"]},"additional_info":{"diagnostics_image_name":"/azureiotedge-diagnostics:1.5.21","network_name":"azure-iot-edge","proxy":null,"upstream_hostname":"itdeiot-de-08.rsu.local","upstream_port":"Mqtt"}},"container-engine-dns":{"result":{"result":"warning","details":["Container engine is not configured with DNS server setting, which may impact connectivity to IoT Hub.\nPlease see https://aka.ms/iotedge-prod-checklist-dns for best practices.\nYou can ignore this warning if you are setting DNS server per module in the Edge deployment."]},"additional_info":{"container_engine_config_path":"/etc/docker/daemon.json","dns":null}},"container-engine-ipv6":{"result":{"result":"ignored"},"additional_info":{"actual_use_ipv6":null,"expected_use_ipv6":false}},"container-engine-logrotate":{"result":{"result":"warning","details":["Container engine is not configured to rotate module logs which may cause it run out of disk space.\nPlease see https://aka.ms/iotedge-prod-checklist-logs for best practices.\nYou can ignore this warning if you are setting log policy per module in the Edge deployment."]},"additional_info":{"daemon_config":{"log-driver":"gelf","log-opts":{"max-file":null,"max-size":null}}}},"container-engine-uri":{"result":{"result":"ok"},"additional_info":{"docker_host_arg":"unix:///var/run/docker.sock","docker_server_version":"28.3.1-1"}},"container-local-time":{"result":{"result":"error","details":["Could not query local time inside container","docker returned exit status: 125, stderr = Unable to find image 'itdeiot-de-08.rsu.local/azureiotedge-diagnostics:1.5.21' locally\ndocker: Error response from daemon: manifest for itdeiot-de-08.rsu.local/azureiotedge-diagnostics:1.5.21 not found: manifest unknown: manifest unknown\n\nRun 'docker run --help' for more information\n"]},"additional_info":{"actual_duration":null,"diff":null,"expected_duration":null}},"container-resolve-parent-hostname":{"result":{"result":"error","details":["Failed to resolve parent hostname itdeiot-de-08.rsu.local","docker returned exit status: 125, stderr = Unable to find image 'itdeiot-de-08.rsu.local/azureiotedge-diagnostics:1.5.21' locally\ndocker: Error response from daemon: manifest for itdeiot-de-08.rsu.local/azureiotedge-diagnostics:1.5.21 not found: manifest unknown: manifest unknown\n\nRun 'docker run --help' for more information\n"]},"additional_info":{}},"edge-agent-storage-mounted-from-host":{"result":{"result":"ok"},"additional_info":{"container_directories":["/var/run/iotedge/mgmt.sock","/var/run/iotedge/workload.sock","/iotedge/storage"],"storage_directory":"/iotedge/storage/edgeAgent"}},"edge-hub-storage-mounted-from-host":{"result":{"result":"ok"},"additional_info":{"container_directories":["/iotedge/storage","/var/run/iotedge/workload.sock"],"storage_directory":"/iotedge/storage/edgeHub"}},"est-identity-and-bootstrap-certificate-expiry":{"result":{"result":"ignored"},"additional_info":{"bootstrap_certificate_info":null,"identity_certificate_info":null}},"est-server-https":{"result":{"result":"ok"},"additional_info":{}},"host-connect-dps-endpoint":{"result":{"result":"ignored"},"additional_info":{"dps_endpoint":null,"dps_hostname":null,"proxy":null}},"host-connect-iothub-amqp":{"result":{"result":"ok"},"additional_info":{"iothub_hostname":"itdeiot-de-08.rsu.local","port_number":5671,"proxy":null}},"host-connect-iothub-https":{"result":{"result":"ok"},"additional_info":{"iothub_hostname":"itdeiot-de-08.rsu.local","port_number":443,"proxy":null}},"host-connect-iothub-mqtt":{"result":{"result":"error","details":["Failed to do TLS Handshake, Connection Attempt Timed out in 70 Seconds","deadline has elapsed"]},"additional_info":{"iothub_hostname":"itdeiot-de-08.rsu.local","port_number":8883,"proxy":null}},"host-local-time":{"result":{"result":"warning","details":["Could not query NTP server","could not resolve NTP pool hostname: failed to lookup address information: Name or service not known","failed to lookup address information: Name or service not known"]},"additional_info":{"offset":null}},"hostname":{"result":{"result":"ok"},"additional_info":{"config_hostname":"x3439-07861-ipc1.rsu.local","machine_hostname":"x3439-07861-ipc1.rsu.local"}},"identity-certificate-expiry":{"result":{"result":"ignored"},"additional_info":{"certificate_info":null,"provisioning_mode":"manual-other"}},"identityd-config-well-formed":{"result":{"result":"ok"},"additional_info":{}},"identityd-running":{"result":{"result":"ok"},"additional_info":{}},"key-pairs-read":{"result":{"result":"ok"},"additional_info":{}},"keyd-config-well-formed":{"result":{"result":"ok"},"additional_info":{}},"keyd-running":{"result":{"result":"ok"},"additional_info":{}},"local-ca-certificate-expiry":{"result":{"result":"ok"},"additional_info":{"certificate_info":{"cert_id":"local-ca","cert_name":"Local CA","not_after":"2029-09-09T13:06:17Z","not_before":"2024-09-09T13:06:17Z"}}},"parent_hostname":{"result":{"result":"ok"},"additional_info":{"config_parent_hostname":"itdeiot-de-08.rsu.local"}},"proxy-settings":{"result":{"result":"ok"},"additional_info":{}},"tpmd-config-well-formed":{"result":{"result":"ok"},"additional_info":{}},"tpmd-running":{"result":{"result":"ignored"},"additional_info":{}}}}

Device Information

• Host OS: Oracle Linux Server 9.6
• Architecture: amd64
• Container OS: Linux containers (mostly alpine linux)

Runtime Versions

• aziot-edged: 1.5.21
• Edge Agent: 1.5.27
• Edge Hub: 1.5.27
• Docker/Moby: 28.5.1-1

Logs

Support bundles for both parent and child devices
support_bundle_child_2025_10_30_12_35_12_UTC.zip

support_bundle_parent_2025_10_30_12_35_22_UTC.zip

Additional Information

Please provide any additional information that may be helpful in understanding the issue.