Device to Cloud Connectivity Issues after migrating from IoTEdge 1.0.10.4 to 1.5

[bmasotta]

Expected Behavior

We have a running iotedge solution running in arm32v7 + docker in docker setup.
This solution has 2 custom modules that connects to the local edge hub by using CreateByEnvironment() (module client)
One of the modules (let's call it "telemetry edge module TEM") receives incoming telemetry, connects the associated azure iot child devices (device client) and sends it's telemetry/twin updates.
This was working fine in the old iotedge version 1.0.10.4.

Child Devices are created inside TEM module as follows:

var authMethod = new DeviceAuthenticationWithRegistrySymmetricKey(deviceData.DeviceId, deviceData.SimmetricKey);
return DeviceClient.Create(edgeIoTHubData.HostName, edgeIoTHubData.GatewayHostName, authMethod, null);

Current Behavior

This was working ok until migrating from IoTEdge from 1.0.10.4 to 1.5
After migration we started to get timeouts all the time. The TEM module gets disconnected from edgeHub
Edge hub reports missed acks from the module. And the module reports timeouts both for twin updates and device to cloud messages.

Steps to Reproduce

Normal operation. No special steps

Context (Environment)

6 simulated devices connected in the TEM module using the edge hub as gateway.
each device sends to the cloud received telemetry and twin patch (at every 10 seconds)

Output of iotedge check

35 check(s) succeeded.
2 check(s) were skipped due to errors from other checks. Re-run with --verbose for more details.
(running verbose does not add additional details)

Timeouts and Disconnections related to custom module

the module that disconnects or presents the problems is the one that does device to cloud messaging
changed protocol from mqtt to amqp, seemed to improve just a little
there's another module that does not present problems, generates telemetry and send to this first module by the use of a route
(no cloud connectivity)

Device Information

• Host OS: mLinux 6.3.5
• Architecture: Arm32v7
• Container OS: arm32v7/ubuntu:22.04 (dind container)
• Docker on Host: 19.03.15

Runtime Versions

• aziot-edged: iotedge 1.5.21
• Edge Agent: 1.5
• Edge Hub: 1.5
• Dind container Docker version 28.5.1-1, build e180ab8ab82d22b7895a3e6e110cf6dd5c45f1d7

Logs

aziot-edged logs

no errors found

edge-agent logs

no errors found

edge-hub logs

"Unable to send twin update
<6> 2025-11-10 19:20:08.805 +00:00 [INF] - Removing device connection for device e33790d8-1c96-4319-a6c8-0e57645223a1 with removeCloudConnection flag 'True'.
<6> 2025-11-10 19:20:08.946 +00:00 [INF] - Closing link Events for 0ba66e6f-fec0-4b7c-b035-4215def394b5
<6> 2025-11-10 19:20:09.014 +00:00 [INF] - Closing receiver in cloud proxy ed3f408d-d43b-47d1-b653-ab552289b6c0 for e33790d8-1c96-4319-a6c8-0e57645223a1
<6> 2025-11-10 19:20:09.029 +00:00 [INF] - Closed cloud proxy ed3f408d-d43b-47d1-b653-ab552289b6c0 for e33790d8-1c96-4319-a6c8-0e57645223a1
<6> 2025-11-10 19:20:09.043 +00:00 [INF] - Device connection removed for device e33790d8-1c96-4319-a6c8-0e57645223a1
<6> 2025-11-10 19:20:09.071 +00:00 [INF] - Remove device connection for device e33790d8-1c96-4319-a6c8-0e57645223a1
<4> 2025-11-10 19:20:09.084 +00:00 [WRN] - Unable to send twin update to e33790d8-1c96-4319-a6c8-0e57645223a1 because ModuleMessages link was not found.
<4> 2025-11-10 19:20:09.101 +00:00 [WRN] - Error processing message in TwinReceiving link for e33790d8-1c96-4319-a6c8-0e57645223a1
Microsoft.Azure.Amqp.AmqpException: 'receiver10945' is closed
   at Microsoft.Azure.Amqp.AmqpObject.ThrowIfClosed()
   at Microsoft.Azure.Devices.Edge.Hub.Amqp.LinkHandlers.ReceivingLinkHandler.ProcessMessageAsync(AmqpMessage amqpMessage)
<6> 2025-11-10 19:20:09.706 +00:00 [INF] - Closing link TwinSending for 0ba66e6f-fec0-4b7c-b035-4215def394b5

"Did not receive ack for message"
<4> 2025-11-10 19:20:29.509 +00:00 [WRN] - Did not receive ack for message fe7b6392-e300-475b-86e0-52da33428f11 from device/module 100ce16a-bba4-4f22-929f-4f5d49315d31/TEM
<4> 2025-11-10 19:20:29.524 +00:00 [WRN] - Error sending messages to module 100ce16a-bba4-4f22-929f-4f5d49315d31/TEM
System.TimeoutException: Message completion response not received
   at Microsoft.Azure.Devices.Edge.Hub.Core.Device.DeviceMessageHandler.SendMessageAsync(IMessage message, String input) in /mnt/vss/_work/1/s/edge-hub/core/src/Microsoft.Azure.Devices.Edge.Hub.Core/device/DeviceMessageHandler.cs:line 498
   at Microsoft.Azure.Devices.Edge.Hub.Core.Routing.ModuleEndpoint.ModuleMessageProcessor.ProcessAsync(ICollection`1 routingMessages, IDeviceProxy dp, CancellationToken token) in /mnt/vss/_work/1/s/edge-hub/core/src/Microsoft.Azure.Devices.Edge.Hub.Core/routing/ModuleEndpoint.cs:line 166

custom module logs (TEM)

Error while processing incoming message: System.TimeoutException: The operation timed out.  ---> Microsoft.Azure.Devices.Client.Exceptions.IotHubCommunicationException: Transient network error occurred, please retry.  ---> System.TimeoutException: The operation has timed out.    at Microsoft.Azure.Devices.Client.Transport.Amqp.AmqpTransportHandler.RoundTripTwinMessageAsync(AmqpTwinMessageType amqpTwinMessageType, TwinCollection reportedProperties, CancellationToken cancellationToken)    at Microsoft.Azure.Devices.Client.Transport.Amqp.AmqpTransportHandler.SendTwinPatchAsync(TwinCollection reportedProperties, CancellationToken cancellationToken)    at Microsoft.Azure.Devices.Client.Transport.ErrorDelegatingHandler.<>c__DisplayClass27_0.<<ExecuteWithErrorHandlingAsync>b__0>d.MoveNext() --- End of stack trace from previous location ---    at Microsoft.Azure.Devices.Client.Transport.ErrorDelegatingHandler.ExecuteWithErrorHandlingAsync[T](Func`1 asyncOperation)    --- End of inner exception stack trace ---    at Microsoft.Azure.Devices.Client.Transport.ErrorDelegatingHandler.ExecuteWithErrorHandlingAsync[T](Func`1 asyncOperation)    at Microsoft.Azure.Devices.Client.Transport.RetryDelegatingHandler.<>c__DisplayClass37_0.<<SendTwinPatchAsync>b__0>d.MoveNext() --- End of stack trace from previous location ---    at Microsoft.Azure.Devices.Client.TransientFaultHandling.RetryPolicy.<>c__DisplayClass34_0.<<RunWithRetryAsync>g__TaskWrapper|0>d.MoveNext() --- End of stack trace from previous location ---    at Microsoft.Azure.Devices.Client.TransientFaultHandling.RetryPolicy.RunWithRetryAsync[T](Func`1 taskFunc, ShouldRetry shouldRetry, Func`2 isTransient, Action`3 onRetrying, Boolean fastFirstRetry, CancellationToken cancellationToken)    at Microsoft.Azure.Devices.Client.TransientFaultHandling.RetryPolicy.RunWithRetryAsync(Func`1 taskFunc, ShouldRetry shouldRetry, Func`2 isTransient, Action`3 onRetrying, Boolean fastFirstRetry, CancellationToken cancellationToken)    at Microsoft.Azure.Devices.Client.Transport.RetryDelegatingHandler.SendTwinPatchAsync(TwinCollection reportedProperties, CancellationToken cancellationToken)    at Microsoft.Azure.Devices.Client.InternalClient.UpdateReportedPropertiesAsync(TwinCollection reportedProperties)    --- End of inner exception stack trace ---    at Microsoft.Azure.Devices.Client.InternalClient.UpdateReportedPropertiesAsync(TwinCollection reportedProperties)    at .Device.SetReportedPropertiesPatch(String jsonPatch) in