Add 60s watchdog for nrf52 to prevent stuck nrf devices like RAK4631, RAK3401

[weebl2000]

In order to restore after battery voltage sags too low. Or any other reason the device might hang for that matter.

In the process of being tested, keeping it as draft until more results are in. Tagging @IoTThinks too.

How to test: ideally with a power supply, first boot at say 3.6V, then drop voltage down to 1.5-2.0V, your device will brown out, and hopefully come back to life if you up the voltage to 3.6V again.

Chose to set runtime voltage about 200mV below the bootlock voltage.

Build this firmware for your device HERE

possibly related issue: #1974

References:

• https://devzone.nordicsemi.com/nordic/nordic-blog/b/blog/posts/optimizing-power-on-nrf52-designs interesting article about the power design on nrf52.

[IoTThinks]

I believe watchdog is a mitigation fix.
In your opinion, what is the root cause in NRF52?
Let me check if old MeshCore (1.11...) has this issue.

ESP32 seems to recover well in this scenario.

[weebl2000]

I believe watchdog is a mitigation fix. In your opinion, what is the root cause in NRF52? Let me check if old MeshCore (1.11...) has this issue.

ESP32 seems to recover well in this scenario.

Yes, this is not the fix of the root cause.

[IoTThinks]

• https://devzone.nordicsemi.com/nordic/nordic-blog/b/blog/posts/optimizing-power-on-nrf52-designs interesting article about the power design on nrf52.

Nice read.
I love to understand the root cause and "one fix one kill" :D

I guess this issue may due to NRF52 limitation or bootlock or even I2C sensors.

[ipduffy]

In src/helpers/nrf52/SerialBLEInterface.cpp in the onConnect function, the _isDeviceConnected flag is kept as false during the initial BLE connection (line 32). If the onSecured function does not succeed (which can happen due to various reasons) then the _isDeviceConnected flag will never be set to true, and the device will enter a limbo state where it never fully connects successfully. The watchdog loop that runs (lines 334-346) only runs if the connection_handle is BLE_CONNECTION_INVALID, which it would not be in this particular case. Recommend setting a timeout on the onSecured function and calling disconnect if the timeout is exceeded.

[ipduffy]

In src/helpers/nrf52/SerialBLEInterface.cpp in the onConnect function, the _isDeviceConnected flag is kept as false during the initial BLE connection (line 32). If the onSecured function does not succeed (which can happen due to various reasons) then the _isDeviceConnected flag will never be set to true, and the device will enter a limbo state where it never fully connects successfully. The watchdog loop that runs (lines 334-346) only runs if the connection_handle is BLE_CONNECTION_INVALID, which it would not be in this particular case. Recommend setting a timeout on the onSecured function and calling disconnect if the timeout is exceeded.

I submitted a PR to address this in #2089

[weebl2000]

In src/helpers/nrf52/SerialBLEInterface.cpp in the onConnect function, the _isDeviceConnected flag is kept as false during the initial BLE connection (line 32). If the onSecured function does not succeed (which can happen due to various reasons) then the _isDeviceConnected flag will never be set to true, and the device will enter a limbo state where it never fully connects successfully. The watchdog loop that runs (lines 334-346) only runs if the connection_handle is BLE_CONNECTION_INVALID, which it would not be in this particular case. Recommend setting a timeout on the onSecured function and calling disconnect if the timeout is exceeded.

I submitted a PR to address this in #2089

Can you open a PR to my fork? I can merge into the existing PR that way. I'm very busy this week and the next but I'll try to have a look soon.

(only if the PR is related BTW, I haven't looked into the remarks yet 😬😅)

[ipduffy]

In src/helpers/nrf52/SerialBLEInterface.cpp in the onConnect function, the _isDeviceConnected flag is kept as false during the initial BLE connection (line 32). If the onSecured function does not succeed (which can happen due to various reasons) then the _isDeviceConnected flag will never be set to true, and the device will enter a limbo state where it never fully connects successfully. The watchdog loop that runs (lines 334-346) only runs if the connection_handle is BLE_CONNECTION_INVALID, which it would not be in this particular case. Recommend setting a timeout on the onSecured function and calling disconnect if the timeout is exceeded.

I submitted a PR to address this in #2089

Can you open a PR to my fork? I can merge into the existing PR that way. I'm very busy this week and the next but I'll try to have a look soon.

(only if the PR is related BTW, I haven't looked into the remarks yet 😬😅)

Sure thing, do you want me to PR to the nrf52-watchdog-brownout branch?

[weebl2000]

Sure thing, do you want me to PR to the nrf52-watchdog-brownout branch?

Yes, sounds great!

[MGJ520]

Everyone, I also encountered a mysterious crash on one of my solar forwarding nodes, which happened almost within an hour. It may be related to voltage, but meshtatsic is running well.

meshtastic/firmware#9699

meshtastic/firmware#8485

meshtastic/firmware#7882

meshtastic/firmware#5909

meshtastic/firmware#5491

[weebl2000]

So far no issues reported/observed.

[dz0ny]

Both the E1000 and L1 tracker as they appear to freeze from time to time, wdt is really needed feature.

Currently, the only way to recover the E1000 from a freeze is by forcing it into DFU mode.

[fschrempf]

Related PR: #1417

[IoTThinks]

So this issue is created by NRF52 power management?

[dz0ny]

So this issue is created by NRF52 power management?

No.

Freeze is created by something in the stack or app, this ensures that watchdog resets the device. I observed freeze multiple times on relay where interrupt(TX) was just stuck in sending state. So not low power regime.

[IoTThinks]

@dz0ny The fix seems to be in the code of NRF52 Power Management.
That's why I have some doubts.