Random IO errors when booting

[jwbonner]

Random when booting, the system can end up in a state with several IO related issues:

• No IMU data is published (the iodaemon NT client is not present)
• No battery voltage data is sent to the DS, though it otherwise claims to be connected
• When enabling the robot, the RSL does not flash

Thus far we've only seen this on our new SystemCore unit (see #178). It seems to happen maybe 20% of the time and can be fixed by rebooting. No faults are displayed in the web UI when this occurs, including for "I/O". I've also attached the log from iodaemon when this happens, which includes some type of USB error. We've been able to recreate this on alpha 5 (with a fixed Mrccomm binary because of #176) and on alpha 7.

iodaemon.log

[rzblue]

Could you also show the logs from limelight_picoflasherprocess.service, and the dmesg logs?

[jwbonner]

picoflasherprocess.log

dmesg.log

[danursuline]

We were able to get a video, but no logs. If it happens again we will see if we can check dmesg, etc .

https://drive.google.com/file/d/1CagfQWJesO2ZINf-7MK0ut2X8_gRKXkU/view?usp=drivesdk

We also noticed that on the display the CAN had an X next to it.

[Alone1206]

This also happens if the device loses power for a moment and regains it during boot-up, causing it to restart before completing the boot process. We’ve observed this occur about 5-10 times out of 100 power cycles. When this happens our screen looks almost the same as what danursuline posted above.

[bhjelstrom]

Thanks for all of the updates. It looks like there's a software-only fix for this.