Hello all, anyone have issues with their Xbee 3 LTE-M/NB-IoT modems failing in the field? I have a few running L0.0.00.00.05.11,A.02.16 and Digi firmware 11417 that have stopped transmitting in the field. Upon checking the modem with an XBIB-CU-TH board, all cellular-related data is blank whereas before it was all there. Checked the SIM in another known, working modem and it’s fine. When issuing a AT+CGSN results in “ERROR” along with all the other cellular network-related commands.
Also it is known that the modem can get in to an error state if a “Safe shutdown” isn’t initiated prior to pulling power. I am using this modem in micropython mode and am issuing an “Airplane Mode” before shutting the modem down. Would like to know why this is happening and how to go about troubleshooting the issue if it only occurs every couple of months…
Indeed it was “bricked”, meaning no cellular-related info to include the IMEI and it’s not recoverable through a ublox firmware re-load. As for the problem, we are being told by Digi that all modems that end up in this state are due to messed up flash memory caused by an unsafe shutdown.
We were using the “Airplane Mode” to safely shut down the modem as outlined in the Xbee manual but evidently that’s not cutting it. The only actual working method of safely shutting down the modem is to use the “SD” command which is contrary to the manual.
Unfortunately Digi doesn’t see this issue as a real problem because only 0.01% have failed this way. This just tells me that most of their users are not cyclically powering down their modems to save power. If they were, they’d have a lot more of these failures.
Due to the nature of Flash memory, the Embedded File System (EFS) may incur memory corruption if
the safe & graceful shutdown procedure outline in the SARA-R4 series system integration manual [3]
is not followed. It is possible that memory corruption may result in the device being inoperable or
impact specific device data info.
Unfortunately, this particular model, SARA-R410M-02B-01, used in the Digi LTE-M/NB-IoT modem doesn’t seem to have the EFS backup and restore feature. I am guessing when Digi picked the u-blox module for their modem design, they probably weren’t aware of or paying enough attention on this issue.
I can confirm your claim. I have an entire box of bricked XB3 LTE-M/NB-IoT modules. We have many instances where the python program will freeze up and when at this point, the only solution is to reset the device. The python watchdog doesn’t appear to work when we hit this frozen state. The freezing appears to be encountered with some sort of cellular issue (poor signal?). It’s difficult to diagnose bc we can’t see precisely what the cause is between python and the Sara modem. We have had many troubles trying to figure out a way to safely, and remotely, reset the devices when the software freezes up. Our devices are never near a human that can reset them manually. Sometimes, it’s a power cycle that is the only solution. Most of the time, simply power cycling the device doesn’t hurt anything, but every now and then, the issue that is the topic of this thread, bricks the XB3.
The other problem we had is that all of our devices are solar powered and for obvious reasons, every now and then we have a low power condition (a car parked on the solar panel, snow, etc). We monitor the power levels and send the XB3 into deep sleep for long periods to conserve power. However, in some cases when a physical condition is impeding the ability to charge the battery, the battery level is so low, that when the device wakes up for the day, it enters what I can a “doom loop” at startup. The XB3 starts its normal routine, but as soon as the cellular radio is powered, the XB3 draws maximum rated current and there just isn’t enough power left in the battery and the XB3 essentially “resets” again. Causing a doom cycle of power resets. This condition is sure to scramble and corrupt the memory.
Does anyone have any tricks to tackle freeze-ups and how to reset the device without having a human physically interact? I did add a timeout to the MQTT library - that seemed to help a lot.
Any ideas for keeping the XB3 shut down in the event of inadequate power? We set up ours to allow them to wake only once a day, but say the weather cools and temperature drops, what was an OK power situation quickly devolves into a no-go situation - this can happen over a few hours. I’ve considered adding a secondary watchdog, low voltage ICs, but many of these ideas, which I have physically tested, consume too much additional power and add cost. I would love some suggestion or methods other have employed. I would buy someone lunch or lunches to discuss. I find myself on a metaphorical island on these technical topics.