CC6 App trashing File System

We have had an App running on our custom CC6 based hardware for over a year pretty succesfully.

One or two occasional hangups requiring reboots but certainly not too bad for our beta site.

We have recently made some minor changes to the yocto build, mostly tidying up.

We have made quite a few changes to our application.

Over the past week our newest batch of 5 hardware devices, newly built and flashed and with the latest app installed have ‘bricked’ themselves under ‘normal’ operation. A sudden lockup and the device becomes unresponsive.

Resorting to connecting a USB terminal ( IP USB stops being available ) and doing a hardware reset brings up a badly corrupted root file system. This prevents any booting beyond displaying the Linux penguin on our LCD graphic display.
There is no response to trying to interrupt the boot to allow reflashing from an SD card and the boot eventually freezes. Bricked.

The flash image appears to be ok. works for extented development sessions. The freexe appears to be random in response to various or general IO.

How do we recover these units?
How do we even begin to work out where the fault is?

pics of the fails -

Do you have access to serial console? You might be able to stop the boot by pressing any key when u-boot starts, this should drop you down to u-boot prompt. If this is disabled then Does your hardwre have bootstrapping options? If so bootstrap to boot from SD card or USB:

Sadly although I can connect to the serial console, no amount of key bashing has managed to drop into u-boot.

It looks like I’m going to have to resort to boot from USB as our board only has that option. Hopefully I can then rebuild the system

that is really only problem #1 as it is not a real solution to have hardware that may well do it again in a day … or a month.