I am trying to implement the following functionality using the Python logging package:
log messages to the flash drive
if the flash drive is not available, switch handler to the BufferringHandler and log into buffer,
once the flash drive is plugged in and available store the logs from BufferHandler into that flash drive and switch the handler into RotateFileHandler.
Which approach would you suggest to use while implementing this functionality?
One that come into my mind is to have one process or
thread to check periodically if the flashdrive is available, and have a flag that indicates if we can store the logs into the flash drive. But I don’t particularly like this approach.
Would you do it different way? Maybe someone has already done something similar and can submit an example of the code?
Are the application’s requirements that it has to support a flash drive being removed/added any time? That would be tricky!
If you just have to worry about it on application start whether a flash drive exists, you could detect if there’s a flash drive and write to it through whatever logger you have in mind.
If there isn’t a flash drive attached, the internal file system can store some data (Not much though, as its also taken up by the python application that’s doing the storing).
Anyway you do it, I’d be careful using the RotateFileHandler from the python library. I’ve used on low I/O bandwidth environments and due to its buffering aspect, it tends to do a lot of writes to flash relative to other file handlers. This isn’t a big deal if your application doesn’t write logs very often, but if it does, you may get into a situation where you are logging messages faster then the file I/O can keep up with.
The situation of detecting flashdrive at the boot time is easy. The actual functionality requires support for flash drive being removed/added any time.
I know this is tricky. Therefore I am asking if there is another way to do it than using threads.
Regarding the logging times it would probably 2-3 logs every second or if needed than I can change it to every 2 seconds.
You would need at least a semaphore (a threading lock).
I would think you would need to close the file between every write - just yanking a flash drive with an open file could cause corruption. It almost seems to me you would need two files, which you write to every time one by one, thus yanking out the flash during write would only corrupt one copy - but you won’t know which until you looked at it on another computer.
The data loggers I have done all “buffer” to RAM for a few minutes, then open the flash file and write a single large binary block in a single write, then close the file. They also are NOT a thread, but a semaphore protected module several tasks share.
If power is lost, my few minutes of log will be lost, but assuming the power remains off for hours - one loses hours of data, not just those minutes so worrying out it tends to be crocodile tears.