Data aquisition of a real-time bursty transient in Dia

I have an data aquisition application that will need to sample an analog channel (either local or xbee), at a reasonably slow rate of 1 sample per second (to save power), then bump up to a faster rate of around 5-10 samples per second when the value is determined to be above a configurable trip-point. Data needs to be sampled and cached until the value goes back down below a second trip-point.

I then need to perform a few basic statistical analysis of the cached data to compute its average/max/etc… This statistical data metric should then be sent upstream to be handled by a presentation module and eventually pushed up to something like iDigi.

A secondary requirement is that the entire data log cache be available to upload for diagnostic purposes and possibly a heavier statistical regression analysis.

In my case, this burst of data will last around 2minutes which will result in around 1-2K+ discrete samples. The time until the next cycle of data is totally random. Typically it will be minutes or even hours. But it could be a few seconds later. I really dont want to push 2000 data samples when a single metric or two will suffice.

One could simply sample every 100msec and then use a presentation module such as idigi_db or idigi_upload to push the entire cached data to the cloud. This would result in a LOT of data that is not typically needed to be sent. These two presentation modules are also based on a sampling TIME instead of channel VALUE.

I am new to Digi/Dia/iDigi and am just learning what it is and what is available. I have created a python-level script to read the local analog input from a X4H into a list, calculate its average, and push that to idigi using the idigi_data command. It seems like with a little work, that approach will work.

But eventually I will probably want to be able to read the values from an Xbee wireless sensor device. My options, as I see it are to either

  1. figure out how to do that from Python

  2. create a custom Dia driver(s)
    2A) custom device
    2B) custom translator device
    2C) custom logger
    2D) custom presentation

  3. create a custom end-point programmable XBee remote device

I am reasonably sure that I can do #1 if I keep dissecting the code samples I am finding scatted around the Digi site. (but suggestions or example/demo code would be appreciated)

I also think that #3 may be the ultimate solution because it would reduce the packets on the xbee PAN, but first things first. I need to work my way to that solution.

Which leaves #2… I am currently looking into modifying the idigi_upload presentation driver sample to change it from TIME-based to EVENT-based. I would still have to do the statistical calculations on a list of samples received from the Xbee AIO and Local AIO device drivers. This is a admittedly a simplistic approach.

Questions:

  1. Has anyone on this forum done something similar to this?

  2. Where does one usually put this type of algorythmic manipulation of locally collected data samples in the DIA environment?

  3. Is there a similar function demo or sample available somewhere in the Digi support collateral?

Thanx, Gary

(digi_upload reference link: http://www.digi.com/wiki/developer/index.php/Dia_Event_Uploader )
(digi_upload related forum link: http://www.digi.com/support/forum/viewthread_thread,8302 )

Hi gwebbtlc,

I have done similar project which is already in production testing phase. The application is not reading analog values but digital one (Xbee DIO). My sample rate is 500ms because I have input with changing rate of 1 second.

For application I use modified xbee_dio.py and idigi_db.py files. I took idigi_db because it has two conditions for sending data away. First is number of samples and the other is timeout. I have modified upload protocol with XML-RPC for my application on Linux.

I have already tested a lot but, still have some issues with timing. It seems that time() function is time consuming and should be used wisely. See my problem http://www.digi.com/support/forum/viewthread_thread,8290.

Buffering on ZB Xbee module is not available. It is possible on 802.15.4 firmware but then you can’t use iDigi DIA framework. The only way is to use Programmable Xbee module and program the module so the samples are buffered on Xbee and then send to the GW. The buffer size on Xbee is limited. I believe some 40 samples can be packed in single ZigBee frame.

Boris