[Errno 7023] ENFILE when calling update_datastream or add_datapoint

Digi Remote Manager Web Services (API)
, , ,
1

I’ve taken the drm_http_requests example for Xbee3 LTE-M/NB-IoT modules on pyCharm, which worked perfectly, and attempted to modify it for my application.

I’m trying to write to 3 data streams per packet received via UART. I receive these packets at about 1 per minute.

I extract from each packet 1 device_id and 3 stream_ids. Each device_id and stream_id could be unique or could already exist. I then sequentially call update_datastream and add_datapoint to for each stream_id. The goal is to add one datapoint to each stream.

Works great about 70% of the time. The other 30% I receive Errno 7023 ENFILE. I’ve also occasionally seen 7005 EIO. The error can occur on either update_datastream or add_datapoint. Most of the time I’ve been able to just retry the operation and it succeeds on the retry. I’ve seen instances where the code halts execution entirely though, and this is of grave concern.

Documentation on this Errno indicates a socket leak, and with the frequency of the error I suspect its happening most/all of the time.

What is the cause of the socket leaking and how can I prevent it using the remotemanager library?

Here is the portion of the code that uploads to DRM. It’s really just 3 sets of update_datastream and add_datapoint with retries for both operations.

        # Datastream 1
        try:
            rm.update_datastream(stream_id1, stream_info1)
            print("Updated Data Stream 1")

            try:
                status = rm.add_datapoint(stream_id1, value1)  # post data to Device Cloud
                print("Data Uploaded")
            except Exception as e:
                status = type(e).__name__ + ': ' + str(e)
                print(status)

                sleep(1)

                rm.update_datastream(stream_id1, stream_info1)
                print("Updated Data Stream 1")

                try:
                    status = rm.add_datapoint(stream_id1, value1) 
                    print("Data Uploaded")
                except Exception as e:
                    status = type(e).__name__ + ': ' + str(e)
                    print(status)

        except Exception as e:
            status = type(e).__name__ + ': ' + str(e)
            print('

exception:', e)

            sleep(1)

            rm.update_datastream(stream_id1, stream_info1)
            print("Updated Data Stream 1")

            try:
                status = rm.add_datapoint(stream_id1, value1)  # post data to Device Cloud
                print("Data Uploaded")
            except Exception as e:
                status = type(e).__name__ + ': ' + str(e)
                print(status)

        sleep(1)
        # Datastream 2
        try:
            rm.update_datastream(stream_id2, stream_info2)
            print("Updated Data Stream 2")

            try:
                status = rm.add_datapoint(stream_id2, value2)  # post data to Device Cloud
                print("Data Uploaded")
            except Exception as e:
                status = type(e).__name__ + ': ' + str(e)
                print(status)

                sleep(1)

                rm.update_datastream(stream_id2, stream_info2)
                print("Updated Data Stream 2")

                try:
                    status = rm.add_datapoint(stream_id2, value2) 
                    print("Data Uploaded")
                except Exception as e:
                    status = type(e).__name__ + ': ' + str(e)
                    print(status)

        except Exception as e:
            status = type(e).__name__ + ': ' + str(e)
            print('

exception:', e)

            sleep(1)

            rm.update_datastream(stream_id2, stream_info2)
            print("Updated Data Stream 2")

            try:
                status = rm.add_datapoint(stream_id2, value2)  # post data to Device Cloud
                print("Data Uploaded")
            except Exception as e:
                status = type(e).__name__ + ': ' + str(e)
                print(status)

        sleep(1)
        # Datastream 3
        try:
            rm.update_datastream(stream_id3, stream_info3)
            print("Updated Data Stream 3")

            try:
                status = rm.add_datapoint(stream_id3, value3)  # post data to Device Cloud
                print("Data Uploaded")
            except Exception as e:
                status = type(e).__name__ + ': ' + str(e)
                print(status)

                sleep(1)

                rm.update_datastream(stream_id3, stream_info3)
                print("Updated Data Stream 3")

                try:
                    status = rm.add_datapoint(stream_id3, value3) 
                    print("Data Uploaded")
                except Exception as e:
                    status = type(e).__name__ + ': ' + str(e)
                    print(status)

        except Exception as e:
            status = type(e).__name__ + ': ' + str(e)
            print('

exception:', e)

            sleep(1)

            rm.update_datastream(stream_id3, stream_info3)
            print("Updated Data Stream 3")

            try:
                status = rm.add_datapoint(stream_id3, value3)  # post data to Device Cloud
                print("Data Uploaded")
            except Exception as e:
                status = type(e).__name__ + ': ' + str(e)
                print(status)

    # wait between cycles
    sleep(1)
    gc.collect()  # Added on 11/9 to attempt to fix ERRNO 7023 (unsuccessful)
2

See https://www.digi.com/resources/documentation/Digidocs/90002258/#troubleshooting/ts_socket_leaks.htm?TocPath=Troubleshooting%257C_____10

3

I’ve already gone over this document and determined it appears to be a socket leak. How is this condition of pending RX data occurring in my application and what do I need to do to prevent it?

That document also says I need to perform a hard/soft reset to fix the issue, but that doesn’t appear to be the case either. I can just retry and succeed.

4

Try opening the socket and keeping it open instead of opening a new one each time. That can be done using the timeout values.

5

I’ve added into the urequests library s.close() and gc.collect() at the end of each call to request().
So the socket is opened and closed each call to request. Any way you can elaborate as to why is this the wrong way to do it or what exactly is causing these exceptions? (7005 EIO and 7023 ENFILE)
What you’re suggesting is I rewrite/modify the urequests library Request() function?
Isn’t this a pretty standard and widely used library?

Perhaps I’m mistaken, but there must be many applications similar to mine that utilize this library as is.

I’m a little worried I’m going to go through the work rewriting/modifying the urequests library only to run into the same exceptions.