Problem using HTTPLIB to POST to a remote server on an X2 [httplib.request()]

Python
1

Greetings all,

First off thanks to ‘descalon’ for answering the last question I had. I’m moving this question to a different thread for cleanliness, since it’s unrelated to the last problem I had.

Unfortunately I’ve come across another issue, this one a bit more devious to solve. I’ve tried resolving it on the X2 to no avail, so I’m hoping the X4 that’s being delivered (hopefully today) will fix the problem.

I am trying to perform an HTTP post to a remote server with the following code [“packet” is a set of binary data, but not where the problem is]:


headers = {"Content-type": "application/octet-stream"} 
conn = httplib.HTTPConnection("xxx.xxx.xxx.xxx:80") 
url = "/DData/Data.aspx"
conn.request("POST", url, packet, headers)

obviously I’ve hidden the IP address to post to. This code runs fine on my PC, however if I try to run it by hand on the Connectport X2 I see the following error:


>>> conn.request("POST", url, packet, headers)
Traceback (most recent call last):
  File "", line 1, in ?
  File "WEB/python/embedded_kit_gateway.zip/httplib.py", line 804, in request
  File "WEB/python/embedded_kit_gateway.zip/httplib.py", line 821, in _send_requ
est
  File "WEB/python/embedded_kit_gateway.zip/httplib.py", line 752, in putrequest

LookupError: unknown encoding: idna

I realize that means that the IDNA encoding module is not being loaded. After looking through the httplib.py code identified by the error, the following line is the problem:

self.putheader('Host', "%s:%s" % (self.host.encode("idna"), self.port))

I’d hate to think that the X2 isn’t able to support the HTTPLIB library, even with the dependencies loaded.

The contents of embedded_kit_gateway.zip include the idna.py module and all of the modules needed to run HTTPLIB, and I’ve included “sys.path.append(“WEB/python/embedded_kit_gateway.zip”)” in the code, however the module is still not loading. I tried loading the IDNA module by hand with all the correct dependencies loaded as well (stringprep.py and unicodedata.py) and I received the following message:


>>> import idna.py
Traceback (most recent call last):
  File "", line 1, in ?
  File "WEB/python/mymodules.zip/idna.py", line 3, in ?
  File "WEB/python/mymodules.zip/stringprep.py", line 8, in ?
  File "WEB/python/mymodules.zip/unicodedata.py", line 12, in ?
  File "WEB/python/mymodules.zip/unicodedata.py", line 10, in __load
AttributeError: 'module' object has no attribute 'load_dynamic'

I’m guessing this is why the IDNA module is not loading, but some sort of exception handling in the “encodings” module is suppressing the output of the IDNA module load error and continuing on without it.

I guess my primary goal is to successfully POST to a remote URL using the HTTPLib’s request() call. Does anyone have any experience with this? I’ve spent well over a day trying to figure out the correct dependencies, only to falter on the “AttributeError” which I can’t overcome. Any and all help would be most welcome.

1 Like
2

Hello John,

I have tried to reproduce your problem and my sample throws the same error as you have posted here. I’ve tried to find any further information in the Digi Wiki page but didn’t find anything related. If you import the encodings.idna in your python example the idna problem should be gone, but it will throw a new one related to the unicodedata module. Unfortunately it is a built-in module and the ConnectPort X firmwares don’t include it.

Anyway, I have found something that could help you. The iDigi Dia software package contains some custom Python libraries for Digi devices. One of them is an httplib module for Digi devices. I tried it with my example and didn’t get any import/encoding error, so maybe it works for you.

I’ve attached the library, copy it inside your project path and import it (import digi_httplib) instead the default one (import httplib).

Good luck!

1 Like
3

Thanks for the response. I did a DIFF between the digi_httplib and the standard one and sure enough, the ONLY difference is that digi_httplib removes the calls IDNA encodings (see below)

digi_httplib.py


                if netloc:
                    self.putheader('Host', netloc)
                elif self.port == HTTP_PORT:
                    self.putheader('Host', self.host)
                else:
                    self.putheader('Host', "%s:%s" % (self.host))

standard python httplib:


                if netloc:
                    self.putheader('Host', netloc.encode("idna"))
                elif self.port == HTTP_PORT:
                    self.putheader('Host', self.host.encode("idna"))
                else:
                    self.putheader('Host', "%s:%s" % (self.host.encode("idna"), self.port))

Looks like someone came across this problem before…

I tried it, and of course it worked. Thanks for the info. I’m not one for modifying standard libraries, so I actually came up with a workaround - after inspecting httplib.py, I realized that the way it builds the HTTP header packet is by reading what was provided in the header tag and adding what’s missing. The offending code above is run in the “host” portion of the HTTP header generator. It turns out that if you supply a “Host” tag in the header, this portion of the code is skipped and the POST to the server works fine. See below:


headers = {"Content-type": "application/octet-stream", [b]"Host" : ""[/b]}
conn = httplib.HTTPConnection("xxx.xxx.xxx.xxx:80")
url = "/DData/Data.aspx"
conn.request("POST", url, packet, headers)
response = conn.getresponse()
print response.status, response.reason
data = response.read()
1 Like