Memory Tips

I’m writing some python scripts for a ConnectPort X2 with 8mb memory.

When the ConnectPort boots it takes 5mb leaving me with 3.

I’ve written a very simple script that reads an XML settings file using ElementTree. I’ve packaged my script module and the ElementTree library (only the single file that I need) into a zip file and loaded it onto the server.

When I run the script, just adding my 17k zip file to the system path (via sys.path.append(…) ) and importing my settings class takes up 2.5mb of memory. I haven’t even parsed the xml file yet!

Basically I’m running out of memory and haven’t even done any real work yet.

I tried running the garbage collector but that did not help.

Any idea what is taking up so much memory and how to reduce it?

Hi bobshort,

I took a quick comparison of on boot memory and after loading the default python.zip, and it consumes approximately 500k of memory. My first impression is the other memory being used is from the settings class your are importing.

If possible, you may want to use a memory profiler such as pysizer (http://pysizer.8325.org/) to determine where the memory is being consumed. Once you know where it is being used, you will have a number of approaches to deal with it, such as pruning down the library to only include used functionality, or using a XML parser with a smaller memory footprint.

Hope this gives you a direction to go to!

This is the downside to the wide availability of “common public tools”. I was also on a project which parses XML, and the common Python libraries require up to 2MB+ just to parse a 40K text file.

You could try the pyexpat XML parser - which was the solution we picked in that other project (it only required a little over 1MB to parse the 40-60K source).

Plus as Max suggests, the very instant you are done with the parser you need to del (free) all resources and run garbage collection.

If your XML is small, the CHEAT I would use is just walk through it as a TEXT file, using ‘split()’ to break out the tokens manually. It won’t be as sexy a result as a true XML tree parse, but if you only have a dozen elements, then creating a while loop to scan for those 12 terms the old-fashioned way takes very little memory.

Thanks for the reply!

I tried using ConfigParser to parse a “.ini” style of configuration file rather than XML. That used up 1.5mb rather than 2.5mb. It is still a lot but is significantly better.

What are are other people using for storing config settings? I was thinking about trying a simple csv file that I parse without any extra libraries. I’m interested in hearing what other people have had success with.

I also need to push data over an SSL socket. I’m afraid it is going to be memory and CPU intensive. Does anyone have experience using sockets with SSL on the connectport?

One of the ways I have done configuration files in the past is using the python interpreter for it. Since you are using already the python interpreter, it incurs little extra memory usage beyond what is defined in the file. You also then have direct access to all the types that one would normally have, and allow some extra pre-processing of the data if you require it.

The downside to that is if people not familiar with python syntax may not be able to work with it, depending on the complexity of the file. With that said, here’s a brief example:

config_file.py:

data1 = ‘hello’
data2 = 127.1234567
data3 = [‘a’, ‘b’, ‘c’, ‘d’]
data4 = foo(‘Init’)
data5 = {parameter1:1, parameter2:2}

Where you would load the settings:

import config_file
print config_file.data1

Once you are done loading the configuration file, you could de-allocate the memory using the ‘del’ operator on the config_file object. Then follow up with a garbage collection, and it should all be freed.

The overall amount of memory used will of course depend on how many settings you have, but using configuration files with a dozen or so strings and floats/ints defined take up a fairly negligible amount.

Thanks for the tips guys!

I think using an interpreted python settings file should do the trick.

Coming to embedded python with 3mb of memory from server side Java/J2EE with gigs and gigs of memory is a bit of a paradigm switch. It certainly is exposing a lot of my bad habits :slight_smile: