Memory Leak in ConnectPort X2B Program

Python
1

Hello everyone,

I have a small program running on my ConnectPort X2B, that collects Measurements from Zigbee Devices and writes them in the system flash drive(i know not a good idea, but i wanted to see results fast). When i have the program up and running the free memory of the system reduces constantly and after a day or two the system breakes down.
I am new to python programming so i suspect my program code has some kind of memory leak. Is there a debug option or something I should keep in mind?


from socket import *
from select import * 
import time
import os
import os.path

#Add a new Measurement to the temporary list
def add(addr,data,timestamp):
	#print "Adding Measurement"
	#print "Listsize:", len(meterdata[0])
	present = 0
	for i in range(len(meterdata[0])):
		if meterdata[0][i] == addr:
			present = 1
			meterdata[1][i] = data
			meterdata[2][i] = timestamp
	if present == 0:
		#print "New Meter!"
		meterdata[0].append(addr)
		meterdata[1].append(data)
		meterdata[2].append(timestamp)
	#print "Adding Measurement done!"


def printmeterdata():
	#print "Printing Measurements:"
	for i in range(len(meterdata[0])):
		print meterdata[0][i], " | ", meterdata[1][i] , " | ", meterdata[2][i] 


#Write Measurements to flash
def writetofile():
	FILE = open("WEB/python/data.py","w")
	for i in range(len(meterdata[0])):
		FILE.write("METER_START
")
		FILE.write("ADDR_START
")
		FILE.write(meterdata[0][i])
		FILE.write("
ADDR_END
")
		FILE.write("TIME_START
")
		FILE.write(meterdata[2][i])
		FILE.write("
TIME_END
")		
		FILE.write("DATA_START
")
		FILE.write(meterdata[1][i])
		FILE.write("
DATA_END
")		
		FILE.write("METER_END
")
	FILE.close()

#Socket Information
sd = socket(AF_ZIGBEE, SOCK_DGRAM, ZBS_PROT_APS)
sd.bind(("", 0xe8, 0, 0))
sd.setblocking(0)
#Insert Meters here
staticlist = []
staticlist.append(("02:11:a6:00:41:5d:19:dd!", 0x5e, 0xc105, 0x703))
#Static Variables
meterdata = [[],[],[]]

payload = "GET
"
erg = ""
stepcount = 1 

try:
#Lifeline to kill program if running
    FILEDUMMY = open("WEB/python/lifeline.py","w")
    FILEDUMMY.write("Delete me to stop me!")
    FILEDUMMY.close()

    if os.path.exists("WEB/python/exception.py"):
                os.remove("WEB/python/exception.py")

    while 1: 
        #print "step nr " , stepcount
	try:

		#ask static devices for measurements
		if stepcount % 10 == 0:
			for listelem in staticlist:
				#print "Sending GET to: " , listelem[0]
				rlist, wlist = ([], [])
				wlist = [sd]
				rlist, wlist, xlist = select(rlist, wlist, [])
				count = sd.sendto(payload, listelem)
	
		rlist, wlist = ([], [])	
	        rlist = [sd]
		#receive package
		erg, src_addr = sd.recvfrom(255)
		#print "Package from :"
		#print src_addr[0]		
		if erg[0:4] == "PID=":
			#print "I AM HERE Packet received!"
			rlist, wlist = ([], [])
			wlist = [sd]
			rlist, wlist, xlist = select(rlist, wlist, [])
			count = sd.sendto(payload, 0, src_addr)
		else:
			#print "Measurement received!"
			timestring = "%s " % time.localtime()
			add(src_addr[0],erg,timestring)
			#print "Measurement added!"
			#printmeterdata()
			#print "Data printed"
			writetofile()
			#print "File written!"

	except Exception, f:
		donothing = 1
		#print f

	time.sleep(0.1)

	#Stop if lifeline is gone
	if os.path.exists("WEB/python/lifeline.py"):
                dummy = 1
        else:   
                raise Exception, "Lifeline deleted!"

  	stepcount = stepcount + 1
	if stepcount < 0:
	    raise Exception, "End of program!"

except Exception, e:
    print e
    sd.close()
    if os.path.exists("WEB/python/lifeline.py"):
    	os.remove("WEB/python/lifeline.py")
    FILE2 = open("WEB/python/exception.py","w")
    FILE2.write("Exception occured!")
    timestring2 = "%s " % time.localtime()
    errorstring = "%s " % e
    FILE2.write(timestring2)
    FILE2.write(errorstring)
2

Hi Timmiotoool,

I suspect the size of meterdata could be the problem. The add function never resets meterdata[0], and will continue to append items to that meterdata[0] list. On the other hand, if you don’t have many different meters, the list could stay small.

The issue could also be driven by fragmentation in memory. Some ways to reduce this would be to keep around the FILE object between function calls, and instead call ‘flush()’ on the object instead of closing it. This would result in not creating the default memory buffer every time the writetofile() function is called, among other things.

On a side note, instead of using statements like ‘donothing = 1’ or ‘dummy = 1’, you can use the python no-op command, ‘pass’.

Hope this helps,
Max

3

You also are not calling gc.collect() - you need to. Python does auto-garbage colection but on it’s own sweet time-schedule. It tends to run gc.collect only after lots of alloc/frees, not a few huge ones. it also doesn’t run gc.collect when RAM is low - it just crashes.

So I call gc.collect after I close any socket & and after setting ‘sock’ to None. Probably every socket you close leaves a 50K chunk of garbage which remains for days. So make a habit of setting all variable (handle to memory) to None after use in a loop, else the data isn’t freed until you run the next loop.

See this page:
http://www.digi.com/wiki/developer/index.php/Python_Garbage_Collection

4

Well first of all thank you for your advice :slight_smile:
To tell the truth, I didn’t know i have to set variables to none after I’ve used them^^

My program got better but it is still leaking memory.

The advice to leave the File Object open didn’t work, as the program didn’t write in the file at all(the flush command didnt do the trick, even combined with os.fsync…)

The gc.collect() command only finds unused memory in the first loop-run but afterwards nothing.

Do you see any other possible leaks in the program, then i would be happy to know.

Greeting Tim

from socket import *
from select import * 
import time
import os
import os.path
import gc

#Add a new Measurement to the temporary list
def add(addr,data,timestamp):
	#print "Adding Measurement"
	#print "Listsize:", len(meterdata[0])
	present = 0
	for i in range(len(meterdata[0])):
		if meterdata[0][i] == addr:
			present = 1
			meterdata[1][i] = data
			meterdata[2][i] = timestamp
	if present == 0:
		#print "New Meter!"
		meterdata[0].append(addr)
		meterdata[1].append(data)
		meterdata[2].append(timestamp)
	#print "Adding Measurement done!"
	present = None


def printmeterdata():
	#print "Printing Measurements:"
	for i in range(len(meterdata[0])):
		print meterdata[0][i], " | ", meterdata[1][i] , " | ", meterdata[2][i] 


#Write Measurements to flash
def writetofile():
	FILE = open("WEB/python/data.py","w")
	for i in range(len(meterdata[0])):
		FILE.write("METER_START
")
		FILE.write("ADDR_START
")
		FILE.write(meterdata[0][i])
		FILE.write("
ADDR_END
")
		FILE.write("TIME_START
")
		FILE.write(meterdata[2][i])
		FILE.write("
TIME_END
")		
		FILE.write("DATA_START
")
		FILE.write(meterdata[1][i])
		FILE.write("
DATA_END
")		
		FILE.write("METER_END
")
    	FILE.close()
	FILE = None


#Socket Information
sd = socket(AF_ZIGBEE, SOCK_DGRAM, ZBS_PROT_APS)
sd.bind(("", 0xe8, 0, 0))
sd.setblocking(0)
#Insert Meters here
staticlist = []
staticlist.append(("02:11:a6:00:41:5d:19:dd!", 0x5e, 0xc105, 0x703))
#Static Variables
meterdata = [[],[],[]]

payload = "GET
"
erg = ""
stepcount = 1 

try:
#Lifeline to kill program if running
    FILEDUMMY = open("WEB/python/lifeline.py","w")
    FILEDUMMY.write("Delete me to stop me!")
    FILEDUMMY.close()

    

    if os.path.exists("WEB/python/exception.py"):
                os.remove("WEB/python/exception.py")

    while 1: 
        #print "step nr " , stepcount
	try:

		#ask static devices for measurements
		if stepcount % 10 == 0:
			for listelem in staticlist:
				#print "Sending GET to: " , listelem[0]
				rlist, wlist = ([], [])
				wlist = [sd]
				rlist, wlist, xlist = select(rlist, wlist, [])
				count = sd.sendto(payload, listelem)
				rlist = None
				wlist = None
				xlist = None
				count = None
	
		rlist, wlist = ([], [])	
	        rlist = [sd]
		#receive package
		erg, src_addr = sd.recvfrom(255)
		#print "Package from :"
		#print src_addr[0]		
		if erg[0:4] == "PID=":
			#print "I AM HERE Packet received!"
			rlist, wlist = ([], [])
			wlist = [sd]
			rlist, wlist, xlist = select(rlist, wlist, [])
			count = sd.sendto(payload, 0, src_addr)
		else:
			#print "Measurement received!"
			timestring = "%s " % time.localtime()
			add(src_addr[0],erg,timestring)
			timestring = None
			#print "Measurement added!"
			#printmeterdata()
			#print "Data printed"
			writetofile()
			#print "File written!"
		rlist = None
		wlist = None
		xlist = None
		count = None
		src_addr = None	
		erg = None	

	except Exception, f:
		f = None
		pass
		#print f

	time.sleep(0.9)

	collected = gc.collect()
	if collected > 0:
	       	print "Garbage collector: collected %d objects." % (collected)
	

	#Stop if lifeline is gone
	if os.path.exists("WEB/python/lifeline.py"):
                pass
        else:   
                raise Exception, "Lifeline deleted!"

  	stepcount = stepcount + 1
	if stepcount < 0:
	    raise Exception, "End of program!"

except Exception, e:
    print e
    sd.close()
    if os.path.exists("WEB/python/lifeline.py"):
    	os.remove("WEB/python/lifeline.py")
    FILE2 = open("WEB/python/exception.py","w")
    FILE2.write("Exception occured!")
    timestring2 = "%s " % time.localtime()
    errorstring = "%s " % e
    FILE2.write(timestring2)
    FILE2.write(errorstring)
    FILE2.close()
5

It’s hard to read your code with it all smushed to the left, but it looks like every time you go through your while loop you’re adding a new item to the meterdata list. I.E. I think:

for i in range(len(meterdata[0])):
if meterdata[0] == addr:

Should be:

for i in range(len(meterdata[0])):
if meterdata[0][I] == addr:

**edit - a lower case I won’t show up above