Thursday, 17 January 2019

How to force Django models to be released from memory

I want to use a management command to run a one-time analysis of the buildings in Massachusetts. I have reduced the offending code to an 8 line snippet that demonstrates the problem I encounter. The comments just explain why I want to do this at all. I am running the code below verbatim, in an otherwise-blank management command

zips = ZipCode.objects.filter(state='MA').order_by('id')
for zip in zips.iterator():
    buildings = Building.objects.filter(boundary__within=zip.boundary)
    important_buildings = []
    for building in buildings.iterator():
        # Some conditionals would go here
        important_buildings.append(building)
    # Several types of analysis would be done on important_buildings, here
    important_buildings = None

When I run this exact code, I find that memory usage steadily increases with each iteration outer loop (I use print('mem', process.memory_info().rss) to check memory usage).

It seems like the important_buildings list is hogging up memory, even after going out of scope. If I replace important_buildings.append(building) with _ = building.pk, it no longer consumes much memory, but I do need that list for some of the analysis.

So, my question is: How can I force Python to release the list of Django models when it goes out of scope?



from How to force Django models to be released from memory

No comments:

Post a Comment