Friday, 23 April 2021

Why Does My Package Take up so Much Memory

I am working on Ptera Software, an open-source aerodynamics solver. This is the first package I have distributed, and I'm having some issues related to memory management.

Specifically, importing my package takes up an absurd amount of memory. The last time I checked, it took around 136 MB of RAM. PyPI lists the package size as 118 MB, which also seems crazy high. For reference, NumPy is only 87 MB.

At first, I thought that maybe I had accidentally included some huge file in the package. So I downloaded every version's tar.gz files from PyPI and extracted them. None was over 1 MB unzipped.

This leads me to believe that there's something wrong with how I am importing my requirements. My REQUIREMENTS.txt file looks like this:

matplotlib >= 3.2.2, < 4.0.0
numpy >= 1.18.5, < 2.0.0
pyvista >= 0.29.0, < 1.0.0
scipy >= 1.5, < 2.0
numba >= 0.53, <1.0

It could also be that I messed up my __init__.py file. It looks like this:

from pterasoftware import aerodynamics
from pterasoftware import airfoils
from pterasoftware import geometry
from pterasoftware import meshing
from pterasoftware import movement
from pterasoftware import operating_point
from pterasoftware import output
from pterasoftware import problems
from pterasoftware import steady_horseshoe_vortex_lattice_method
from pterasoftware import steady_ring_vortex_lattice_method
from pterasoftware import unsteady_ring_vortex_lattice_method

The directory structure is as so:

├───pterasoftware
│   ├───airfoils
│   │   └───naca0012.dat
│   ├───__init__.py
│   ├───aerodynamics.py
│   ├───geometry.py
│   ├───meshing.py
│   ├───movement.py
│   ├───operating_point.py
│   ├───output.py
│   ├───problems.py
│   ├───steady_horsehoe_vortex_lattice_method.py
│   ├───steady_ring_vortex_lattice_method.py
│   └───unsteady_ring_vortex_lattice_method.py

I know that importing large packages like numpy, matplotlib, and scipy can be memory heavy. However, I know plenty of packages that use these resources, which don't take anywhere near 136 MB to import. What am I missing here?

Here's the code I use to test the memory allocated while importing the package:

from memory_profiler import profile


@profile
def find_import_memory_usage():
    import pterasoftware as ps


if __name__ == "__main__":
    find_import_memory_usage()


from Why Does My Package Take up so Much Memory

No comments:

Post a Comment