Sunday 27 September 2020

Compiled python version: ModuleNotFoundError at import

I have a project in a conda environment that runs with python 3.7.7 (on linux). When I recompile the same version of python (3.7.7) and put/replace the executable at the same location, I expect the program to run the same way, but the import fails.

With the original version of python:

(condaenv) mypc:~/Proj$ /home/me/.conda/envs/condaenv/bin/python3.7.bak
Python 3.7.7 (default, Mar 26 2020, 15:48:22) 
[GCC 7.3.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import gym
>>> quit()

With the compiled version:

(condaenv) mypc:~/Proj$ /home/me/.conda/condaenv/proj/bin/python3.7
Python 3.7.7 (default, Sep 24 2020, 16:28:06) 
[GCC 9.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import gym
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'gym'
>>> quit()

The environment variables are supposed to be the same as I run from the same location, on the same terminal, without altering it between the two calls. Since the first import works without any problem, the packages are well installed.

The system imports such as sys work fine in both version, but I had to export LD_LIBRARY_PATH before for the compiled version, which was not the case for the "normal" version. But keeping the LD_LIBRARY_PATH unchanged between the two calls don't change anything.

What happens that the second call can't find the matching package? What am I missing?

-- Edit 1 --

The compiled version shared objects dependencies are the following:

$ ldd python3.7
        linux-vdso.so.1 (0x00007ffed457a000)
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f63e53a6000)
        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f63e53a0000)
        libutil.so.1 => /lib/x86_64-linux-gnu/libutil.so.1 (0x00007f63e539b000)
        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f63e524c000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f63e505a000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f63e577f000)

The original python are the following:

$ ldd python3.7.bak 
        linux-vdso.so.1 (0x00007ffc0d1c8000)
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f89ab6bc000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f89ab4ca000)
        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f89ab4c4000)
        libutil.so.1 => /lib/x86_64-linux-gnu/libutil.so.1 (0x00007f89ab4bf000)
        librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f89ab4b4000)
        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f89ab365000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f89aba80000)

The versions differs a bit because they are not compiled with the same version of the compiler (I guess). I am however surprised that the new version doesn't require librt.so but the original version does. It could be because of the compilation flags though, which prevent the optimisation. I added --without-pymalloc --with-pydebug --with-valgrind during the configure step. But I don't think it should interfere with python normal behaviour with library.



from Compiled python version: ModuleNotFoundError at import

No comments:

Post a Comment