Tuesday, 3 September 2019

Why wouldn't the tox see the environment variable during install_command?

Given the tox==3.13.2, see details

++pip install tox
Requirement already satisfied: tox in /home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages (3.13.2)

and the tox.ini:

[tox]
envlist =
    py-cythonized


[testenv]
; simplify numpy installation
setenv =
    LAPACK=
    ATLAS=None
    PYTHONWARNINGS=ignore

; Copy all environment variables to the tox test environment
passenv = *

deps =
    py-cythonized: Cython >= 0.28.5

changedir = nltk/test
commands =
    ; scipy and scikit-learn requires numpy even to run setup.py so
    ; they can't be installed in one command
    pip install scipy scikit-learn

    ; python runtests.py --with-coverage --cover-inclusive --cover-package=nltk --cover-html --cover-html-dir={envdir}/docs []
    python runtests.py []

And specifically:

# Test Cython compiled installation.
[testenv:py-cythonized]
setenv =
    CYTHONIZE_NLTK = "true"
    NLTK_DATA = {homedir}/nltk_data/
extras = all
passenv =
    CYTHONIZE_NLTK
commands =
    {toxinidir}/tools/travis/coverage-pylint.sh

There's the CYTHONIZE_NLTK variable that should be passed when testing with tox -e py-cythonize tox.ini -vv

The CYTHONIZE_NLTK variable is checked by my setup.py to add extended modules ext_modules:

MODULES_TO_COMPILE = [
    #'nltk.ccg.*', # Fails on https://travis-ci.org/nltk/nltk/jobs/529589821#L2077
    'nltk.chat.*',
    'nltk.chunk.*',

def compile_modules(modules):
    """
    Compile the named modules using Cython, using the clearer Python 3 semantics.
    """
    import Cython
    from Cython.Build import cythonize
    files = [name.replace('.', os.path.sep) + '.py' for name in modules]
    print("Compiling %d modules using Cython %s" % (len(modules), Cython.__version__))
    return cythonize(files, language_level=3)


print('PRINTING ENVIRON...')
print(os.environ)
if os.getenv('CYTHONIZE_NLTK') == 'true':
    ext_modules = compile_modules(MODULES_TO_COMPILE)
else:
    ext_modules = None

setup(
    name="nltk",
    extras_require=extras_require,
    packages=find_packages(),
    ext_modules=ext_modules,
    zip_safe=False,  # since normal files will be present too?
    entry_points=console_scripts,
)

But from the CI, it looks like the CYTHONIZE_NLTK wasn't recognized when tox does the install_command:

[5305] /home/travis/build/nltk/nltk$ /home/travis/build/nltk/nltk/.tox/py-cythonized/bin/python -m pip install --exists-action w '/home/travis/build/nltk/nltk/.tox/.tmp/package/1/nltk-3.5.zip[all]'

More details on https://travis-ci.org/nltk/nltk/jobs/578224159#L1988



from Why wouldn't the tox see the environment variable during install_command?

No comments:

Post a Comment