Thursday, 9 September 2021

Loading custom conda envs not working in SageMaker

I have installed miniconda on my AWS SageMaker persistent EBS instance. Here is my starting script:

#!/bin/bash

set -e

# OVERVIEW
# This script installs a custom, persistent installation of conda on the Notebook Instance's EBS volume, and ensures
# that these custom environments are available as kernels in Jupyter.
# 
# The on-start script uses the custom conda environment created in the on-create script and uses the ipykernel package
# to add that as a kernel in Jupyter.
#
# For another example, see:
# https://docs.aws.amazon.com/sagemaker/latest/dg/nbi-add-external.html#nbi-isolated-environment

sudo -u ec2-user -i <<'EOF'
unset SUDO_UID
WORKING_DIR=/home/ec2-user/SageMaker/

for env in $WORKING_DIR/miniconda/envs/*; do
    BASENAME=$(basename "$env")
    source "$WORKING_DIR/miniconda/bin/activate"
    source activate "$BASENAME"
    pip install ipykernel boto3
    python -m ipykernel install --user --name "$BASENAME" --display-name "Custom ($BASENAME)"
done
# Optionally, uncomment these lines to disable SageMaker-provided Conda functionality.
# echo "c.EnvironmentKernelSpecManager.use_conda_directly = False" >> /home/ec2-user/.jupyter/jupyter_notebook_config.py
# rm /home/ec2-user/.condarc
EOF

echo "Restarting the Jupyter server.."
restart jupyter-server

I use this in order to load my custom envs. However, when I access the JupyterLab interface, even if I see that the activated kernel is the Custom one, the only version of python running on my notebook kernel is /home/ec2-user/anaconda3/envs/JupyterSystemEnv/bin/python:

enter image description here

I also inspected the CloudWatch logs, and I see this error log: Could not find conda environment: [custom_env].

But, when I run the commands of the starting script within the JupyterLab terminal, conda succeeds in finding those envs. So the question is: what am I missing?

Thanks a lot.



from Loading custom conda envs not working in SageMaker

No comments:

Post a Comment