Tuesday 31 December 2019

snakemake cluster script ImportError snakemake.utils

I have a strange issue that comes and goes randomly and I really can't figure out when and why.
I am running a snakemake pipeline like this:

conda activate $myEnv    
snakemake -s $snakefile --configfile test.conf.yml --cluster "python $qsub_script" --latency-wait 60 --use-conda -p -j 10 --jobscript "$job_script"

I installed snakemake 5.9.1 (also tried downgrading to 5.5.4) within a conda environment.
This works fine if I just run this command, but when I qsub this command to the PBS cluster I'm using, I get an error. My qsub script looks like this:

#PBS stuff...

source ~/.bashrc
hostname
conda activate PGC_de_novo

cd $workDir
snakefile="..."
qsub_script="pbs_qsub_snakemake_wrapper.py"
job_script="..."
snakemake -s $snakefile --configfile test.conf.yml --cluster "python $qsub_script" --latency-wait 60 --use-conda -p -j 10 --jobscript "$job_script" >out 2>err

And the error message I get is:

...
Traceback (most recent call last):
  File "/path/to/pbs_qsub_snakemake_wrapper.py", line 6, in <module>
    from snakemake.utils import read_job_properties
ImportError: No module named snakemake.utils
Error submitting jobscript (exit code 1):
...

So it looks like for some reason my cluster script doesn't find snakemake, although snakemake is clearly installed. As I said, this problem keeps coming and going. It'd stay for a few hours, then go away for now aparent reason. I guess this indicates an environment problem, but I really can't figure out what, and ran out of ideas. I've tried:

  • different conda versions
  • different snakemake versions
  • different nodes on the cluster
  • ssh to the node it just failed on and try to reproduce the error

but nothing. Any ideas where to look? Thanks!



from snakemake cluster script ImportError snakemake.utils

No comments:

Post a Comment