Slurm Tips for Toil - Revision history

Anovak at 14:50, 21 August 2026

2026-08-21T14:50:10Z

← Older revision		Revision as of 07:50, 21 August 2026
Line 24:		Line 24:
	This may in turn need you to log out and back in.		This may in turn need you to log out and back in.

	* When upgrading Toil, if you are using a Pip installation in a virtual environment, remember that ''--upgrade doesn't work on installations from Git!''. Pip [https://github.com/pypa/pip/issues/5780#issuecomment-2541153550 isn't able to recognize that a new Git commit should replace an old one of the same pre-release version]. So, when upgrading Toil with Pip, remember to uninstall it first:		* When upgrading Toil, if you are using a Pip installation in a virtual environment, remember that '''--upgrade doesn't work on installations from Git!''' Pip [https://github.com/pypa/pip/issues/5780#issuecomment-2541153550 isn't able to recognize that a new Git commit should replace an old one of the same pre-release version]. So, when upgrading Toil with Pip, remember to uninstall it first:

	pip uninstall toil		pip uninstall toil

Anovak at 14:49, 21 August 2026

2026-08-21T14:49:48Z

← Older revision		Revision as of 07:49, 21 August 2026
Line 24:		Line 24:
	This may in turn need you to log out and back in.		This may in turn need you to log out and back in.

	* When upgrading Toil, if you are using a Pip installation in a virtual environment, remember that ''--upgrade doesn't work on installations from Git!''. Pip [https://github.com/pypa/pip/issues/5780#issuecomment-2541153550 isn't able to recognize that a new Git commit should replace an old one of the same pre-release version]. So, when upgrading Toil with Pip, remember to uninstall it ~~firts~~:		* When upgrading Toil, if you are using a Pip installation in a virtual environment, remember that ''--upgrade doesn't work on installations from Git!''. Pip [https://github.com/pypa/pip/issues/5780#issuecomment-2541153550 isn't able to recognize that a new Git commit should replace an old one of the same pre-release version]. So, when upgrading Toil with Pip, remember to uninstall it first:

	pip uninstall toil		pip uninstall toil

Anovak at 14:49, 21 August 2026

2026-08-21T14:49:23Z

← Older revision		Revision as of 07:49, 21 August 2026
Line 4:		Line 4:

	pipx install 'toil[wdl]'		pipx install 'toil[wdl]'

			Or with support for all features:

			pipx install 'toil[all]'

	To use a development version of Toil, you can install from source instead:		To use a development version of Toil, you can install from source instead:
Line 19:		Line 23:

	This may in turn need you to log out and back in.		This may in turn need you to log out and back in.

			* When upgrading Toil, if you are using a Pip installation in a virtual environment, remember that ''--upgrade doesn't work on installations from Git!''. Pip [https://github.com/pypa/pip/issues/5780#issuecomment-2541153550 isn't able to recognize that a new Git commit should replace an old one of the same pre-release version]. So, when upgrading Toil with Pip, remember to uninstall it firts:

			pip uninstall toil
			pip install 'toil[all]@git+https://github.com/DataBiosphere/toil.git'

	* For Toil options, you will want '''--batchSystem slurm''' to make it use Slurm and '''--batchLogsDir ./logs''' (or some other location on a shared filesystem) for the Slurm logs to not get lost.		* For Toil options, you will want '''--batchSystem slurm''' to make it use Slurm and '''--batchLogsDir ./logs''' (or some other location on a shared filesystem) for the Slurm logs to not get lost.

Anovak at 15:39, 26 May 2026

2026-05-26T15:39:19Z

← Older revision		Revision as of 08:39, 26 May 2026
Line 7:		Line 7:
	To use a development version of Toil, you can install from source instead:		To use a development version of Toil, you can install from source instead:

	pipx 'toil[wdl]@git+https://github.com/DataBiosphere/toil.git'		pipx install 'toil[wdl]@git+https://github.com/DataBiosphere/toil.git'

	Or for a particular branch:		Or for a particular branch:

Anovak at 22:11, 11 December 2025

2025-12-11T22:11:55Z

← Older revision		Revision as of 15:11, 11 December 2025
Line 26:		Line 26:
	* If using '''toil-wdl-runner''', you might want to add '''--jobStore ./jobStore''' to make sure the job store is in a defined, shared location so that you can use '''--restart''' later.		* If using '''toil-wdl-runner''', you might want to add '''--jobStore ./jobStore''' to make sure the job store is in a defined, shared location so that you can use '''--restart''' later.

	* If using '''toil-wdl-runner''', you will want to set the '''SINGULARITY_CACHEDIR''' and '''MINIWDL__SINGULARITY__IMAGE_CACHE''' environment variables for your workflow to locations on shared storage, and possibly to the default cache locations in your home directory. Otherwise Toil will set them to temporary node-local directories for each node, and thus re-download images for each workflow run, and for each cluster node. To avoid this, you could, for example, before your run or in your '''~/.bashrc''' you could:		* If using '''toil-wdl-runner''', you will want to set the '''SINGULARITY_CACHEDIR''' and '''MINIWDL__SINGULARITY__IMAGE_CACHE''' environment variables for your workflow to locations on shared storage, and possibly to the default cache locations in your home directory. Otherwise Toil will set them to temporary per-workflow node-local directories for each node, and thus re-download images for each workflow run, and for each cluster node. To avoid this, you could, for example, before your run or in your '''~/.bashrc''' you could:

	export SINGULARITY_CACHEDIR=$HOME/.singularity/cache		export SINGULARITY_CACHEDIR=$HOME/.singularity/cache
	export MINIWDL__SINGULARITY__IMAGE_CACHE=$HOME/.cache/miniwdl		export MINIWDL__SINGULARITY__IMAGE_CACHE=$HOME/.cache/miniwdl

	If you run a lot of workflows, or a workflow with a lot of containers, you will run out of space in your home directory. In that case, you can try using persistent per-node storage for your image caches instead:		If you run a lot of workflows, or a workflow with a lot of containers, you will run out of space in your home directory. In that case, you can try using semi-persistent per-node storage for your image caches instead:

	export SINGULARITY_CACHEDIR="/data/tmp/$(whoami)/cache/singularity"		export SINGULARITY_CACHEDIR="/data/tmp/$(whoami)/cache/singularity"
	export MINIWDL__SINGULARITY__IMAGE_CACHE="/data/tmp/$(whoami)/cache/miniwdl"		export MINIWDL__SINGULARITY__IMAGE_CACHE="/data/tmp/$(whoami)/cache/miniwdl"

Anovak at 22:11, 11 December 2025

2025-12-11T22:11:10Z

← Older revision		Revision as of 15:11, 11 December 2025
Line 26:		Line 26:
	* If using '''toil-wdl-runner''', you might want to add '''--jobStore ./jobStore''' to make sure the job store is in a defined, shared location so that you can use '''--restart''' later.		* If using '''toil-wdl-runner''', you might want to add '''--jobStore ./jobStore''' to make sure the job store is in a defined, shared location so that you can use '''--restart''' later.

	* If using '''toil-wdl-runner''', you will want to set the '''SINGULARITY_CACHEDIR''' and '''MINIWDL__SINGULARITY__IMAGE_CACHE''' environment variables for your workflow to locations on shared storage, and possibly to the default cache locations in your home directory. Otherwise Toil will set them to node-local directories for each node, and thus re-download images for each workflow run, and for each cluster node. To avoid this, you could, for example, before your run or in your '''~/.bashrc''' you could:		* If using '''toil-wdl-runner''', you will want to set the '''SINGULARITY_CACHEDIR''' and '''MINIWDL__SINGULARITY__IMAGE_CACHE''' environment variables for your workflow to locations on shared storage, and possibly to the default cache locations in your home directory. Otherwise Toil will set them to temporary node-local directories for each node, and thus re-download images for each workflow run, and for each cluster node. To avoid this, you could, for example, before your run or in your '''~/.bashrc''' you could:

	export SINGULARITY_CACHEDIR=$HOME/.singularity/cache		export SINGULARITY_CACHEDIR=$HOME/.singularity/cache
	export MINIWDL__SINGULARITY__IMAGE_CACHE=$HOME/.cache/miniwdl		export MINIWDL__SINGULARITY__IMAGE_CACHE=$HOME/.cache/miniwdl

			If you run a lot of workflows, or a workflow with a lot of containers, you will run out of space in your home directory. In that case, you can try using persistent per-node storage for your image caches instead:

			export SINGULARITY_CACHEDIR="/data/tmp/$(whoami)/cache/singularity"
			export MINIWDL__SINGULARITY__IMAGE_CACHE="/data/tmp/$(whoami)/cache/miniwdl"

Anovak at 19:12, 14 February 2025

2025-02-14T19:12:39Z

← Older revision		Revision as of 12:12, 14 February 2025
Line 3:		Line 3:
	* Install Toil with WDL support with:		* Install Toil with WDL support with:

	~~pip3~~ install ~~--upgrade~~ 'toil[wdl]'		pipx install 'toil[wdl]'

	To use a development version of Toil, you can install from source instead:		To use a development version of Toil, you can install from source instead:

	~~pip3 install~~ 'toil[wdl]@git+https://github.com/DataBiosphere/toil.git'		pipx 'toil[wdl]@git+https://github.com/DataBiosphere/toil.git'

	Or for a particular branch:		Or for a particular branch:

	~~pip3~~ install 'toil[wdl]@git+https://github.com/DataBiosphere/toil.git@issues/123-abc'		pipx install 'toil[wdl]@git+https://github.com/DataBiosphere/toil.git@issues/123-abc'

	* You will then need to ~~make sure your '''~/.local/bin''' directory is on your PATH. Open up your '''~/.bashrc''' file and add~~:		If you don't have <code>pipx</code>, you would first need to:

	~~export PATH=$PATH:$HOME/.local/bin~~		python3 -m pip install --user pipx
			python3 -m pipx ensurepath

	~~Then make sure~~ to log out and back in ~~again~~.		This may in turn need you to log out and back in.

	* For Toil options, you will want '''--batchSystem slurm''' to make it use Slurm and '''--batchLogsDir ./logs''' (or some other location on a shared filesystem) for the Slurm logs to not get lost.		* For Toil options, you will want '''--batchSystem slurm''' to make it use Slurm and '''--batchLogsDir ./logs''' (or some other location on a shared filesystem) for the Slurm logs to not get lost.

Anovak: Change to new extras syntax from https://github.com/pypa/pip/pull/11617

2025-02-11T20:49:14Z

Change to new extras syntax from https://github.com/pypa/pip/pull/11617

← Older revision		Revision as of 13:49, 11 February 2025
Line 7:		Line 7:
	To use a development version of Toil, you can install from source instead:		To use a development version of Toil, you can install from source instead:

	pip3 install 'git+https://github.com/DataBiosphere/toil.git~~#egg=toil[wdl]~~'		pip3 install 'toil[wdl]@git+https://github.com/DataBiosphere/toil.git'

	Or for a particular branch:		Or for a particular branch:

	pip3 install 'git+https://github.com/DataBiosphere/toil.git@issues/123-abc~~#egg=toil[wdl]~~'		pip3 install 'toil[wdl]@git+https://github.com/DataBiosphere/toil.git@issues/123-abc'

	* You will then need to make sure your '''~/.local/bin''' directory is on your PATH. Open up your '''~/.bashrc''' file and add:		* You will then need to make sure your '''~/.local/bin''' directory is on your PATH. Open up your '''~/.bashrc''' file and add:

Anovak: Add quotes to protect brackets

2025-02-11T20:42:22Z

Add quotes to protect brackets

← Older revision		Revision as of 13:42, 11 February 2025
Line 3:		Line 3:
	* Install Toil with WDL support with:		* Install Toil with WDL support with:

	pip3 install --upgrade toil[wdl]		pip3 install --upgrade 'toil[wdl]'

	To use a development version of Toil, you can install from source instead:		To use a development version of Toil, you can install from source instead:

	pip3 install git+https://github.com/DataBiosphere/toil.git#egg=toil[wdl]		pip3 install 'git+https://github.com/DataBiosphere/toil.git#egg=toil[wdl]'

	Or for a particular branch:		Or for a particular branch:

	pip3 install git+https://github.com/DataBiosphere/toil.git@issues/123-abc#egg=toil[wdl]		pip3 install 'git+https://github.com/DataBiosphere/toil.git@issues/123-abc#egg=toil[wdl]'

	* You will then need to make sure your '''~/.local/bin''' directory is on your PATH. Open up your '''~/.bashrc''' file and add:		* You will then need to make sure your '''~/.local/bin''' directory is on your PATH. Open up your '''~/.bashrc''' file and add:

Anovak at 19:08, 16 February 2024

2024-02-16T19:08:32Z

← Older revision		Revision as of 12:08, 16 February 2024
Line 1:		Line 1:
	Here are some tips for running Toil workflows on the Phoenix Slurm cluster. Mostly you might want to run WDL workflows, but you can use some of these for other workflows like Cactus. You can also consult [https://github.com/DataBiosphere/toil/blob/master/docs/running~~/wdl~~.rst~~#running-wdl-with-toil~~ the Toil documentation on WDL workflows].		Here are some tips for running Toil workflows on the Phoenix Slurm cluster. Mostly you might want to run WDL workflows, but you can use some of these for other workflows like Cactus. You can also consult [https://github.com/DataBiosphere/toil/blob/master/docs/wdl/running.rst the Toil documentation on WDL workflows].

	* Install Toil with WDL support with:		* Install Toil with WDL support with: