Slurm Tips for Toil - Revision history

Anovak at 22:11, 11 December 2025

2025-12-11T22:11:55Z

← Older revision		Revision as of 15:11, 11 December 2025
Line 26:		Line 26:
	* If using '''toil-wdl-runner''', you might want to add '''--jobStore ./jobStore''' to make sure the job store is in a defined, shared location so that you can use '''--restart''' later.		* If using '''toil-wdl-runner''', you might want to add '''--jobStore ./jobStore''' to make sure the job store is in a defined, shared location so that you can use '''--restart''' later.

	* If using '''toil-wdl-runner''', you will want to set the '''SINGULARITY_CACHEDIR''' and '''MINIWDL__SINGULARITY__IMAGE_CACHE''' environment variables for your workflow to locations on shared storage, and possibly to the default cache locations in your home directory. Otherwise Toil will set them to temporary node-local directories for each node, and thus re-download images for each workflow run, and for each cluster node. To avoid this, you could, for example, before your run or in your '''~/.bashrc''' you could:		* If using '''toil-wdl-runner''', you will want to set the '''SINGULARITY_CACHEDIR''' and '''MINIWDL__SINGULARITY__IMAGE_CACHE''' environment variables for your workflow to locations on shared storage, and possibly to the default cache locations in your home directory. Otherwise Toil will set them to temporary per-workflow node-local directories for each node, and thus re-download images for each workflow run, and for each cluster node. To avoid this, you could, for example, before your run or in your '''~/.bashrc''' you could:

	export SINGULARITY_CACHEDIR=$HOME/.singularity/cache		export SINGULARITY_CACHEDIR=$HOME/.singularity/cache
	export MINIWDL__SINGULARITY__IMAGE_CACHE=$HOME/.cache/miniwdl		export MINIWDL__SINGULARITY__IMAGE_CACHE=$HOME/.cache/miniwdl

	If you run a lot of workflows, or a workflow with a lot of containers, you will run out of space in your home directory. In that case, you can try using persistent per-node storage for your image caches instead:		If you run a lot of workflows, or a workflow with a lot of containers, you will run out of space in your home directory. In that case, you can try using semi-persistent per-node storage for your image caches instead:

	export SINGULARITY_CACHEDIR="/data/tmp/$(whoami)/cache/singularity"		export SINGULARITY_CACHEDIR="/data/tmp/$(whoami)/cache/singularity"
	export MINIWDL__SINGULARITY__IMAGE_CACHE="/data/tmp/$(whoami)/cache/miniwdl"		export MINIWDL__SINGULARITY__IMAGE_CACHE="/data/tmp/$(whoami)/cache/miniwdl"

Anovak at 22:11, 11 December 2025

2025-12-11T22:11:10Z

← Older revision		Revision as of 15:11, 11 December 2025
Line 26:		Line 26:
	* If using '''toil-wdl-runner''', you might want to add '''--jobStore ./jobStore''' to make sure the job store is in a defined, shared location so that you can use '''--restart''' later.		* If using '''toil-wdl-runner''', you might want to add '''--jobStore ./jobStore''' to make sure the job store is in a defined, shared location so that you can use '''--restart''' later.

	* If using '''toil-wdl-runner''', you will want to set the '''SINGULARITY_CACHEDIR''' and '''MINIWDL__SINGULARITY__IMAGE_CACHE''' environment variables for your workflow to locations on shared storage, and possibly to the default cache locations in your home directory. Otherwise Toil will set them to node-local directories for each node, and thus re-download images for each workflow run, and for each cluster node. To avoid this, you could, for example, before your run or in your '''~/.bashrc''' you could:		* If using '''toil-wdl-runner''', you will want to set the '''SINGULARITY_CACHEDIR''' and '''MINIWDL__SINGULARITY__IMAGE_CACHE''' environment variables for your workflow to locations on shared storage, and possibly to the default cache locations in your home directory. Otherwise Toil will set them to temporary node-local directories for each node, and thus re-download images for each workflow run, and for each cluster node. To avoid this, you could, for example, before your run or in your '''~/.bashrc''' you could:

	export SINGULARITY_CACHEDIR=$HOME/.singularity/cache		export SINGULARITY_CACHEDIR=$HOME/.singularity/cache
	export MINIWDL__SINGULARITY__IMAGE_CACHE=$HOME/.cache/miniwdl		export MINIWDL__SINGULARITY__IMAGE_CACHE=$HOME/.cache/miniwdl

			If you run a lot of workflows, or a workflow with a lot of containers, you will run out of space in your home directory. In that case, you can try using persistent per-node storage for your image caches instead:

			export SINGULARITY_CACHEDIR="/data/tmp/$(whoami)/cache/singularity"
			export MINIWDL__SINGULARITY__IMAGE_CACHE="/data/tmp/$(whoami)/cache/miniwdl"

Anovak at 19:12, 14 February 2025

2025-02-14T19:12:39Z

← Older revision		Revision as of 12:12, 14 February 2025
Line 3:		Line 3:
	* Install Toil with WDL support with:		* Install Toil with WDL support with:

	~~pip3~~ install ~~--upgrade~~ 'toil[wdl]'		pipx install 'toil[wdl]'

	To use a development version of Toil, you can install from source instead:		To use a development version of Toil, you can install from source instead:

	~~pip3 install~~ 'toil[wdl]@git+https://github.com/DataBiosphere/toil.git'		pipx 'toil[wdl]@git+https://github.com/DataBiosphere/toil.git'

	Or for a particular branch:		Or for a particular branch:

	~~pip3~~ install 'toil[wdl]@git+https://github.com/DataBiosphere/toil.git@issues/123-abc'		pipx install 'toil[wdl]@git+https://github.com/DataBiosphere/toil.git@issues/123-abc'

	* You will then need to ~~make sure your '''~/.local/bin''' directory is on your PATH. Open up your '''~/.bashrc''' file and add~~:		If you don't have <code>pipx</code>, you would first need to:

	~~export PATH=$PATH:$HOME/.local/bin~~		python3 -m pip install --user pipx
			python3 -m pipx ensurepath

	~~Then make sure~~ to log out and back in ~~again~~.		This may in turn need you to log out and back in.

	* For Toil options, you will want '''--batchSystem slurm''' to make it use Slurm and '''--batchLogsDir ./logs''' (or some other location on a shared filesystem) for the Slurm logs to not get lost.		* For Toil options, you will want '''--batchSystem slurm''' to make it use Slurm and '''--batchLogsDir ./logs''' (or some other location on a shared filesystem) for the Slurm logs to not get lost.

Anovak: Change to new extras syntax from https://github.com/pypa/pip/pull/11617

2025-02-11T20:49:14Z

Change to new extras syntax from https://github.com/pypa/pip/pull/11617

← Older revision		Revision as of 13:49, 11 February 2025
Line 7:		Line 7:
	To use a development version of Toil, you can install from source instead:		To use a development version of Toil, you can install from source instead:

	pip3 install 'git+https://github.com/DataBiosphere/toil.git~~#egg=toil[wdl]~~'		pip3 install 'toil[wdl]@git+https://github.com/DataBiosphere/toil.git'

	Or for a particular branch:		Or for a particular branch:

	pip3 install 'git+https://github.com/DataBiosphere/toil.git@issues/123-abc~~#egg=toil[wdl]~~'		pip3 install 'toil[wdl]@git+https://github.com/DataBiosphere/toil.git@issues/123-abc'

	* You will then need to make sure your '''~/.local/bin''' directory is on your PATH. Open up your '''~/.bashrc''' file and add:		* You will then need to make sure your '''~/.local/bin''' directory is on your PATH. Open up your '''~/.bashrc''' file and add:

Anovak: Add quotes to protect brackets

2025-02-11T20:42:22Z

Add quotes to protect brackets

← Older revision		Revision as of 13:42, 11 February 2025
Line 3:		Line 3:
	* Install Toil with WDL support with:		* Install Toil with WDL support with:

	pip3 install --upgrade toil[wdl]		pip3 install --upgrade 'toil[wdl]'

	To use a development version of Toil, you can install from source instead:		To use a development version of Toil, you can install from source instead:

	pip3 install git+https://github.com/DataBiosphere/toil.git#egg=toil[wdl]		pip3 install 'git+https://github.com/DataBiosphere/toil.git#egg=toil[wdl]'

	Or for a particular branch:		Or for a particular branch:

	pip3 install git+https://github.com/DataBiosphere/toil.git@issues/123-abc#egg=toil[wdl]		pip3 install 'git+https://github.com/DataBiosphere/toil.git@issues/123-abc#egg=toil[wdl]'

	* You will then need to make sure your '''~/.local/bin''' directory is on your PATH. Open up your '''~/.bashrc''' file and add:		* You will then need to make sure your '''~/.local/bin''' directory is on your PATH. Open up your '''~/.bashrc''' file and add:

Anovak at 19:08, 16 February 2024

2024-02-16T19:08:32Z

← Older revision		Revision as of 12:08, 16 February 2024
Line 1:		Line 1:
	Here are some tips for running Toil workflows on the Phoenix Slurm cluster. Mostly you might want to run WDL workflows, but you can use some of these for other workflows like Cactus. You can also consult [https://github.com/DataBiosphere/toil/blob/master/docs/running~~/wdl~~.rst~~#running-wdl-with-toil~~ the Toil documentation on WDL workflows].		Here are some tips for running Toil workflows on the Phoenix Slurm cluster. Mostly you might want to run WDL workflows, but you can use some of these for other workflows like Cactus. You can also consult [https://github.com/DataBiosphere/toil/blob/master/docs/wdl/running.rst the Toil documentation on WDL workflows].

	* Install Toil with WDL support with:		* Install Toil with WDL support with:

Anovak: Show how to install with extras and a branch

2023-11-20T21:54:12Z

Show how to install with extras and a branch

← Older revision		Revision as of 14:54, 20 November 2023
Line 8:		Line 8:

	pip3 install git+https://github.com/DataBiosphere/toil.git#egg=toil[wdl]		pip3 install git+https://github.com/DataBiosphere/toil.git#egg=toil[wdl]

			Or for a particular branch:

			pip3 install git+https://github.com/DataBiosphere/toil.git@issues/123-abc#egg=toil[wdl]

	* You will then need to make sure your '''~/.local/bin''' directory is on your PATH. Open up your '''~/.bashrc''' file and add:		* You will then need to make sure your '''~/.local/bin''' directory is on your PATH. Open up your '''~/.bashrc''' file and add:

Anovak at 14:08, 27 June 2023

2023-06-27T14:08:56Z

← Older revision		Revision as of 07:08, 27 June 2023
Line 1:		Line 1:
	Here are some tips for running Toil workflows on the Phoenix Slurm cluster. Mostly you might want to run WDL workflows, but you can use some of these for other workflows like Cactus. You can also consult [https://github.com/DataBiosphere/toil/blob/master/docs/running/wdl.rst#running-wdl-with-toil the Toil documentation on WDL workflows].		Here are some tips for running Toil workflows on the Phoenix Slurm cluster. Mostly you might want to run WDL workflows, but you can use some of these for other workflows like Cactus. You can also consult [https://github.com/DataBiosphere/toil/blob/master/docs/running/wdl.rst#running-wdl-with-toil the Toil documentation on WDL workflows].

	* ~~Because the new~~ WDL ~~interpreter in~~ Toil ~~isn't yet in any release~~, you ~~would want to~~ install ~~Toil~~ from source ~~with WDL support with~~:		* Install Toil with WDL support with:

			pip3 install --upgrade toil[wdl]

			To use a development version of Toil, you can install from source instead:

	pip3 install git+https://github.com/DataBiosphere/toil.git#egg=toil[wdl]		pip3 install git+https://github.com/DataBiosphere/toil.git#egg=toil[wdl]

Anovak at 21:29, 5 June 2023

2023-06-05T21:29:47Z

← Older revision		Revision as of 14:29, 5 June 2023
Line 12:		Line 12:

	* For Toil options, you will want '''--batchSystem slurm''' to make it use Slurm and '''--batchLogsDir ./logs''' (or some other location on a shared filesystem) for the Slurm logs to not get lost.		* For Toil options, you will want '''--batchSystem slurm''' to make it use Slurm and '''--batchLogsDir ./logs''' (or some other location on a shared filesystem) for the Slurm logs to not get lost.

			* You may be able to speed up your workflow with '''--caching true''', to cache data on nodes to be shared among multiple simultaneous tasks.

	* If using '''toil-wdl-runner''', you might want to add '''--jobStore ./jobStore''' to make sure the job store is in a defined, shared location so that you can use '''--restart''' later.		* If using '''toil-wdl-runner''', you might want to add '''--jobStore ./jobStore''' to make sure the job store is in a defined, shared location so that you can use '''--restart''' later.

Anovak: Created page with "Here are some tips for running Toil workflows on the Phoenix Slurm cluster. Mostly you might want to run WDL workflows, but you can use some of these for other workflows like..."

2023-05-15T21:34:40Z

Created page with "Here are some tips for running Toil workflows on the Phoenix Slurm cluster. Mostly you might want to run WDL workflows, but you can use some of these for other workflows like..."

New page

Here are some tips for running Toil workflows on the Phoenix Slurm cluster. Mostly you might want to run WDL workflows, but you can use some of these for other workflows like Cactus. You can also consult [https://github.com/DataBiosphere/toil/blob/master/docs/running/wdl.rst#running-wdl-with-toil the Toil documentation on WDL workflows].

* Because the new WDL interpreter in Toil isn't yet in any release, you would want to install Toil from source with WDL support with:

pip3 install git+https://github.com/DataBiosphere/toil.git#egg=toil[wdl]

* You will then need to make sure your '''~/.local/bin''' directory is on your PATH. Open up your '''~/.bashrc''' file and add:

export PATH=$PATH:$HOME/.local/bin

Then make sure to log out and back in again.

* For Toil options, you will want '''--batchSystem slurm''' to make it use Slurm and '''--batchLogsDir ./logs''' (or some other location on a shared filesystem) for the Slurm logs to not get lost.

* If using '''toil-wdl-runner''', you might want to add '''--jobStore ./jobStore''' to make sure the job store is in a defined, shared location so that you can use '''--restart''' later.

* If using '''toil-wdl-runner''', you will want to set the '''SINGULARITY_CACHEDIR''' and '''MINIWDL__SINGULARITY__IMAGE_CACHE''' environment variables for your workflow to locations on shared storage, and possibly to the default cache locations in your home directory. Otherwise Toil will set them to node-local directories for each node, and thus re-download images for each workflow run, and for each cluster node. To avoid this, you could, for example, before your run or in your '''~/.bashrc''' you could:

export SINGULARITY_CACHEDIR=$HOME/.singularity/cache
export MINIWDL__SINGULARITY__IMAGE_CACHE=$HOME/.cache/miniwdl