Phoenix WDL Tutorial - Revision history

Anovak: /* Frequently Asked Questions */ Add an error someone asked about

2025-11-18T15:09:06Z

Frequently Asked Questions: Add an error someone asked about

← Older revision		Revision as of 08:09, 18 November 2025
Line 673:		Line 673:

	Toil [https://github.com/DataBiosphere/toil/commit/2de6eea2cc2e688b53062a98687445f0cca56669 recently gained support] for deleting files at the ''end'' of WDL workflows. So if you have a large file that you only need for part of your workflow, consider writing the part that creates and uses it as a separate sub-<code>workflow</code> and invoking it with <code>call</code>. Then the file will be cleaned up when the child workflow ends, leaving more space for files created in the parent workflow.		Toil [https://github.com/DataBiosphere/toil/commit/2de6eea2cc2e688b53062a98687445f0cca56669 recently gained support] for deleting files at the ''end'' of WDL workflows. So if you have a large file that you only need for part of your workflow, consider writing the part that creates and uses it as a separate sub-<code>workflow</code> and invoking it with <code>call</code>. Then the file will be cleaned up when the child workflow ends, leaving more space for files created in the parent workflow.

			===I am unable to allocate resources!===
			You might get an error like:

			srun: error: Unable to allocate resources: Invalid account or account/partition combination specified

			If this happens, you probably haven't been granted access to the Phoenix cluster, or at least to the partition you are trying to use. You should email <code>cluster-admin@soe.ucsc.edu</code> to ask for access.

	=Additional WDL resources=		=Additional WDL resources=

Anovak: /* Writing the file */

2025-03-18T20:48:24Z

Writing the file

← Older revision		Revision as of 13:48, 18 March 2025
Line 417:		Line 417:
	memory: "0.5 GB"		memory: "0.5 GB"
	disks: "local-disk 1 SSD"		disks: "local-disk 1 SSD"
	docker: "ubuntu:22.04"		docker: "ubuntu:24.04"
	}		}
	}		}
Line 477:		Line 477:
	memory: "0.5 GB"		memory: "0.5 GB"
	disks: "local-disk 1 SSD"		disks: "local-disk 1 SSD"
	docker: "ubuntu:22.04"		docker: "ubuntu:24.04"
	}		}
	}		}
Line 541:		Line 541:
	memory: "0.5 GB"		memory: "0.5 GB"
	disks: "local-disk 1 SSD"		disks: "local-disk 1 SSD"
	docker: "ubuntu:22.04"		docker: "ubuntu:24.04"
	}		}
	}		}

Anovak: /* Writing your own workflow */

2025-03-18T20:46:04Z

Writing your own workflow

← Older revision		Revision as of 13:46, 18 March 2025
Line 555:		Line 555:
	Then run it on the cluster with Toil:		Then run it on the cluster with Toil:

	~~mkdir -p logs~~		toil-wdl-runner --jobStore ./fizzbuzz_store --batchSystem slurm --slurmTime 00:10:00 --caching false --batchLogsDir ./logs fizzbuzz.wdl fizzbuzz.json -o fizzbuzz_out -m fizzbuzz_out.json
	~~export TOIL_SLURM_ARGS="--time=02:00:00 --partition=medium"~~
	toil-wdl-runner --jobStore ./fizzbuzz_store --batchSystem slurm --caching false --batchLogsDir ./logs fizzbuzz.wdl fizzbuzz.json -o fizzbuzz_out -m fizzbuzz_out.json

	Or locally:		Or locally:

Anovak: /* How do I delete files in WDL? */

2025-03-18T18:43:57Z

How do I delete files in WDL?

← Older revision		Revision as of 11:43, 18 March 2025
Line 674:		Line 674:
	WDL doesn't have a built-in way to delete files; if you run a task that deletes a file, it will still exist in Toil's job store storage.		WDL doesn't have a built-in way to delete files; if you run a task that deletes a file, it will still exist in Toil's job store storage.

	Toil [https://github.com/DataBiosphere/toil/commit/2de6eea2cc2e688b53062a98687445f0cca56669 recently gained support] for deleting files at the ''end'' of WDL workflows. So if you have a large file that you only need for part of your workflow, consider writing that ~~part~~ as a separate sub-<code>workflow</code> and invoking it with <code>call</code>. Then the file will be cleaned up when the child workflow ends, leaving more space ~~fro~~ files created in the parent workflow.		Toil [https://github.com/DataBiosphere/toil/commit/2de6eea2cc2e688b53062a98687445f0cca56669 recently gained support] for deleting files at the ''end'' of WDL workflows. So if you have a large file that you only need for part of your workflow, consider writing the part that creates and uses it as a separate sub-<code>workflow</code> and invoking it with <code>call</code>. Then the file will be cleaned up when the child workflow ends, leaving more space for files created in the parent workflow.

	=Additional WDL resources=		=Additional WDL resources=

Anovak: /* How do I delete files in WDL? */

2025-03-18T18:43:13Z

How do I delete files in WDL?

← Older revision		Revision as of 11:43, 18 March 2025
Line 674:		Line 674:
	WDL doesn't have a built-in way to delete files; if you run a task that deletes a file, it will still exist in Toil's job store storage.		WDL doesn't have a built-in way to delete files; if you run a task that deletes a file, it will still exist in Toil's job store storage.

	Toil [https://github.com/DataBiosphere/toil/commit/2de6eea2cc2e688b53062a98687445f0cca56669 recently gained support] for deleting files at the 'end' of WDL workflows. So if you have a large file that you only need for part of your workflow, consider writing that part as a separate sub-<code>workflow</code> and invoking it with <code>call</code>. Then the file will be cleaned up when the child workflow ends, leaving more space fro files created in the parent workflow.		Toil [https://github.com/DataBiosphere/toil/commit/2de6eea2cc2e688b53062a98687445f0cca56669 recently gained support] for deleting files at the ''end'' of WDL workflows. So if you have a large file that you only need for part of your workflow, consider writing that part as a separate sub-<code>workflow</code> and invoking it with <code>call</code>. Then the file will be cleaned up when the child workflow ends, leaving more space fro files created in the parent workflow.

	=Additional WDL resources=		=Additional WDL resources=

Anovak: /* How do I delete files in WDL? */

2025-03-18T18:43:03Z

How do I delete files in WDL?

← Older revision		Revision as of 11:43, 18 March 2025
Line 674:		Line 674:
	WDL doesn't have a built-in way to delete files; if you run a task that deletes a file, it will still exist in Toil's job store storage.		WDL doesn't have a built-in way to delete files; if you run a task that deletes a file, it will still exist in Toil's job store storage.

	Toil [https://github.com/DataBiosphere/toil/commit/2de6eea2cc2e688b53062a98687445f0cca56669 recently gained support] for deleting files at the end of WDL workflows. So if you have a large file that you only need for part of your workflow, consider writing that part as a separate sub-<code>workflow</code> and invoking it with <code>call</code>. Then the file will be cleaned up when the child workflow ends, leaving more space fro files created in the parent workflow.		Toil [https://github.com/DataBiosphere/toil/commit/2de6eea2cc2e688b53062a98687445f0cca56669 recently gained support] for deleting files at the 'end' of WDL workflows. So if you have a large file that you only need for part of your workflow, consider writing that part as a separate sub-<code>workflow</code> and invoking it with <code>call</code>. Then the file will be cleaned up when the child workflow ends, leaving more space fro files created in the parent workflow.

	=Additional WDL resources=		=Additional WDL resources=

Anovak: /* Frequently Asked Questions */

2025-03-18T18:42:50Z

Frequently Asked Questions

← Older revision		Revision as of 11:42, 18 March 2025
Line 670:		Line 670:

	If you are looking for the log for a worker process that did not finish (i.e. that crashed), make sure to look on the machine that the worker actually ran on, not on the head node.		If you are looking for the log for a worker process that did not finish (i.e. that crashed), make sure to look on the machine that the worker actually ran on, not on the head node.

			===How do I delete files in WDL?===
			WDL doesn't have a built-in way to delete files; if you run a task that deletes a file, it will still exist in Toil's job store storage.

			Toil [https://github.com/DataBiosphere/toil/commit/2de6eea2cc2e688b53062a98687445f0cca56669 recently gained support] for deleting files at the end of WDL workflows. So if you have a large file that you only need for part of your workflow, consider writing that part as a separate sub-<code>workflow</code> and invoking it with <code>call</code>. Then the file will be cleaned up when the child workflow ends, leaving more space fro files created in the parent workflow.

	=Additional WDL resources=		=Additional WDL resources=

Anovak: /* Installing Toil with WDL support */

2025-02-14T19:09:08Z

Installing Toil with WDL support

← Older revision		Revision as of 12:09, 14 February 2025
Line 50:		Line 50:
	python3 -m pipx ensurepath		python3 -m pipx ensurepath

	This may instruct you to log out and log back in or take some other action to adopt the new <code>PATH</code> settings.		This may instruct you to log out and log back in or take some other action to adopt the new <code>PATH</code> settings.

	When installing Toil, you need to specify that you want WDL support. To do this, you can run:		When installing Toil, you need to specify that you want WDL support. To do this, you can run:
Line 64:		Line 64:
	This will install Toil in the <code>.local</code> directory inside your home directory, which we write as <code>~/.local</code>. The program to run WDL workflows, <code>toil-wdl-runner</code>, will be at <code>~/.local/bin/toil-wdl-runner</code>. The <code>python3 -m pipx ensurepath</code> command should have added the <code>~/.local</code> directory to your <code>PATH</code> environment variable, to ensure you can find these commands.		This will install Toil in the <code>.local</code> directory inside your home directory, which we write as <code>~/.local</code>. The program to run WDL workflows, <code>toil-wdl-runner</code>, will be at <code>~/.local/bin/toil-wdl-runner</code>. The <code>python3 -m pipx ensurepath</code> command should have added the <code>~/.local</code> directory to your <code>PATH</code> environment variable, to ensure you can find these commands.

	~~After that~~, ~~log out and log back in~~, ~~to restart bash~~ and ~~pick up the change~~.		If you see something from <code>pipx</code> like:

			- cwltoil (symlink missing or pointing to unexpected location)

			Then <code>pipx uninstall toil</code>, remove the offending file from <code>~/.local/bin</code>, and try again.

	To make sure it worked, you can run:		To make sure it worked, you can run:

Anovak at 19:03, 14 February 2025

2025-02-14T19:03:18Z

← Older revision		Revision as of 12:03, 14 February 2025
Line 50:		Line 50:
	python3 -m pipx ensurepath		python3 -m pipx ensurepath

	When installing, you need to specify that you want WDL support. To do this, you can run:		This may instruct you to log out and log back in or take some other action to adopt the new <code>PATH</code> settings.

			When installing Toil, you need to specify that you want WDL support. To do this, you can run:

	pipx install 'toil[wdl]'		pipx install 'toil[wdl]'
Line 57:		Line 59:

	pipx install 'toil[wdl,aws,google]'		pipx install 'toil[wdl,aws,google]'

			To change what extras are used when you have an existing Toil installation, you will need to use the <code>--force</code> option.

	This will install Toil in the <code>.local</code> directory inside your home directory, which we write as <code>~/.local</code>. The program to run WDL workflows, <code>toil-wdl-runner</code>, will be at <code>~/.local/bin/toil-wdl-runner</code>. The <code>python3 -m pipx ensurepath</code> command should have added the <code>~/.local</code> directory to your <code>PATH</code> environment variable, to ensure you can find these commands.		This will install Toil in the <code>.local</code> directory inside your home directory, which we write as <code>~/.local</code>. The program to run WDL workflows, <code>toil-wdl-runner</code>, will be at <code>~/.local/bin/toil-wdl-runner</code>. The <code>python3 -m pipx ensurepath</code> command should have added the <code>~/.local</code> directory to your <code>PATH</code> environment variable, to ensure you can find these commands.

Anovak: /* Installing Toil with WDL support */

2025-02-14T19:00:28Z

Installing Toil with WDL support

← Older revision		Revision as of 12:00, 14 February 2025
Line 43:		Line 43:
	==Installing Toil with WDL support==		==Installing Toil with WDL support==

	Once you are on the head node, you can install Toil, a program for running workflows. ~~When installing, you need to specify that you want WDL support. To do this, you can run:~~		Once you are on the head node, you can install Toil, a program for running workflows.

	~~pip~~ install ~~--upgrade --user 'toil~~[~~wdl~~]'		Toil is written in Python, and the modern way to install Python command line tools is with pipx. So [https://pipx.pypa.io/latest/installation/ install pipx]:

	~~If you also want to use AWS S3 <code>s3://</code> and/or Google <code>gs://</code> URLs for data~~, you ~~will~~ need to ~~also install Toil with the <code>aws</code> and <code>google</code> extras~~, ~~respectively~~:		python3 -m pip install --user pipx
			python3 -m pipx ensurepath

			When installing, you need to specify that you want WDL support. To do this, you can run:

	~~pip~~ install ~~--upgrade --user~~ 'toil[wdl~~,aws,google~~]'		pipx install 'toil[wdl]'

	~~This will install Toil in the~~ <code>~~.local~~</code> ~~directory inside your home directory, which we write as~~ <code>~/~~.local~~</code>~~. The program~~ to ~~run WDL workflows,~~ <code>~~toil-wdl-runner~~</code>~~, will be at~~ <code>~~~/.local/bin/toil-wdl-runner~~</code>.		If you also want to use AWS S3 <code>s3://</code> and/or Google <code>gs://</code> URLs for data, you will need to also install Toil with the <code>aws</code> and <code>google</code> extras, respectively:

	~~By default, the command interpreter will not look there, so if you type <code>~~toil-wdl~~-runner</code>, it will complain that the command is not found. To fix this~~, ~~you need to configure the command interpreter (bash) to look where Toil is installed. To do this~~, ~~run:~~		pipx install 'toil[wdl,aws,google]'

	~~echo 'export PATH="${HOME}~~/.local/bin~~:${PATH}"'~~ >>~/.~~bashrc~~		This will install Toil in the <code>.local</code> directory inside your home directory, which we write as <code>~/.local</code>. The program to run WDL workflows, <code>toil-wdl-runner</code>, will be at <code>~/.local/bin/toil-wdl-runner</code>. The <code>python3 -m pipx ensurepath</code> command should have added the <code>~/.local</code> directory to your <code>PATH</code> environment variable, to ensure you can find these commands.

	After that, log out and log back in, to restart bash and pick up the change.		After that, log out and log back in, to restart bash and pick up the change.