View source for Phoenix WDL Tutorial

=Tutorial: Getting Started with WDL Workflows on Phoenix=

Instead of giant shell scripts that only work on one grad student's laptop, modern, reusable bioinformatics experiments should be written as workflows, in a language like Workflow Description language (WDL). Workflows succinctly describe their own execution requirements, and which pieces depend on which other pieces, making your analyses reproducible by people other than you.

Workflows are also easily scaled up and down: you can develop and test your workflow on a small test data set on one machine, and then run it on real data on the cluster without having to worry about whether the right tasks will run in the right order.

This tutorial will help you get started writing and running workflows on the Genomics Institute's Phoenix cluster. By the end, you will be able to run workflows on Phoenix with [https://toil.readthedocs.io/en/latest/ Toil], write your own workflows in WDL, and debug workflows when something goes wrong.

==Setup==

Before we begin, you will need a computer to work at, which you are able to install software on, and the ability to connect to other machines over SSH.

===Getting VPN access===

We are going to work on the Phoenix cluster, but this cluster is kept behind the Prism firewall, where all of our controlled-access data lives. So, to get access to the cluster, you need to get access to the VPN (Virtual Private Network) system that we use to allow people through the firewall.

To get VPN access, follow the instructions at https://giwiki.gi.ucsc.edu/index.php/Requirement_for_users_to_get_GI_VPN_access. Note that this process involves making a one-on-one appointment with one of our admins to help you set up your VPN client, so make sure to do it in advance of when you need to use the cluster.

===Connecting to Phoenix===

Once you have VPN access, you can connect to the "head node" of the Phoenix cluster. This node is where everyone logs in, but you should *not* run actual work on this node; it exists only to give you access to the files on the cluster and to the commands to control cluster jobs.

To connect to the head node:

1. Connect to the VPN.
2. SSH to <code>phoenix.prism</code>. At the command line, run:

 ssh phoenix.prism

If your username on the cluster (say, <code>flastname</code>) is different than your username on your computer (which might be <code>firstname</code>), you might instead have to run:

 ssh flastname@phoenix.prism

The first time you connect, you will see a message like:

 The authenticity of host 'phoenix.prism (10.50.1.66)' can't be established.
 ED25519 key fingerprint is SHA256:SUgdBXgsWwUJXxAz/BpGzlGFLOsFtZzeqQ3kzdl3iuI.
 This key is not known by any other names
 Are you sure you want to continue connecting (yes/no/[fingerprint])?

This is your computer asking you to help it decide if it is talking to the genuine <code>phoenix.prism</code>, and not an imposter. You will want to make sure that the "key fingerprint" is indeed <code>SHA256:SUgdBXgsWwUJXxAz/BpGzlGFLOsFtZzeqQ3kzdl3iuI</code>. If it is not, someone (probably the GI sysadmins, but possibly a cabal of hackers) has replaced the head node, and you should verify that this was supposed to happen. If the fingerprints do match, type <code>yes</code> to accept and remember that the server is who it says it is.

===Installing Toil with WDL support===

Once you are on the head node, you can install Toil, a program for running workflows. When installing, you need to specify that you want WDL support. To do this, you can run:

 pip install --upgrade --user toil[wdl]

This will install Toil in the <code>.local</code> directory inside your home directory, which we write as </code>~/.local</code>. The program to run WDL workflows, <code>toil-wdl-runner</code>, will be at <code>~/.local/bin/toil-wdl-runner</code>.

By default, the command interpreter *will not* look there, so if you type <code>toil-wdl-runner</code>, it will complain that the command is not found. To fix this, you need to configure the command interpreter (bash) to look where Toil is installed. To do this, run:

 echo 'export PATH="${HOME}/.local/bin:${PATH}"' >>~/.bashrc

After that, **log out and log back in**, to restart bash and pick up the change.

To make sure it worked, you can run:

 toil-wdl-runner --help

If everything worked correctly, it will print a long list of the various option flags that the <code>toil-wdl-runner</code> command supports.

If you ever want to upgrade Toil to a new release, you can repeat the <code>pip</code> command above.

===Configuring Toil for Phoenix===

Toil is set up to work in a large number of different environments, and doesn't necessarily rely on the existence of things like a shared cluster filesystem. However, on the Phoenix cluster, we have a shared filesystem, and so we should configure Toil to use it for caching the Docker container images used for running workflow steps. So, use these commands to make sure that Toil knows where it ought to put its caches:

 echo 'export SINGULARITY_CACHEDIR="${HOME}/.singularity/cache"' >>~/.bashrc
 echo 'export MINIWDL__SINGULARITY__IMAGE_CACHE="${HOME}/.cache/miniwdl"' >>~/.bashrc

After that, **log out and log back in again**, to apply the changes.

If you don't do this, Toil will re-download each container image, on each node, for each run of each workflow. That wastes a lot of time, and can exhaust the [https://docs.docker.com/docker-hub/download-rate-limit/#whats-the-download-rate-limit-on-docker-hub limits on how many containers you are allowed to download each day], so it is important not to skip this step.

==Running an existing workflow==

First, let's use <code>toil-wdl-runner</code> to run an existing demonstration workflow. We're going to use the MiniWDL self-test workflow, from the [https://github.com/chanzuckerberg/miniwdl#readme MiniWDL project].

First, download the workflow. While Toil can run workflows directly from a URL, your commands will be shorter if the workflow is available locally.

 wget https://raw.githubusercontent.com/DataBiosphere/toil/d686daca091849e681d2f3f3a349001ca83d2e3e/src/toil/test/wdl/miniwdl_self_test/self_test.wdl

===Preparing an input file===

Near the top of the WDL file, there's a section like this:

 workflow hello_caller {
     input {
         File who
     }

This means that there is a workflow named <code>hello_caller</code> in this file, and it takes as input a file variable named <code>who</code>. For this particular workflow, the file is supposed to have a list of names, one per line, and the workflow is going to greet each one.

So first, we have to make that list of names. Let's make it in <code>names.txt</code>

 echo "Mridula Resurrección" >names.txt
 echo "Gershom Šarlota" >>names.txt
 echo "Ritchie Ravi" >>names.txt

Then, we need to create an *inputs file*, which is a JSON (JavaScript Object Notation) file describing what value to use for each input when running the workflow. (You can also reach down into the workflow and override individual task settings, but for now we'll just set the inputs.) So, make another file next to <code>names.txt</code> that references it by relative path, like this:

 echo '{"hello_caller.who": "./names.txt"}' >inputs.json

Note that, for a key, we're using the workflow name, a dot, and then the input name. For a value, we're using a quoted string of the filename, relative to the location of the inputs file. Absolute paths and URLs will also work for files; more information on the input file syntax is in [https://github.com/openwdl/wdl/blob/main/versions/1.1/SPEC.md#json-input-format the JSON Input Format section of the WDL specification].

===Testing at small scale single-machine===

We are now ready to run the workflow!

You don't want to run workflows on the head node. So, use Slurm to get an interactive session on one of the cluster's worker nodes, by running:

 srun -c 2 --mem 8G --pty bash -i

This will start a new shell; to leave it and go back to the head node you can use <code>exit</code>.

In your new shell, run this Toil command:

 toil-wdl-runner self_test.wdl inputs.json -o local_run

This will, by default, use the <code>single_machine</code> Toil "batch system" to run all of the workflow's tasks locally. Output will be sent to a new directory named <code>local_run</code>.

This will print a lot of logging to standard error, and to standard output it will print:

 {"hello_caller.message_files": ["local_run/Mridula Resurrecci\u00f3n.txt", "local_run/Gershom \u0160arlota.txt", "local_run/Ritchie Ravi.txt"], "hello_caller.messages": ["Hello, Mridula Resurrecci\u00f3n!", "Hello, Gershom \u0160arlota!", "Hello, Ritchie Ravi!"]}

The <code>local_run</code> directory will contain the described text files (with Unicode escape sequences like <code>\u00f3</code> replaced by their corresponding characters), each containing a greeting for the corresponding person.

To leave your interactive Slurm session and return to the head node, use <code>exit</code>.

===Running at larger scale===

Back on the head node, let's prepare to run a larger run. Greeting 3 people isn't cool, let's greet one hundred people!

Go get this handy list of people and cut it to length:

 wget https://gist.githubusercontent.com/smsohan/ae142977b5099dba03f6e0d909108e97/raw/f6e319b1a0f6a0f87f93f73b3acd24795361aeba/1000_names.txt
 head -n100 1000_names.txt >100_names.txt

And make a new inputs file:

 echo '{"hello_caller.who": "./100_names.txt"}' >inputs_big.json

Now, we will run the same workflow, but with the new inputs, and against the Slurm cluster.

To run against the Slurm cluster, we need to use the <code>--jobStore</code> option to point Toil to a shared directory it can create where it can store information that the cluster nodes can read. We will add the <code>--batchLogsDir</code> option to tell Toil to store the logs from the individual Slurm jobs in a folder on the shared filesystem. We'll also use the <code>-m</code> option to save the output JSON to a file instead of printing it.

 mkdir -p logs
 toil-wdl-runner --jobStore ./big_store --batchSystem slurm --batchLogsDir ./logs self_test.wdl inputs_big.json -o slurm_run -m slurm_run.json

This will tick for a while, but eventually you should end up with 100 greeting files in the <code>slurm_run</code> directory.

==Writing your own workflow==

In addition to running existing workflows, you probably want to be able to write your own. This part of the tutorial will walk you through writing a workflow. We're going to write a workflow for [https://en.wikipedia.org/wiki/Fizz_buzz Fizz Buzz]. 

===Writing the file===

All WDL files need to start with a <code>version</code> statement (unless they are very old <code>draft-2</code> files. Toil supports <code>draft-2</code>, WDL 1.0, and WDL 1.1, while Cromwell (another popular WDL runner used on Terra) supports only <code>draft-2</code> and 1.0.

So let's start a new WDL 1.0 workflow. Open up a file named <code>fizzbuzz.wdl</code> and start with a version statement:

 version 1.0

Then, add an empty <code>workflow</code> named <code>FizzBuzz</code>.
 
 version 1.0
 workflow FizzBuzz {
 }

Workflows usually need some kind of user input, so let's give our workflow an <code>input</code> section.

 version 1.0
 workflow FizzBuzz {
     input {
         # How many FizzBuzz numbers do we want to make?
         Int item_count
         # Every multiple of this number, we produce "Fizz"
         Int to_fizz = 3
         # Every multiple of this number, we produce "Buzz"
         Int to_buzz = 5
         # Optional replacement for the string to print when a multiple of both
         String? fizzbuzz_override
     }
 }

Notice that each input has a type, a name, and an optional default value. If the type ends in <code>?</code>, the value is optional, and it may be <code>null</code>. If an input is *not* optional, and there is no default value, then the user's inputs file *must* specify a value for it in order for the workflow to run.

Now we'll start on the body of the workflow, to be inserted just after the inputs section. 

The first thing we're going to need to do is create an array of all the numbers up to the <code>item_count</code>. We can do this by calling the WDL <code>range()</code> function, and assigning the result to an <code>Array[Int]</code> variable.

 Array[Int] numbers = range(item_count)

WDL 1.0 has [https://github.com/openwdl/wdl/blob/main/versions/1.0/SPEC.md#standard-library a wide variety of functions in its standard library], and WDL 1.1 has even more.

Once we create an array of all the numbers, we can use a <code>scatter</code> to operate on each. WDL does not have loops; instead it has scatters, which work a bit like a <code>map()</code> in Python. The body of the scatter runs for each value in the input array, all in parallel. We're going to increment all the numbers, since FizzBuzz starts at 1 but WDL <code>range()</code> starts at 0.

 Array[Int] numbers = range(item_count)
 scatter (i in numbers) {
     Int one_based = i + 1
 }

Inside the body of the scatter, we are going to put some conditionals to determine if we should produce <code>"Fizz"</code>, <code>"Buzz"</code>, or <code>"FizzBuzz"</code>. To support our <code>fizzbuzz_override</code>, we use an array of it and a default value, and use the WDL <code>select_first()</code> function to find the first non-null value in that array.

Each execution of a scatter is allowed to declare variables, and outside the scatter those variables are combined into arrays of all the results. But each variable can be declared only *once* in the scatter, even with conditionals. So we're going to use <code>select_first()</code> at the end and take advantage of variables from un-executed conditionals being <code>null</code>. 

Note that WDL supports conditional *expressions* with a <code>then</code> and an <code>else</code>, but conditional *statements* only have a body, not an <code>else</code> branch. If you need an else you will have to check the negated condition.

So first, let's handle the special cases.

 Array[Int] numbers = range(item_count)
 scatter (i in numbers) {
    Int one_based = i + 1
    if (one_based % to_fizz == 0) {
        String fizz = "Fizz"
        if (one_based % to_buzz == 0) {
            String fizzbuzz = select_first([fizzbuzz_override, "FizzBuzz"])
        }
    }
    if (one_based % to_buzz == 0) {
        String buzz = "Buzz"
    }
    if (one_based % to_fizz != 0 && one_based % to_buzz != 0) {
        # Just a normal number.
    }
 }

Now for the normal numbers, we need to convert our number into a string. In WDL 1.1, and in WDL 1.0 on Cromwell, you can use a <code>${}</code> substitution syntax in quoted strings anywhere, not just in command line commands. Toil technically will support this too, but it's not in the spec, and the tutorial needs an excuse for you to call a task. So we're going to insert a call to a <code>stringify_number</code> task, to be written later.

To call a task (or another workflow), we use a <code>call</code> statement and give it some inputs. Then we can fish the output values out of the task with </code>.</code> access, only if we don't make a noise instead.


 Array[Int] numbers = range(item_count)
 scatter (i in numbers) {
    Int one_based = i + 1
    if (one_based % to_fizz == 0) {
        String fizz = "Fizz"
        if (one_based % to_buzz == 0) {
            String fizzbuzz = select_first([fizzbuzz_override, "FizzBuzz"])
        }
    }
    if (one_based % to_buzz == 0) {
        String buzz = "Buzz"
    }
    if (one_based % to_fizz != 0 && one_based % to_buzz != 0) {
        # Just a normal number.
        call stringify_number {
            input:
                the_number = one_based
        }
    }
    String result = select_first([fizzbuzz, fizz, buzz, stringify_number.the_string])
 }

We can put the code into the workflow now, and set about writing the task.

 version 1.0
 workflow FizzBuzz {
     input {
         # How many FizzBuzz numbers do we want to make?
         Int item_count
         # Every multiple of this number, we produce "Fizz"
         Int to_fizz = 3
         # Every multiple of this number, we produce "Buzz"
         Int to_buzz = 5
         # Optional replacement for the string to print when a multiple of both
         String? fizzbuzz_override
     }
     Array[Int] numbers = range(item_count)
     scatter (i in numbers) {
         Int one_based = i + 1
         
         if (one_based % to_fizz == 0) {
             String fizz = "Fizz"
             if (one_based % to_buzz == 0) {
                 String fizzbuzz = select_first([fizzbuzz_override, "FizzBuzz"])
             }
          }
         if (one_based % to_buzz == 0) {
             String buzz = "Buzz"
         }
         if (one_based % to_fizz != 0 && one_based % to_buzz != 0) {
             # Just a normal number.
             call stringify_number {
                 input:
                     the_number = one_based
             }
         }
         String result = select_first([fizzbuzz, fizz, buzz, stringify_number.the_string]
     }
 }

Our task should go after the workflow in the file. It looks a lot like a workflow except it uses <code>task</code>.

 task stringify_number {
 }

We're going to want it to take in an integer <code>the_number</code>, and we're going to want it to output a string <code>the_string</code>. So let's fill that in in <code>input</code> and <code>output</code> sections.

 task stringify_number {
     input {
         Int the_number
     }
     # ???
     output {
         String the_string # = ???
     }
 }

Now, unlike workflows, tasks can have a <code>command</code> section, which gives a command to run. This section is now usually set off with triple angle brackets, and inside it you can use <code>~{}</code>, that is, Bash-like substitution but with a tilde, to place WDL variables into your command script. So let's add a command that will echo back the number so we can see it as a string.

 task stringify_number {
     input {
         Int the_number
     }
     command <<<
        # This is a Bash script.
        # So we should do good Bash script things like stop on errors
        set -e

        # Now print our number as a string
        echo ~{the_number}
     >>>
     output {
         String the_string # = ???
     }
 }

Now we need to capture the result of the command script. The WDL <code>stdout()</code> returns a WDL <code>File</code> containing the standard output printed by the task's command. We want to read that back into a string, which we can do with the WDL <code>read_string()</code> function (which also [https://github.com/openwdl/wdl/blob/main/versions/1.0/SPEC.md#string-read_stringstringfile removes trailing newlines]).

 task stringify_number {
     input {
         Int the_number
     }
     command <<<
        # This is a Bash script.
        # So we should do good Bash script things like stop on errors
        set -e
        # Now print our number as a string
        echo ~{the_number}
     >>>
     output {
         String the_string = read_string(stdout())
     }
 }

We're also going to want to add a <code>runtime</code> section to our task, to specify resource requirements. We're also going to tell it to run in a Docker container, to make sure that absolutely nothing can go wrong with our delicate <code>echo</code> command. In a real workflow, you probably want to set up optiopnal inputs for all the tasks to let you control the resource requirements, but here we will just hardcode them.

 task stringify_number {
     input {
         Int the_number
     }
     command <<<
         # This is a Bash script.
         # So we should do good Bash script things like stop on errors
         set -e
         # Now print our number as a string
         echo ~{the_number}
     >>>
     output {
         String the_string = read_string(stdout())
     }
     runtime {
         cpu: 1
         memory: "0.5 GB"
         disks: "local-disk 1 SSD"
         docker: "ubuntu:22.04"
     }
 }

The <code>disks</code> section is a little weird; it isn't in the WDL spec, but Toil supports Cromwell-style strings that ask for a <code>local-disk</code> of a certain number of gigabytes, which may suggest that it be <code>SSD</code> storage.

Then we can put our task into our WDL file:

 version 1.0
 workflow FizzBuzz {
     input {
         # How many FizzBuzz numbers do we want to make?
         Int item_count
         # Every multiple of this number, we produce "Fizz"
         Int to_fizz = 3
         # Every multiple of this number, we produce "Buzz"
         Int to_buzz = 5
         # Optional replacement for the string to print when a multiple of both
         String? fizzbuzz_override
     }
     Array[Int] numbers = range(item_count)
     scatter (i in numbers) {
         Int one_based = i + 1
         if (one_based % to_fizz == 0) {
             String fizz = "Fizz"
             if (one_based % to_buzz == 0) {
                 String fizzbuzz = select_first([fizzbuzz_override, "FizzBuzz"])
             }
          }
         if (one_based % to_buzz == 0) {
             String buzz = "Buzz"
         }
         if (one_based % to_fizz != 0 && one_based % to_buzz != 0) {
             # Just a normal number.
             call stringify_number {
                 input:
                     the_number = one_based
             }
         }
         String result = select_first([fizzbuzz, fizz, buzz, stringify_number.the_string]
     }
 }
 task stringify_number {
     input {
         Int the_number
     }
     command <<<
         # This is a Bash script.
         # So we should do good Bash script things like stop on errors
         set -e
         # Now print our number as a string
         echo ~{the_number}
     >>>
     output {
         String the_string = read_string(stdout())
     }
     runtime {
         cpu: 1
         memory: "0.5 GB"
         disks: "local-disk 1 SSD"
         docker: "ubuntu:22.04"
     }
 }

Now the only thing missing is a workflow-level <code>output</code> section. Technically, in WDL 1.0 you aren's supposed to need this, but you do need it in 1.1 and Toil doesn't actually support not having one yet, so we're going to make one. We need to collect together all the strings that came out of the different tasks in our scatter into an <code>Array[String]</code>. We'll add the <code>output</code> section at the end of the <code>workflow</code> section, above the task.


 version 1.0
 workflow FizzBuzz {
     input {
         # How many FizzBuzz numbers do we want to make?
         Int item_count
         # Every multiple of this number, we produce "Fizz"
         Int to_fizz = 3
         # Every multiple of this number, we produce "Buzz"
         Int to_buzz = 5
         # Optional replacement for the string to print when a multiple of both
         String? fizzbuzz_override
     }
     Array[Int] numbers = range(item_count)
     scatter (i in numbers) {
         Int one_based = i + 1
         if (one_based % to_fizz == 0) {
             String fizz = "Fizz"
             if (one_based % to_buzz == 0) {
                 String fizzbuzz = select_first([fizzbuzz_override, "FizzBuzz"])
             }
          }
         if (one_based % to_buzz == 0) {
             String buzz = "Buzz"
         }
         if (one_based % to_fizz != 0 && one_based % to_buzz != 0) {
             # Just a normal number.
             call stringify_number {
                 input:
                     the_number = one_based
             }
         }
         String result = select_first([fizzbuzz, fizz, buzz, stringify_number.the_string]
     }
     output {
        Array[String] fizzbuzz_results = result
     }
 }
 task stringify_number {
     input {
         Int the_number
     }
     command <<<
         # This is a Bash script.
         # So we should do good Bash script things like stop on errors
         set -e
         # Now print our number as a string
         echo ~{the_number}
     >>>
     output {
         String the_string = read_string(stdout())
     }
     runtime {
         cpu: 1
         memory: "0.5 GB"
         disks: "local-disk 1 SSD"
         docker: "ubuntu:22.04"
     }
 }

Because the <code>result</code> variable is defined inside a <code>scatter</code>, when we reference it outside the scatter we see it as being an array.

Now all that remains is to run the workflow! As before, make an inputs file to specify the workflow inputs:

 echo '{"FizzBuzz.item_count": 20}' >fizzbuzz.json

Then run it on the cluster with Toil:

 mkdir -p logs
 toil-wdl-runner --jobStore ./fizzbuzz_store --batchSystem slurm --batchLogsDir ./logs fizzbuzz.wdl fizzbuzz.json -o fizzbuzz_out -m fizzbuzz_out.json

Or locally:

    toil-wdl-runner fizzbuzz.wdl fizzbuzz.json -o fizzbuzz_out -m fizzbuzz_out.json

==Debugging Workflows==

===Frequently Asked Questions===

====I am getting warnings that <code>XDG_RUNTIME_DIR is set to nonexistent directory /run/user/$UID; your environment may be out of spec!</code>====

This happens because Slurm is providing Toil with an <code>XDG_RUNTIME_DIR</code> environment variable that points to a directory that doesn't exist, which the XDG spec says it shouldn't be doing. This is a known bug in the GI Slurm configuration, and Toil is letting you know that it is working around it.

====Toil said it was <code>Redirecting logging to /data/tmp/somewhere/worker_log.txt</code> but I can't find that file!====

The Toil worker logs are automatically cleaned up when the worker finishes. If you want to see the individual worker logs in the Toil log, use the <code>--logDebug</code> option to Toil.

If you are looking for the log for a worker process that did not finish (i.e. that crashed), make sure to look on the machine that the worker actually ran on, not on the head node.

==Additional WDL resources==

For more information on writing and running WDL workflows, see:

* [https://docs.openwdl.org/en/stable/ The WDL dcoumentation]
* [https://www.youtube.com/playlist?list=PL4Q4HssKcxYv5syJKUKRrD8Fbd-_CnxTM The "Learn WDL" video course on YouTube]
* [https://github.com/openwdl/wdl/blob/main/versions/1.1/SPEC.md The WDL 1.1 language specification]