Slurm Tips for vg: Difference between revisions

From UCSC Genomics Institute Computing Infrastructure Information

No edit summary
(→‎Setting Up: Up[date source code storage location and fix build instructions to cover dealing with dependencies on Slurm)
 
(5 intermediate revisions by the same user not shown)
Line 3: Line 3:
==Setting Up==
==Setting Up==


1. After connecting to the VPN, connect to the cluster head node:
1. After connecting to the VPN, connect to an interactive node:


  ssh phoenix.prism
  ssh razzmatazz.prism


This node is relatively small, so you shouldn't run real work on it, but it is the place you need to be to submit Slurm jobs.
This node is relatively small, so you shouldn't run real work on it, but it is the place you need to be to submit Slurm jobs.


2. Make yourself a user directory under '''/private/groups''', which is where large data must be stored. For example, if you are in the Paten lab:
2. Make yourself a user directory under '''/private/groups''', which is where large data (but NOT code) must be stored. For example, if you are in the Paten lab:


  mkdir /private/groups/patenlab/$USER
  mkdir /private/groups/patenlab/$USER


3. (Optional) Link it over to your home directory, so it is easy to use storage there to store your repos. The '''/private/groups''' storage may be faster than the home directory storage.
3. Make sure you have SSH keys created and add them to Github.


mkdir -p /private/groups/patenlab/$USER/workspace
  cat ~/.ssh/id_ed25519.pub || (ssh-keygen -t ed25519 && cat  ~/.ssh/id_ed25519.pub)
ln -s /private/groups/patenlab/$USER/workspace ~/workspace
 
4. Make sure you have SSH keys created and add them to Github.
 
  cat ~/.ssh/id_ed25519.pub || (ssh-keygen -t dsa && cat  ~/.ssh/id_ed25519.pub)
  # Paste into https://github.com/settings/ssh/new
  # Paste into https://github.com/settings/ssh/new


5. Make a place to put your clone, and clone vg:
4. Make a place to put your clone, and clone vg:


  mkdir -p ~/workspace
  mkdir -p ~/workspace
Line 30: Line 25:
  cd vg
  cd vg


6. vg's dependencies should already be installed on the cluster nodes. If any of them seem to be missing, tell cluster-admin@soe.ucsc.edu to install them.
5. Make a symlink to a directory you can put large files in that is easily accessed from your vg directory.
 
mkdir /private/groups/patenlab/$USER/data
ln -s mkdir /private/groups/patenlab/$USER/data ~/workspace/vg/data
 
6. Build vg. To build on an interactive node:
 
make -j16
 
To build as a cluster job (possibly faster):


7. Build vg as a Slurm job. This will send the build out to the cluster as a 64-core, 80G memory job, and keep the output logs in your terminal.
srun -c 64 --mem=80G --time=00:30:00 make -j64


srun -c 64 --mem=80G make -j64
If you run into build failures a cluster job, try building on the interactive node instead. If that works, we might have forgotten to install all the vg dependencies on the cluster nodes. We want vg's dependencies to be installed on the cluster nodes and all the interactive nodes, but sometimes we add new dependencies and they don't get rolled out everywhere. If any dependencies seem to be missing, ask cluster-admin@soe.ucsc.edu to install them.


This will leave your vg binary at '''~/workspace/vg/bin/vg'''.
Either build approach will leave your vg binary at '''~/workspace/vg/bin/vg'''.


==Misc Tips==
==Misc Tips==


* If you want an interactive session with appreciable resources, you can schedule one with '''srun'''. For example, to get 16 cores and 120G memory all for you, run:
* For a lightweight job that outputs to your terminal or that can be waited for in a Bash script, run an individual command directly from <code>srun</code>:
 
srun -c1 --mem 2G --partition short --time 1:00:00 sleep 10
 
* If you need to run a few commands in the same shell, use <code>sbatch --wrap</code>:
 
sbatch -c1 --mem 2G --partition short --time 1:00:00 --wrap ". venv/bin/activate; ./script1.py && ./script2.py"
 
* To watch a batch job's output live, look at the <code>Submitted batch job 5244464</code> line from <code>sbatch</code> and run:
 
tail -f slurm-5244464.out
 
* '''Danger!''' If you ''really'' need an interactive session with appreciable resources, you can schedule one with <code>srun --pty</code>. But it is '''very easy''' to waste resources like this, since the job will happily sit there not doing anything until it hits the timeout. Only do this for testing! For real work, use one of the other methods!
 
srun -c 16 --mem 120G --time=08:00:00 --partition=medium --pty bash -i
 
* To send out a job without making a script file for it, use '''sbatch --wrap "your command here"'''.
 
* You can use arguments from SBATCH lines on the command line!


srun -c 16 --mem 120G --pty bash -i
* You can use [https://github.com/CLIP-HPC/SlurmCommander#readme Slurm Commander] to watch the state of the cluster with the '''scom''' command.

Latest revision as of 07:33, 19 October 2026

This page explains how to set up a development environment for vg on the Phoenix cluster.

Setting Up

1. After connecting to the VPN, connect to an interactive node: 

ssh razzmatazz.prism

This node is relatively small, so you shouldn't run real work on it, but it is the place you need to be to submit Slurm jobs.

2. Make yourself a user directory under /private/groups, which is where large data (but NOT code) must be stored. For example, if you are in the Paten lab: 

mkdir /private/groups/patenlab/$USER

3. Make sure you have SSH keys created and add them to Github. 

cat ~/.ssh/id_ed25519.pub || (ssh-keygen -t ed25519 && cat  ~/.ssh/id_ed25519.pub)
# Paste into https://github.com/settings/ssh/new

4. Make a place to put your clone, and clone vg: 

mkdir -p ~/workspace
cd ~/workspace
git clone --recursive git@github.com:vgteam/vg.git
cd vg

5. Make a symlink to a directory you can put large files in that is easily accessed from your vg directory. 

mkdir /private/groups/patenlab/$USER/data
ln -s mkdir /private/groups/patenlab/$USER/data ~/workspace/vg/data

6. Build vg. To build on an interactive node: 

make -j16

To build as a cluster job (possibly faster): 

srun -c 64 --mem=80G --time=00:30:00 make -j64

If you run into build failures a cluster job, try building on the interactive node instead. If that works, we might have forgotten to install all the vg dependencies on the cluster nodes. We want vg's dependencies to be installed on the cluster nodes and all the interactive nodes, but sometimes we add new dependencies and they don't get rolled out everywhere. If any dependencies seem to be missing, ask cluster-admin@soe.ucsc.edu to install them.

Either build approach will leave your vg binary at ~/workspace/vg/bin/vg.

Misc Tips

  • For a lightweight job that outputs to your terminal or that can be waited for in a Bash script, run an individual command directly from srun:
srun -c1 --mem 2G --partition short --time 1:00:00 sleep 10
  • If you need to run a few commands in the same shell, use sbatch --wrap:
sbatch -c1 --mem 2G --partition short --time 1:00:00 --wrap ". venv/bin/activate; ./script1.py && ./script2.py"
  • To watch a batch job's output live, look at the Submitted batch job 5244464 line from sbatch and run:
tail -f slurm-5244464.out
  • Danger! If you really need an interactive session with appreciable resources, you can schedule one with srun --pty. But it is very easy to waste resources like this, since the job will happily sit there not doing anything until it hits the timeout. Only do this for testing! For real work, use one of the other methods!
srun -c 16 --mem 120G --time=08:00:00 --partition=medium --pty bash -i
  • To send out a job without making a script file for it, use sbatch --wrap "your command here".
  • You can use arguments from SBATCH lines on the command line!
  • You can use Slurm Commander to watch the state of the cluster with the scom command.
Retrieved from "http://giwiki.gi.ucsc.edu/index.php?title=Slurm_Tips_for_vg&oldid=836"