Slurm Tips for vg - Revision history

Anovak: /* Setting Up */ Up[date source code storage location and fix build instructions to cover dealing with dependencies on Slurm

2026-10-19T15:33:00Z

Setting Up: Up[date source code storage location and fix build instructions to cover dealing with dependencies on Slurm

← Older revision		Revision as of 07:33, 19 October 2026
Line 9:		Line 9:
	This node is relatively small, so you shouldn't run real work on it, but it is the place you need to be to submit Slurm jobs.		This node is relatively small, so you shouldn't run real work on it, but it is the place you need to be to submit Slurm jobs.

	2. Make yourself a user directory under '''/private/groups''', which is where large data must be stored. For example, if you are in the Paten lab:		2. Make yourself a user directory under '''/private/groups''', which is where large data (but NOT code) must be stored. For example, if you are in the Paten lab:

	mkdir /private/groups/patenlab/$USER		mkdir /private/groups/patenlab/$USER

	3~~. (Optional) Link it over to your home directory, so it is easy to use storage there to store your repos. The '''/private/groups''' storage may be faster than the home directory storage.~~		3. Make sure you have SSH keys created and add them to Github.

	~~mkdir -p /private/groups/patenlab/$USER/workspace~~
	~~ln -s /private/groups/patenlab/$USER/workspace ~/workspace~~

	4. Make sure you have SSH keys created and add them to Github.

	cat ~/.ssh/id_ed25519.pub \|\| (ssh-keygen -t ed25519 && cat ~/.ssh/id_ed25519.pub)		cat ~/.ssh/id_ed25519.pub \|\| (ssh-keygen -t ed25519 && cat ~/.ssh/id_ed25519.pub)
	# Paste into https://github.com/settings/ssh/new		# Paste into https://github.com/settings/ssh/new

	5. Make a place to put your clone, and clone vg:		4. Make a place to put your clone, and clone vg:

	mkdir -p ~/workspace		mkdir -p ~/workspace
Line 30:		Line 25:
	cd vg		cd vg

	6. vg~~'s dependencies should already be installed on the cluster nodes~~. ~~If any of them seem to be missing, tell cluster~~-~~admin@soe~~.~~ucsc.edu to install them~~.		5. Make a symlink to a directory you can put large files in that is easily accessed from your vg directory.

			mkdir /private/groups/patenlab/$USER/data
			ln -s mkdir /private/groups/patenlab/$USER/data ~/workspace/vg/data

			6. Build vg. To build on an interactive node:

	~~7. Build vg~~ as a ~~Slurm job. This will send the build out to the~~ cluster ~~as a 64-core, 80G memory~~ job~~, and keep the output logs in your terminal.~~		make -j16

			To build as a cluster job (possibly faster):

	srun -c 64 --mem=80G --time=00:30:00 make -j64		srun -c 64 --mem=80G --time=00:30:00 make -j64

	~~This~~ will leave your vg binary at '''~/workspace/vg/bin/vg'''.		If you run into build failures a cluster job, try building on the interactive node instead. If that works, we might have forgotten to install all the vg dependencies on the cluster nodes. We want vg's dependencies to be installed on the cluster nodes and all the interactive nodes, but sometimes we add new dependencies and they don't get rolled out everywhere. If any dependencies seem to be missing, ask cluster-admin@soe.ucsc.edu to install them.

			Either build approach will leave your vg binary at '''~/workspace/vg/bin/vg'''.

	==Misc Tips==		==Misc Tips==

Anovak: /* Setting Up */

2025-01-16T16:58:39Z

Setting Up

← Older revision		Revision as of 08:58, 16 January 2025
Line 3:		Line 3:
	==Setting Up==		==Setting Up==

	1. After connecting to the VPN, connect to ~~the cluster head~~ node:		1. After connecting to the VPN, connect to an interactive node:

	ssh ~~phoenix~~.prism		ssh razzmatazz.prism

	This node is relatively small, so you shouldn't run real work on it, but it is the place you need to be to submit Slurm jobs.		This node is relatively small, so you shouldn't run real work on it, but it is the place you need to be to submit Slurm jobs.

Anovak: /* Misc Tips */ Discourage interactive shells

2024-10-11T17:05:25Z

Misc Tips: Discourage interactive shells

@@ Line 40: / Line 40: @@
 ==Misc Tips==
-* If you want an interactive session with appreciable resources, you can schedule one with '''srun'''. For example, to get 16 cores and 120G memory all for you, run:
+* For a lightweight job that outputs to your terminal or that can be waited for in a Bash script, run an individual command directly from <code>srun</code>:
   srun -c 16 --mem 120G --time=08:00:00 --partition=medium --pty bash -i

Anovak: /* Setting Up */

2024-01-05T15:35:25Z

Setting Up

← Older revision		Revision as of 07:35, 5 January 2024
Line 20:		Line 20:
	4. Make sure you have SSH keys created and add them to Github.		4. Make sure you have SSH keys created and add them to Github.

	cat ~/.ssh/id_ed25519.pub \|\| (ssh-keygen -t ~~dsa~~ && cat ~/.ssh/id_ed25519.pub)		cat ~/.ssh/id_ed25519.pub \|\| (ssh-keygen -t ed25519 && cat ~/.ssh/id_ed25519.pub)
	# Paste into https://github.com/settings/ssh/new		# Paste into https://github.com/settings/ssh/new

Anovak at 15:11, 5 January 2024

2024-01-05T15:11:59Z

← Older revision		Revision as of 07:11, 5 January 2024
Line 34:		Line 34:
	7. Build vg as a Slurm job. This will send the build out to the cluster as a 64-core, 80G memory job, and keep the output logs in your terminal.		7. Build vg as a Slurm job. This will send the build out to the cluster as a 64-core, 80G memory job, and keep the output logs in your terminal.

	srun -c 64 --mem=80G make -j64		srun -c 64 --mem=80G --time=00:30:00 make -j64

	This will leave your vg binary at '''~/workspace/vg/bin/vg'''.		This will leave your vg binary at '''~/workspace/vg/bin/vg'''.
Line 42:		Line 42:
	* If you want an interactive session with appreciable resources, you can schedule one with '''srun'''. For example, to get 16 cores and 120G memory all for you, run:		* If you want an interactive session with appreciable resources, you can schedule one with '''srun'''. For example, to get 16 cores and 120G memory all for you, run:

	srun -c 16 --mem 120G --pty bash -i		srun -c 16 --mem 120G --time=08:00:00 --partition=medium --pty bash -i

	* To send out a job without making a script file for it, use '''sbatch --wrap "your command here"'''.		* To send out a job without making a script file for it, use '''sbatch --wrap "your command here"'''.

Anovak: /* Misc Tips */

2023-05-15T21:43:24Z

Misc Tips

← Older revision		Revision as of 13:43, 15 May 2023
Line 43:		Line 43:

	srun -c 16 --mem 120G --pty bash -i		srun -c 16 --mem 120G --pty bash -i

			* To send out a job without making a script file for it, use '''sbatch --wrap "your command here"'''.

			* You can use arguments from SBATCH lines on the command line!

			* You can use [https://github.com/CLIP-HPC/SlurmCommander#readme Slurm Commander] to watch the state of the cluster with the '''scom''' command.

Anovak at 21:40, 15 May 2023

2023-05-15T21:40:42Z

← Older revision		Revision as of 13:40, 15 May 2023
Line 1:		Line 1:
	This page explains how to set up a development environment for [https://github.com/vgteam/vg vg] on the Phoenix cluster.		This page explains how to set up a development environment for [https://github.com/vgteam/vg vg] on the Phoenix cluster.

	1. Make yourself a user directory under '''/private/groups''', which is where large data must be stored. For example, if you are in the Paten lab:		==Setting Up==

			1. After connecting to the VPN, connect to the cluster head node:

			ssh phoenix.prism

			This node is relatively small, so you shouldn't run real work on it, but it is the place you need to be to submit Slurm jobs.

			2. Make yourself a user directory under '''/private/groups''', which is where large data must be stored. For example, if you are in the Paten lab:

	mkdir /private/groups/patenlab/$USER		mkdir /private/groups/patenlab/$USER

	2. (Optional) Link it over to your home directory, so it is easy to use storage there to store your repos. The '''/private/groups''' storage may be faster than the home directory storage.		3. (Optional) Link it over to your home directory, so it is easy to use storage there to store your repos. The '''/private/groups''' storage may be faster than the home directory storage.

	mkdir -p /private/groups/patenlab/$USER/workspace		mkdir -p /private/groups/patenlab/$USER/workspace
	ln -s /private/groups/patenlab/$USER/workspace ~/workspace		ln -s /private/groups/patenlab/$USER/workspace ~/workspace

	3. Make sure you have SSH keys created and add them to Github.		4. Make sure you have SSH keys created and add them to Github.

	cat ~/.ssh/id_ed25519.pub \|\| (ssh-keygen -t dsa && cat ~/.ssh/id_ed25519.pub)		cat ~/.ssh/id_ed25519.pub \|\| (ssh-keygen -t dsa && cat ~/.ssh/id_ed25519.pub)
	# Paste into https://github.com/settings/ssh/new		# Paste into https://github.com/settings/ssh/new

	4. Make a place to put your clone, and clone vg:		5. Make a place to put your clone, and clone vg:

	mkdir -p ~/workspace		mkdir -p ~/workspace
Line 22:		Line 30:
	cd vg		cd vg

	5. vg's dependencies should already be installed on the cluster nodes. If any of them seem to be missing, tell cluster-admin@soe.ucsc.edu to install them.		6. vg's dependencies should already be installed on the cluster nodes. If any of them seem to be missing, tell cluster-admin@soe.ucsc.edu to install them.

	6. Build vg as a Slurm job. This will send the build out to the cluster as a 64-core, 80G memory job, and keep the output logs in your terminal.		7. Build vg as a Slurm job. This will send the build out to the cluster as a 64-core, 80G memory job, and keep the output logs in your terminal.

	srun -c 64 --mem=80G make -j64		srun -c 64 --mem=80G make -j64

	This will leave your vg binary at '''~/workspace/vg/bin/vg'''.		This will leave your vg binary at '''~/workspace/vg/bin/vg'''.

			==Misc Tips==

			* If you want an interactive session with appreciable resources, you can schedule one with '''srun'''. For example, to get 16 cores and 120G memory all for you, run:

			srun -c 16 --mem 120G --pty bash -i

Anovak: Created page with "This page explains how to set up a development environment for [https://github.com/vgteam/vg vg] on the Phoenix cluster. 1. Make yourself a user directory under '''/private/g..."

2023-05-15T21:33:26Z

Created page with "This page explains how to set up a development environment for [https://github.com/vgteam/vg vg] on the Phoenix cluster. 1. Make yourself a user directory under '''/private/g..."

New page

This page explains how to set up a development environment for [https://github.com/vgteam/vg vg] on the Phoenix cluster.

1. Make yourself a user directory under '''/private/groups''', which is where large data must be stored. For example, if you are in the Paten lab:

mkdir /private/groups/patenlab/$USER

2. (Optional) Link it over to your home directory, so it is easy to use storage there to store your repos. The '''/private/groups''' storage may be faster than the home directory storage.

mkdir -p /private/groups/patenlab/$USER/workspace
ln -s /private/groups/patenlab/$USER/workspace ~/workspace

3. Make sure you have SSH keys created and add them to Github.

cat ~/.ssh/id_ed25519.pub || (ssh-keygen -t dsa && cat ~/.ssh/id_ed25519.pub)
# Paste into https://github.com/settings/ssh/new

4. Make a place to put your clone, and clone vg:

mkdir -p ~/workspace
cd ~/workspace
git clone --recursive git@github.com:vgteam/vg.git
cd vg

5. vg's dependencies should already be installed on the cluster nodes. If any of them seem to be missing, tell cluster-admin@soe.ucsc.edu to install them.

6. Build vg as a Slurm job. This will send the build out to the cluster as a 64-core, 80G memory job, and keep the output logs in your terminal.

srun -c 64 --mem=80G make -j64

This will leave your vg binary at '''~/workspace/vg/bin/vg'''.