Overview of using Slurm - Revision history

Weiler: /* Submit a Slurm Batch Job */

2024-05-17T21:01:33Z

Submit a Slurm Batch Job

← Older revision		Revision as of 14:01, 17 May 2024
Line 5:		Line 5:
	== Submit a Slurm Batch Job ==		== Submit a Slurm Batch Job ==

	In order to submit a Slurm batch job list, you will need to create a directory that you will have read and write access to on all the nodes (which will often be a shared space). Let's say I have a batch named "experiment-1". I would create that directory in my groups area:		In order to submit a Slurm batch job list, you will need to create a directory that you will have both read and write access to on all the nodes (which will often be a shared space). Let's say I have a batch named "experiment-1". I would create that directory in my groups area:

	% mkdir -p /private/groups/clusteradmin/weiler/slurm-jobs/experiment-1		% mkdir -p /private/groups/clusteradmin/weiler/slurm-jobs/experiment-1

Weiler: /* CGROUPS and Resource Management */

2024-05-17T21:00:35Z

CGROUPS and Resource Management

← Older revision		Revision as of 14:00, 17 May 2024
Line 84:		Line 84:
	Our installation of Slurm will utilize Linux CGROUPS, which puts a hard resource cap on jobs. If you define that your job will need 4GB of RAM, and it uses 5GB, it will fail with an OOM exception. Likewise with CPU or GPU resources; if your job ends up using more than you specify, it will fail. Or the "--time" batch file option, your job will fail if it takes longer than what you specify there. This is to keep the nodes from crashing from runaway jobs that use more resources than you think they will.		Our installation of Slurm will utilize Linux CGROUPS, which puts a hard resource cap on jobs. If you define that your job will need 4GB of RAM, and it uses 5GB, it will fail with an OOM exception. Likewise with CPU or GPU resources; if your job ends up using more than you specify, it will fail. Or the "--time" batch file option, your job will fail if it takes longer than what you specify there. This is to keep the nodes from crashing from runaway jobs that use more resources than you think they will.

	So... TEST YOUR JOBS! Find ~~our~~ how much in resources a single job needs before you launch 100 of them.		So... TEST YOUR JOBS! Find out how much in resources a single job needs before you launch 100 of them.

	== TEST YOUR JOBS! ==		== TEST YOUR JOBS! ==

	Let me say that one more time. Test your jobs before launching a bunch of them! If it fails, you don't want it to fail 100 or more times. You can also get a good idea of how much RAM and CPU it will need so you can better define your batch files. It's critical to get a good idea of how many resources each of your jobs will use and define your job file appropriately.		Let me say that one more time. Test your jobs before launching a bunch of them! If it fails, you don't want it to fail 100 or more times. You can also get a good idea of how much RAM and CPU it will need so you can better define your batch files. It's critical to get a good idea of how many resources each of your jobs will use and define your job file appropriately.

Weiler at 15:06, 11 May 2024

2024-05-11T15:06:33Z

← Older revision		Revision as of 08:06, 11 May 2024
Line 1:		Line 1:
	When using Slurm, you will need to log into the ~~Slurm head node~~ (~~currently phoenix.prism~~). Once you have ssh'd in there, you can execute slurm batch or interactive commands.		When using Slurm, you will need to log into one of the interactive compute servers in the PRISM area (such as emerald, mustard, crimson or razzmatazz). Once you have ssh'd in there, you can execute slurm batch or interactive commands.

	You might also want to consult the [[Quick Reference Guide]].		You might also want to consult the [[Quick Reference Guide]].

Weiler at 17:39, 29 November 2023

2023-11-29T17:39:48Z

← Older revision		Revision as of 10:39, 29 November 2023
Line 21:		Line 21:
	#		#
	# Partition - This is the queue it goes in:		# Partition - This is the queue it goes in:
	#SBATCH --partition=~~main~~		#SBATCH --partition=short
	#		#
	# Where to send email (optional)		# Where to send email (optional)

Weiler: /* Submit a Slurm Batch Job */

2023-06-15T16:22:50Z

Submit a Slurm Batch Job

← Older revision		Revision as of 09:22, 15 June 2023
Line 32:		Line 32:
	#SBATCH --mem=4gb		#SBATCH --mem=4gb
	#		#
	# Number of tasks (one for each ~~GPU~~ desired for use case) (example):		# Number of tasks (one for each CPU desired for use case) (example):
	#SBATCH --ntasks=1		#SBATCH --ntasks=1
	#		#

Weiler at 22:11, 12 June 2023

2023-06-12T22:11:37Z

← Older revision		Revision as of 15:11, 12 June 2023
Line 1:		Line 1:
	When using Slurm, you will need to log into the Slurm head node (currently phoenix~~-01~~.~~gi.ucsc.edu, a one node cluster at the moment~~). Once you have ssh'd in there, you can execute slurm batch or interactive commands.		When using Slurm, you will need to log into the Slurm head node (currently phoenix.prism). Once you have ssh'd in there, you can execute slurm batch or interactive commands.

	You might also want to consult the [[Quick Reference Guide]].		You might also want to consult the [[Quick Reference Guide]].
Line 85:		Line 85:

	So... TEST YOUR JOBS! Find our how much in resources a single job needs before you launch 100 of them.		So... TEST YOUR JOBS! Find our how much in resources a single job needs before you launch 100 of them.

			== TEST YOUR JOBS! ==

			Let me say that one more time. Test your jobs before launching a bunch of them! If it fails, you don't want it to fail 100 or more times. You can also get a good idea of how much RAM and CPU it will need so you can better define your batch files. It's critical to get a good idea of how many resources each of your jobs will use and define your job file appropriately.

Anovak: Cross-link to quick reference

2023-05-11T15:52:28Z

Cross-link to quick reference

← Older revision		Revision as of 08:52, 11 May 2023
Line 1:		Line 1:
	When using Slurm, you will need to log into the Slurm head node (currently phoenix-01.gi.ucsc.edu, a one node cluster at the moment). Once you have ssh'd in there, you can execute slurm batch or interactive commands.		When using Slurm, you will need to log into the Slurm head node (currently phoenix-01.gi.ucsc.edu, a one node cluster at the moment). Once you have ssh'd in there, you can execute slurm batch or interactive commands.

			You might also want to consult the [[Quick Reference Guide]].

	== Submit a Slurm Batch Job ==		== Submit a Slurm Batch Job ==

Weiler at 20:19, 3 May 2023

2023-05-03T20:19:52Z

← Older revision		Revision as of 13:19, 3 May 2023
Line 37:		Line 37:
	#SBATCH --cpus-per-task=1		#SBATCH --cpus-per-task=1
	#		#
	# Number of GPUs, this can be in the format of "--gres=gpu:[1-8]", or "--gres=gpu:A5500:[1-8] with the type included (optional)		# Number of GPUs, this can be in the format of "--gres=gpu:[1-8]", or "--gres=gpu:A5500:[1-8]" with the type included (optional)
	#SBATCH --gres=gpu:1		#SBATCH --gres=gpu:1
	#		#

Weiler at 20:19, 3 May 2023

2023-05-03T20:19:22Z

← Older revision		Revision as of 13:19, 3 May 2023
Line 37:		Line 37:
	#SBATCH --cpus-per-task=1		#SBATCH --cpus-per-task=1
	#		#
	# Number of GPUs, this can be in the format of "gpu:[1-4]", or "gpu:A5500:[1-4] with the type included (optional)		# Number of GPUs, this can be in the format of "--gres=gpu:[1-8]", or "--gres=gpu:A5500:[1-8] with the type included (optional)
	#SBATCH --gres=gpu:1		#SBATCH --gres=gpu:1
	#		#

Weiler at 03:12, 2 May 2023

2023-05-02T03:12:06Z

← Older revision		Revision as of 20:12, 1 May 2023
Line 37:		Line 37:
	#SBATCH --cpus-per-task=1		#SBATCH --cpus-per-task=1
	#		#
	# Number of GPUs, this can be in the format of "gpu:[1-4]", or "gpu:~~K80~~:[1-4] with the type included (optional)		# Number of GPUs, this can be in the format of "gpu:[1-4]", or "gpu:A5500:[1-4] with the type included (optional)
	#SBATCH --gres=gpu:1		#SBATCH --gres=gpu:1
	#		#