Using Docker under Slurm - Revision history

Anovak: Explain how to pass the right GPUs

2024-12-04T14:45:05Z

Explain how to pass the right GPUs

← Older revision		Revision as of 07:45, 4 December 2024
Line 28:		Line 28:

	The --memory argument is in megabytes (hence the 'm' at the end). So the above example will set a memory limit of 1GB.		The --memory argument is in megabytes (hence the 'm' at the end). So the above example will set a memory limit of 1GB.

			== Docker and GPUs ==

			If you are using GPUs with Docker, you need to make sure that your Docker container requests access to the ''correct'' GPUs: the ones which Slurm assigned to your job. These will be passed in the <code>SLURM_STEP_GPUS</code> (for GPUs for a single step) or <code>SLURM_JOB_GPUS</code> (for GPUs for a whole job) environment variables. They need to be passed to Docker like this:

			docker run --gpus="\"device=${SLURM_STEP_GPUS:-$SLURM_JOB_GPUS}\"" nvidia/cuda nvidia-smi

			'''Note the escaped quotes'''; the Docker command needs to have double-quotes ''inside'' the argument value. The <code>${:-}</code> syntax will use <code>SLURM_STEP_GPUS</code> if it is set and <code>SLURM_JOB_GPUS</code> if it isn't; if you know which will be set for your job, you can use just that one.

			If you are using Nextflow, you will need to set <code>docker.runOptions</code> to include this flag.

			docker.runOptions="--gpus \\\"device=$SLURM_JOB_GPUS\\\""

			If you are using Toil to run CWL or WDL, the correct GPUs will be passed to containers automatically.

	== Cleaning Scripts ==		== Cleaning Scripts ==

Weiler at 16:42, 30 June 2023

2023-06-30T16:42:56Z

← Older revision		Revision as of 09:42, 30 June 2023
Line 1:		Line 1:
			__TOC__

	Sometimes it is convenient to ask Slurm to run your job in a docker container. This is just fine, however, you will need to fully test your job in a docker container beforehand (on mustard or emerald, for example) to see how much RAM and CPU resources it requires, so you can accurately describe in your slurm job submission file how many resources it needs.		Sometimes it is convenient to ask Slurm to run your job in a docker container. This is just fine, however, you will need to fully test your job in a docker container beforehand (on mustard or emerald, for example) to see how much RAM and CPU resources it requires, so you can accurately describe in your slurm job submission file how many resources it needs.

Weiler at 16:42, 30 June 2023

2023-06-30T16:42:37Z

@@ Line 1: / Line 1: @@
 Sometimes it is convenient to ask Slurm to run your job in a docker container.  This is just fine, however, you will need to fully test your job in a docker container beforehand (on mustard or emerald, for example) to see how much RAM and CPU resources it requires, so you can accurately describe in your slurm job submission file how many resources it needs.
 You can run your container on mustard then look at 'top' to see how much RAM and CPU it needs.
@@ Line 17: / Line 19: @@
   $ docker image rm be6777ad00cf
-We also have auto-cleaning scripts running that will delete any containers and images that were created/pulled more than 7 days ago.  This inlcudes the cluster nodes and also the phoenix head node itself.  If you need a place to have your images/containers remain longer than that, please put them on mustard, emerald, crimson or razzmatazz.
+== Resource Limits ==

Weiler at 21:22, 27 June 2023

2023-06-27T21:22:32Z

← Older revision		Revision as of 14:22, 27 June 2023
Line 17:		Line 17:
	$ docker image rm be6777ad00cf		$ docker image rm be6777ad00cf

	We also have auto-cleaning scripts running that will delete any containers and images that were created/pulled more than 7 days ago. This inlcudes the cluster nodes and also the phoenix head node itself. If you need a place to have your images/containers remain longer than that, please put them on mustard,emerald, crimson or razzmatazz.		We also have auto-cleaning scripts running that will delete any containers and images that were created/pulled more than 7 days ago. This inlcudes the cluster nodes and also the phoenix head node itself. If you need a place to have your images/containers remain longer than that, please put them on mustard, emerald, crimson or razzmatazz.

Weiler: Created page with "Sometimes it is convenient to ask Slurm to run your job in a docker container. This is just fine, however, you will need to fully test your job in a docker container beforeha..."

2023-06-27T21:21:56Z

Created page with "Sometimes it is convenient to ask Slurm to run your job in a docker container. This is just fine, however, you will need to fully test your job in a docker container beforeha..."

New page

Sometimes it is convenient to ask Slurm to run your job in a docker container. This is just fine, however, you will need to fully test your job in a docker container beforehand (on mustard or emerald, for example) to see how much RAM and CPU resources it requires, so you can accurately describe in your slurm job submission file how many resources it needs.

You can run your container on mustard then look at 'top' to see how much RAM and CPU it needs.

You also will need to be aware that you will need to pull your docker image from a registry, like DockerHub or Quay. And you should also run you docker container with the '--rm' flag, so the container cleans itself up after running. So your workflow would look something like this:

1: Pull image from DockerHub
2: docker run --rm docker/welcome-to-docker

Optionally you can clean up your image as well, but only if you don't have many jobs using that image on the same node. For example, if I wanted to remove the image laballed "weiler/mytools":

$ docker image ls
REPOSITORY TAG IMAGE ID CREATED SIZE
weiler/mytools latest be6777ad00cf 19 hours ago 396MB
somedude/tools latest 9b1d1f6fbf6f 3 weeks ago 607MB

$ docker image rm be6777ad00cf

We also have auto-cleaning scripts running that will delete any containers and images that were created/pulled more than 7 days ago. This inlcudes the cluster nodes and also the phoenix head node itself. If you need a place to have your images/containers remain longer than that, please put them on mustard,emerald, crimson or razzmatazz.