GPU Resources - Revision history

Anovak: Add partition and time

2024-06-28T16:32:15Z

Add partition and time

← Older revision		Revision as of 09:32, 28 June 2024
Line 1:		Line 1:
	When submitting jobs, you can ask for GPUs in one of two ways. One is:		When submitting jobs, you can ask for GPUs in one of two ways. One is:

			#SBATCH --partition=gpu
	#SBATCH --gres=gpu:1		#SBATCH --gres=gpu:1

	That will ask for 1 GPU generically on a node with a free GPU. This request is more specific:		That will ask for 1 GPU generically on a node with a free GPU. This request is more specific:

			#SBATCH --partition=gpu
	#SBATCH --gres=gpu:A5500:3		#SBATCH --gres=gpu:A5500:3

Line 41:		Line 43:

	singularity pull docker://tensorflow/tensorflow:latest-gpu		singularity pull docker://tensorflow/tensorflow:latest-gpu
	srun -c 8 --mem 10G --gres=gpu:1 singularity run --nv docker://tensorflow/tensorflow:latest-gpu python -c 'from tensorflow.python.client import device_lib; print(device_lib.list_local_devices())'		srun -c 8 --mem 10G --partition=gpu --time=00:20:00 --gres=gpu:1 singularity run --nv docker://tensorflow/tensorflow:latest-gpu python -c 'from tensorflow.python.client import device_lib; print(device_lib.list_local_devices())'

	This will produce output showing that the Tensorflow container is indeed able to talk to one GPU:		This will produce output showing that the Tensorflow container is indeed able to talk to one GPU:
Line 89:		Line 91:
	An example of a working command is:		An example of a working command is:

	srun -c 1 --mem 4G --gres=gpu:2 bash -c 'docker run --rm --gpus=\"device=$SLURM_STEP_GPUS\" nvidia/cuda:12.1.1-base-ubuntu22.04 nvidia-smi'		srun -c 1 --mem 4G --partition=gpu --time=00:20:00 --gres=gpu:2 bash -c 'docker run --rm --gpus=\"device=$SLURM_STEP_GPUS\" nvidia/cuda:12.1.1-base-ubuntu22.04 nvidia-smi'

	Note that the double-quotes are included in the argument to '''--gpus''' as seen by the Docker client, and that '''bash''' and single-quotes are used to ensure that '''$SLURM_STEP_GPUS''' is evaluated within the job itself, and not on the head node.		Note that the double-quotes are included in the argument to '''--gpus''' as seen by the Docker client, and that '''bash''' and single-quotes are used to ensure that '''$SLURM_STEP_GPUS''' is evaluated within the job itself, and not on the head node.

Weiler at 13:51, 26 October 2023

2023-10-26T13:51:29Z

← Older revision		Revision as of 06:51, 26 October 2023
Line 13:		Line 13:
	nVidia RTX A5500 : 24GB RAM		nVidia RTX A5500 : 24GB RAM
	nVidia A100 : 80GB RAM		nVidia A100 : 80GB RAM
	~~nVidia GeForce RTX 2080 Ti : 11GB RAM~~
	~~nVidia GeForce RTX 1080 Ti : 11GB RAM~~

	For the most part, Slurm takes care of making sure that each job only sees and used the GPUs assigned to it. Within the job, '''CUDA_VISIBLE_DEVICES''' will be set in the environment, but it will always be set to a list of your requested number of GPUs, starting at 0. Slurm re-numbers the GPUs assigned to each job to appear to start at 0, within the job. If you need access to the "real" GPU numbers (to log or to pass along to Docker), they are available in the '''SLURM_JOB_GPUS''' (for '''sbatch''') or '''SLURM_STEP_GPUS''' (for '''srun''') environment variable.		For the most part, Slurm takes care of making sure that each job only sees and used the GPUs assigned to it. Within the job, '''CUDA_VISIBLE_DEVICES''' will be set in the environment, but it will always be set to a list of your requested number of GPUs, starting at 0. Slurm re-numbers the GPUs assigned to each job to appear to start at 0, within the job. If you need access to the "real" GPU numbers (to log or to pass along to Docker), they are available in the '''SLURM_JOB_GPUS''' (for '''sbatch''') or '''SLURM_STEP_GPUS''' (for '''srun''') environment variable.

Anovak at 19:32, 22 May 2023

2023-05-22T19:32:57Z

← Older revision		Revision as of 12:32, 22 May 2023
Line 16:		Line 16:
	nVidia GeForce RTX 1080 Ti : 11GB RAM		nVidia GeForce RTX 1080 Ti : 11GB RAM

	For the most part, Slurm takes care of making sure that each job only sees and used the GPUs assigned to it. Within the job, '''CUDA_VISIBLE_DEVICES''' will be set in the environment, but it will always be set to a list of your requested number of GPUs, starting at 0. Slurm re-numbers the GPUs assigned to each job to appear to start at 0, within the job. If you need access to the "real" GPU numbers (to log or to pass along to Docker), they are available in the '''SLURM_STEP_GPUS''' environment variable.		For the most part, Slurm takes care of making sure that each job only sees and used the GPUs assigned to it. Within the job, '''CUDA_VISIBLE_DEVICES''' will be set in the environment, but it will always be set to a list of your requested number of GPUs, starting at 0. Slurm re-numbers the GPUs assigned to each job to appear to start at 0, within the job. If you need access to the "real" GPU numbers (to log or to pass along to Docker), they are available in the '''SLURM_JOB_GPUS''' (for '''sbatch''') or '''SLURM_STEP_GPUS''' (for '''srun''') environment variable.

	==Running GPU Workloads==		==Running GPU Workloads==
Line 87:		Line 87:
	The '''nvidia''' runtime is set up and will automatically be used.		The '''nvidia''' runtime is set up and will automatically be used.

	While Slurm configures each Slurm job with a cgroup that directs it to the correct GPUs, '''using Docker to run another container escapes Slurm's confinement''', and using '''--gpus=1''' will ''always'' use the ''first'' GPU in the system, whether that GPU is assigned to your job or not. When using Docker, you ''must'' consult the '''SLRUM_STEP_GPUS''' environment variable and pass that along to your container. You should also impose limits on all other resources used by your Docker container, so that your whole job stays within the resources allocated by Slurm's scheduler. (TODO: find out how cgroups handles oversubscription between a Docker container and the Slurm container that launched it).		While Slurm configures each Slurm job with a cgroup that directs it to the correct GPUs, '''using Docker to run another container escapes Slurm's confinement''', and using '''--gpus=1''' will ''always'' use the ''first'' GPU in the system, whether that GPU is assigned to your job or not. When using Docker, you ''must'' consult the '''SLURM_JOB_GPUS''' (for '''sbatch''') or '''SLRUM_STEP_GPUS''' (for '''srun''') environment variable and pass that along to your container. You should also impose limits on all other resources used by your Docker container, so that your whole job stays within the resources allocated by Slurm's scheduler. (TODO: find out how cgroups handles oversubscription between a Docker container and the Slurm container that launched it).

	An example of a working command is:		An example of a working command is:

Anovak at 15:19, 22 May 2023

2023-05-22T15:19:32Z

← Older revision		Revision as of 08:19, 22 May 2023
Line 15:		Line 15:
	nVidia GeForce RTX 2080 Ti : 11GB RAM		nVidia GeForce RTX 2080 Ti : 11GB RAM
	nVidia GeForce RTX 1080 Ti : 11GB RAM		nVidia GeForce RTX 1080 Ti : 11GB RAM

			For the most part, Slurm takes care of making sure that each job only sees and used the GPUs assigned to it. Within the job, '''CUDA_VISIBLE_DEVICES''' will be set in the environment, but it will always be set to a list of your requested number of GPUs, starting at 0. Slurm re-numbers the GPUs assigned to each job to appear to start at 0, within the job. If you need access to the "real" GPU numbers (to log or to pass along to Docker), they are available in the '''SLURM_STEP_GPUS''' environment variable.

	==Running GPU Workloads==		==Running GPU Workloads==

Anovak: /* Running Containers in Singularity */

2023-05-22T15:11:47Z

Running Containers in Singularity

← Older revision		Revision as of 08:11, 22 May 2023
Line 38:		Line 38:
	====Running Containers in Singularity====		====Running Containers in Singularity====

	You can run containers on the cluster using Singularity, and give them access to GPUs using the '''--nv''' option. For example:		You can run containers on the cluster using Singularity, and give them access to the GPUs that Slurm has selected using the '''--nv''' option. For example:

	singularity pull docker://tensorflow/tensorflow:latest-gpu		singularity pull docker://tensorflow/tensorflow:latest-gpu
Line 68:		Line 68:
	xla_global_id: 416903419		xla_global_id: 416903419
	]		]

			Slurm's containment of the Slurm job to the correct set of GPUs is also passed through to the Singularity container; there is no need to specifically direct Singularity to use the right GPUs unless you are doing something unusual.

	====Running Containers in Slurm====		====Running Containers in Slurm====

Anovak: /* Running Containers in Singularity */

2023-05-22T15:07:48Z

Running Containers in Singularity

← Older revision		Revision as of 08:07, 22 May 2023
Line 41:		Line 41:

	singularity pull docker://tensorflow/tensorflow:latest-gpu		singularity pull docker://tensorflow/tensorflow:latest-gpu
	srun -c 8 --mem 10G --gres=gpu:~~1 --exclude phoenix-~~01 singularity run --nv docker://tensorflow/tensorflow:latest-gpu python -c 'from tensorflow.python.client import device_lib; print(device_lib.list_local_devices())'		srun -c 8 --mem 10G --gres=gpu:1 singularity run --nv docker://tensorflow/tensorflow:latest-gpu python -c 'from tensorflow.python.client import device_lib; print(device_lib.list_local_devices())'

	This will produce output showing that the Tensorflow container is indeed able to talk to one GPU:		This will produce output showing that the Tensorflow container is indeed able to talk to one GPU:

Anovak: /* Running Containers in Docker */ Note how to point Docker at the right GPUs.

2023-05-22T15:07:02Z

Running Containers in Docker: Note how to point Docker at the right GPUs.

← Older revision		Revision as of 08:07, 22 May 2023
Line 81:		Line 81:
	You might be used to running containers with Docker, or containerized GPU workloads with the nVidia Container Runtime or Toolkit. Docker is installed on all the nodes and the daemon is running; if the '''docker''' command does not work for you, ask cluster-admin to add you to the right groups.		You might be used to running containers with Docker, or containerized GPU workloads with the nVidia Container Runtime or Toolkit. Docker is installed on all the nodes and the daemon is running; if the '''docker''' command does not work for you, ask cluster-admin to add you to the right groups.

	The '''nvidia''' runtime is set up~~, so you should~~ be ~~able to do:~~		The '''nvidia''' runtime is set up and will automatically be used.

	~~srun -c 1 --mem 4G --gres=gpu:1 docker~~ run ~~--rm --runtime=nvidia~~ --gpus=1 ~~nvidia/cuda:12~~.1.~~1-base-ubuntu22~~.~~04 nvidia-smi~~		While Slurm configures each Slurm job with a cgroup that directs it to the correct GPUs, '''using Docker to run another container escapes Slurm's confinement''', and using '''--gpus=1''' will ''always'' use the ''first'' GPU in the system, whether that GPU is assigned to your job or not. When using Docker, you ''must'' consult the '''SLRUM_STEP_GPUS''' environment variable and pass that along to your container. You should also impose limits on all other resources used by your Docker container, so that your whole job stays within the resources allocated by Slurm's scheduler. (TODO: find out how cgroups handles oversubscription between a Docker container and the Slurm container that launched it).

	~~You shouldn't need the '''--runtime''' argument in normal operation, just '''--gpus'''.~~		An example of a working command is:

	~~Further testing is needed~~ to ~~determine if Slurm is able~~ to ~~assign individual GPUs to individual jobs in a way~~ that ~~Docker respects~~.		srun -c 1 --mem 4G --gres=gpu:2 bash -c 'docker run --rm --gpus=\"device=$SLURM_STEP_GPUS\" nvidia/cuda:12.1.1-base-ubuntu22.04 nvidia-smi'

			Note that the double-quotes are included in the argument to '''--gpus''' as seen by the Docker client, and that '''bash''' and single-quotes are used to ensure that '''$SLURM_STEP_GPUS''' is evaluated within the job itself, and not on the head node.

Anovak: /* Running Containers in Docker */

2023-05-18T21:43:01Z

Running Containers in Docker

← Older revision		Revision as of 14:43, 18 May 2023
Line 86:		Line 86:

	You shouldn't need the '''--runtime''' argument in normal operation, just '''--gpus'''.		You shouldn't need the '''--runtime''' argument in normal operation, just '''--gpus'''.

			Further testing is needed to determine if Slurm is able to assign individual GPUs to individual jobs in a way that Docker respects.

Anovak at 21:39, 18 May 2023

2023-05-18T21:39:08Z

← Older revision		Revision as of 14:39, 18 May 2023
Line 81:		Line 81:
	You might be used to running containers with Docker, or containerized GPU workloads with the nVidia Container Runtime or Toolkit. Docker is installed on all the nodes and the daemon is running; if the '''docker''' command does not work for you, ask cluster-admin to add you to the right groups.		You might be used to running containers with Docker, or containerized GPU workloads with the nVidia Container Runtime or Toolkit. Docker is installed on all the nodes and the daemon is running; if the '''docker''' command does not work for you, ask cluster-admin to add you to the right groups.

	~~We are working on getting the nVidia~~ runtime set up to ~~make~~ the '''--~~gpus~~''' ~~option to~~ '''~~docker run~~''' ~~work~~.		The '''nvidia''' runtime is set up, so you should be able to do:

			srun -c 1 --mem 4G --gres=gpu:1 docker run --rm --runtime=nvidia --gpus=1 nvidia/cuda:12.1.1-base-ubuntu22.04 nvidia-smi

			You shouldn't need the '''--runtime''' argument in normal operation, just '''--gpus'''.

Anovak: /* Running Containers in Docker */

2023-05-15T19:55:47Z

Running Containers in Docker

← Older revision		Revision as of 12:55, 15 May 2023
Line 79:		Line 79:
	====Running Containers in Docker====		====Running Containers in Docker====

	You might be used to running containers with Docker, or containerized GPU workloads with the nVidia Container Runtime or Toolkit. Docker is installed on all the nodes and the daemon is running; if the `docker` command does not work for you, ask cluster-admin to add you to the right groups.		You might be used to running containers with Docker, or containerized GPU workloads with the nVidia Container Runtime or Toolkit. Docker is installed on all the nodes and the daemon is running; if the '''docker''' command does not work for you, ask cluster-admin to add you to the right groups.

	We are working on getting the nVidia runtime set up to make the '''--gpus''' option to '''docker run''' work.		We are working on getting the nVidia runtime set up to make the '''--gpus''' option to '''docker run''' work.