UCSC Genomics Institute Computing Infrastructure Information - Recent changes [en]

Firewalled Computing Resources Overview

Weiler — Tue, 09 Jun 2026 14:02:09 GMT

The Phoenix Cluster

← Older revision		Revision as of 07:02, 9 June 2026
Line 66:		Line 66:
	! Primary Storage		! Primary Storage
	! Secondary Storage		! Secondary Storage
			! Archive Storage
	! Network Backbone		! Network Backbone
	\|-		\|-
Line 74:		Line 75:
	\| 1.7 PB of Clustered Ceph NVMe Solid State Storage		\| 1.7 PB of Clustered Ceph NVMe Solid State Storage
	\| 2.5 PB Spinning Disk Storage (Ceph + ZFS)		\| 2.5 PB Spinning Disk Storage (Ceph + ZFS)
			\| 4.6 PB Spinning Disk Storage (ZFS)
	\| 10 Gb/s (nodes) / 200 Gb/s+ (storage)		\| 10 Gb/s (nodes) / 200 Gb/s+ (storage)
	\|}		\|}

Slurm Best Practices

Tkell — Tue, 02 Jun 2026 20:45:17 GMT

Slurm Interactive Sessions

← Older revision		Revision as of 13:45, 2 June 2026
Line 64:		Line 64:
	== Slurm Interactive Sessions ==		== Slurm Interactive Sessions ==

	A SLURM interactive session reserves resources on compute nodes allowing you to use them interactively as you would the login nodes.		A SLURM interactive session reserves resources on compute nodes allowing you to use them interactively as you would the login nodes. Interactive sessions are limited to a maximum time of 5 hours.

	'''WARNING''': An interactive session will, once it starts, use the entire requested block of CPU time and other resources unless earlier exited, even if unused. Don't forget to exit an interactive session once finished. And also, VERY important to remember, don't start an interactive session with the maximum available walltime just to pin the node for yourself so your jobs don't have to wait in the queue. This is extremely bad etiquette, and people often notice when idle interactive sessions exist and complain about it. '''Just don't do it.''' If you have a need for an interactive session, launch it, run your code and then exit as soon as you are done.		'''WARNING''': An interactive session will, once it starts, use the entire requested block of CPU time and other resources unless earlier exited, even if unused. Don't forget to exit an interactive session once finished. And also, VERY important to remember, don't start an interactive session with the maximum available walltime just to pin the node for yourself so your jobs don't have to wait in the queue. This is extremely bad etiquette, and people often notice when idle interactive sessions exist and complain about it. '''Just don't do it.''' If you have a need for an interactive session, launch it, run your code and then exit as soon as you are done.

Setting Up The VPN on MacOS

Tkell — Tue, 02 Jun 2026 18:26:41 GMT

← Older revision		Revision as of 11:26, 2 June 2026
(One intermediate revision by the same user not shown)
Line 37:		Line 37:
	After installation, in your Finder, you may want to navigate to the Applications folder and drag the Tunnelblick icon to your dock for easy launching.		After installation, in your Finder, you may want to navigate to the Applications folder and drag the Tunnelblick icon to your dock for easy launching.

	After launching Tunnelblick from the Applications folder, you will see a small "tunnel" icon on the top right of your screen, next to the date and WiFi icon. Drag the configuration file ('''prism-duo.ovpn''') on your Desktop to the little Tunnelblick icon on the top right of your screen to install the new profile. It will ask you to type in your ~~laptop~~ password (do that). It will also ask you if you want to install for "Only You" or "All Users". Select "Only You".		After launching Tunnelblick from the Applications folder, you will see a small "tunnel" icon on the top right of your screen, next to the date and WiFi icon. Drag the configuration file ('''prism-duo.ovpn''') on your Desktop to the little Tunnelblick icon on the top right of your screen to install the new profile. It will ask you to type in your computer password (do that). It will also ask you if you want to install for "Only You" or "All Users". Select "Only You".

	== Login to the VPN ==		== Login to the VPN ==
Line 59:		Line 59:

	https://giwiki.gi.ucsc.edu/index.php?title=Firewalled_Computing_Resources_Overview		https://giwiki.gi.ucsc.edu/index.php?title=Firewalled_Computing_Resources_Overview

			== Troubleshooting ==

			If you're able to connect to the VPN, but can't reach any GI resources behind the firewall, your local network settings may be interfering.

			* Try restarting your computer.
			* Make sure you don't have any other VPN software active.

			If you are still unable to connect, try these steps:

			* Disconnect from the Tunnelblick VPN.
			* Open System Settings > Wi-Fi (or Network if you're connected to a wired network connection)
			* Find the network you're connected to and click "Details...", then navigate to the "DNS" section
			* Delete any DNS servers that have been manually added, such as 8.8.8.8 or 1.1.1.1.
			* Click OK then reconnect to the VPN

			You can also enable "Allow changes to manually-set network settings" in Tunnelblick > Configurations > Advanced.


	As always, if you have any questions, please email '''cluster-admin@soe.ucsc.edu''' for help.		As always, if you have any questions, please email '''cluster-admin@soe.ucsc.edu''' for help.

Firewalled Environment Storage Overview

Weiler — Sat, 30 May 2026 17:43:47 GMT

Storage

← Older revision		Revision as of 10:43, 30 May 2026
(13 intermediate revisions by the same user not shown)
Line 10:		Line 10:
	! /private/home		! /private/home
	! /private/groups		! /private/groups
			! /private/warm-archive
	\|-		\|-
	\| style="font-weight:bold; text-align:left;" \| Default ~~Soft Quota~~		\| style="font-weight:bold; text-align:left;" \| Default Quota
	~~\| 100 GB~~
	~~\| 15 TB~~
	\|-
	~~\| style="font-weight:bold; text-align:left;" \| Default Hard~~ Quota
	\| 100 GB		\| 100 GB
	\| 15 TB		\| 15 TB
			\| 50 TB
	\|-		\|-
	\| style="font-weight:bold; text-align:left;" \| Total Capacity		\| style="font-weight:bold; text-align:left;" \| Total Capacity
	\| 19 TB		\| 19 TB
	\| 1.7 PB		\| 1.7 PB
			\| 4.6 PB
	\|- style="text-align:left;"		\|- style="text-align:left;"
	\| style="font-weight:bold;" \| Access Speed		\| style="font-weight:bold;" \| Access Speed
	\| Very Fast (NVMe Flash Media)		\| Very Fast (NVMe Flash Media)
	\| Very Fast (NVMe Flash Media)		\| Very Fast (NVMe Flash Media, Distributed Filesystem)
			\| Slower (Spinning Disks)
	\|- style="text-align:left;"		\|- style="text-align:left;"
	\| style="font-weight:bold;" \| Intended Use		\| style="font-weight:bold;" \| Intended Use
	\| This space should be used for login scripts, small bits of code or software repos, etc. No large data should be stored here.		\| This space should be used for login scripts, small bits of code or software repos, etc. No large data should be stored here.
	\| This space should be used for large computational/shared data, large software installations and the like.		\| This space should be used for large computational/shared (hot) data, large software installations and the like.
			\| Archival Use Only. Not meant for active cluster computation. Warm/Cold data only.
	\|}		\|}

	'''Home Directories (/private/home/username)'''		'''Home Directories (/private/home/username)'''

	Your home directory will be located as "/private/home/username" and has a 100GB ~~soft quota and a 100GB hard~~ quota. Your home directory is meant for small scripts and login data, or a git repo. Please do not try to store large data there or computer on large jobs using data in your home directory.		Your home directory will be located as "/private/home/username" and has a 100GB quota. Your home directory is meant for small scripts and login data, or a git repo. Please do not try to store large data there or computer on large jobs using data in your home directory.

	'''Groups Directories (/private/groups/~~groupname~~)'''		'''Groups Directories (/private/groups/labname)'''

	The group storage directories are created per PI, and each group directory has a default 15TB hard quota. For example, if David Haussler is the PI that you report to directly, then the directory would exist as /private/groups/hausslerlab. Request access to that group directory and you will then be able to write to it. Each of those group directories are shared by the lab it belongs to, so you must be wary of everyone's data usage and share the 15TB available per group accordingly.		The group storage directories are created per PI, and each group directory has a default 15TB hard quota. For example, if David Haussler is the PI that you report to directly, then the directory would exist as /private/groups/hausslerlab. Request access to that group directory and you will then be able to write to it. Each of those group directories are shared by the lab it belongs to, so you must be wary of everyone's data usage and share the 15TB available per group accordingly.

	On the ~~compute servers you can check your group's~~ current quota usage ~~by using the 'getfattr' command~~. ~~You can only check the quota of a group you are part of~~ (~~you would be a member of the UNIX group of the same name~~). ~~If you wanted to check the quota usage of /private/groups/hausslerlab for~~ example~~, you would do~~:		The groups storage directories have a file in the root of them that contains current quota usage info. These files (.quota_info) are updated once an hour. For example:

	~~$ getfattr -n ceph.dir.rbytes~~ /private/groups/~~hausslerlab~~		cat /private/groups/corbettlab/.quota_info

			==================================================
			CephFS Storage Quota Report: corbettlab
			==================================================
			Filesystem Tier: CephFS
			Space Used: 72T
			Total Quota: 92T
			Available: 21T

	~~getfattr~~: ~~Removing leading~~ '/' ~~from absolute path names~~		Last Updated: 2026-05-30 09:22:31 PDT
	~~# file:~~ private/~~groups~~/~~hausslerlab~~		==================================================
	~~ceph.dir.rbytes="6522955553147"~~
			Every lab directory has a '''.quota_info''' file in its root.

			'''Archive Directories (/private/warm-archive/labname)'''

	~~That number is in bytes. So divide~~ by ~~1,000,000~~,~~000,000~~ and ~~you get '6.522 TB'~~. ~~That~~ is ~~how much data is currently being used~~.		The archive storage directories are created per PI '''by request''', and each group directory has a default 50TB hard quota. For example, if David Haussler is the PI that you report to directly, then the directory would exist as /private/warm-archive/hausslerlab.

	~~To check~~ the ~~max~~ quota ~~limit, use this command~~:		The archive storage directories have a file in the root of them that contains current quota usage info. These files (.quota_info) are updated once an hour. For example:

	~~$ getfattr -n ceph.quota.max_bytes~~ /private/~~groups~~/~~hausslerlab~~		cat /private/warm-archive/corbettlab/.quota_info

			==================================================
			ZFS Storage Quota Report: corbettlab
			==================================================
			Project ID: 105
			Space Used: 148G
			Total Quota: 50T
			Available: 50T

	~~getfattr~~: ~~Removing leading '/' from absolute path names~~		Last Updated: 2026-05-30 08:45:57 PDT
	~~# file~~: ~~private/groups/hausslerlab~~		==================================================
	~~ceph.quota.max_bytes~~=~~"15000000000000"~~

	~~And 15000000000000 divided by 1,000,000,000,000 is 15 TB~~.		Every lab directory has a '''.quota_info''' file in its root.

	== Storage Quota Alerting ==		== Storage Quota Alerting ==

	If you and/or folks in your lab would like an automated alert when the /private/groups/labname quota is getting to a certain percentage of fullness, we can set that up for you and others in your lab. Just email '''cluster-admin@soe.ucsc.edu''' with the following information:		If you and/or folks in your lab would like an automated alert when the '''/private/groups/labname''' quota is getting to a certain percentage of fullness, we can set that up for you and others in your lab. Just email '''cluster-admin@soe.ucsc.edu''' with the following information:

	1: Which directory you would like to watch quotas on (i.e. /private/groups/somelab)		1: Which directory you would like to watch quotas on (i.e. /private/groups/somelab)

Genomics Institute Computing Information

Weiler — Fri, 29 May 2026 21:24:35 GMT

GI Firewalled Computing Environment (PRISM)

← Older revision		Revision as of 14:24, 29 May 2026
Line 13:		Line 13:
	*[[Visual Studio Code (vscode) Configuration Tweaks]]		*[[Visual Studio Code (vscode) Configuration Tweaks]]
	*[http://logserv.gi.ucsc.edu/cgi-bin/private-groups.cgi '''/private/groups''' Data Usage Graphs]		*[http://logserv.gi.ucsc.edu/cgi-bin/private-groups.cgi '''/private/groups''' Data Usage Graphs]
			*[http://logserv.gi.ucsc.edu/cgi-bin/private-warm-archive.cgi '''/private/warm-archive''' Data Usage Graphs]
	*[[Resetting your VPN/PRISM Password]]		*[[Resetting your VPN/PRISM Password]]

Slurm Tips for Toil

Anovak — Tue, 26 May 2026 15:39:19 GMT

← Older revision		Revision as of 08:39, 26 May 2026
Line 7:		Line 7:
	To use a development version of Toil, you can install from source instead:		To use a development version of Toil, you can install from source instead:

	pipx 'toil[wdl]@git+https://github.com/DataBiosphere/toil.git'		pipx install 'toil[wdl]@git+https://github.com/DataBiosphere/toil.git'

	Or for a particular branch:		Or for a particular branch:

Create a Jupyter Virtual Environment

Weiler — Sat, 23 May 2026 14:02:56 GMT

Create Your Virtual Environment

← Older revision		Revision as of 07:02, 23 May 2026
(One intermediate revision by the same user not shown)
Line 19:		Line 19:
	Next create the environment!		Next create the environment!

	cd ~~jupyterhub_venv~~		cd jupyter_venv
	python3 -m venv jupyterhub		python3 -m venv jupyterhub

Access the JupyterHub Slurm Server

Weiler — Sat, 23 May 2026 01:11:04 GMT

The Live Notebook

← Older revision		Revision as of 18:11, 22 May 2026
(3 intermediate revisions by the same user not shown)
Line 39:		Line 39:
	Once you start your notebook (give it some time, it can take 10-20 seconds to start based on your environment and resource requests), you should see your notebook appear as something like this in your web browser:		Once you start your notebook (give it some time, it can take 10-20 seconds to start based on your environment and resource requests), you should see your notebook appear as something like this in your web browser:

	[[File:~~Notebook~~.png\|1200px\|frame\|left\|Login Page]]		[[File:Live_notebook.png\|1200px\|frame\|left\|Login Page]]

			If the slurm queue is full, you may be waiting a while.

	This is a notebook actually running on one of the cluster nodes, but the connection is proxied through the head node, and you are seeing it in your web browser. You can actually see your job running in slurm by looking at the output of the '''squeue''' command, if you are curious.		This is a notebook actually running on one of the cluster nodes, but the connection is proxied through the head node, and you are seeing it in your web browser. You can actually see your job running in slurm by looking at the output of the '''squeue''' command, if you are curious.
Line 46:		Line 48:

	When you are done, don't forget to terminate your notebook to free up the Slurm resources it was using! To close your notebook, click on '''File -> Hub Control Panel'''. Then from there click the red '''Stop My Server''' button. Your notebook will terminate and the associated Slurm job will also exit.		When you are done, don't forget to terminate your notebook to free up the Slurm resources it was using! To close your notebook, click on '''File -> Hub Control Panel'''. Then from there click the red '''Stop My Server''' button. Your notebook will terminate and the associated Slurm job will also exit.

			If you simply click "Logout", or "File -> Logout", it will log you out, but your notebook is still running! So next time you log in, it will take you right back to that notebook. If you are truly done, then kill your notebook via '''Stop My Server''' as described above.

			If slurm sees the job is still running when the job's wall time is exceeded, slurm will kill your notebook automatically.

File:Live notebook.png

Weiler — Sat, 23 May 2026 00:57:24 GMT

Weiler uploaded File:Live notebook.png

New page

Access the JupyterHub Slurm Server

Weiler — Sat, 23 May 2026 00:55:35 GMT

← Older revision		Revision as of 17:55, 22 May 2026
Line 31:		Line 31:
	If you need a more specific environment, click on the '''Advanced''' tab, which should yield this screen:		If you need a more specific environment, click on the '''Advanced''' tab, which should yield this screen:

	[[File:~~Advanced_server~~.png\|600px\|frame\|left\|Advanced Server]]		[[File:Advanced_options.png\|600px\|frame\|left\|Advanced Server]]

	This allows for a more fine grained resource definition when requesting resources for your notebook. It also allows for a custom Jupyter environment to be defined, if you need one. The Default environment is fairly basic, so it may make sense to create your own custom environment. If you would like to do that, refer to [[Create a Jupyter Virtual Environment]].		This allows for a more fine grained resource definition when requesting resources for your notebook. It also allows for a custom Jupyter environment to be defined, if you need one. The Default environment is fairly basic, so it may make sense to create your own custom environment. If you would like to do that, refer to [[Create a Jupyter Virtual Environment]].

File:Advanced options.png

Weiler — Sat, 23 May 2026 00:55:15 GMT

Weiler uploaded File:Advanced options.png

New page

Access the JupyterHub Slurm Server

Weiler — Sat, 23 May 2026 00:54:04 GMT

← Older revision		Revision as of 17:54, 22 May 2026
Line 23:		Line 23:
	You will see this page after logging in, which is the "Simple" setup page:		You will see this page after logging in, which is the "Simple" setup page:

	[[File:~~Simple_server~~.png\|600px\|frame\|left\|Simple Server]]		[[File:Simple_options.png\|600px\|frame\|left\|Simple Server]]

	Here you can define more specific job parameters.		Here you can define more specific job parameters.

File:Simple options.png

Weiler — Sat, 23 May 2026 00:53:45 GMT

Weiler uploaded File:Simple options.png

New page

Access the JupyterHub Slurm Server

Weiler — Sat, 23 May 2026 00:52:09 GMT

← Older revision		Revision as of 17:52, 22 May 2026
Line 13:		Line 13:
	There you will see a login page:		There you will see a login page:

	[[File:~~Jupyter_login~~.png\|1200px\|frame\|left\|Login Page]]		[[File:Signin.png\|1200px\|frame\|left\|Login Page]]

	Login using your regular UNIX username and password.		Login using your regular UNIX username and password.

File:Signin.png

Weiler — Sat, 23 May 2026 00:51:42 GMT

Weiler uploaded File:Signin.png

New page

Create a Jupyter Virtual Environment

Weiler — Sat, 23 May 2026 00:49:52 GMT

← Older revision		Revision as of 17:49, 22 May 2026
Line 58:		Line 58:
	You will want to click on the '''Advanced''' tab at the top. When defining your notebook resources, under "Jupyter environment", select "Custom". Then define your description and give the path to your '''/bin''' directory in your environment, as such:		You will want to click on the '''Advanced''' tab at the top. When defining your notebook resources, under "Jupyter environment", select "Custom". Then define your description and give the path to your '''/bin''' directory in your environment, as such:

	[[File:~~Custom2~~.png\|800px\|frame\|left\|Custom Environment]]		[[File:Custom_env.png\|800px\|frame\|left\|Custom Environment]]

	It will then launch a notebook based on that environment, and all modules and packages installed in that environment will be available.		It will then launch a notebook based on that environment, and all modules and packages installed in that environment will be available.

File:Custom env.png

Weiler — Sat, 23 May 2026 00:49:28 GMT

Weiler uploaded File:Custom env.png

New page

Change Default File Browser Directory

Weiler — Sat, 23 May 2026 00:42:56 GMT

Created page with "__TOC__ When starting a notebook, by default, the "File Browser" directory on the left panel of the Jupyter Notebook window in your home directory. If you would prefer to have that default directory be something else, follow these steps. == Stop All Current Running Notebooks == Stop all your notebooks before proceeding. In this case, click "File" -> "Hub Control Panel", then click the red "Stop Server" button. == Create User Jupyter Config File == Create a custom..."

New page

__TOC__

When starting a notebook, by default, the "File Browser" directory on the left panel of the Jupyter Notebook window in your home directory. If you would prefer to have that default directory be something else, follow these steps.

== Stop All Current Running Notebooks ==

Stop all your notebooks before proceeding. In this case, click "File" -> "Hub Control Panel", then click the red "Stop Server" button.

== Create User Jupyter Config File ==

Create a custom Jupyter config file by ssh'ing over to '''mustard.prism''' and then doing:

jupyter server --generate-config

It will create a directory in your home directory called '''.jupyter'''. There is a file in that directory called:

jupyter_server_config.py

We need to edit that file, and add a line to the top of the file. The line we add will direct the Jupyter Notebook to use a different root directory when spawning, often a groups directory like '''/private/groups/hausslerlab''' as an example. Open the file using '''vim''' or whatever your favorite text editor is:

vim ~/.jupyter/jupyter_server_config.py

Add this line to the top of that file:

c.ServerApp.root_dir = '/private/groups/hausslerlab'

'''/private/groups/hausslerlab''' should be changed to whatever you want your default directory to be.

== Launch a New Notebook Instance ==

Launch a new cluster notebook instance, and you should see your new default directory on the left!

== OPTIONAL: Symlinks in your Home Directory ==

There is another way to do this that doesn't involve creating a config file and editing it. Simply add a symlink in your home directory to your lab's groups directory, and you can click on that in the file browser and it will take you to your lab directory.

cd ~
ln -s /private/groups/hausslerlab hausslerlab

Again, '''/private/groups/hausslerlab''' should be wherever you want the link to take you.

Genomics Institute Computing Information

Weiler — Sat, 23 May 2026 00:40:21 GMT

JupyterHub Notebooks Spawned via Slurm

← Older revision		Revision as of 17:40, 22 May 2026
(One intermediate revision by the same user not shown)
Line 40:		Line 40:
	*[[Access the JupyterHub Slurm Server]]		*[[Access the JupyterHub Slurm Server]]
	*[[Create a Jupyter Virtual Environment]]		*[[Create a Jupyter Virtual Environment]]
	*[[~~Using Shared Jupyter Virtual Environments~~]]		*[[Change Default File Browser Directory]]

	== Slurm at the Genomics Institute ==		== Slurm at the Genomics Institute ==

Create a Jupyter Virtual Environment

Weiler — Sat, 23 May 2026 00:33:43 GMT

Created page with "__TOC__ If you need a specific Jupyter environment, because maybe you need specific Python modules or something like that, you've come to the right place. == Find a Location to Install == You will want to install your Jupyter Virtual Environment somewhere accessible from the PRISM Compute Nodes and all the cluster nodes, so it will need to either be in your home directory or in '''/private/groups/[lab_name]''' somewhere. For this example I'll use my home directory, '..."

New page

__TOC__

If you need a specific Jupyter environment, because maybe you need specific Python modules or something like that, you've come to the right place.

== Find a Location to Install ==

You will want to install your Jupyter Virtual Environment somewhere accessible from the PRISM Compute Nodes and all the cluster nodes, so it will need to either be in your home directory or in '''/private/groups/[lab_name]''' somewhere. For this example I'll use my home directory, '''/private/home/weiler'''.

== Create Your Virtual Environment ==

I'll be creating this Virtual Environment in my home directory (but create it wherever you deem necessary). So:

cd /private/home/weiler

Then create the directory for the environment to go in:

mkdir jupyter_venv

Next create the environment!

cd jupyterhub_venv
python3 -m venv jupyterhub

It will create an environment in '''/private/home/weiler/jupyter_env/jupyterhub'''.

Next we activate the environment:

source ~/jupyter_env/jupyterhub/bin/activate

Your prompt should now look something like this:

(jupyterhub) [weiler@mustard jupyter_venv]#

== Install Requisite and Optional Packages ==

From there you will need to minimally install these three packages:

(jupyterhub) [weiler@mustard jupyter_venv]# pip3 install jupyterhub jupyterlab batchspawner

You will '''need''' those three packages to start. So definitely install those. You can additionally install any other python packages, such as:

(jupyterhub) [weiler@mustard jupyter_venv]# pip3 install numpy pandas

== Exit Your Virtual Environment ==

To exit out of your virtual environment, simply do:

deactivate

and you should return to your regular shell.

== Call Your Virtual Environment In JupyterHub ==

Once you login to JupyterHub on the cluster here:

https://jupyterhub.prism

You will want to click on the '''Advanced''' tab at the top. When defining your notebook resources, under "Jupyter environment", select "Custom". Then define your description and give the path to your '''/bin''' directory in your environment, as such:

[[File:Custom2.png|800px|frame|left|Custom Environment]]

It will then launch a notebook based on that environment, and all modules and packages installed in that environment will be available.

Access the JupyterHub Slurm Server

Weiler — Sat, 23 May 2026 00:24:49 GMT

← Older revision		Revision as of 17:24, 22 May 2026
Line 7:		Line 7:
	You can login to the JupyterHub interface here:		You can login to the JupyterHub interface here:

	https://~~jupyter~~.~~mckenna.edu~~		https://jupyterhub.prism

	Your web browser may complain about a name mismatch on the SSL certificate but you can click past it, the connection is still secure.		Your web browser may complain about a name mismatch on the SSL certificate but you can click past it, the connection is still secure.

Access the JupyterHub Slurm Server

Weiler — Sat, 23 May 2026 00:23:52 GMT

Created page with "__TOC__ We have a JupyterHub server running on top of Slurm that is very useful for spawning Jupyter Notebooks. == Login to the Hub == You can login to the JupyterHub interface here: https://jupyter.mckenna.edu Your web browser may complain about a name mismatch on the SSL certificate but you can click past it, the connection is still secure. There you will see a login page: 1200px|frame|left|Login Page Login using your regular UNIX us..."

New page

__TOC__

We have a JupyterHub server running on top of Slurm that is very useful for spawning Jupyter Notebooks.

== Login to the Hub ==

You can login to the JupyterHub interface here:

https://jupyter.mckenna.edu

Your web browser may complain about a name mismatch on the SSL certificate but you can click past it, the connection is still secure.

There you will see a login page:

[[File:Jupyter_login.png|1200px|frame|left|Login Page]]

Login using your regular UNIX username and password.

== Configure Your Notebook ==

Once you login, you can configure your notebook for its required resources in Slurm, and then launch your notebook, which will appear on a node Slurm chooses. If a node is free right then, your notebook will launch immediately. If the Slurm queue is full, you may have to wait for resources to become available.

You will see this page after logging in, which is the "Simple" setup page:

[[File:Simple_server.png|600px|frame|left|Simple Server]]

Here you can define more specific job parameters.

You will want to mostly accept the defaults if running a simple notebook. But if your notebook requires more resources, then kick up the max time or CPU/GPU count as appropriate. Remember Slurm will kill the notebook if it persists past the '''Job Duration''' parameter. Simply click the '''Start''' button on the bottom to launch the notebook.

If you need a more specific environment, click on the '''Advanced''' tab, which should yield this screen:

[[File:Advanced_server.png|600px|frame|left|Advanced Server]]

This allows for a more fine grained resource definition when requesting resources for your notebook. It also allows for a custom Jupyter environment to be defined, if you need one. The Default environment is fairly basic, so it may make sense to create your own custom environment. If you would like to do that, refer to [[Create a Jupyter Virtual Environment]].

== The Live Notebook ==

Once you start your notebook (give it some time, it can take 10-20 seconds to start based on your environment and resource requests), you should see your notebook appear as something like this in your web browser:

[[File:Notebook.png|1200px|frame|left|Login Page]]

This is a notebook actually running on one of the cluster nodes, but the connection is proxied through the head node, and you are seeing it in your web browser. You can actually see your job running in slurm by looking at the output of the '''squeue''' command, if you are curious.

== Terminate Your Notebook When Finished ==

When you are done, don't forget to terminate your notebook to free up the Slurm resources it was using! To close your notebook, click on '''File -> Hub Control Panel'''. Then from there click the red '''Stop My Server''' button. Your notebook will terminate and the associated Slurm job will also exit.

Genomics Institute Computing Information

Weiler — Sat, 23 May 2026 00:22:50 GMT

← Older revision		Revision as of 17:22, 22 May 2026
Line 36:		Line 36:
	*[[AWS Best Practices]]		*[[AWS Best Practices]]
	*[[AWS S3 Lifecycle Management]]		*[[AWS S3 Lifecycle Management]]

			== JupyterHub Notebooks Spawned via Slurm ==
			*[[Access the JupyterHub Slurm Server]]
			*[[Create a Jupyter Virtual Environment]]
			*[[Using Shared Jupyter Virtual Environments]]

	== Slurm at the Genomics Institute ==		== Slurm at the Genomics Institute ==