UCSC Genomics Institute Computing Infrastructure Information - User contributions [en]

Firewalled Computing Resources Overview

2026-06-09T14:02:09Z

Weiler: /* The Phoenix Cluster */

== Doing Work and Computing ==

When doing research, running jobs and the like, please be careful of your resource consumption on the server you are on. Don't run too many threads or cores at once if such a thing overruns the RAM available or the disk IO available. If you are not sure of your potential RAM, CPU or disk impact, start small with one or two processes and work your way up from there. Also, before running your stuff, check what else is already happening on the server by using the 'top' command to see who else and what else is running and what kind of resources are already being consumed. If, after starting a process, you realize that the server slows down considerably or becomes unusable, kill
your processes and re-evaluate what you need to make things work. These servers are shared resources - be a good neighbor!

== Server Types and Management ==

After confirming your VPN software is working, you can ssh into one of the shared compute servers behind the VPN. The DNS suffix for all machines is ".prism". So, "mustard" would have a full DNS name of "mustard.prism":

{| class="wikitable" style="text-align:center;"
|- style="font-weight:bold; text-align:left;"
! Node Name
! Operating System 
! CPU Cores
! Memory
! Network Bandwidth
! Scratch Space
|-
| style="text-align:left;" | mustard
| style="text-align:left;" | Ubuntu 22.04
| 160
| 1.5 TB
| 10 Gb/s
| 9 TB
|-
| style="text-align:left;" | emerald
| style="text-align:left;" | Ubuntu 22.04
| 64
| 1 TB
| 10 Gb/s
| 690 GB
|-
| style="text-align:left;" | crimson
| style="text-align:left;" | Ubuntu 22.04
| 32
| 256 GB
| 10 Gb/s
| 5.5 TB
|-
| style="text-align:left;" | razzmatazz
| style="text-align:left;" | Ubuntu 22.04
| 32
| 256 GB
| 10 Gb/s
| 5.5 TB
|}

These ''shared'' servers are managed by the Genomics Institute Cluster Admin group. If you need software installed on any of these servers, please make your request by emailing cluster-admin@soe.ucsc.edu. The above mentioned scratch space is usually located in '''/data/scratch''' on each server.

== The Firewall ==

All servers are behind a firewall in this environment, and as such, you must connect to the VPN in order to access them. They will not be accessible from the greater Internet without VPN. Although you will be able to connect outbound from them to other servers on the internet to copy data in, sync git repos, stuff like that. It is only inbound connections that will be blocked. All machines behind the firewall have the private domain name suffix of "*.prism".

== The Phoenix Cluster ==

This is a cluster of 25 Ubuntu 22.04 nodes, some of which have GPUs in them. Each node generally has about 2TB RAM and 256 cores, although the cluster is heterogeneous and has multiple node types. You interact with the Phoenix Cluster via the Slurm Job Scheduler. You must specifically request access to use Slurm on the Phoenix Cluster, just email '''cluster-admin@soe.ucsc.edu''' for access.

This HPC cluster is a fairly modern cluster comprised of:

{| class="wikitable" style="text-align:center;"
|- style="font-weight:bold;"
! Total CPU Cores
! Total Memory
! Local Scratch Storage
! Six GPU Nodes
! Primary Storage
! Secondary Storage
! Archive Storage
! Network Backbone
|-
| 7040
| 49.9 TB
| 16 TB (NVMe) per node, 400 TB total
| 8 GPUs per node (A100, A5500), 48 GPUs total
| 1.7 PB of Clustered Ceph NVMe Solid State Storage
| 2.5 PB Spinning Disk Storage (Ceph + ZFS)
| 4.6 PB Spinning Disk Storage (ZFS)
| 10 Gb/s (nodes) / 200 Gb/s+ (storage)
|}

Specific breakdown of node architecture:

{| class="wikitable" style="text-align:center;"
|- style="font-weight:bold;"
! Node Name
! Operating System 
! CPU Cores
! GPUs/Type
! Memory
! Network Bandwidth
! Scratch Space
|-
| style="text-align:left;" | phoenix-00
| style="text-align:left;" | Ubuntu 22.04
| 256
| 8 / Nvidia A100
| 1 TB
| 10 Gb/s
| 16 TB NVMe
|-
| style="text-align:left;" | phoenix-[01-05]
| style="text-align:left;" | Ubuntu 22.04
| 256
| 8 / Nvidia A5500
| 2 TB
| 10 Gb/s
| 16 TB NVMe
|-
| style="text-align:left;" | phoenix-[06-08]
| style="text-align:left;" | Ubuntu 22.04
| 256
| N/A
| 2 TB
| 10 Gb/s
| 16 TB NVMe
|-
| style="text-align:left;" | phoenix-[09-10]
| style="text-align:left;" | Ubuntu 22.04
| 384
| N/A
| 2.3 TB
| 10 Gb/s
| 16 TB NVMe
|-
| style="text-align:left;" | phoenix-[11-21]
| style="text-align:left;" | Ubuntu 22.04
| 256
| N/A
| 2 TB
| 10 Gb/s
| 16 TB NVMe
|-
| style="text-align:left;" | phoenix-[22-24]
| style="text-align:left;" | Ubuntu 22.04
| 384
| N/A
| 2.3 TB
| 10 Gb/s
| 16 TB NVMe
|}

The cluster head node is '''phoenix.prism'''. However, you cannot directly login to phoenix.prism in order to protect the scheduler from errant or runaway jobs there, so jobs can be submitted from any interactive compute server (mustard, emerald, razzmatazz or crimson). To learn more about how to use Slurm, refer to:

https://giwiki.gi.ucsc.edu/index.php/Genomics_Institute_Computing_Information#Slurm_at_the_Genomics_Institute

For scratch on the cluster, TMPDIR will be set to /data/tmp (which is local to each cluster node). That area is cleaned often so don't store any data there that isn't being used by your jobs.

==Graphical Diagram of the Firewalled Area==

This is a general representation of how things look:

[[File:Ucsc_gi_private_diagram.png|900px]]

Firewalled Environment Storage Overview

2026-05-30T17:43:47Z

Weiler: /* Storage */

== Storage ==

Our servers mount two types of ''shared'' storage; home directories and group storage directories. These home directories will mount over the network to all shared compute servers and the phoenix cluster, so any server you login to will have these filesystems available:

'''Filesystem Specifications'''

{| class="wikitable" style="text-align:center;"
|- style="font-weight:bold; text-align:left;"
! Filesystem 
! /private/home
! /private/groups
! /private/warm-archive
|-
| style="font-weight:bold; text-align:left;" | Default Quota
| 100 GB
| 15 TB
| 50 TB
|-
| style="font-weight:bold; text-align:left;" | Total Capacity
| 19 TB
| 1.7 PB
| 4.6 PB
|- style="text-align:left;"
| style="font-weight:bold;" | Access Speed
| Very Fast (NVMe Flash Media)
| Very Fast (NVMe Flash Media, Distributed Filesystem)
| Slower (Spinning Disks)
|- style="text-align:left;"
| style="font-weight:bold;" | Intended Use
| This space should be used for login scripts, small bits of code or software repos, etc. No large data should be stored here.
| This space should be used for large computational/shared (hot) data, large software installations and the like.
| Archival Use Only. Not meant for active cluster computation. Warm/Cold data only.
|}

'''Home Directories (/private/home/username)'''

Your home directory will be located as "/private/home/username" and has a 100GB quota. Your home directory is meant for small scripts and login data, or a git repo. Please do not try to store large data there or computer on large jobs using data in your home directory.

'''Groups Directories (/private/groups/labname)'''

The group storage directories are created per PI, and each group directory has a default 15TB hard quota. For example, if David Haussler is the PI that you report to directly, then the directory would exist as /private/groups/hausslerlab. Request access to that group directory and you will then be able to write to it. Each of those group directories are shared by the lab it belongs to, so you must be wary of everyone's data usage and share the 15TB available per group accordingly.

The groups storage directories have a file in the root of them that contains current quota usage info. These files (.quota_info) are updated once an hour. For example:

cat /private/groups/corbettlab/.quota_info

==================================================
CephFS Storage Quota Report: corbettlab
==================================================
Filesystem Tier: CephFS
Space Used: 72T
Total Quota: 92T
Available: 21T

Last Updated: 2026-05-30 09:22:31 PDT
==================================================

Every lab directory has a '''.quota_info''' file in its root.

'''Archive Directories (/private/warm-archive/labname)'''

The archive storage directories are created per PI '''by request''', and each group directory has a default 50TB hard quota. For example, if David Haussler is the PI that you report to directly, then the directory would exist as /private/warm-archive/hausslerlab.

The archive storage directories have a file in the root of them that contains current quota usage info. These files (.quota_info) are updated once an hour. For example:

cat /private/warm-archive/corbettlab/.quota_info

==================================================
ZFS Storage Quota Report: corbettlab
==================================================
Project ID: 105
Space Used: 148G
Total Quota: 50T
Available: 50T

Last Updated: 2026-05-30 08:45:57 PDT
==================================================

Every lab directory has a '''.quota_info''' file in its root.

== Storage Quota Alerting ==

If you and/or folks in your lab would like an automated alert when the '''/private/groups/labname''' quota is getting to a certain percentage of fullness, we can set that up for you and others in your lab. Just email '''cluster-admin@soe.ucsc.edu''' with the following information:

1: Which directory you would like to watch quotas on (i.e. /private/groups/somelab)
2: What % full you would like an email alert at
3: What email addresses you want on the alert list

After setup, our alerting system will alert folks on that email list ''every 4 hours'' until the quota in question is reduced to an amount under the alerting % threshold you asked for. So it is a bit noisy, but will force folks to delete data in order to stop the alerts. When the system notices that the quota usage has decreased to under the alert threshold, you will receive one final email with an "OK" notification that things are OK now.

== /data/scratch Space on the Servers ==

Each server will generally have a local /data/scratch filesystem that you can use to store temporary files. '''BE ADVISED''' that /data/scratch is not backed up, and the data there could disappear in the event of a disk failure or anything else. Do not store important data there. If it is important, it should be moved somewhere else very soon after creation.

== Backups ==

/private/groups is backed up monthly on the first of the month (which usually takes a week to complete). Please note that the following directories in the tree '''WILL NOT''' be backed up:

tmp/
temp/
TMP/
TEMP/
cache/
.cache/
scratch/
*.tmp/

So if you have data that you know isn't important and should be excluded from the backups, put them in a directory suffixed with ".tmp". Such as this example:

/private/groups/clusteradmin/mybams.tmp/

Firewalled Environment Storage Overview

2026-05-30T16:24:54Z

Weiler: /* Storage */

== Storage ==

Our servers mount two types of ''shared'' storage; home directories and group storage directories. These home directories will mount over the network to all shared compute servers and the phoenix cluster, so any server you login to will have these filesystems available:

'''Filesystem Specifications'''

{| class="wikitable" style="text-align:center;"
|- style="font-weight:bold; text-align:left;"
! Filesystem 
! /private/home
! /private/groups
! /private/warm-archive
|-
| style="font-weight:bold; text-align:left;" | Default Quota
| 100 GB
| 15 TB
| 50 TB
|-
| style="font-weight:bold; text-align:left;" | Total Capacity
| 19 TB
| 1.7 PB
| 4.6 PB
|- style="text-align:left;"
| style="font-weight:bold;" | Access Speed
| Very Fast (NVMe Flash Media)
| Very Fast (NVMe Flash Media, Distributed Filesystem)
| Slower (Spinning Disks)
|- style="text-align:left;"
| style="font-weight:bold;" | Intended Use
| This space should be used for login scripts, small bits of code or software repos, etc. No large data should be stored here.
| This space should be used for large computational/shared (hot) data, large software installations and the like.
| Archival Use Only. Not meant for active cluster computation. Warm/Cold data only.
|}

'''Home Directories (/private/home/username)'''

Your home directory will be located as "/private/home/username" and has a 100GB quota. Your home directory is meant for small scripts and login data, or a git repo. Please do not try to store large data there or computer on large jobs using data in your home directory.

'''Groups Directories (/private/groups/labname)'''

The group storage directories are created per PI, and each group directory has a default 15TB hard quota. For example, if David Haussler is the PI that you report to directly, then the directory would exist as /private/groups/hausslerlab. Request access to that group directory and you will then be able to write to it. Each of those group directories are shared by the lab it belongs to, so you must be wary of everyone's data usage and share the 15TB available per group accordingly.

The groups storage directories have a file in the root of them that contains current quota usage info. These files (.quota_info) are updated once an hour. For example:

cat /private/groups/corbettlab/.quota_info

==================================================
CephFS Storage Quota Report: corbettlab
==================================================
Filesystem Tier: CephFS
Space Used: 72T
Total Quota: 92T
Available: 21T

Last Updated: 2026-05-30 09:22:31 PDT
==================================================

Every lab directory has a .quota_info file in its root.

'''Archive Directories (/private/warm-archive/labname)'''

The archive storage directories are created per PI '''by request''', and each group directory has a default 50TB hard quota. For example, if David Haussler is the PI that you report to directly, then the directory would exist as /private/warm-archive/hausslerlab.

The archive storage directories have a file in the root of them that contains current quota usage info. These files (.quota_info) are updated once an hour. For example:

cat /private/warm-archive/corbettlab/.quota_info

==================================================
ZFS Storage Quota Report: corbettlab
==================================================
Project ID: 105
Space Used: 148G
Total Quota: 50T
Available: 50T

Last Updated: 2026-05-30 08:45:57 PDT
==================================================

Every lab directory has a '''.quota_info''' file in its root.

== Storage Quota Alerting ==

If you and/or folks in your lab would like an automated alert when the '''/private/groups/labname''' quota is getting to a certain percentage of fullness, we can set that up for you and others in your lab. Just email '''cluster-admin@soe.ucsc.edu''' with the following information:

1: Which directory you would like to watch quotas on (i.e. /private/groups/somelab)
2: What % full you would like an email alert at
3: What email addresses you want on the alert list

After setup, our alerting system will alert folks on that email list ''every 4 hours'' until the quota in question is reduced to an amount under the alerting % threshold you asked for. So it is a bit noisy, but will force folks to delete data in order to stop the alerts. When the system notices that the quota usage has decreased to under the alert threshold, you will receive one final email with an "OK" notification that things are OK now.

== /data/scratch Space on the Servers ==

Each server will generally have a local /data/scratch filesystem that you can use to store temporary files. '''BE ADVISED''' that /data/scratch is not backed up, and the data there could disappear in the event of a disk failure or anything else. Do not store important data there. If it is important, it should be moved somewhere else very soon after creation.

== Backups ==

/private/groups is backed up monthly on the first of the month (which usually takes a week to complete). Please note that the following directories in the tree '''WILL NOT''' be backed up:

tmp/
temp/
TMP/
TEMP/
cache/
.cache/
scratch/
*.tmp/

So if you have data that you know isn't important and should be excluded from the backups, put them in a directory suffixed with ".tmp". Such as this example:

/private/groups/clusteradmin/mybams.tmp/

Firewalled Environment Storage Overview

2026-05-30T15:53:36Z

Weiler: /* Storage Quota Alerting */

== Storage ==

Our servers mount two types of ''shared'' storage; home directories and group storage directories. These home directories will mount over the network to all shared compute servers and the phoenix cluster, so any server you login to will have these filesystems available:

'''Filesystem Specifications'''

{| class="wikitable" style="text-align:center;"
|- style="font-weight:bold; text-align:left;"
! Filesystem 
! /private/home
! /private/groups
! /private/warm-archive
|-
| style="font-weight:bold; text-align:left;" | Default Quota
| 100 GB
| 15 TB
| 50 TB
|-
| style="font-weight:bold; text-align:left;" | Total Capacity
| 19 TB
| 1.7 PB
| 4.6 PB
|- style="text-align:left;"
| style="font-weight:bold;" | Access Speed
| Very Fast (NVMe Flash Media)
| Very Fast (NVMe Flash Media, Distributed Filesystem)
| Slower (Spinning Disks)
|- style="text-align:left;"
| style="font-weight:bold;" | Intended Use
| This space should be used for login scripts, small bits of code or software repos, etc. No large data should be stored here.
| This space should be used for large computational/shared (hot) data, large software installations and the like.
| Archival Use Only. Not meant for active cluster computation. Warm/Cold data only.
|}

'''Home Directories (/private/home/username)'''

Your home directory will be located as "/private/home/username" and has a 100GB quota. Your home directory is meant for small scripts and login data, or a git repo. Please do not try to store large data there or computer on large jobs using data in your home directory.

'''Groups Directories (/private/groups/labname)'''

The group storage directories are created per PI, and each group directory has a default 15TB hard quota. For example, if David Haussler is the PI that you report to directly, then the directory would exist as /private/groups/hausslerlab. Request access to that group directory and you will then be able to write to it. Each of those group directories are shared by the lab it belongs to, so you must be wary of everyone's data usage and share the 15TB available per group accordingly.

On the compute servers you can check your group's current '''/private/groups''' quota usage by using the 'getfattr' command. You can only check the quota of a group you are part of (you would be a member of the UNIX group of the same name). If you wanted to check the quota usage of /private/groups/hausslerlab for example, you would do:

$ getfattr -n ceph.dir.rbytes /private/groups/hausslerlab

getfattr: Removing leading '/' from absolute path names
# file: private/groups/hausslerlab
ceph.dir.rbytes="6522955553147"

That number is in bytes. So divide by 1,000,000,000,000 and you get '6.522 TB'. That is how much data is currently being used.

To check the max quota limit, use this command:

$ getfattr -n ceph.quota.max_bytes /private/groups/hausslerlab

getfattr: Removing leading '/' from absolute path names
# file: private/groups/hausslerlab
ceph.quota.max_bytes="15000000000000"

And 15000000000000 divided by 1,000,000,000,000 is 15 TB.

'''Archive Directories (/private/warm-archive/labname)'''

The archive storage directories are created per PI '''by request''', and each group directory has a default 50TB hard quota. For example, if David Haussler is the PI that you report to directly, then the directory would exist as /private/warm-archive/hausslerlab.

The archive storage directories have a file in the root of them that contains current quota usage info. These files (.quota_info) are updated once an hour. For example:

cat /private/warm-archive/corbettlab/.quota_info

==================================================
ZFS Storage Quota Report: corbettlab
==================================================
Project ID: 105
Space Used: 148G
Total Quota: 50T
Available: 50T

Last Updated: 2026-05-30 08:45:57 PDT
==================================================

Every lab directory has a '''.quota_info''' file in its root.

== Storage Quota Alerting ==

If you and/or folks in your lab would like an automated alert when the '''/private/groups/labname''' quota is getting to a certain percentage of fullness, we can set that up for you and others in your lab. Just email '''cluster-admin@soe.ucsc.edu''' with the following information:

1: Which directory you would like to watch quotas on (i.e. /private/groups/somelab)
2: What % full you would like an email alert at
3: What email addresses you want on the alert list

After setup, our alerting system will alert folks on that email list ''every 4 hours'' until the quota in question is reduced to an amount under the alerting % threshold you asked for. So it is a bit noisy, but will force folks to delete data in order to stop the alerts. When the system notices that the quota usage has decreased to under the alert threshold, you will receive one final email with an "OK" notification that things are OK now.

== /data/scratch Space on the Servers ==

Each server will generally have a local /data/scratch filesystem that you can use to store temporary files. '''BE ADVISED''' that /data/scratch is not backed up, and the data there could disappear in the event of a disk failure or anything else. Do not store important data there. If it is important, it should be moved somewhere else very soon after creation.

== Backups ==

/private/groups is backed up monthly on the first of the month (which usually takes a week to complete). Please note that the following directories in the tree '''WILL NOT''' be backed up:

tmp/
temp/
TMP/
TEMP/
cache/
.cache/
scratch/
*.tmp/

So if you have data that you know isn't important and should be excluded from the backups, put them in a directory suffixed with ".tmp". Such as this example:

/private/groups/clusteradmin/mybams.tmp/

Firewalled Environment Storage Overview

2026-05-30T15:53:14Z

Weiler: /* Storage */

== Storage ==

Our servers mount two types of ''shared'' storage; home directories and group storage directories. These home directories will mount over the network to all shared compute servers and the phoenix cluster, so any server you login to will have these filesystems available:

'''Filesystem Specifications'''

{| class="wikitable" style="text-align:center;"
|- style="font-weight:bold; text-align:left;"
! Filesystem 
! /private/home
! /private/groups
! /private/warm-archive
|-
| style="font-weight:bold; text-align:left;" | Default Quota
| 100 GB
| 15 TB
| 50 TB
|-
| style="font-weight:bold; text-align:left;" | Total Capacity
| 19 TB
| 1.7 PB
| 4.6 PB
|- style="text-align:left;"
| style="font-weight:bold;" | Access Speed
| Very Fast (NVMe Flash Media)
| Very Fast (NVMe Flash Media, Distributed Filesystem)
| Slower (Spinning Disks)
|- style="text-align:left;"
| style="font-weight:bold;" | Intended Use
| This space should be used for login scripts, small bits of code or software repos, etc. No large data should be stored here.
| This space should be used for large computational/shared (hot) data, large software installations and the like.
| Archival Use Only. Not meant for active cluster computation. Warm/Cold data only.
|}

'''Home Directories (/private/home/username)'''

Your home directory will be located as "/private/home/username" and has a 100GB quota. Your home directory is meant for small scripts and login data, or a git repo. Please do not try to store large data there or computer on large jobs using data in your home directory.

'''Groups Directories (/private/groups/labname)'''

The group storage directories are created per PI, and each group directory has a default 15TB hard quota. For example, if David Haussler is the PI that you report to directly, then the directory would exist as /private/groups/hausslerlab. Request access to that group directory and you will then be able to write to it. Each of those group directories are shared by the lab it belongs to, so you must be wary of everyone's data usage and share the 15TB available per group accordingly.

On the compute servers you can check your group's current '''/private/groups''' quota usage by using the 'getfattr' command. You can only check the quota of a group you are part of (you would be a member of the UNIX group of the same name). If you wanted to check the quota usage of /private/groups/hausslerlab for example, you would do:

$ getfattr -n ceph.dir.rbytes /private/groups/hausslerlab

getfattr: Removing leading '/' from absolute path names
# file: private/groups/hausslerlab
ceph.dir.rbytes="6522955553147"

That number is in bytes. So divide by 1,000,000,000,000 and you get '6.522 TB'. That is how much data is currently being used.

To check the max quota limit, use this command:

$ getfattr -n ceph.quota.max_bytes /private/groups/hausslerlab

getfattr: Removing leading '/' from absolute path names
# file: private/groups/hausslerlab
ceph.quota.max_bytes="15000000000000"

And 15000000000000 divided by 1,000,000,000,000 is 15 TB.

'''Archive Directories (/private/warm-archive/labname)'''

The archive storage directories are created per PI '''by request''', and each group directory has a default 50TB hard quota. For example, if David Haussler is the PI that you report to directly, then the directory would exist as /private/warm-archive/hausslerlab.

The archive storage directories have a file in the root of them that contains current quota usage info. These files (.quota_info) are updated once an hour. For example:

cat /private/warm-archive/corbettlab/.quota_info

==================================================
ZFS Storage Quota Report: corbettlab
==================================================
Project ID: 105
Space Used: 148G
Total Quota: 50T
Available: 50T

Last Updated: 2026-05-30 08:45:57 PDT
==================================================

Every lab directory has a '''.quota_info''' file in its root.

== Storage Quota Alerting ==

If you and/or folks in your lab would like an automated alert when the /private/groups/labname quota is getting to a certain percentage of fullness, we can set that up for you and others in your lab. Just email '''cluster-admin@soe.ucsc.edu''' with the following information:

1: Which directory you would like to watch quotas on (i.e. /private/groups/somelab)
2: What % full you would like an email alert at
3: What email addresses you want on the alert list

After setup, our alerting system will alert folks on that email list ''every 4 hours'' until the quota in question is reduced to an amount under the alerting % threshold you asked for. So it is a bit noisy, but will force folks to delete data in order to stop the alerts. When the system notices that the quota usage has decreased to under the alert threshold, you will receive one final email with an "OK" notification that things are OK now.

== /data/scratch Space on the Servers ==

Each server will generally have a local /data/scratch filesystem that you can use to store temporary files. '''BE ADVISED''' that /data/scratch is not backed up, and the data there could disappear in the event of a disk failure or anything else. Do not store important data there. If it is important, it should be moved somewhere else very soon after creation.

== Backups ==

/private/groups is backed up monthly on the first of the month (which usually takes a week to complete). Please note that the following directories in the tree '''WILL NOT''' be backed up:

tmp/
temp/
TMP/
TEMP/
cache/
.cache/
scratch/
*.tmp/

So if you have data that you know isn't important and should be excluded from the backups, put them in a directory suffixed with ".tmp". Such as this example:

/private/groups/clusteradmin/mybams.tmp/

Firewalled Environment Storage Overview

2026-05-30T15:52:31Z

Weiler: /* Storage */

== Storage ==

Our servers mount two types of ''shared'' storage; home directories and group storage directories. These home directories will mount over the network to all shared compute servers and the phoenix cluster, so any server you login to will have these filesystems available:

'''Filesystem Specifications'''

{| class="wikitable" style="text-align:center;"
|- style="font-weight:bold; text-align:left;"
! Filesystem 
! /private/home
! /private/groups
! /private/warm-archive
|-
| style="font-weight:bold; text-align:left;" | Default Quota
| 100 GB
| 15 TB
| 50 TB
|-
| style="font-weight:bold; text-align:left;" | Total Capacity
| 19 TB
| 1.7 PB
| 4.6 PB
|- style="text-align:left;"
| style="font-weight:bold;" | Access Speed
| Very Fast (NVMe Flash Media)
| Very Fast (NVMe Flash Media, Distributed Filesystem)
| Slower (Spinning Disks)
|- style="text-align:left;"
| style="font-weight:bold;" | Intended Use
| This space should be used for login scripts, small bits of code or software repos, etc. No large data should be stored here.
| This space should be used for large computational/shared (hot) data, large software installations and the like.
| Archival Use Only. Not meant for active cluster computation. Warm/Cold data only.
|}

'''Home Directories (/private/home/username)'''

Your home directory will be located as "/private/home/username" and has a 100GB quota. Your home directory is meant for small scripts and login data, or a git repo. Please do not try to store large data there or computer on large jobs using data in your home directory.

'''Groups Directories (/private/groups/labname)'''

The group storage directories are created per PI, and each group directory has a default 15TB hard quota. For example, if David Haussler is the PI that you report to directly, then the directory would exist as /private/groups/hausslerlab. Request access to that group directory and you will then be able to write to it. Each of those group directories are shared by the lab it belongs to, so you must be wary of everyone's data usage and share the 15TB available per group accordingly.

On the compute servers you can check your group's current '''/private/groups''' quota usage by using the 'getfattr' command. You can only check the quota of a group you are part of (you would be a member of the UNIX group of the same name). If you wanted to check the quota usage of /private/groups/hausslerlab for example, you would do:

$ getfattr -n ceph.dir.rbytes /private/groups/hausslerlab

getfattr: Removing leading '/' from absolute path names
# file: private/groups/hausslerlab
ceph.dir.rbytes="6522955553147"

That number is in bytes. So divide by 1,000,000,000,000 and you get '6.522 TB'. That is how much data is currently being used.

To check the max quota limit, use this command:

$ getfattr -n ceph.quota.max_bytes /private/groups/hausslerlab

getfattr: Removing leading '/' from absolute path names
# file: private/groups/hausslerlab
ceph.quota.max_bytes="15000000000000"

And 15000000000000 divided by 1,000,000,000,000 is 15 TB.

'''Archive Directories (/private/warm-archive/labname)'''

The archive storage directories are created per PI '''by request''', and each group directory has a default 50TB hard quota. For example, if David Haussler is the PI that you report to directly, then the directory would exist as /private/warm-archive/hausslerlab.

The archive storage directories have a file in the root of them that contains current quota usage info. These files (.quota_info) are updated once an hour. For example:

cat /private/warm-archive/corbettlab/.quota_info

==================================================
ZFS Storage Quota Report: corbettlab
==================================================
Project ID: 105
Space Used: 148G
Total Quota: 100T
Available: 100T

Last Updated: 2026-05-30 08:45:57 PDT
==================================================

Every lab directory has a '''.quota_info''' file in its root.

== Storage Quota Alerting ==

If you and/or folks in your lab would like an automated alert when the /private/groups/labname quota is getting to a certain percentage of fullness, we can set that up for you and others in your lab. Just email '''cluster-admin@soe.ucsc.edu''' with the following information:

1: Which directory you would like to watch quotas on (i.e. /private/groups/somelab)
2: What % full you would like an email alert at
3: What email addresses you want on the alert list

After setup, our alerting system will alert folks on that email list ''every 4 hours'' until the quota in question is reduced to an amount under the alerting % threshold you asked for. So it is a bit noisy, but will force folks to delete data in order to stop the alerts. When the system notices that the quota usage has decreased to under the alert threshold, you will receive one final email with an "OK" notification that things are OK now.

== /data/scratch Space on the Servers ==

Each server will generally have a local /data/scratch filesystem that you can use to store temporary files. '''BE ADVISED''' that /data/scratch is not backed up, and the data there could disappear in the event of a disk failure or anything else. Do not store important data there. If it is important, it should be moved somewhere else very soon after creation.

== Backups ==

/private/groups is backed up monthly on the first of the month (which usually takes a week to complete). Please note that the following directories in the tree '''WILL NOT''' be backed up:

tmp/
temp/
TMP/
TEMP/
cache/
.cache/
scratch/
*.tmp/

So if you have data that you know isn't important and should be excluded from the backups, put them in a directory suffixed with ".tmp". Such as this example:

/private/groups/clusteradmin/mybams.tmp/

Firewalled Environment Storage Overview

2026-05-30T15:03:03Z

Weiler: /* Storage */

== Storage ==

Our servers mount two types of ''shared'' storage; home directories and group storage directories. These home directories will mount over the network to all shared compute servers and the phoenix cluster, so any server you login to will have these filesystems available:

'''Filesystem Specifications'''

{| class="wikitable" style="text-align:center;"
|- style="font-weight:bold; text-align:left;"
! Filesystem 
! /private/home
! /private/groups
! /private/warm-archive
|-
| style="font-weight:bold; text-align:left;" | Default Quota
| 100 GB
| 15 TB
| 50 TB
|-
| style="font-weight:bold; text-align:left;" | Total Capacity
| 19 TB
| 1.7 PB
| 4.6 PB
|- style="text-align:left;"
| style="font-weight:bold;" | Access Speed
| Very Fast (NVMe Flash Media)
| Very Fast (NVMe Flash Media, Distributed Filesystem)
| Slower (Spinning Disks)
|- style="text-align:left;"
| style="font-weight:bold;" | Intended Use
| This space should be used for login scripts, small bits of code or software repos, etc. No large data should be stored here.
| This space should be used for large computational/shared (hot) data, large software installations and the like.
| Archival Use Only. Not meant for active cluster computation. Warm/Cold data only.
|}

'''Home Directories (/private/home/username)'''

Your home directory will be located as "/private/home/username" and has a 100GB quota. Your home directory is meant for small scripts and login data, or a git repo. Please do not try to store large data there or computer on large jobs using data in your home directory.

'''Groups Directories (/private/groups/labname)'''

The group storage directories are created per PI, and each group directory has a default 15TB hard quota. For example, if David Haussler is the PI that you report to directly, then the directory would exist as /private/groups/hausslerlab. Request access to that group directory and you will then be able to write to it. Each of those group directories are shared by the lab it belongs to, so you must be wary of everyone's data usage and share the 15TB available per group accordingly.

On the compute servers you can check your group's current '''/private/groups''' quota usage by using the 'getfattr' command. You can only check the quota of a group you are part of (you would be a member of the UNIX group of the same name). If you wanted to check the quota usage of /private/groups/hausslerlab for example, you would do:

$ getfattr -n ceph.dir.rbytes /private/groups/hausslerlab

getfattr: Removing leading '/' from absolute path names
# file: private/groups/hausslerlab
ceph.dir.rbytes="6522955553147"

That number is in bytes. So divide by 1,000,000,000,000 and you get '6.522 TB'. That is how much data is currently being used.

To check the max quota limit, use this command:

$ getfattr -n ceph.quota.max_bytes /private/groups/hausslerlab

getfattr: Removing leading '/' from absolute path names
# file: private/groups/hausslerlab
ceph.quota.max_bytes="15000000000000"

And 15000000000000 divided by 1,000,000,000,000 is 15 TB.

'''Archive Directories (/private/warm-archive/labname)'''

The archive storage directories are created per PI '''by request''', and each group directory has a default 50TB hard quota. For example, if David Haussler is the PI that you report to directly, then the directory would exist as /private/warm-archive/hausslerlab.

== Storage Quota Alerting ==

If you and/or folks in your lab would like an automated alert when the /private/groups/labname quota is getting to a certain percentage of fullness, we can set that up for you and others in your lab. Just email '''cluster-admin@soe.ucsc.edu''' with the following information:

1: Which directory you would like to watch quotas on (i.e. /private/groups/somelab)
2: What % full you would like an email alert at
3: What email addresses you want on the alert list

After setup, our alerting system will alert folks on that email list ''every 4 hours'' until the quota in question is reduced to an amount under the alerting % threshold you asked for. So it is a bit noisy, but will force folks to delete data in order to stop the alerts. When the system notices that the quota usage has decreased to under the alert threshold, you will receive one final email with an "OK" notification that things are OK now.

== /data/scratch Space on the Servers ==

Each server will generally have a local /data/scratch filesystem that you can use to store temporary files. '''BE ADVISED''' that /data/scratch is not backed up, and the data there could disappear in the event of a disk failure or anything else. Do not store important data there. If it is important, it should be moved somewhere else very soon after creation.

== Backups ==

/private/groups is backed up monthly on the first of the month (which usually takes a week to complete). Please note that the following directories in the tree '''WILL NOT''' be backed up:

tmp/
temp/
TMP/
TEMP/
cache/
.cache/
scratch/
*.tmp/

So if you have data that you know isn't important and should be excluded from the backups, put them in a directory suffixed with ".tmp". Such as this example:

/private/groups/clusteradmin/mybams.tmp/

Firewalled Environment Storage Overview

2026-05-29T21:36:07Z

Weiler: /* Storage */

== Storage ==

Our servers mount two types of ''shared'' storage; home directories and group storage directories. These home directories will mount over the network to all shared compute servers and the phoenix cluster, so any server you login to will have these filesystems available:

'''Filesystem Specifications'''

{| class="wikitable" style="text-align:center;"
|- style="font-weight:bold; text-align:left;"
! Filesystem 
! /private/home
! /private/groups
! /private/warm-archive
|-
| style="font-weight:bold; text-align:left;" | Default Quota
| 100 GB
| 15 TB
| 50 TB
|-
| style="font-weight:bold; text-align:left;" | Total Capacity
| 19 TB
| 1.7 PB
| 4.6 PB
|- style="text-align:left;"
| style="font-weight:bold;" | Access Speed
| Very Fast (NVMe Flash Media)
| Very Fast (NVMe Flash Media, Distributed Filesystem)
| Slower (Spinning Disks)
|- style="text-align:left;"
| style="font-weight:bold;" | Intended Use
| This space should be used for login scripts, small bits of code or software repos, etc. No large data should be stored here.
| This space should be used for large computational/shared (hot) data, large software installations and the like.
| Archival Use Only. Not meant for active cluster computation. Warm/Cold data only.
|}

'''Home Directories (/private/home/username)'''

Your home directory will be located as "/private/home/username" and has a 100GB quota. Your home directory is meant for small scripts and login data, or a git repo. Please do not try to store large data there or computer on large jobs using data in your home directory.

'''Groups Directories (/private/groups/labname)'''

The group storage directories are created per PI, and each group directory has a default 15TB hard quota. For example, if David Haussler is the PI that you report to directly, then the directory would exist as /private/groups/hausslerlab. Request access to that group directory and you will then be able to write to it. Each of those group directories are shared by the lab it belongs to, so you must be wary of everyone's data usage and share the 15TB available per group accordingly.

'''Archive Directories (/private/warm-archive/labname)'''

The archive storage directories are created per PI '''by request''', and each group directory has a default 50TB hard quota. For example, if David Haussler is the PI that you report to directly, then the directory would exist as /private/warm-archive/hausslerlab.

On the compute servers you can check your group's current '''/private/groups''' quota usage by using the 'getfattr' command. You can only check the quota of a group you are part of (you would be a member of the UNIX group of the same name). If you wanted to check the quota usage of /private/groups/hausslerlab for example, you would do:

$ getfattr -n ceph.dir.rbytes /private/groups/hausslerlab

getfattr: Removing leading '/' from absolute path names
# file: private/groups/hausslerlab
ceph.dir.rbytes="6522955553147"

That number is in bytes. So divide by 1,000,000,000,000 and you get '6.522 TB'. That is how much data is currently being used.

To check the max quota limit, use this command:

$ getfattr -n ceph.quota.max_bytes /private/groups/hausslerlab

getfattr: Removing leading '/' from absolute path names
# file: private/groups/hausslerlab
ceph.quota.max_bytes="15000000000000"

And 15000000000000 divided by 1,000,000,000,000 is 15 TB.

== Storage Quota Alerting ==

If you and/or folks in your lab would like an automated alert when the /private/groups/labname quota is getting to a certain percentage of fullness, we can set that up for you and others in your lab. Just email '''cluster-admin@soe.ucsc.edu''' with the following information:

1: Which directory you would like to watch quotas on (i.e. /private/groups/somelab)
2: What % full you would like an email alert at
3: What email addresses you want on the alert list

After setup, our alerting system will alert folks on that email list ''every 4 hours'' until the quota in question is reduced to an amount under the alerting % threshold you asked for. So it is a bit noisy, but will force folks to delete data in order to stop the alerts. When the system notices that the quota usage has decreased to under the alert threshold, you will receive one final email with an "OK" notification that things are OK now.

== /data/scratch Space on the Servers ==

Each server will generally have a local /data/scratch filesystem that you can use to store temporary files. '''BE ADVISED''' that /data/scratch is not backed up, and the data there could disappear in the event of a disk failure or anything else. Do not store important data there. If it is important, it should be moved somewhere else very soon after creation.

== Backups ==

/private/groups is backed up monthly on the first of the month (which usually takes a week to complete). Please note that the following directories in the tree '''WILL NOT''' be backed up:

tmp/
temp/
TMP/
TEMP/
cache/
.cache/
scratch/
*.tmp/

So if you have data that you know isn't important and should be excluded from the backups, put them in a directory suffixed with ".tmp". Such as this example:

/private/groups/clusteradmin/mybams.tmp/

Firewalled Environment Storage Overview

2026-05-29T21:35:22Z

Weiler: /* Storage */

== Storage ==

Our servers mount two types of ''shared'' storage; home directories and group storage directories. These home directories will mount over the network to all shared compute servers and the phoenix cluster, so any server you login to will have these filesystems available:

'''Filesystem Specifications'''

{| class="wikitable" style="text-align:center;"
|- style="font-weight:bold; text-align:left;"
! Filesystem 
! /private/home
! /private/groups
! /private/warm-archive
|-
| style="font-weight:bold; text-align:left;" | Default Quota
| 100 GB
| 15 TB
| 50 TB
|-
| style="font-weight:bold; text-align:left;" | Total Capacity
| 19 TB
| 1.7 PB
| 4.6 PB
|- style="text-align:left;"
| style="font-weight:bold;" | Access Speed
| Very Fast (NVMe Flash Media)
| Very Fast (NVMe Flash Media)
| Slower (Spinning Disks)
|- style="text-align:left;"
| style="font-weight:bold;" | Intended Use
| This space should be used for login scripts, small bits of code or software repos, etc. No large data should be stored here.
| This space should be used for large computational/shared (hot) data, large software installations and the like.
| Archival Use Only. Not meant for active cluster computation. Warm/Cold data only.
|}

'''Home Directories (/private/home/username)'''

Your home directory will be located as "/private/home/username" and has a 100GB quota. Your home directory is meant for small scripts and login data, or a git repo. Please do not try to store large data there or computer on large jobs using data in your home directory.

'''Groups Directories (/private/groups/labname)'''

The group storage directories are created per PI, and each group directory has a default 15TB hard quota. For example, if David Haussler is the PI that you report to directly, then the directory would exist as /private/groups/hausslerlab. Request access to that group directory and you will then be able to write to it. Each of those group directories are shared by the lab it belongs to, so you must be wary of everyone's data usage and share the 15TB available per group accordingly.

'''Archive Directories (/private/warm-archive/labname)'''

The archive storage directories are created per PI '''by request''', and each group directory has a default 50TB hard quota. For example, if David Haussler is the PI that you report to directly, then the directory would exist as /private/warm-archive/hausslerlab.

On the compute servers you can check your group's current '''/private/groups''' quota usage by using the 'getfattr' command. You can only check the quota of a group you are part of (you would be a member of the UNIX group of the same name). If you wanted to check the quota usage of /private/groups/hausslerlab for example, you would do:

$ getfattr -n ceph.dir.rbytes /private/groups/hausslerlab

getfattr: Removing leading '/' from absolute path names
# file: private/groups/hausslerlab
ceph.dir.rbytes="6522955553147"

That number is in bytes. So divide by 1,000,000,000,000 and you get '6.522 TB'. That is how much data is currently being used.

To check the max quota limit, use this command:

$ getfattr -n ceph.quota.max_bytes /private/groups/hausslerlab

getfattr: Removing leading '/' from absolute path names
# file: private/groups/hausslerlab
ceph.quota.max_bytes="15000000000000"

And 15000000000000 divided by 1,000,000,000,000 is 15 TB.

== Storage Quota Alerting ==

If you and/or folks in your lab would like an automated alert when the /private/groups/labname quota is getting to a certain percentage of fullness, we can set that up for you and others in your lab. Just email '''cluster-admin@soe.ucsc.edu''' with the following information:

1: Which directory you would like to watch quotas on (i.e. /private/groups/somelab)
2: What % full you would like an email alert at
3: What email addresses you want on the alert list

After setup, our alerting system will alert folks on that email list ''every 4 hours'' until the quota in question is reduced to an amount under the alerting % threshold you asked for. So it is a bit noisy, but will force folks to delete data in order to stop the alerts. When the system notices that the quota usage has decreased to under the alert threshold, you will receive one final email with an "OK" notification that things are OK now.

== /data/scratch Space on the Servers ==

Each server will generally have a local /data/scratch filesystem that you can use to store temporary files. '''BE ADVISED''' that /data/scratch is not backed up, and the data there could disappear in the event of a disk failure or anything else. Do not store important data there. If it is important, it should be moved somewhere else very soon after creation.

== Backups ==

/private/groups is backed up monthly on the first of the month (which usually takes a week to complete). Please note that the following directories in the tree '''WILL NOT''' be backed up:

tmp/
temp/
TMP/
TEMP/
cache/
.cache/
scratch/
*.tmp/

So if you have data that you know isn't important and should be excluded from the backups, put them in a directory suffixed with ".tmp". Such as this example:

/private/groups/clusteradmin/mybams.tmp/

Firewalled Environment Storage Overview

2026-05-29T21:34:31Z

Weiler: /* Storage */

== Storage ==

Our servers mount two types of ''shared'' storage; home directories and group storage directories. These home directories will mount over the network to all shared compute servers and the phoenix cluster, so any server you login to will have these filesystems available:

'''Filesystem Specifications'''

{| class="wikitable" style="text-align:center;"
|- style="font-weight:bold; text-align:left;"
! Filesystem 
! /private/home
! /private/groups
! /private/warm-archive
|-
| style="font-weight:bold; text-align:left;" | Default Quota
| 100 GB
| 15 TB
| 50 TB
|-
| style="font-weight:bold; text-align:left;" | Total Capacity
| 19 TB
| 1.7 PB
| 4.6 PB
|- style="text-align:left;"
| style="font-weight:bold;" | Access Speed
| Very Fast (NVMe Flash Media)
| Very Fast (NVMe Flash Media)
| Slower (Spinning Disks)
|- style="text-align:left;"
| style="font-weight:bold;" | Intended Use
| This space should be used for login scripts, small bits of code or software repos, etc. No large data should be stored here.
| This space should be used for large computational/shared data, large software installations and the like.
| Archival Use Only. Not meant for active cluster computation. Cold data only.
|}

'''Home Directories (/private/home/username)'''

Your home directory will be located as "/private/home/username" and has a 100GB quota. Your home directory is meant for small scripts and login data, or a git repo. Please do not try to store large data there or computer on large jobs using data in your home directory.

'''Groups Directories (/private/groups/labname)'''

The group storage directories are created per PI, and each group directory has a default 15TB hard quota. For example, if David Haussler is the PI that you report to directly, then the directory would exist as /private/groups/hausslerlab. Request access to that group directory and you will then be able to write to it. Each of those group directories are shared by the lab it belongs to, so you must be wary of everyone's data usage and share the 15TB available per group accordingly.

'''Archive Directories (/private/warm-archive/labname)'''

The archive storage directories are created per PI '''by request''', and each group directory has a default 50TB hard quota. For example, if David Haussler is the PI that you report to directly, then the directory would exist as /private/warm-archive/hausslerlab.

On the compute servers you can check your group's current '''/private/groups''' quota usage by using the 'getfattr' command. You can only check the quota of a group you are part of (you would be a member of the UNIX group of the same name). If you wanted to check the quota usage of /private/groups/hausslerlab for example, you would do:

$ getfattr -n ceph.dir.rbytes /private/groups/hausslerlab

getfattr: Removing leading '/' from absolute path names
# file: private/groups/hausslerlab
ceph.dir.rbytes="6522955553147"

That number is in bytes. So divide by 1,000,000,000,000 and you get '6.522 TB'. That is how much data is currently being used.

To check the max quota limit, use this command:

$ getfattr -n ceph.quota.max_bytes /private/groups/hausslerlab

getfattr: Removing leading '/' from absolute path names
# file: private/groups/hausslerlab
ceph.quota.max_bytes="15000000000000"

And 15000000000000 divided by 1,000,000,000,000 is 15 TB.

== Storage Quota Alerting ==

If you and/or folks in your lab would like an automated alert when the /private/groups/labname quota is getting to a certain percentage of fullness, we can set that up for you and others in your lab. Just email '''cluster-admin@soe.ucsc.edu''' with the following information:

1: Which directory you would like to watch quotas on (i.e. /private/groups/somelab)
2: What % full you would like an email alert at
3: What email addresses you want on the alert list

After setup, our alerting system will alert folks on that email list ''every 4 hours'' until the quota in question is reduced to an amount under the alerting % threshold you asked for. So it is a bit noisy, but will force folks to delete data in order to stop the alerts. When the system notices that the quota usage has decreased to under the alert threshold, you will receive one final email with an "OK" notification that things are OK now.

== /data/scratch Space on the Servers ==

Each server will generally have a local /data/scratch filesystem that you can use to store temporary files. '''BE ADVISED''' that /data/scratch is not backed up, and the data there could disappear in the event of a disk failure or anything else. Do not store important data there. If it is important, it should be moved somewhere else very soon after creation.

== Backups ==

/private/groups is backed up monthly on the first of the month (which usually takes a week to complete). Please note that the following directories in the tree '''WILL NOT''' be backed up:

tmp/
temp/
TMP/
TEMP/
cache/
.cache/
scratch/
*.tmp/

So if you have data that you know isn't important and should be excluded from the backups, put them in a directory suffixed with ".tmp". Such as this example:

/private/groups/clusteradmin/mybams.tmp/

Firewalled Environment Storage Overview

2026-05-29T21:33:18Z

Weiler: /* Storage */

== Storage ==

Our servers mount two types of ''shared'' storage; home directories and group storage directories. These home directories will mount over the network to all shared compute servers and the phoenix cluster, so any server you login to will have these filesystems available:

'''Filesystem Specifications'''

{| class="wikitable" style="text-align:center;"
|- style="font-weight:bold; text-align:left;"
! Filesystem 
! /private/home
! /private/groups
! /private/warm-archive
|-
| style="font-weight:bold; text-align:left;" | Default Quota
| 100 GB
| 15 TB
| 50 TB
|-
| style="font-weight:bold; text-align:left;" | Total Capacity
| 19 TB
| 1.7 PB
| 4.6 PB
|- style="text-align:left;"
| style="font-weight:bold;" | Access Speed
| Very Fast (NVMe Flash Media)
| Very Fast (NVMe Flash Media)
| Slower (Spinning Disks)
|- style="text-align:left;"
| style="font-weight:bold;" | Intended Use
| This space should be used for login scripts, small bits of code or software repos, etc. No large data should be stored here.
| This space should be used for large computational/shared data, large software installations and the like.
| Archival Use Only. Not meant for active cluster computation.
|}

'''Home Directories (/private/home/username)'''

Your home directory will be located as "/private/home/username" and has a 100GB quota. Your home directory is meant for small scripts and login data, or a git repo. Please do not try to store large data there or computer on large jobs using data in your home directory.

'''Groups Directories (/private/groups/labname)'''

The group storage directories are created per PI, and each group directory has a default 15TB hard quota. For example, if David Haussler is the PI that you report to directly, then the directory would exist as /private/groups/hausslerlab. Request access to that group directory and you will then be able to write to it. Each of those group directories are shared by the lab it belongs to, so you must be wary of everyone's data usage and share the 15TB available per group accordingly.

'''Archive Directories (/private/warm-archive/labname)'''

The archive storage directories are created per PI '''by request''', and each group directory has a default 50TB hard quota. For example, if David Haussler is the PI that you report to directly, then the directory would exist as /private/warm-archive/hausslerlab.

On the compute servers you can check your group's current '''/private/groups''' quota usage by using the 'getfattr' command. You can only check the quota of a group you are part of (you would be a member of the UNIX group of the same name). If you wanted to check the quota usage of /private/groups/hausslerlab for example, you would do:

$ getfattr -n ceph.dir.rbytes /private/groups/hausslerlab

getfattr: Removing leading '/' from absolute path names
# file: private/groups/hausslerlab
ceph.dir.rbytes="6522955553147"

That number is in bytes. So divide by 1,000,000,000,000 and you get '6.522 TB'. That is how much data is currently being used.

To check the max quota limit, use this command:

$ getfattr -n ceph.quota.max_bytes /private/groups/hausslerlab

getfattr: Removing leading '/' from absolute path names
# file: private/groups/hausslerlab
ceph.quota.max_bytes="15000000000000"

And 15000000000000 divided by 1,000,000,000,000 is 15 TB.

== Storage Quota Alerting ==

If you and/or folks in your lab would like an automated alert when the /private/groups/labname quota is getting to a certain percentage of fullness, we can set that up for you and others in your lab. Just email '''cluster-admin@soe.ucsc.edu''' with the following information:

1: Which directory you would like to watch quotas on (i.e. /private/groups/somelab)
2: What % full you would like an email alert at
3: What email addresses you want on the alert list

After setup, our alerting system will alert folks on that email list ''every 4 hours'' until the quota in question is reduced to an amount under the alerting % threshold you asked for. So it is a bit noisy, but will force folks to delete data in order to stop the alerts. When the system notices that the quota usage has decreased to under the alert threshold, you will receive one final email with an "OK" notification that things are OK now.

== /data/scratch Space on the Servers ==

Each server will generally have a local /data/scratch filesystem that you can use to store temporary files. '''BE ADVISED''' that /data/scratch is not backed up, and the data there could disappear in the event of a disk failure or anything else. Do not store important data there. If it is important, it should be moved somewhere else very soon after creation.

== Backups ==

/private/groups is backed up monthly on the first of the month (which usually takes a week to complete). Please note that the following directories in the tree '''WILL NOT''' be backed up:

tmp/
temp/
TMP/
TEMP/
cache/
.cache/
scratch/
*.tmp/

So if you have data that you know isn't important and should be excluded from the backups, put them in a directory suffixed with ".tmp". Such as this example:

/private/groups/clusteradmin/mybams.tmp/

Firewalled Environment Storage Overview

2026-05-29T21:32:51Z

Weiler: /* Storage */

== Storage ==

Our servers mount two types of ''shared'' storage; home directories and group storage directories. These home directories will mount over the network to all shared compute servers and the phoenix cluster, so any server you login to will have these filesystems available:

'''Filesystem Specifications'''

{| class="wikitable" style="text-align:center;"
|- style="font-weight:bold; text-align:left;"
! Filesystem 
! /private/home
! /private/groups
! /private/warm-archive
|-
| style="font-weight:bold; text-align:left;" | Default Quota
| 100 GB
| 15 TB
| 50 TB
|-
| style="font-weight:bold; text-align:left;" | Total Capacity
| 19 TB
| 1.7 PB
| 4.6 PB
|- style="text-align:left;"
| style="font-weight:bold;" | Access Speed
| Very Fast (NVMe Flash Media)
| Very Fast (NVMe Flash Media)
| Slower (Spinning Disks)
|- style="text-align:left;"
| style="font-weight:bold;" | Intended Use
| This space should be used for login scripts, small bits of code or software repos, etc. No large data should be stored here.
| This space should be used for large computational/shared data, large software installations and the like.
| Archival Use Only. Not meant for active cluster computation.
|}

'''Home Directories (/private/home/username)'''

Your home directory will be located as "/private/home/username" and has a 100GB quota. Your home directory is meant for small scripts and login data, or a git repo. Please do not try to store large data there or computer on large jobs using data in your home directory.

'''Groups Directories (/private/groups/labname)'''

The group storage directories are created per PI, and each group directory has a default 15TB hard quota. For example, if David Haussler is the PI that you report to directly, then the directory would exist as /private/groups/hausslerlab. Request access to that group directory and you will then be able to write to it. Each of those group directories are shared by the lab it belongs to, so you must be wary of everyone's data usage and share the 15TB available per group accordingly.

'''Archive Directories (/private/warm-archive/labname)'''

The archive storage directories are created per PI, and each group directory has a default 50TB hard quota. For example, if David Haussler is the PI that you report to directly, then the directory would exist as /private/warm-archive/hausslerlab.

On the compute servers you can check your group's current '''/private/groups''' quota usage by using the 'getfattr' command. You can only check the quota of a group you are part of (you would be a member of the UNIX group of the same name). If you wanted to check the quota usage of /private/groups/hausslerlab for example, you would do:

$ getfattr -n ceph.dir.rbytes /private/groups/hausslerlab

getfattr: Removing leading '/' from absolute path names
# file: private/groups/hausslerlab
ceph.dir.rbytes="6522955553147"

That number is in bytes. So divide by 1,000,000,000,000 and you get '6.522 TB'. That is how much data is currently being used.

To check the max quota limit, use this command:

$ getfattr -n ceph.quota.max_bytes /private/groups/hausslerlab

getfattr: Removing leading '/' from absolute path names
# file: private/groups/hausslerlab
ceph.quota.max_bytes="15000000000000"

And 15000000000000 divided by 1,000,000,000,000 is 15 TB.

== Storage Quota Alerting ==

If you and/or folks in your lab would like an automated alert when the /private/groups/labname quota is getting to a certain percentage of fullness, we can set that up for you and others in your lab. Just email '''cluster-admin@soe.ucsc.edu''' with the following information:

1: Which directory you would like to watch quotas on (i.e. /private/groups/somelab)
2: What % full you would like an email alert at
3: What email addresses you want on the alert list

After setup, our alerting system will alert folks on that email list ''every 4 hours'' until the quota in question is reduced to an amount under the alerting % threshold you asked for. So it is a bit noisy, but will force folks to delete data in order to stop the alerts. When the system notices that the quota usage has decreased to under the alert threshold, you will receive one final email with an "OK" notification that things are OK now.

== /data/scratch Space on the Servers ==

Each server will generally have a local /data/scratch filesystem that you can use to store temporary files. '''BE ADVISED''' that /data/scratch is not backed up, and the data there could disappear in the event of a disk failure or anything else. Do not store important data there. If it is important, it should be moved somewhere else very soon after creation.

== Backups ==

/private/groups is backed up monthly on the first of the month (which usually takes a week to complete). Please note that the following directories in the tree '''WILL NOT''' be backed up:

tmp/
temp/
TMP/
TEMP/
cache/
.cache/
scratch/
*.tmp/

So if you have data that you know isn't important and should be excluded from the backups, put them in a directory suffixed with ".tmp". Such as this example:

/private/groups/clusteradmin/mybams.tmp/

Firewalled Environment Storage Overview

2026-05-29T21:30:59Z

Weiler: /* Storage */

== Storage ==

Our servers mount two types of ''shared'' storage; home directories and group storage directories. These home directories will mount over the network to all shared compute servers and the phoenix cluster, so any server you login to will have these filesystems available:

'''Filesystem Specifications'''

{| class="wikitable" style="text-align:center;"
|- style="font-weight:bold; text-align:left;"
! Filesystem 
! /private/home
! /private/groups
! /private/warm-archive
|-
| style="font-weight:bold; text-align:left;" | Default Quota
| 100 GB
| 15 TB
| 50 TB
|-
| style="font-weight:bold; text-align:left;" | Total Capacity
| 19 TB
| 1.7 PB
| 4.6 PB
|- style="text-align:left;"
| style="font-weight:bold;" | Access Speed
| Very Fast (NVMe Flash Media)
| Very Fast (NVMe Flash Media)
| Slower (Spinning Disks)
|- style="text-align:left;"
| style="font-weight:bold;" | Intended Use
| This space should be used for login scripts, small bits of code or software repos, etc. No large data should be stored here.
| This space should be used for large computational/shared data, large software installations and the like.
| Archival Use Only. Not meant for active cluster computation.
|}

'''Home Directories (/private/home/username)'''

Your home directory will be located as "/private/home/username" and has a 100GB quota. Your home directory is meant for small scripts and login data, or a git repo. Please do not try to store large data there or computer on large jobs using data in your home directory.

'''Groups Directories (/private/groups/groupname)'''

The group storage directories are created per PI, and each group directory has a default 15TB hard quota. For example, if David Haussler is the PI that you report to directly, then the directory would exist as /private/groups/hausslerlab. Request access to that group directory and you will then be able to write to it. Each of those group directories are shared by the lab it belongs to, so you must be wary of everyone's data usage and share the 15TB available per group accordingly.

'''Archive Directories (/private/warm-archive)'''

The archive storage directories are created per PI, and each group directory has a default 50TB hard quota. For example, if David Haussler is the PI that you report to directly, then the directory would exist as /private/warm-archive/hausslerlab.

On the compute servers you can check your group's current '''/private/groups''' quota usage by using the 'getfattr' command. You can only check the quota of a group you are part of (you would be a member of the UNIX group of the same name). If you wanted to check the quota usage of /private/groups/hausslerlab for example, you would do:

$ getfattr -n ceph.dir.rbytes /private/groups/hausslerlab

getfattr: Removing leading '/' from absolute path names
# file: private/groups/hausslerlab
ceph.dir.rbytes="6522955553147"

That number is in bytes. So divide by 1,000,000,000,000 and you get '6.522 TB'. That is how much data is currently being used.

To check the max quota limit, use this command:

$ getfattr -n ceph.quota.max_bytes /private/groups/hausslerlab

getfattr: Removing leading '/' from absolute path names
# file: private/groups/hausslerlab
ceph.quota.max_bytes="15000000000000"

And 15000000000000 divided by 1,000,000,000,000 is 15 TB.

== Storage Quota Alerting ==

If you and/or folks in your lab would like an automated alert when the /private/groups/labname quota is getting to a certain percentage of fullness, we can set that up for you and others in your lab. Just email '''cluster-admin@soe.ucsc.edu''' with the following information:

1: Which directory you would like to watch quotas on (i.e. /private/groups/somelab)
2: What % full you would like an email alert at
3: What email addresses you want on the alert list

After setup, our alerting system will alert folks on that email list ''every 4 hours'' until the quota in question is reduced to an amount under the alerting % threshold you asked for. So it is a bit noisy, but will force folks to delete data in order to stop the alerts. When the system notices that the quota usage has decreased to under the alert threshold, you will receive one final email with an "OK" notification that things are OK now.

== /data/scratch Space on the Servers ==

Each server will generally have a local /data/scratch filesystem that you can use to store temporary files. '''BE ADVISED''' that /data/scratch is not backed up, and the data there could disappear in the event of a disk failure or anything else. Do not store important data there. If it is important, it should be moved somewhere else very soon after creation.

== Backups ==

/private/groups is backed up monthly on the first of the month (which usually takes a week to complete). Please note that the following directories in the tree '''WILL NOT''' be backed up:

tmp/
temp/
TMP/
TEMP/
cache/
.cache/
scratch/
*.tmp/

So if you have data that you know isn't important and should be excluded from the backups, put them in a directory suffixed with ".tmp". Such as this example:

/private/groups/clusteradmin/mybams.tmp/

Firewalled Environment Storage Overview

2026-05-29T21:29:09Z

Weiler: /* Storage */

== Storage ==

Our servers mount two types of ''shared'' storage; home directories and group storage directories. These home directories will mount over the network to all shared compute servers and the phoenix cluster, so any server you login to will have these filesystems available:

'''Filesystem Specifications'''

{| class="wikitable" style="text-align:center;"
|- style="font-weight:bold; text-align:left;"
! Filesystem 
! /private/home
! /private/groups
! /private/warm-archive
|-
| style="font-weight:bold; text-align:left;" | Default Quota
| 100 GB
| 15 TB
| 50 TB
|-
| style="font-weight:bold; text-align:left;" | Total Capacity
| 19 TB
| 1.7 PB
| 4.6 PB
|- style="text-align:left;"
| style="font-weight:bold;" | Access Speed
| Very Fast (NVMe Flash Media)
| Very Fast (NVMe Flash Media)
| Slower (Spinning Disks)
|- style="text-align:left;"
| style="font-weight:bold;" | Intended Use
| This space should be used for login scripts, small bits of code or software repos, etc. No large data should be stored here.
| This space should be used for large computational/shared data, large software installations and the like.
| Archival Use Only. Not meant for active cluster computation.
|}

'''Home Directories (/private/home/username)'''

Your home directory will be located as "/private/home/username" and has a 100GB soft quota and a 100GB hard quota. Your home directory is meant for small scripts and login data, or a git repo. Please do not try to store large data there or computer on large jobs using data in your home directory.

'''Groups Directories (/private/groups/groupname)'''

The group storage directories are created per PI, and each group directory has a default 15TB hard quota. For example, if David Haussler is the PI that you report to directly, then the directory would exist as /private/groups/hausslerlab. Request access to that group directory and you will then be able to write to it. Each of those group directories are shared by the lab it belongs to, so you must be wary of everyone's data usage and share the 15TB available per group accordingly.

'''Archive Directories (/private/warm-archive)'''

The archive storage directories are created per PI, and each group directory has a default 50TB hard quota. For example, if David Haussler is the PI that you report to directly, then the directory would exist as /private/warm-archive/hausslerlab.

On the compute servers you can check your group's current '''/private/groups''' quota usage by using the 'getfattr' command. You can only check the quota of a group you are part of (you would be a member of the UNIX group of the same name). If you wanted to check the quota usage of /private/groups/hausslerlab for example, you would do:

$ getfattr -n ceph.dir.rbytes /private/groups/hausslerlab

getfattr: Removing leading '/' from absolute path names
# file: private/groups/hausslerlab
ceph.dir.rbytes="6522955553147"

That number is in bytes. So divide by 1,000,000,000,000 and you get '6.522 TB'. That is how much data is currently being used.

To check the max quota limit, use this command:

$ getfattr -n ceph.quota.max_bytes /private/groups/hausslerlab

getfattr: Removing leading '/' from absolute path names
# file: private/groups/hausslerlab
ceph.quota.max_bytes="15000000000000"

And 15000000000000 divided by 1,000,000,000,000 is 15 TB.

== Storage Quota Alerting ==

If you and/or folks in your lab would like an automated alert when the /private/groups/labname quota is getting to a certain percentage of fullness, we can set that up for you and others in your lab. Just email '''cluster-admin@soe.ucsc.edu''' with the following information:

1: Which directory you would like to watch quotas on (i.e. /private/groups/somelab)
2: What % full you would like an email alert at
3: What email addresses you want on the alert list

After setup, our alerting system will alert folks on that email list ''every 4 hours'' until the quota in question is reduced to an amount under the alerting % threshold you asked for. So it is a bit noisy, but will force folks to delete data in order to stop the alerts. When the system notices that the quota usage has decreased to under the alert threshold, you will receive one final email with an "OK" notification that things are OK now.

== /data/scratch Space on the Servers ==

Each server will generally have a local /data/scratch filesystem that you can use to store temporary files. '''BE ADVISED''' that /data/scratch is not backed up, and the data there could disappear in the event of a disk failure or anything else. Do not store important data there. If it is important, it should be moved somewhere else very soon after creation.

== Backups ==

/private/groups is backed up monthly on the first of the month (which usually takes a week to complete). Please note that the following directories in the tree '''WILL NOT''' be backed up:

tmp/
temp/
TMP/
TEMP/
cache/
.cache/
scratch/
*.tmp/

So if you have data that you know isn't important and should be excluded from the backups, put them in a directory suffixed with ".tmp". Such as this example:

/private/groups/clusteradmin/mybams.tmp/

Firewalled Environment Storage Overview

2026-05-29T21:28:28Z

Weiler: /* Storage */

== Storage ==

Our servers mount two types of ''shared'' storage; home directories and group storage directories. These home directories will mount over the network to all shared compute servers and the phoenix cluster, so any server you login to will have these filesystems available:

'''Filesystem Specifications'''

{| class="wikitable" style="text-align:center;"
|- style="font-weight:bold; text-align:left;"
! Filesystem 
! /private/home
! /private/groups
! /private/warm-archive
|-
| style="font-weight:bold; text-align:left;" | Default Soft Quota
| 100 GB
| 15 TB
| 50 TB
|-
| style="font-weight:bold; text-align:left;" | Default Hard Quota
| 100 GB
| 15 TB
| 50 TB
|-
| style="font-weight:bold; text-align:left;" | Total Capacity
| 19 TB
| 1.7 PB
| 4.6 PB
|- style="text-align:left;"
| style="font-weight:bold;" | Access Speed
| Very Fast (NVMe Flash Media)
| Very Fast (NVMe Flash Media)
| Slower (Spinning Disks)
|- style="text-align:left;"
| style="font-weight:bold;" | Intended Use
| This space should be used for login scripts, small bits of code or software repos, etc. No large data should be stored here.
| This space should be used for large computational/shared data, large software installations and the like.
| Archival Use Only. Not meant for active cluster computation.
|}

'''Home Directories (/private/home/username)'''

Your home directory will be located as "/private/home/username" and has a 100GB soft quota and a 100GB hard quota. Your home directory is meant for small scripts and login data, or a git repo. Please do not try to store large data there or computer on large jobs using data in your home directory.

'''Groups Directories (/private/groups/groupname)'''

The group storage directories are created per PI, and each group directory has a default 15TB hard quota. For example, if David Haussler is the PI that you report to directly, then the directory would exist as /private/groups/hausslerlab. Request access to that group directory and you will then be able to write to it. Each of those group directories are shared by the lab it belongs to, so you must be wary of everyone's data usage and share the 15TB available per group accordingly.

'''Archive Directories (/private/warm-archive)'''

The archive storage directories are created per PI, and each group directory has a default 50TB hard quota. For example, if David Haussler is the PI that you report to directly, then the directory would exist as /private/warm-archive/hausslerlab.

On the compute servers you can check your group's current '''/private/groups''' quota usage by using the 'getfattr' command. You can only check the quota of a group you are part of (you would be a member of the UNIX group of the same name). If you wanted to check the quota usage of /private/groups/hausslerlab for example, you would do:

$ getfattr -n ceph.dir.rbytes /private/groups/hausslerlab

getfattr: Removing leading '/' from absolute path names
# file: private/groups/hausslerlab
ceph.dir.rbytes="6522955553147"

That number is in bytes. So divide by 1,000,000,000,000 and you get '6.522 TB'. That is how much data is currently being used.

To check the max quota limit, use this command:

$ getfattr -n ceph.quota.max_bytes /private/groups/hausslerlab

getfattr: Removing leading '/' from absolute path names
# file: private/groups/hausslerlab
ceph.quota.max_bytes="15000000000000"

And 15000000000000 divided by 1,000,000,000,000 is 15 TB.

== Storage Quota Alerting ==

If you and/or folks in your lab would like an automated alert when the /private/groups/labname quota is getting to a certain percentage of fullness, we can set that up for you and others in your lab. Just email '''cluster-admin@soe.ucsc.edu''' with the following information:

1: Which directory you would like to watch quotas on (i.e. /private/groups/somelab)
2: What % full you would like an email alert at
3: What email addresses you want on the alert list

After setup, our alerting system will alert folks on that email list ''every 4 hours'' until the quota in question is reduced to an amount under the alerting % threshold you asked for. So it is a bit noisy, but will force folks to delete data in order to stop the alerts. When the system notices that the quota usage has decreased to under the alert threshold, you will receive one final email with an "OK" notification that things are OK now.

== /data/scratch Space on the Servers ==

Each server will generally have a local /data/scratch filesystem that you can use to store temporary files. '''BE ADVISED''' that /data/scratch is not backed up, and the data there could disappear in the event of a disk failure or anything else. Do not store important data there. If it is important, it should be moved somewhere else very soon after creation.

== Backups ==

/private/groups is backed up monthly on the first of the month (which usually takes a week to complete). Please note that the following directories in the tree '''WILL NOT''' be backed up:

tmp/
temp/
TMP/
TEMP/
cache/
.cache/
scratch/
*.tmp/

So if you have data that you know isn't important and should be excluded from the backups, put them in a directory suffixed with ".tmp". Such as this example:

/private/groups/clusteradmin/mybams.tmp/

Genomics Institute Computing Information

2026-05-29T21:24:35Z

Weiler: /* GI Firewalled Computing Environment (PRISM) */

Welcome to the Genomic Institute Computing Information Repository! Browse the below topics for help in the area you are curious about.

== GI Public Computing Environment ==
*[[How to access the public servers]]
*[[Resetting your Public UNIX Password]]

== GI Firewalled Computing Environment (PRISM) ==
*[[Access to the Firewalled Compute Servers]]
*[[Firewalled Computing Resources Overview]]
*[[Firewalled Environment Storage Overview]]
*[[Firewalled User Account and Storage Cost]]
*[[Grafana Performance Metrics]]
*[[Visual Studio Code (vscode) Configuration Tweaks]]
*[http://logserv.gi.ucsc.edu/cgi-bin/private-groups.cgi '''/private/groups''' Data Usage Graphs]
*[http://logserv.gi.ucsc.edu/cgi-bin/private-warm-archive.cgi '''/private/warm-archive''' Data Usage Graphs]
*[[Resetting your VPN/PRISM Password]]

==VPN Access==
*[[Requirement for users to get GI VPN access]]
*[[Setting Up The VPN on MacOS]]
*[[Setting Up The VPN on Windows]]
*[[Setting Up The VPN on Linux]]
*[[Multi Factor Authentication (MFA) Frequently Asked Questions]]
*[[Duo Pushes Aren't Being Sent to My Phone!]]

== NIH dbGaP Access Requirements ==
*[[Requirements for dbGaP Access]]

== giCloud Openstack ==
*[[Overview of giCloud in the Genomics Institute]]
*[[Quick Start Instructions to Get Rolling with OpenStack]]

== Amazon Web Services Information ==
*[[Overview of Getting and Using an AWS IAM Account]]
*[[AWS Account List and Numbers]]
*[[AWS Shared Bucket Usage Graphs]]
*[[AWS Best Practices]]
*[[AWS S3 Lifecycle Management]]

== JupyterHub Notebooks Spawned via Slurm ==
*[[Access the JupyterHub Slurm Server]]
*[[Create a Jupyter Virtual Environment]]
*[[Change Default File Browser Directory]]

== Slurm at the Genomics Institute ==
*[[Overview of using Slurm]]
*[[Slurm Best Practices]]
*[[Cluster Etiquette]]
*[[Annotated Slurm Script]]
*[[Job Arrays]]
*[[GPU Resources]]
*[[Quick Reference Guide]]
*[[Convenient Slurm Commands]]
*[[Slurm Queues (Partitions) and Resource Management]]
*[[Slurm Tips for vg]]
*[[Slurm Tips for Toil]]
*[[Using Docker under Slurm]]
*[[Phoenix WDL Tutorial]]
*[[Slurm Insights Dashboard]]

==General Docker Information==
*[[Running a Container as a non-root User]]

== Problems or technical support ==

If you have any questions or problems with the GI computing environment, please send an email to '''cluster-admin@soe.ucsc.edu'''

Create a Jupyter Virtual Environment

2026-05-23T14:02:56Z

Weiler: /* Create Your Virtual Environment */

__TOC__

If you need a specific Jupyter environment, because maybe you need specific Python modules or something like that, you've come to the right place.

== Find a Location to Install ==

You will want to install your Jupyter Virtual Environment somewhere accessible from the PRISM Compute Nodes and all the cluster nodes, so it will need to either be in your home directory or in '''/private/groups/[lab_name]''' somewhere. For this example I'll use my home directory, '''/private/home/weiler'''.

== Create Your Virtual Environment ==

I'll be creating this Virtual Environment in my home directory (but create it wherever you deem necessary). So:

cd /private/home/weiler

Then create the directory for the environment to go in:

mkdir jupyter_venv

Next create the environment!

cd jupyter_venv
python3 -m venv jupyterhub

It will create an environment in '''/private/home/weiler/jupyter_env/jupyterhub'''.

Next we activate the environment:

source ~/jupyter_env/jupyterhub/bin/activate

Your prompt should now look something like this:

(jupyterhub) [weiler@mustard jupyter_venv]#

== Install Requisite and Optional Packages ==

From there you will need to minimally install these three packages:

(jupyterhub) [weiler@mustard jupyter_venv]# pip3 install jupyterhub jupyterlab batchspawner

You will '''need''' those three packages to start. So definitely install those. You can additionally install any other python packages, such as:

(jupyterhub) [weiler@mustard jupyter_venv]# pip3 install numpy pandas

== Exit Your Virtual Environment ==

To exit out of your virtual environment, simply do:

deactivate

and you should return to your regular shell.

== Call Your Virtual Environment In JupyterHub ==

Once you login to JupyterHub on the cluster here:

https://jupyterhub.prism

You will want to click on the '''Advanced''' tab at the top. When defining your notebook resources, under "Jupyter environment", select "Custom". Then define your description and give the path to your '''/bin''' directory in your environment, as such:

[[File:Custom_env.png|800px|frame|left|Custom Environment]]

It will then launch a notebook based on that environment, and all modules and packages installed in that environment will be available.

Create a Jupyter Virtual Environment

2026-05-23T14:02:34Z

Weiler: /* Create Your Virtual Environment */

__TOC__

If you need a specific Jupyter environment, because maybe you need specific Python modules or something like that, you've come to the right place.

== Find a Location to Install ==

You will want to install your Jupyter Virtual Environment somewhere accessible from the PRISM Compute Nodes and all the cluster nodes, so it will need to either be in your home directory or in '''/private/groups/[lab_name]''' somewhere. For this example I'll use my home directory, '''/private/home/weiler'''.

== Create Your Virtual Environment ==

I'll be creating this Virtual Environment in my home directory (but create it wherever you deem necessary). So:

cd /private/home/weiler

Then create the directory for the environment to go in:

mkdir jupyterhub_venv

Next create the environment!

cd jupyterhub_venv
python3 -m venv jupyterhub

It will create an environment in '''/private/home/weiler/jupyter_env/jupyterhub'''.

Next we activate the environment:

source ~/jupyter_env/jupyterhub/bin/activate

Your prompt should now look something like this:

(jupyterhub) [weiler@mustard jupyter_venv]#

== Install Requisite and Optional Packages ==

From there you will need to minimally install these three packages:

(jupyterhub) [weiler@mustard jupyter_venv]# pip3 install jupyterhub jupyterlab batchspawner

You will '''need''' those three packages to start. So definitely install those. You can additionally install any other python packages, such as:

(jupyterhub) [weiler@mustard jupyter_venv]# pip3 install numpy pandas

== Exit Your Virtual Environment ==

To exit out of your virtual environment, simply do:

deactivate

and you should return to your regular shell.

== Call Your Virtual Environment In JupyterHub ==

Once you login to JupyterHub on the cluster here:

https://jupyterhub.prism

You will want to click on the '''Advanced''' tab at the top. When defining your notebook resources, under "Jupyter environment", select "Custom". Then define your description and give the path to your '''/bin''' directory in your environment, as such:

[[File:Custom_env.png|800px|frame|left|Custom Environment]]

It will then launch a notebook based on that environment, and all modules and packages installed in that environment will be available.

Access the JupyterHub Slurm Server

2026-05-23T01:11:04Z

Weiler: /* The Live Notebook */

__TOC__

We have a JupyterHub server running on top of Slurm that is very useful for spawning Jupyter Notebooks.

== Login to the Hub ==

You can login to the JupyterHub interface here:

https://jupyterhub.prism

Your web browser may complain about a name mismatch on the SSL certificate but you can click past it, the connection is still secure.

There you will see a login page:

[[File:Signin.png|1200px|frame|left|Login Page]]

Login using your regular UNIX username and password.

== Configure Your Notebook ==

Once you login, you can configure your notebook for its required resources in Slurm, and then launch your notebook, which will appear on a node Slurm chooses. If a node is free right then, your notebook will launch immediately. If the Slurm queue is full, you may have to wait for resources to become available.

You will see this page after logging in, which is the "Simple" setup page:

[[File:Simple_options.png|600px|frame|left|Simple Server]]

Here you can define more specific job parameters.

You will want to mostly accept the defaults if running a simple notebook. But if your notebook requires more resources, then kick up the max time or CPU/GPU count as appropriate. Remember Slurm will kill the notebook if it persists past the '''Job Duration''' parameter. Simply click the '''Start''' button on the bottom to launch the notebook.

If you need a more specific environment, click on the '''Advanced''' tab, which should yield this screen:

[[File:Advanced_options.png|600px|frame|left|Advanced Server]]

This allows for a more fine grained resource definition when requesting resources for your notebook. It also allows for a custom Jupyter environment to be defined, if you need one. The Default environment is fairly basic, so it may make sense to create your own custom environment. If you would like to do that, refer to [[Create a Jupyter Virtual Environment]].

== The Live Notebook ==

Once you start your notebook (give it some time, it can take 10-20 seconds to start based on your environment and resource requests), you should see your notebook appear as something like this in your web browser:

[[File:Live_notebook.png|1200px|frame|left|Login Page]]

If the slurm queue is full, you may be waiting a while.

This is a notebook actually running on one of the cluster nodes, but the connection is proxied through the head node, and you are seeing it in your web browser. You can actually see your job running in slurm by looking at the output of the '''squeue''' command, if you are curious.

== Terminate Your Notebook When Finished ==

When you are done, don't forget to terminate your notebook to free up the Slurm resources it was using! To close your notebook, click on '''File -> Hub Control Panel'''. Then from there click the red '''Stop My Server''' button. Your notebook will terminate and the associated Slurm job will also exit.

If you simply click "Logout", or "File -> Logout", it will log you out, but your notebook is still running! So next time you log in, it will take you right back to that notebook. If you are truly done, then kill your notebook via '''Stop My Server''' as described above.

If slurm sees the job is still running when the job's wall time is exceeded, slurm will kill your notebook automatically.

Access the JupyterHub Slurm Server

2026-05-23T01:10:18Z

Weiler: /* Terminate Your Notebook When Finished */

__TOC__

We have a JupyterHub server running on top of Slurm that is very useful for spawning Jupyter Notebooks.

== Login to the Hub ==

You can login to the JupyterHub interface here:

https://jupyterhub.prism

Your web browser may complain about a name mismatch on the SSL certificate but you can click past it, the connection is still secure.

There you will see a login page:

[[File:Signin.png|1200px|frame|left|Login Page]]

Login using your regular UNIX username and password.

== Configure Your Notebook ==

Once you login, you can configure your notebook for its required resources in Slurm, and then launch your notebook, which will appear on a node Slurm chooses. If a node is free right then, your notebook will launch immediately. If the Slurm queue is full, you may have to wait for resources to become available.

You will see this page after logging in, which is the "Simple" setup page:

[[File:Simple_options.png|600px|frame|left|Simple Server]]

Here you can define more specific job parameters.

You will want to mostly accept the defaults if running a simple notebook. But if your notebook requires more resources, then kick up the max time or CPU/GPU count as appropriate. Remember Slurm will kill the notebook if it persists past the '''Job Duration''' parameter. Simply click the '''Start''' button on the bottom to launch the notebook.

If you need a more specific environment, click on the '''Advanced''' tab, which should yield this screen:

[[File:Advanced_options.png|600px|frame|left|Advanced Server]]

This allows for a more fine grained resource definition when requesting resources for your notebook. It also allows for a custom Jupyter environment to be defined, if you need one. The Default environment is fairly basic, so it may make sense to create your own custom environment. If you would like to do that, refer to [[Create a Jupyter Virtual Environment]].

== The Live Notebook ==

Once you start your notebook (give it some time, it can take 10-20 seconds to start based on your environment and resource requests), you should see your notebook appear as something like this in your web browser:

[[File:Live_notebook.png|1200px|frame|left|Login Page]]

This is a notebook actually running on one of the cluster nodes, but the connection is proxied through the head node, and you are seeing it in your web browser. You can actually see your job running in slurm by looking at the output of the '''squeue''' command, if you are curious.

== Terminate Your Notebook When Finished ==

When you are done, don't forget to terminate your notebook to free up the Slurm resources it was using! To close your notebook, click on '''File -> Hub Control Panel'''. Then from there click the red '''Stop My Server''' button. Your notebook will terminate and the associated Slurm job will also exit.

If you simply click "Logout", or "File -> Logout", it will log you out, but your notebook is still running! So next time you log in, it will take you right back to that notebook. If you are truly done, then kill your notebook via '''Stop My Server''' as described above.

If slurm sees the job is still running when the job's wall time is exceeded, slurm will kill your notebook automatically.

Access the JupyterHub Slurm Server

2026-05-23T01:09:06Z

Weiler: /* Terminate Your Notebook When Finished */

__TOC__

We have a JupyterHub server running on top of Slurm that is very useful for spawning Jupyter Notebooks.

== Login to the Hub ==

You can login to the JupyterHub interface here:

https://jupyterhub.prism

Your web browser may complain about a name mismatch on the SSL certificate but you can click past it, the connection is still secure.

There you will see a login page:

[[File:Signin.png|1200px|frame|left|Login Page]]

Login using your regular UNIX username and password.

== Configure Your Notebook ==

Once you login, you can configure your notebook for its required resources in Slurm, and then launch your notebook, which will appear on a node Slurm chooses. If a node is free right then, your notebook will launch immediately. If the Slurm queue is full, you may have to wait for resources to become available.

You will see this page after logging in, which is the "Simple" setup page:

[[File:Simple_options.png|600px|frame|left|Simple Server]]

Here you can define more specific job parameters.

You will want to mostly accept the defaults if running a simple notebook. But if your notebook requires more resources, then kick up the max time or CPU/GPU count as appropriate. Remember Slurm will kill the notebook if it persists past the '''Job Duration''' parameter. Simply click the '''Start''' button on the bottom to launch the notebook.

If you need a more specific environment, click on the '''Advanced''' tab, which should yield this screen:

[[File:Advanced_options.png|600px|frame|left|Advanced Server]]

This allows for a more fine grained resource definition when requesting resources for your notebook. It also allows for a custom Jupyter environment to be defined, if you need one. The Default environment is fairly basic, so it may make sense to create your own custom environment. If you would like to do that, refer to [[Create a Jupyter Virtual Environment]].

== The Live Notebook ==

Once you start your notebook (give it some time, it can take 10-20 seconds to start based on your environment and resource requests), you should see your notebook appear as something like this in your web browser:

[[File:Live_notebook.png|1200px|frame|left|Login Page]]

This is a notebook actually running on one of the cluster nodes, but the connection is proxied through the head node, and you are seeing it in your web browser. You can actually see your job running in slurm by looking at the output of the '''squeue''' command, if you are curious.

== Terminate Your Notebook When Finished ==

When you are done, don't forget to terminate your notebook to free up the Slurm resources it was using! To close your notebook, click on '''File -> Hub Control Panel'''. Then from there click the red '''Stop My Server''' button. Your notebook will terminate and the associated Slurm job will also exit.

If you simply click "Logout", or "File -> Logout", it will log you out, but your notebook is still running! So next time you log in, it will take you right back to that notebook. If you are truly done, then kill your notebook via '''Stop My Server''' as described above.

Access the JupyterHub Slurm Server

2026-05-23T00:57:42Z

Weiler:

__TOC__

We have a JupyterHub server running on top of Slurm that is very useful for spawning Jupyter Notebooks.

== Login to the Hub ==

You can login to the JupyterHub interface here:

https://jupyterhub.prism

Your web browser may complain about a name mismatch on the SSL certificate but you can click past it, the connection is still secure.

There you will see a login page:

[[File:Signin.png|1200px|frame|left|Login Page]]

Login using your regular UNIX username and password.

== Configure Your Notebook ==

Once you login, you can configure your notebook for its required resources in Slurm, and then launch your notebook, which will appear on a node Slurm chooses. If a node is free right then, your notebook will launch immediately. If the Slurm queue is full, you may have to wait for resources to become available.

You will see this page after logging in, which is the "Simple" setup page:

[[File:Simple_options.png|600px|frame|left|Simple Server]]

Here you can define more specific job parameters.

You will want to mostly accept the defaults if running a simple notebook. But if your notebook requires more resources, then kick up the max time or CPU/GPU count as appropriate. Remember Slurm will kill the notebook if it persists past the '''Job Duration''' parameter. Simply click the '''Start''' button on the bottom to launch the notebook.

If you need a more specific environment, click on the '''Advanced''' tab, which should yield this screen:

[[File:Advanced_options.png|600px|frame|left|Advanced Server]]

This allows for a more fine grained resource definition when requesting resources for your notebook. It also allows for a custom Jupyter environment to be defined, if you need one. The Default environment is fairly basic, so it may make sense to create your own custom environment. If you would like to do that, refer to [[Create a Jupyter Virtual Environment]].

== The Live Notebook ==

Once you start your notebook (give it some time, it can take 10-20 seconds to start based on your environment and resource requests), you should see your notebook appear as something like this in your web browser:

[[File:Live_notebook.png|1200px|frame|left|Login Page]]

This is a notebook actually running on one of the cluster nodes, but the connection is proxied through the head node, and you are seeing it in your web browser. You can actually see your job running in slurm by looking at the output of the '''squeue''' command, if you are curious.

== Terminate Your Notebook When Finished ==

When you are done, don't forget to terminate your notebook to free up the Slurm resources it was using! To close your notebook, click on '''File -> Hub Control Panel'''. Then from there click the red '''Stop My Server''' button. Your notebook will terminate and the associated Slurm job will also exit.

File:Live notebook.png

2026-05-23T00:57:24Z

Weiler:

Access the JupyterHub Slurm Server

2026-05-23T00:55:35Z

Weiler:

__TOC__

We have a JupyterHub server running on top of Slurm that is very useful for spawning Jupyter Notebooks.

== Login to the Hub ==

You can login to the JupyterHub interface here:

https://jupyterhub.prism

Your web browser may complain about a name mismatch on the SSL certificate but you can click past it, the connection is still secure.

There you will see a login page:

[[File:Signin.png|1200px|frame|left|Login Page]]

Login using your regular UNIX username and password.

== Configure Your Notebook ==

Once you login, you can configure your notebook for its required resources in Slurm, and then launch your notebook, which will appear on a node Slurm chooses. If a node is free right then, your notebook will launch immediately. If the Slurm queue is full, you may have to wait for resources to become available.

You will see this page after logging in, which is the "Simple" setup page:

[[File:Simple_options.png|600px|frame|left|Simple Server]]

Here you can define more specific job parameters.

You will want to mostly accept the defaults if running a simple notebook. But if your notebook requires more resources, then kick up the max time or CPU/GPU count as appropriate. Remember Slurm will kill the notebook if it persists past the '''Job Duration''' parameter. Simply click the '''Start''' button on the bottom to launch the notebook.

If you need a more specific environment, click on the '''Advanced''' tab, which should yield this screen:

[[File:Advanced_options.png|600px|frame|left|Advanced Server]]

This allows for a more fine grained resource definition when requesting resources for your notebook. It also allows for a custom Jupyter environment to be defined, if you need one. The Default environment is fairly basic, so it may make sense to create your own custom environment. If you would like to do that, refer to [[Create a Jupyter Virtual Environment]].

== The Live Notebook ==

Once you start your notebook (give it some time, it can take 10-20 seconds to start based on your environment and resource requests), you should see your notebook appear as something like this in your web browser:

[[File:Notebook.png|1200px|frame|left|Login Page]]

This is a notebook actually running on one of the cluster nodes, but the connection is proxied through the head node, and you are seeing it in your web browser. You can actually see your job running in slurm by looking at the output of the '''squeue''' command, if you are curious.

== Terminate Your Notebook When Finished ==

When you are done, don't forget to terminate your notebook to free up the Slurm resources it was using! To close your notebook, click on '''File -> Hub Control Panel'''. Then from there click the red '''Stop My Server''' button. Your notebook will terminate and the associated Slurm job will also exit.

File:Advanced options.png

2026-05-23T00:55:15Z

Weiler:

Access the JupyterHub Slurm Server

2026-05-23T00:54:04Z

Weiler:

__TOC__

We have a JupyterHub server running on top of Slurm that is very useful for spawning Jupyter Notebooks.

== Login to the Hub ==

You can login to the JupyterHub interface here:

https://jupyterhub.prism

Your web browser may complain about a name mismatch on the SSL certificate but you can click past it, the connection is still secure.

There you will see a login page:

[[File:Signin.png|1200px|frame|left|Login Page]]

Login using your regular UNIX username and password.

== Configure Your Notebook ==

Once you login, you can configure your notebook for its required resources in Slurm, and then launch your notebook, which will appear on a node Slurm chooses. If a node is free right then, your notebook will launch immediately. If the Slurm queue is full, you may have to wait for resources to become available.

You will see this page after logging in, which is the "Simple" setup page:

[[File:Simple_options.png|600px|frame|left|Simple Server]]

Here you can define more specific job parameters.

You will want to mostly accept the defaults if running a simple notebook. But if your notebook requires more resources, then kick up the max time or CPU/GPU count as appropriate. Remember Slurm will kill the notebook if it persists past the '''Job Duration''' parameter. Simply click the '''Start''' button on the bottom to launch the notebook.

If you need a more specific environment, click on the '''Advanced''' tab, which should yield this screen:

[[File:Advanced_server.png|600px|frame|left|Advanced Server]]

This allows for a more fine grained resource definition when requesting resources for your notebook. It also allows for a custom Jupyter environment to be defined, if you need one. The Default environment is fairly basic, so it may make sense to create your own custom environment. If you would like to do that, refer to [[Create a Jupyter Virtual Environment]].

== The Live Notebook ==

Once you start your notebook (give it some time, it can take 10-20 seconds to start based on your environment and resource requests), you should see your notebook appear as something like this in your web browser:

[[File:Notebook.png|1200px|frame|left|Login Page]]

This is a notebook actually running on one of the cluster nodes, but the connection is proxied through the head node, and you are seeing it in your web browser. You can actually see your job running in slurm by looking at the output of the '''squeue''' command, if you are curious.

== Terminate Your Notebook When Finished ==

When you are done, don't forget to terminate your notebook to free up the Slurm resources it was using! To close your notebook, click on '''File -> Hub Control Panel'''. Then from there click the red '''Stop My Server''' button. Your notebook will terminate and the associated Slurm job will also exit.

File:Simple options.png

2026-05-23T00:53:45Z

Weiler:

Access the JupyterHub Slurm Server

2026-05-23T00:52:09Z

Weiler:

__TOC__

We have a JupyterHub server running on top of Slurm that is very useful for spawning Jupyter Notebooks.

== Login to the Hub ==

You can login to the JupyterHub interface here:

https://jupyterhub.prism

Your web browser may complain about a name mismatch on the SSL certificate but you can click past it, the connection is still secure.

There you will see a login page:

[[File:Signin.png|1200px|frame|left|Login Page]]

Login using your regular UNIX username and password.

== Configure Your Notebook ==

Once you login, you can configure your notebook for its required resources in Slurm, and then launch your notebook, which will appear on a node Slurm chooses. If a node is free right then, your notebook will launch immediately. If the Slurm queue is full, you may have to wait for resources to become available.

You will see this page after logging in, which is the "Simple" setup page:

[[File:Simple_server.png|600px|frame|left|Simple Server]]

Here you can define more specific job parameters.

You will want to mostly accept the defaults if running a simple notebook. But if your notebook requires more resources, then kick up the max time or CPU/GPU count as appropriate. Remember Slurm will kill the notebook if it persists past the '''Job Duration''' parameter. Simply click the '''Start''' button on the bottom to launch the notebook.

If you need a more specific environment, click on the '''Advanced''' tab, which should yield this screen:

[[File:Advanced_server.png|600px|frame|left|Advanced Server]]

This allows for a more fine grained resource definition when requesting resources for your notebook. It also allows for a custom Jupyter environment to be defined, if you need one. The Default environment is fairly basic, so it may make sense to create your own custom environment. If you would like to do that, refer to [[Create a Jupyter Virtual Environment]].

== The Live Notebook ==

Once you start your notebook (give it some time, it can take 10-20 seconds to start based on your environment and resource requests), you should see your notebook appear as something like this in your web browser:

[[File:Notebook.png|1200px|frame|left|Login Page]]

This is a notebook actually running on one of the cluster nodes, but the connection is proxied through the head node, and you are seeing it in your web browser. You can actually see your job running in slurm by looking at the output of the '''squeue''' command, if you are curious.

== Terminate Your Notebook When Finished ==

When you are done, don't forget to terminate your notebook to free up the Slurm resources it was using! To close your notebook, click on '''File -> Hub Control Panel'''. Then from there click the red '''Stop My Server''' button. Your notebook will terminate and the associated Slurm job will also exit.

File:Signin.png

2026-05-23T00:51:42Z

Weiler:

Create a Jupyter Virtual Environment

2026-05-23T00:49:52Z

Weiler:

__TOC__

If you need a specific Jupyter environment, because maybe you need specific Python modules or something like that, you've come to the right place.

== Find a Location to Install ==

You will want to install your Jupyter Virtual Environment somewhere accessible from the PRISM Compute Nodes and all the cluster nodes, so it will need to either be in your home directory or in '''/private/groups/[lab_name]''' somewhere. For this example I'll use my home directory, '''/private/home/weiler'''.

== Create Your Virtual Environment ==

I'll be creating this Virtual Environment in my home directory (but create it wherever you deem necessary). So:

cd /private/home/weiler

Then create the directory for the environment to go in:

mkdir jupyter_venv

Next create the environment!

cd jupyterhub_venv
python3 -m venv jupyterhub

It will create an environment in '''/private/home/weiler/jupyter_env/jupyterhub'''.

Next we activate the environment:

source ~/jupyter_env/jupyterhub/bin/activate

Your prompt should now look something like this:

(jupyterhub) [weiler@mustard jupyter_venv]#

== Install Requisite and Optional Packages ==

From there you will need to minimally install these three packages:

(jupyterhub) [weiler@mustard jupyter_venv]# pip3 install jupyterhub jupyterlab batchspawner

You will '''need''' those three packages to start. So definitely install those. You can additionally install any other python packages, such as:

(jupyterhub) [weiler@mustard jupyter_venv]# pip3 install numpy pandas

== Exit Your Virtual Environment ==

To exit out of your virtual environment, simply do:

deactivate

and you should return to your regular shell.

== Call Your Virtual Environment In JupyterHub ==

Once you login to JupyterHub on the cluster here:

https://jupyterhub.prism

You will want to click on the '''Advanced''' tab at the top. When defining your notebook resources, under "Jupyter environment", select "Custom". Then define your description and give the path to your '''/bin''' directory in your environment, as such:

[[File:Custom_env.png|800px|frame|left|Custom Environment]]

It will then launch a notebook based on that environment, and all modules and packages installed in that environment will be available.

File:Custom env.png

2026-05-23T00:49:28Z

Weiler:

Change Default File Browser Directory

2026-05-23T00:42:56Z

Weiler: Created page with "__TOC__ When starting a notebook, by default, the "File Browser" directory on the left panel of the Jupyter Notebook window in your home directory. If you would prefer to have that default directory be something else, follow these steps. == Stop All Current Running Notebooks == Stop all your notebooks before proceeding. In this case, click "File" -> "Hub Control Panel", then click the red "Stop Server" button. == Create User Jupyter Config File == Create a custom..."

__TOC__

When starting a notebook, by default, the "File Browser" directory on the left panel of the Jupyter Notebook window in your home directory. If you would prefer to have that default directory be something else, follow these steps.

== Stop All Current Running Notebooks ==

Stop all your notebooks before proceeding. In this case, click "File" -> "Hub Control Panel", then click the red "Stop Server" button.

== Create User Jupyter Config File ==

Create a custom Jupyter config file by ssh'ing over to '''mustard.prism''' and then doing:

jupyter server --generate-config

It will create a directory in your home directory called '''.jupyter'''. There is a file in that directory called:

jupyter_server_config.py

We need to edit that file, and add a line to the top of the file. The line we add will direct the Jupyter Notebook to use a different root directory when spawning, often a groups directory like '''/private/groups/hausslerlab''' as an example. Open the file using '''vim''' or whatever your favorite text editor is:

vim ~/.jupyter/jupyter_server_config.py

Add this line to the top of that file:

c.ServerApp.root_dir = '/private/groups/hausslerlab'

'''/private/groups/hausslerlab''' should be changed to whatever you want your default directory to be.

== Launch a New Notebook Instance ==

Launch a new cluster notebook instance, and you should see your new default directory on the left!

== OPTIONAL: Symlinks in your Home Directory ==

There is another way to do this that doesn't involve creating a config file and editing it. Simply add a symlink in your home directory to your lab's groups directory, and you can click on that in the file browser and it will take you to your lab directory.

cd ~
ln -s /private/groups/hausslerlab hausslerlab

Again, '''/private/groups/hausslerlab''' should be wherever you want the link to take you.

Genomics Institute Computing Information

2026-05-23T00:40:21Z

Weiler: /* JupyterHub Notebooks Spawned via Slurm */

Welcome to the Genomic Institute Computing Information Repository! Browse the below topics for help in the area you are curious about.

== GI Public Computing Environment ==
*[[How to access the public servers]]
*[[Resetting your Public UNIX Password]]

== GI Firewalled Computing Environment (PRISM) ==
*[[Access to the Firewalled Compute Servers]]
*[[Firewalled Computing Resources Overview]]
*[[Firewalled Environment Storage Overview]]
*[[Firewalled User Account and Storage Cost]]
*[[Grafana Performance Metrics]]
*[[Visual Studio Code (vscode) Configuration Tweaks]]
*[http://logserv.gi.ucsc.edu/cgi-bin/private-groups.cgi '''/private/groups''' Data Usage Graphs]
*[[Resetting your VPN/PRISM Password]]

==VPN Access==
*[[Requirement for users to get GI VPN access]]
*[[Setting Up The VPN on MacOS]]
*[[Setting Up The VPN on Windows]]
*[[Setting Up The VPN on Linux]]
*[[Multi Factor Authentication (MFA) Frequently Asked Questions]]
*[[Duo Pushes Aren't Being Sent to My Phone!]]

== NIH dbGaP Access Requirements ==
*[[Requirements for dbGaP Access]]

== giCloud Openstack ==
*[[Overview of giCloud in the Genomics Institute]]
*[[Quick Start Instructions to Get Rolling with OpenStack]]

== Amazon Web Services Information ==
*[[Overview of Getting and Using an AWS IAM Account]]
*[[AWS Account List and Numbers]]
*[[AWS Shared Bucket Usage Graphs]]
*[[AWS Best Practices]]
*[[AWS S3 Lifecycle Management]]

== JupyterHub Notebooks Spawned via Slurm ==
*[[Access the JupyterHub Slurm Server]]
*[[Create a Jupyter Virtual Environment]]
*[[Change Default File Browser Directory]]

== Slurm at the Genomics Institute ==
*[[Overview of using Slurm]]
*[[Slurm Best Practices]]
*[[Cluster Etiquette]]
*[[Annotated Slurm Script]]
*[[Job Arrays]]
*[[GPU Resources]]
*[[Quick Reference Guide]]
*[[Convenient Slurm Commands]]
*[[Slurm Queues (Partitions) and Resource Management]]
*[[Slurm Tips for vg]]
*[[Slurm Tips for Toil]]
*[[Using Docker under Slurm]]
*[[Phoenix WDL Tutorial]]
*[[Slurm Insights Dashboard]]

==General Docker Information==
*[[Running a Container as a non-root User]]

== Problems or technical support ==

If you have any questions or problems with the GI computing environment, please send an email to '''cluster-admin@soe.ucsc.edu'''

Genomics Institute Computing Information

2026-05-23T00:35:07Z

Weiler: /* JupyterHub Notebooks Spawned via Slurm */

Welcome to the Genomic Institute Computing Information Repository! Browse the below topics for help in the area you are curious about.

== GI Public Computing Environment ==
*[[How to access the public servers]]
*[[Resetting your Public UNIX Password]]

== GI Firewalled Computing Environment (PRISM) ==
*[[Access to the Firewalled Compute Servers]]
*[[Firewalled Computing Resources Overview]]
*[[Firewalled Environment Storage Overview]]
*[[Firewalled User Account and Storage Cost]]
*[[Grafana Performance Metrics]]
*[[Visual Studio Code (vscode) Configuration Tweaks]]
*[http://logserv.gi.ucsc.edu/cgi-bin/private-groups.cgi '''/private/groups''' Data Usage Graphs]
*[[Resetting your VPN/PRISM Password]]

==VPN Access==
*[[Requirement for users to get GI VPN access]]
*[[Setting Up The VPN on MacOS]]
*[[Setting Up The VPN on Windows]]
*[[Setting Up The VPN on Linux]]
*[[Multi Factor Authentication (MFA) Frequently Asked Questions]]
*[[Duo Pushes Aren't Being Sent to My Phone!]]

== NIH dbGaP Access Requirements ==
*[[Requirements for dbGaP Access]]

== giCloud Openstack ==
*[[Overview of giCloud in the Genomics Institute]]
*[[Quick Start Instructions to Get Rolling with OpenStack]]

== Amazon Web Services Information ==
*[[Overview of Getting and Using an AWS IAM Account]]
*[[AWS Account List and Numbers]]
*[[AWS Shared Bucket Usage Graphs]]
*[[AWS Best Practices]]
*[[AWS S3 Lifecycle Management]]

== JupyterHub Notebooks Spawned via Slurm ==
*[[Access the JupyterHub Slurm Server]]
*[[Create a Jupyter Virtual Environment]]

== Slurm at the Genomics Institute ==
*[[Overview of using Slurm]]
*[[Slurm Best Practices]]
*[[Cluster Etiquette]]
*[[Annotated Slurm Script]]
*[[Job Arrays]]
*[[GPU Resources]]
*[[Quick Reference Guide]]
*[[Convenient Slurm Commands]]
*[[Slurm Queues (Partitions) and Resource Management]]
*[[Slurm Tips for vg]]
*[[Slurm Tips for Toil]]
*[[Using Docker under Slurm]]
*[[Phoenix WDL Tutorial]]
*[[Slurm Insights Dashboard]]

==General Docker Information==
*[[Running a Container as a non-root User]]

== Problems or technical support ==

If you have any questions or problems with the GI computing environment, please send an email to '''cluster-admin@soe.ucsc.edu'''

Create a Jupyter Virtual Environment

2026-05-23T00:33:43Z

Weiler: Created page with "__TOC__ If you need a specific Jupyter environment, because maybe you need specific Python modules or something like that, you've come to the right place. == Find a Location to Install == You will want to install your Jupyter Virtual Environment somewhere accessible from the PRISM Compute Nodes and all the cluster nodes, so it will need to either be in your home directory or in '''/private/groups/[lab_name]''' somewhere. For this example I'll use my home directory, '..."

__TOC__

If you need a specific Jupyter environment, because maybe you need specific Python modules or something like that, you've come to the right place.

== Find a Location to Install ==

You will want to install your Jupyter Virtual Environment somewhere accessible from the PRISM Compute Nodes and all the cluster nodes, so it will need to either be in your home directory or in '''/private/groups/[lab_name]''' somewhere. For this example I'll use my home directory, '''/private/home/weiler'''.

== Create Your Virtual Environment ==

I'll be creating this Virtual Environment in my home directory (but create it wherever you deem necessary). So:

cd /private/home/weiler

Then create the directory for the environment to go in:

mkdir jupyter_venv

Next create the environment!

cd jupyterhub_venv
python3 -m venv jupyterhub

It will create an environment in '''/private/home/weiler/jupyter_env/jupyterhub'''.

Next we activate the environment:

source ~/jupyter_env/jupyterhub/bin/activate

Your prompt should now look something like this:

(jupyterhub) [weiler@mustard jupyter_venv]#

== Install Requisite and Optional Packages ==

From there you will need to minimally install these three packages:

(jupyterhub) [weiler@mustard jupyter_venv]# pip3 install jupyterhub jupyterlab batchspawner

You will '''need''' those three packages to start. So definitely install those. You can additionally install any other python packages, such as:

(jupyterhub) [weiler@mustard jupyter_venv]# pip3 install numpy pandas

== Exit Your Virtual Environment ==

To exit out of your virtual environment, simply do:

deactivate

and you should return to your regular shell.

== Call Your Virtual Environment In JupyterHub ==

Once you login to JupyterHub on the cluster here:

https://jupyterhub.prism

You will want to click on the '''Advanced''' tab at the top. When defining your notebook resources, under "Jupyter environment", select "Custom". Then define your description and give the path to your '''/bin''' directory in your environment, as such:

[[File:Custom2.png|800px|frame|left|Custom Environment]]

It will then launch a notebook based on that environment, and all modules and packages installed in that environment will be available.

Access the JupyterHub Slurm Server

2026-05-23T00:24:49Z

Weiler:

__TOC__

We have a JupyterHub server running on top of Slurm that is very useful for spawning Jupyter Notebooks.

== Login to the Hub ==

You can login to the JupyterHub interface here:

https://jupyterhub.prism

Your web browser may complain about a name mismatch on the SSL certificate but you can click past it, the connection is still secure.

There you will see a login page:

[[File:Jupyter_login.png|1200px|frame|left|Login Page]]

Login using your regular UNIX username and password.

== Configure Your Notebook ==

Once you login, you can configure your notebook for its required resources in Slurm, and then launch your notebook, which will appear on a node Slurm chooses. If a node is free right then, your notebook will launch immediately. If the Slurm queue is full, you may have to wait for resources to become available.

You will see this page after logging in, which is the "Simple" setup page:

[[File:Simple_server.png|600px|frame|left|Simple Server]]

Here you can define more specific job parameters.

You will want to mostly accept the defaults if running a simple notebook. But if your notebook requires more resources, then kick up the max time or CPU/GPU count as appropriate. Remember Slurm will kill the notebook if it persists past the '''Job Duration''' parameter. Simply click the '''Start''' button on the bottom to launch the notebook.

If you need a more specific environment, click on the '''Advanced''' tab, which should yield this screen:

[[File:Advanced_server.png|600px|frame|left|Advanced Server]]

This allows for a more fine grained resource definition when requesting resources for your notebook. It also allows for a custom Jupyter environment to be defined, if you need one. The Default environment is fairly basic, so it may make sense to create your own custom environment. If you would like to do that, refer to [[Create a Jupyter Virtual Environment]].

== The Live Notebook ==

Once you start your notebook (give it some time, it can take 10-20 seconds to start based on your environment and resource requests), you should see your notebook appear as something like this in your web browser:

[[File:Notebook.png|1200px|frame|left|Login Page]]

This is a notebook actually running on one of the cluster nodes, but the connection is proxied through the head node, and you are seeing it in your web browser. You can actually see your job running in slurm by looking at the output of the '''squeue''' command, if you are curious.

== Terminate Your Notebook When Finished ==

When you are done, don't forget to terminate your notebook to free up the Slurm resources it was using! To close your notebook, click on '''File -> Hub Control Panel'''. Then from there click the red '''Stop My Server''' button. Your notebook will terminate and the associated Slurm job will also exit.

Access the JupyterHub Slurm Server

2026-05-23T00:23:52Z

Weiler: Created page with "__TOC__ We have a JupyterHub server running on top of Slurm that is very useful for spawning Jupyter Notebooks. == Login to the Hub == You can login to the JupyterHub interface here: https://jupyter.mckenna.edu Your web browser may complain about a name mismatch on the SSL certificate but you can click past it, the connection is still secure. There you will see a login page: Login Page Login using your regular UNIX us..."

__TOC__

We have a JupyterHub server running on top of Slurm that is very useful for spawning Jupyter Notebooks.

== Login to the Hub ==

You can login to the JupyterHub interface here:

https://jupyter.mckenna.edu

Your web browser may complain about a name mismatch on the SSL certificate but you can click past it, the connection is still secure.

There you will see a login page:

[[File:Jupyter_login.png|1200px|frame|left|Login Page]]

Login using your regular UNIX username and password.

== Configure Your Notebook ==

Once you login, you can configure your notebook for its required resources in Slurm, and then launch your notebook, which will appear on a node Slurm chooses. If a node is free right then, your notebook will launch immediately. If the Slurm queue is full, you may have to wait for resources to become available.

You will see this page after logging in, which is the "Simple" setup page:

[[File:Simple_server.png|600px|frame|left|Simple Server]]

Here you can define more specific job parameters.

You will want to mostly accept the defaults if running a simple notebook. But if your notebook requires more resources, then kick up the max time or CPU/GPU count as appropriate. Remember Slurm will kill the notebook if it persists past the '''Job Duration''' parameter. Simply click the '''Start''' button on the bottom to launch the notebook.

If you need a more specific environment, click on the '''Advanced''' tab, which should yield this screen:

[[File:Advanced_server.png|600px|frame|left|Advanced Server]]

This allows for a more fine grained resource definition when requesting resources for your notebook. It also allows for a custom Jupyter environment to be defined, if you need one. The Default environment is fairly basic, so it may make sense to create your own custom environment. If you would like to do that, refer to [[Create a Jupyter Virtual Environment]].

== The Live Notebook ==

Once you start your notebook (give it some time, it can take 10-20 seconds to start based on your environment and resource requests), you should see your notebook appear as something like this in your web browser:

[[File:Notebook.png|1200px|frame|left|Login Page]]

This is a notebook actually running on one of the cluster nodes, but the connection is proxied through the head node, and you are seeing it in your web browser. You can actually see your job running in slurm by looking at the output of the '''squeue''' command, if you are curious.

== Terminate Your Notebook When Finished ==

When you are done, don't forget to terminate your notebook to free up the Slurm resources it was using! To close your notebook, click on '''File -> Hub Control Panel'''. Then from there click the red '''Stop My Server''' button. Your notebook will terminate and the associated Slurm job will also exit.

Genomics Institute Computing Information

2026-05-23T00:22:50Z

Weiler:

Welcome to the Genomic Institute Computing Information Repository! Browse the below topics for help in the area you are curious about.

== GI Public Computing Environment ==
*[[How to access the public servers]]
*[[Resetting your Public UNIX Password]]

== GI Firewalled Computing Environment (PRISM) ==
*[[Access to the Firewalled Compute Servers]]
*[[Firewalled Computing Resources Overview]]
*[[Firewalled Environment Storage Overview]]
*[[Firewalled User Account and Storage Cost]]
*[[Grafana Performance Metrics]]
*[[Visual Studio Code (vscode) Configuration Tweaks]]
*[http://logserv.gi.ucsc.edu/cgi-bin/private-groups.cgi '''/private/groups''' Data Usage Graphs]
*[[Resetting your VPN/PRISM Password]]

==VPN Access==
*[[Requirement for users to get GI VPN access]]
*[[Setting Up The VPN on MacOS]]
*[[Setting Up The VPN on Windows]]
*[[Setting Up The VPN on Linux]]
*[[Multi Factor Authentication (MFA) Frequently Asked Questions]]
*[[Duo Pushes Aren't Being Sent to My Phone!]]

== NIH dbGaP Access Requirements ==
*[[Requirements for dbGaP Access]]

== giCloud Openstack ==
*[[Overview of giCloud in the Genomics Institute]]
*[[Quick Start Instructions to Get Rolling with OpenStack]]

== Amazon Web Services Information ==
*[[Overview of Getting and Using an AWS IAM Account]]
*[[AWS Account List and Numbers]]
*[[AWS Shared Bucket Usage Graphs]]
*[[AWS Best Practices]]
*[[AWS S3 Lifecycle Management]]

== JupyterHub Notebooks Spawned via Slurm ==
*[[Access the JupyterHub Slurm Server]]
*[[Create a Jupyter Virtual Environment]]
*[[Using Shared Jupyter Virtual Environments]]

== Slurm at the Genomics Institute ==
*[[Overview of using Slurm]]
*[[Slurm Best Practices]]
*[[Cluster Etiquette]]
*[[Annotated Slurm Script]]
*[[Job Arrays]]
*[[GPU Resources]]
*[[Quick Reference Guide]]
*[[Convenient Slurm Commands]]
*[[Slurm Queues (Partitions) and Resource Management]]
*[[Slurm Tips for vg]]
*[[Slurm Tips for Toil]]
*[[Using Docker under Slurm]]
*[[Phoenix WDL Tutorial]]
*[[Slurm Insights Dashboard]]

==General Docker Information==
*[[Running a Container as a non-root User]]

== Problems or technical support ==

If you have any questions or problems with the GI computing environment, please send an email to '''cluster-admin@soe.ucsc.edu'''

Firewalled Computing Resources Overview

2026-04-29T15:18:53Z

Weiler: /* The Phoenix Cluster */

== Doing Work and Computing ==

When doing research, running jobs and the like, please be careful of your resource consumption on the server you are on. Don't run too many threads or cores at once if such a thing overruns the RAM available or the disk IO available. If you are not sure of your potential RAM, CPU or disk impact, start small with one or two processes and work your way up from there. Also, before running your stuff, check what else is already happening on the server by using the 'top' command to see who else and what else is running and what kind of resources are already being consumed. If, after starting a process, you realize that the server slows down considerably or becomes unusable, kill
your processes and re-evaluate what you need to make things work. These servers are shared resources - be a good neighbor!

== Server Types and Management ==

After confirming your VPN software is working, you can ssh into one of the shared compute servers behind the VPN. The DNS suffix for all machines is ".prism". So, "mustard" would have a full DNS name of "mustard.prism":

{| class="wikitable" style="text-align:center;"
|- style="font-weight:bold; text-align:left;"
! Node Name
! Operating System 
! CPU Cores
! Memory
! Network Bandwidth
! Scratch Space
|-
| style="text-align:left;" | mustard
| style="text-align:left;" | Ubuntu 22.04
| 160
| 1.5 TB
| 10 Gb/s
| 9 TB
|-
| style="text-align:left;" | emerald
| style="text-align:left;" | Ubuntu 22.04
| 64
| 1 TB
| 10 Gb/s
| 690 GB
|-
| style="text-align:left;" | crimson
| style="text-align:left;" | Ubuntu 22.04
| 32
| 256 GB
| 10 Gb/s
| 5.5 TB
|-
| style="text-align:left;" | razzmatazz
| style="text-align:left;" | Ubuntu 22.04
| 32
| 256 GB
| 10 Gb/s
| 5.5 TB
|}

These ''shared'' servers are managed by the Genomics Institute Cluster Admin group. If you need software installed on any of these servers, please make your request by emailing cluster-admin@soe.ucsc.edu. The above mentioned scratch space is usually located in '''/data/scratch''' on each server.

== The Firewall ==

All servers are behind a firewall in this environment, and as such, you must connect to the VPN in order to access them. They will not be accessible from the greater Internet without VPN. Although you will be able to connect outbound from them to other servers on the internet to copy data in, sync git repos, stuff like that. It is only inbound connections that will be blocked. All machines behind the firewall have the private domain name suffix of "*.prism".

== The Phoenix Cluster ==

This is a cluster of 25 Ubuntu 22.04 nodes, some of which have GPUs in them. Each node generally has about 2TB RAM and 256 cores, although the cluster is heterogeneous and has multiple node types. You interact with the Phoenix Cluster via the Slurm Job Scheduler. You must specifically request access to use Slurm on the Phoenix Cluster, just email '''cluster-admin@soe.ucsc.edu''' for access.

This HPC cluster is a fairly modern cluster comprised of:

{| class="wikitable" style="text-align:center;"
|- style="font-weight:bold;"
! Total CPU Cores
! Total Memory
! Local Scratch Storage
! Six GPU Nodes
! Primary Storage
! Secondary Storage
! Network Backbone
|-
| 7040
| 49.9 TB
| 16 TB (NVMe) per node, 400 TB total
| 8 GPUs per node (A100, A5500), 48 GPUs total
| 1.7 PB of Clustered Ceph NVMe Solid State Storage
| 2.5 PB Spinning Disk Storage (Ceph + ZFS)
| 10 Gb/s (nodes) / 200 Gb/s+ (storage)
|}

Specific breakdown of node architecture:

{| class="wikitable" style="text-align:center;"
|- style="font-weight:bold;"
! Node Name
! Operating System 
! CPU Cores
! GPUs/Type
! Memory
! Network Bandwidth
! Scratch Space
|-
| style="text-align:left;" | phoenix-00
| style="text-align:left;" | Ubuntu 22.04
| 256
| 8 / Nvidia A100
| 1 TB
| 10 Gb/s
| 16 TB NVMe
|-
| style="text-align:left;" | phoenix-[01-05]
| style="text-align:left;" | Ubuntu 22.04
| 256
| 8 / Nvidia A5500
| 2 TB
| 10 Gb/s
| 16 TB NVMe
|-
| style="text-align:left;" | phoenix-[06-08]
| style="text-align:left;" | Ubuntu 22.04
| 256
| N/A
| 2 TB
| 10 Gb/s
| 16 TB NVMe
|-
| style="text-align:left;" | phoenix-[09-10]
| style="text-align:left;" | Ubuntu 22.04
| 384
| N/A
| 2.3 TB
| 10 Gb/s
| 16 TB NVMe
|-
| style="text-align:left;" | phoenix-[11-21]
| style="text-align:left;" | Ubuntu 22.04
| 256
| N/A
| 2 TB
| 10 Gb/s
| 16 TB NVMe
|-
| style="text-align:left;" | phoenix-[22-24]
| style="text-align:left;" | Ubuntu 22.04
| 384
| N/A
| 2.3 TB
| 10 Gb/s
| 16 TB NVMe
|}

The cluster head node is '''phoenix.prism'''. However, you cannot directly login to phoenix.prism in order to protect the scheduler from errant or runaway jobs there, so jobs can be submitted from any interactive compute server (mustard, emerald, razzmatazz or crimson). To learn more about how to use Slurm, refer to:

https://giwiki.gi.ucsc.edu/index.php/Genomics_Institute_Computing_Information#Slurm_at_the_Genomics_Institute

For scratch on the cluster, TMPDIR will be set to /data/tmp (which is local to each cluster node). That area is cleaned often so don't store any data there that isn't being used by your jobs.

==Graphical Diagram of the Firewalled Area==

This is a general representation of how things look:

[[File:Ucsc_gi_private_diagram.png|900px]]

Firewalled Environment Storage Overview

2026-04-29T15:10:34Z

Weiler: /* Backups */

== Storage ==

Our servers mount two types of ''shared'' storage; home directories and group storage directories. These home directories will mount over the network to all shared compute servers and the phoenix cluster, so any server you login to will have these filesystems available:

'''Filesystem Specifications'''

{| class="wikitable" style="text-align:center;"
|- style="font-weight:bold; text-align:left;"
! Filesystem 
! /private/home
! /private/groups
|-
| style="font-weight:bold; text-align:left;" | Default Soft Quota
| 100 GB
| 15 TB
|-
| style="font-weight:bold; text-align:left;" | Default Hard Quota
| 100 GB
| 15 TB
|-
| style="font-weight:bold; text-align:left;" | Total Capacity
| 19 TB
| 1.7 PB
|- style="text-align:left;"
| style="font-weight:bold;" | Access Speed
| Very Fast (NVMe Flash Media)
| Very Fast (NVMe Flash Media)
|- style="text-align:left;"
| style="font-weight:bold;" | Intended Use
| This space should be used for login scripts, small bits of code or software repos, etc. No large data should be stored here.
| This space should be used for large computational/shared data, large software installations and the like.
|}

'''Home Directories (/private/home/username)'''

Your home directory will be located as "/private/home/username" and has a 100GB soft quota and a 100GB hard quota. Your home directory is meant for small scripts and login data, or a git repo. Please do not try to store large data there or computer on large jobs using data in your home directory.

'''Groups Directories (/private/groups/groupname)'''

The group storage directories are created per PI, and each group directory has a default 15TB hard quota. For example, if David Haussler is the PI that you report to directly, then the directory would exist as /private/groups/hausslerlab. Request access to that group directory and you will then be able to write to it. Each of those group directories are shared by the lab it belongs to, so you must be wary of everyone's data usage and share the 15TB available per group accordingly.

On the compute servers you can check your group's current quota usage by using the 'getfattr' command. You can only check the quota of a group you are part of (you would be a member of the UNIX group of the same name). If you wanted to check the quota usage of /private/groups/hausslerlab for example, you would do:

$ getfattr -n ceph.dir.rbytes /private/groups/hausslerlab

getfattr: Removing leading '/' from absolute path names
# file: private/groups/hausslerlab
ceph.dir.rbytes="6522955553147"

That number is in bytes. So divide by 1,000,000,000,000 and you get '6.522 TB'. That is how much data is currently being used.

To check the max quota limit, use this command:

$ getfattr -n ceph.quota.max_bytes /private/groups/hausslerlab

getfattr: Removing leading '/' from absolute path names
# file: private/groups/hausslerlab
ceph.quota.max_bytes="15000000000000"

And 15000000000000 divided by 1,000,000,000,000 is 15 TB.

== Storage Quota Alerting ==

If you and/or folks in your lab would like an automated alert when the /private/groups/labname quota is getting to a certain percentage of fullness, we can set that up for you and others in your lab. Just email '''cluster-admin@soe.ucsc.edu''' with the following information:

1: Which directory you would like to watch quotas on (i.e. /private/groups/somelab)
2: What % full you would like an email alert at
3: What email addresses you want on the alert list

After setup, our alerting system will alert folks on that email list ''every 4 hours'' until the quota in question is reduced to an amount under the alerting % threshold you asked for. So it is a bit noisy, but will force folks to delete data in order to stop the alerts. When the system notices that the quota usage has decreased to under the alert threshold, you will receive one final email with an "OK" notification that things are OK now.

== /data/scratch Space on the Servers ==

Each server will generally have a local /data/scratch filesystem that you can use to store temporary files. '''BE ADVISED''' that /data/scratch is not backed up, and the data there could disappear in the event of a disk failure or anything else. Do not store important data there. If it is important, it should be moved somewhere else very soon after creation.

== Backups ==

/private/groups is backed up monthly on the first of the month (which usually takes a week to complete). Please note that the following directories in the tree '''WILL NOT''' be backed up:

tmp/
temp/
TMP/
TEMP/
cache/
.cache/
scratch/
*.tmp/

So if you have data that you know isn't important and should be excluded from the backups, put them in a directory suffixed with ".tmp". Such as this example:

/private/groups/clusteradmin/mybams.tmp/

Firewalled Computing Resources Overview

2026-04-29T13:59:15Z

Weiler: /* The Phoenix Cluster */

== Doing Work and Computing ==

When doing research, running jobs and the like, please be careful of your resource consumption on the server you are on. Don't run too many threads or cores at once if such a thing overruns the RAM available or the disk IO available. If you are not sure of your potential RAM, CPU or disk impact, start small with one or two processes and work your way up from there. Also, before running your stuff, check what else is already happening on the server by using the 'top' command to see who else and what else is running and what kind of resources are already being consumed. If, after starting a process, you realize that the server slows down considerably or becomes unusable, kill
your processes and re-evaluate what you need to make things work. These servers are shared resources - be a good neighbor!

== Server Types and Management ==

After confirming your VPN software is working, you can ssh into one of the shared compute servers behind the VPN. The DNS suffix for all machines is ".prism". So, "mustard" would have a full DNS name of "mustard.prism":

{| class="wikitable" style="text-align:center;"
|- style="font-weight:bold; text-align:left;"
! Node Name
! Operating System 
! CPU Cores
! Memory
! Network Bandwidth
! Scratch Space
|-
| style="text-align:left;" | mustard
| style="text-align:left;" | Ubuntu 22.04
| 160
| 1.5 TB
| 10 Gb/s
| 9 TB
|-
| style="text-align:left;" | emerald
| style="text-align:left;" | Ubuntu 22.04
| 64
| 1 TB
| 10 Gb/s
| 690 GB
|-
| style="text-align:left;" | crimson
| style="text-align:left;" | Ubuntu 22.04
| 32
| 256 GB
| 10 Gb/s
| 5.5 TB
|-
| style="text-align:left;" | razzmatazz
| style="text-align:left;" | Ubuntu 22.04
| 32
| 256 GB
| 10 Gb/s
| 5.5 TB
|}

These ''shared'' servers are managed by the Genomics Institute Cluster Admin group. If you need software installed on any of these servers, please make your request by emailing cluster-admin@soe.ucsc.edu. The above mentioned scratch space is usually located in '''/data/scratch''' on each server.

== The Firewall ==

All servers are behind a firewall in this environment, and as such, you must connect to the VPN in order to access them. They will not be accessible from the greater Internet without VPN. Although you will be able to connect outbound from them to other servers on the internet to copy data in, sync git repos, stuff like that. It is only inbound connections that will be blocked. All machines behind the firewall have the private domain name suffix of "*.prism".

== The Phoenix Cluster ==

This is a cluster of 25 Ubuntu 22.04 nodes, some of which have GPUs in them. Each node generally has about 2TB RAM and 256 cores, although the cluster is heterogeneous and has multiple node types. You interact with the Phoenix Cluster via the Slurm Job Scheduler. You must specifically request access to use Slurm on the Phoenix Cluster, just email '''cluster-admin@soe.ucsc.edu''' for access.

This HPC cluster is a fairly modern cluster comprised of:

{| class="wikitable" style="text-align:center;"
|- style="font-weight:bold;"
! Total CPU Cores
! Total Memory
! Local Scratch Storage
! Six GPU Nodes
! Primary Storage
! Secondary Storage
! Network Backbone
|-
| 7040
| 49.9 TB
| 16 TB (NVMe) per node, 400 TB total
| 8 GPUs per node (A100, A5500), 48 GPUs total
| 1.7 PB of Clustered Ceph NVMe Solid State Storage
| 2.5 PB Spinning Disk Storage (Ceph + ZFS)
| 10G (nodes) / 200G+ (storage)
|}

Specific breakdown of node architecture:

{| class="wikitable" style="text-align:center;"
|- style="font-weight:bold;"
! Node Name
! Operating System 
! CPU Cores
! GPUs/Type
! Memory
! Network Bandwidth
! Scratch Space
|-
| style="text-align:left;" | phoenix-00
| style="text-align:left;" | Ubuntu 22.04
| 256
| 8 / Nvidia A100
| 1 TB
| 10 Gb/s
| 16 TB NVMe
|-
| style="text-align:left;" | phoenix-[01-05]
| style="text-align:left;" | Ubuntu 22.04
| 256
| 8 / Nvidia A5500
| 2 TB
| 10 Gb/s
| 16 TB NVMe
|-
| style="text-align:left;" | phoenix-[06-08]
| style="text-align:left;" | Ubuntu 22.04
| 256
| N/A
| 2 TB
| 10 Gb/s
| 16 TB NVMe
|-
| style="text-align:left;" | phoenix-[09-10]
| style="text-align:left;" | Ubuntu 22.04
| 384
| N/A
| 2.3 TB
| 10 Gb/s
| 16 TB NVMe
|-
| style="text-align:left;" | phoenix-[11-21]
| style="text-align:left;" | Ubuntu 22.04
| 256
| N/A
| 2 TB
| 10 Gb/s
| 16 TB NVMe
|-
| style="text-align:left;" | phoenix-[22-24]
| style="text-align:left;" | Ubuntu 22.04
| 384
| N/A
| 2.3 TB
| 10 Gb/s
| 16 TB NVMe
|}

The cluster head node is '''phoenix.prism'''. However, you cannot directly login to phoenix.prism in order to protect the scheduler from errant or runaway jobs there, so jobs can be submitted from any interactive compute server (mustard, emerald, razzmatazz or crimson). To learn more about how to use Slurm, refer to:

https://giwiki.gi.ucsc.edu/index.php/Genomics_Institute_Computing_Information#Slurm_at_the_Genomics_Institute

For scratch on the cluster, TMPDIR will be set to /data/tmp (which is local to each cluster node). That area is cleaned often so don't store any data there that isn't being used by your jobs.

==Graphical Diagram of the Firewalled Area==

This is a general representation of how things look:

[[File:Ucsc_gi_private_diagram.png|900px]]

Firewalled Computing Resources Overview

2026-04-29T13:47:08Z

Weiler: /* The Phoenix Cluster */

== Doing Work and Computing ==

When doing research, running jobs and the like, please be careful of your resource consumption on the server you are on. Don't run too many threads or cores at once if such a thing overruns the RAM available or the disk IO available. If you are not sure of your potential RAM, CPU or disk impact, start small with one or two processes and work your way up from there. Also, before running your stuff, check what else is already happening on the server by using the 'top' command to see who else and what else is running and what kind of resources are already being consumed. If, after starting a process, you realize that the server slows down considerably or becomes unusable, kill
your processes and re-evaluate what you need to make things work. These servers are shared resources - be a good neighbor!

== Server Types and Management ==

After confirming your VPN software is working, you can ssh into one of the shared compute servers behind the VPN. The DNS suffix for all machines is ".prism". So, "mustard" would have a full DNS name of "mustard.prism":

{| class="wikitable" style="text-align:center;"
|- style="font-weight:bold; text-align:left;"
! Node Name
! Operating System 
! CPU Cores
! Memory
! Network Bandwidth
! Scratch Space
|-
| style="text-align:left;" | mustard
| style="text-align:left;" | Ubuntu 22.04
| 160
| 1.5 TB
| 10 Gb/s
| 9 TB
|-
| style="text-align:left;" | emerald
| style="text-align:left;" | Ubuntu 22.04
| 64
| 1 TB
| 10 Gb/s
| 690 GB
|-
| style="text-align:left;" | crimson
| style="text-align:left;" | Ubuntu 22.04
| 32
| 256 GB
| 10 Gb/s
| 5.5 TB
|-
| style="text-align:left;" | razzmatazz
| style="text-align:left;" | Ubuntu 22.04
| 32
| 256 GB
| 10 Gb/s
| 5.5 TB
|}

These ''shared'' servers are managed by the Genomics Institute Cluster Admin group. If you need software installed on any of these servers, please make your request by emailing cluster-admin@soe.ucsc.edu. The above mentioned scratch space is usually located in '''/data/scratch''' on each server.

== The Firewall ==

All servers are behind a firewall in this environment, and as such, you must connect to the VPN in order to access them. They will not be accessible from the greater Internet without VPN. Although you will be able to connect outbound from them to other servers on the internet to copy data in, sync git repos, stuff like that. It is only inbound connections that will be blocked. All machines behind the firewall have the private domain name suffix of "*.prism".

== The Phoenix Cluster ==

This is a cluster of 25 Ubuntu 22.04 nodes, some of which have GPUs in them. Each node generally has about 2TB RAM and 256 cores, although the cluster is heterogeneous and has multiple node types. You interact with the Phoenix Cluster via the Slurm Job Scheduler. You must specifically request access to use Slurm on the Phoenix Cluster, just email '''cluster-admin@soe.ucsc.edu''' for access.

This HPC cluster is a fairly modern cluster comprised of:

{| class="wikitable" style="text-align:center;"
|- style="font-weight:bold;"
! Total CPU Cores
! Total Memory
! Local Scratch Storage
! Six GPU Nodes
! Primary Storage
! Secondary Storage
! Network Backbone
|-
| 7040
| 49.9 TB
| 16 TB (NVMe) per node, 400 TB total
| 8 GPUs per node (A100, A5500), 48 GPUs total
| 1.7 PB of Clustered Ceph NVMe Solid State Storage
| 2.5 PB Spinning Disk Storage (Ceph + ZFS)
| 10G (nodes) / 100G+ (storage)
|}

Specific breakdown of node architecture:

{| class="wikitable" style="text-align:center;"
|- style="font-weight:bold;"
! Node Name
! Operating System 
! CPU Cores
! GPUs/Type
! Memory
! Network Bandwidth
! Scratch Space
|-
| style="text-align:left;" | phoenix-00
| style="text-align:left;" | Ubuntu 22.04
| 256
| 8 / Nvidia A100
| 1 TB
| 10 Gb/s
| 16 TB NVMe
|-
| style="text-align:left;" | phoenix-[01-05]
| style="text-align:left;" | Ubuntu 22.04
| 256
| 8 / Nvidia A5500
| 2 TB
| 10 Gb/s
| 16 TB NVMe
|-
| style="text-align:left;" | phoenix-[06-08]
| style="text-align:left;" | Ubuntu 22.04
| 256
| N/A
| 2 TB
| 10 Gb/s
| 16 TB NVMe
|-
| style="text-align:left;" | phoenix-[09-10]
| style="text-align:left;" | Ubuntu 22.04
| 384
| N/A
| 2.3 TB
| 10 Gb/s
| 16 TB NVMe
|-
| style="text-align:left;" | phoenix-[11-21]
| style="text-align:left;" | Ubuntu 22.04
| 256
| N/A
| 2 TB
| 10 Gb/s
| 16 TB NVMe
|-
| style="text-align:left;" | phoenix-[22-24]
| style="text-align:left;" | Ubuntu 22.04
| 384
| N/A
| 2.3 TB
| 10 Gb/s
| 16 TB NVMe
|}

The cluster head node is '''phoenix.prism'''. However, you cannot directly login to phoenix.prism in order to protect the scheduler from errant or runaway jobs there, so jobs can be submitted from any interactive compute server (mustard, emerald, razzmatazz or crimson). To learn more about how to use Slurm, refer to:

https://giwiki.gi.ucsc.edu/index.php/Genomics_Institute_Computing_Information#Slurm_at_the_Genomics_Institute

For scratch on the cluster, TMPDIR will be set to /data/tmp (which is local to each cluster node). That area is cleaned often so don't store any data there that isn't being used by your jobs.

==Graphical Diagram of the Firewalled Area==

This is a general representation of how things look:

[[File:Ucsc_gi_private_diagram.png|900px]]

Firewalled Computing Resources Overview

2026-04-29T13:44:53Z

Weiler: /* The Phoenix Cluster */

== Doing Work and Computing ==

When doing research, running jobs and the like, please be careful of your resource consumption on the server you are on. Don't run too many threads or cores at once if such a thing overruns the RAM available or the disk IO available. If you are not sure of your potential RAM, CPU or disk impact, start small with one or two processes and work your way up from there. Also, before running your stuff, check what else is already happening on the server by using the 'top' command to see who else and what else is running and what kind of resources are already being consumed. If, after starting a process, you realize that the server slows down considerably or becomes unusable, kill
your processes and re-evaluate what you need to make things work. These servers are shared resources - be a good neighbor!

== Server Types and Management ==

After confirming your VPN software is working, you can ssh into one of the shared compute servers behind the VPN. The DNS suffix for all machines is ".prism". So, "mustard" would have a full DNS name of "mustard.prism":

{| class="wikitable" style="text-align:center;"
|- style="font-weight:bold; text-align:left;"
! Node Name
! Operating System 
! CPU Cores
! Memory
! Network Bandwidth
! Scratch Space
|-
| style="text-align:left;" | mustard
| style="text-align:left;" | Ubuntu 22.04
| 160
| 1.5 TB
| 10 Gb/s
| 9 TB
|-
| style="text-align:left;" | emerald
| style="text-align:left;" | Ubuntu 22.04
| 64
| 1 TB
| 10 Gb/s
| 690 GB
|-
| style="text-align:left;" | crimson
| style="text-align:left;" | Ubuntu 22.04
| 32
| 256 GB
| 10 Gb/s
| 5.5 TB
|-
| style="text-align:left;" | razzmatazz
| style="text-align:left;" | Ubuntu 22.04
| 32
| 256 GB
| 10 Gb/s
| 5.5 TB
|}

These ''shared'' servers are managed by the Genomics Institute Cluster Admin group. If you need software installed on any of these servers, please make your request by emailing cluster-admin@soe.ucsc.edu. The above mentioned scratch space is usually located in '''/data/scratch''' on each server.

== The Firewall ==

All servers are behind a firewall in this environment, and as such, you must connect to the VPN in order to access them. They will not be accessible from the greater Internet without VPN. Although you will be able to connect outbound from them to other servers on the internet to copy data in, sync git repos, stuff like that. It is only inbound connections that will be blocked. All machines behind the firewall have the private domain name suffix of "*.prism".

== The Phoenix Cluster ==

This is a cluster of 25 Ubuntu 22.04 nodes, some of which have GPUs in them. Each node generally has about 2TB RAM and 256 cores, although the cluster is heterogeneous and has multiple node types. You interact with the Phoenix Cluster via the Slurm Job Scheduler. You must specifically request access to use Slurm on the Phoenix Cluster, just email '''cluster-admin@soe.ucsc.edu''' for access.

This cluster is a fairly modern cluster comprised of:

{| class="wikitable" style="text-align:center;"
|- style="font-weight:bold;"
! Total CPU Cores
! Total Memory
! Local Scratch Storage
! Six GPU Nodes
! Primary Storage
! Secondary Storage
! Network Backbone
|-
| 7040
| 49.9 TB
| 16 TB (NVMe) per node, 400 TB total
| 8 GPUs per node (A100, A5500), 48 GPUs total
| 1.7 PB of Clustered Ceph NVMe Solid State Storage
| 2.5 PB Spinning Disk Storage (Ceph + ZFS)
| 10G (nodes) / 100G+ (storage)
|}

Specific breakdown of node architecture:

{| class="wikitable" style="text-align:center;"
|- style="font-weight:bold;"
! Node Name
! Operating System 
! CPU Cores
! GPUs/Type
! Memory
! Network Bandwidth
! Scratch Space
|-
| style="text-align:left;" | phoenix-00
| style="text-align:left;" | Ubuntu 22.04
| 256
| 8 / Nvidia A100
| 1 TB
| 10 Gb/s
| 16 TB NVMe
|-
| style="text-align:left;" | phoenix-[01-05]
| style="text-align:left;" | Ubuntu 22.04
| 256
| 8 / Nvidia A5500
| 2 TB
| 10 Gb/s
| 16 TB NVMe
|-
| style="text-align:left;" | phoenix-[06-08]
| style="text-align:left;" | Ubuntu 22.04
| 256
| N/A
| 2 TB
| 10 Gb/s
| 16 TB NVMe
|-
| style="text-align:left;" | phoenix-[09-10]
| style="text-align:left;" | Ubuntu 22.04
| 384
| N/A
| 2.3 TB
| 10 Gb/s
| 16 TB NVMe
|-
| style="text-align:left;" | phoenix-[11-21]
| style="text-align:left;" | Ubuntu 22.04
| 256
| N/A
| 2 TB
| 10 Gb/s
| 16 TB NVMe
|-
| style="text-align:left;" | phoenix-[22-24]
| style="text-align:left;" | Ubuntu 22.04
| 384
| N/A
| 2.3 TB
| 10 Gb/s
| 16 TB NVMe
|}

The cluster head node is '''phoenix.prism'''. However, you cannot directly login to phoenix.prism in order to protect the scheduler from errant or runaway jobs there, so jobs can be submitted from any interactive compute server (mustard, emerald, razzmatazz or crimson). To learn more about how to use Slurm, refer to:

https://giwiki.gi.ucsc.edu/index.php/Genomics_Institute_Computing_Information#Slurm_at_the_Genomics_Institute

For scratch on the cluster, TMPDIR will be set to /data/tmp (which is local to each cluster node). That area is cleaned often so don't store any data there that isn't being used by your jobs.

==Graphical Diagram of the Firewalled Area==

This is a general representation of how things look:

[[File:Ucsc_gi_private_diagram.png|900px]]

Firewalled Computing Resources Overview

2026-04-29T13:44:18Z

Weiler: /* The Phoenix Cluster */

== Doing Work and Computing ==

When doing research, running jobs and the like, please be careful of your resource consumption on the server you are on. Don't run too many threads or cores at once if such a thing overruns the RAM available or the disk IO available. If you are not sure of your potential RAM, CPU or disk impact, start small with one or two processes and work your way up from there. Also, before running your stuff, check what else is already happening on the server by using the 'top' command to see who else and what else is running and what kind of resources are already being consumed. If, after starting a process, you realize that the server slows down considerably or becomes unusable, kill
your processes and re-evaluate what you need to make things work. These servers are shared resources - be a good neighbor!

== Server Types and Management ==

After confirming your VPN software is working, you can ssh into one of the shared compute servers behind the VPN. The DNS suffix for all machines is ".prism". So, "mustard" would have a full DNS name of "mustard.prism":

{| class="wikitable" style="text-align:center;"
|- style="font-weight:bold; text-align:left;"
! Node Name
! Operating System 
! CPU Cores
! Memory
! Network Bandwidth
! Scratch Space
|-
| style="text-align:left;" | mustard
| style="text-align:left;" | Ubuntu 22.04
| 160
| 1.5 TB
| 10 Gb/s
| 9 TB
|-
| style="text-align:left;" | emerald
| style="text-align:left;" | Ubuntu 22.04
| 64
| 1 TB
| 10 Gb/s
| 690 GB
|-
| style="text-align:left;" | crimson
| style="text-align:left;" | Ubuntu 22.04
| 32
| 256 GB
| 10 Gb/s
| 5.5 TB
|-
| style="text-align:left;" | razzmatazz
| style="text-align:left;" | Ubuntu 22.04
| 32
| 256 GB
| 10 Gb/s
| 5.5 TB
|}

These ''shared'' servers are managed by the Genomics Institute Cluster Admin group. If you need software installed on any of these servers, please make your request by emailing cluster-admin@soe.ucsc.edu. The above mentioned scratch space is usually located in '''/data/scratch''' on each server.

== The Firewall ==

All servers are behind a firewall in this environment, and as such, you must connect to the VPN in order to access them. They will not be accessible from the greater Internet without VPN. Although you will be able to connect outbound from them to other servers on the internet to copy data in, sync git repos, stuff like that. It is only inbound connections that will be blocked. All machines behind the firewall have the private domain name suffix of "*.prism".

== The Phoenix Cluster ==

This is a cluster of 25 Ubuntu 22.04 nodes, some of which have GPUs in them. Each node generally has about 2TB RAM and 256 cores, although the cluster is heterogeneous and has multiple node types. You interact with the Phoenix Cluster via the Slurm Job Scheduler. You must specifically request access to use Slurm on the Phoenix Cluster, just email '''cluster-admin@soe.ucsc.edu''' for access.

This cluster is a fairly modern cluster comprised of:

{| class="wikitable" style="text-align:center;"
|- style="font-weight:bold;"
! Total CPU Cores
! Total Memory
! Local Scratch Storage
! Six GPU Nodes
! Primary Storage
! Secondary Storage
! Network Backbone
|-
| 7040
| 49.9 TB
| 16 TB (NVMe) per node, 400 TB total
| 8 GPUs per node (A100, A5500), for 48 GPUs total
| 1.7 PB of Clustered Ceph NVMe Solid State Storage
| 2.5 PB Spinning Disk Storage (Ceph + ZFS)
| 10G (nodes) / 100G+ (storage)
|}

Specific breakdown of node architecture:

{| class="wikitable" style="text-align:center;"
|- style="font-weight:bold;"
! Node Name
! Operating System 
! CPU Cores
! GPUs/Type
! Memory
! Network Bandwidth
! Scratch Space
|-
| style="text-align:left;" | phoenix-00
| style="text-align:left;" | Ubuntu 22.04
| 256
| 8 / Nvidia A100
| 1 TB
| 10 Gb/s
| 16 TB NVMe
|-
| style="text-align:left;" | phoenix-[01-05]
| style="text-align:left;" | Ubuntu 22.04
| 256
| 8 / Nvidia A5500
| 2 TB
| 10 Gb/s
| 16 TB NVMe
|-
| style="text-align:left;" | phoenix-[06-08]
| style="text-align:left;" | Ubuntu 22.04
| 256
| N/A
| 2 TB
| 10 Gb/s
| 16 TB NVMe
|-
| style="text-align:left;" | phoenix-[09-10]
| style="text-align:left;" | Ubuntu 22.04
| 384
| N/A
| 2.3 TB
| 10 Gb/s
| 16 TB NVMe
|-
| style="text-align:left;" | phoenix-[11-21]
| style="text-align:left;" | Ubuntu 22.04
| 256
| N/A
| 2 TB
| 10 Gb/s
| 16 TB NVMe
|-
| style="text-align:left;" | phoenix-[22-24]
| style="text-align:left;" | Ubuntu 22.04
| 384
| N/A
| 2.3 TB
| 10 Gb/s
| 16 TB NVMe
|}

The cluster head node is '''phoenix.prism'''. However, you cannot directly login to phoenix.prism in order to protect the scheduler from errant or runaway jobs there, so jobs can be submitted from any interactive compute server (mustard, emerald, razzmatazz or crimson). To learn more about how to use Slurm, refer to:

https://giwiki.gi.ucsc.edu/index.php/Genomics_Institute_Computing_Information#Slurm_at_the_Genomics_Institute

For scratch on the cluster, TMPDIR will be set to /data/tmp (which is local to each cluster node). That area is cleaned often so don't store any data there that isn't being used by your jobs.

==Graphical Diagram of the Firewalled Area==

This is a general representation of how things look:

[[File:Ucsc_gi_private_diagram.png|900px]]

Firewalled Computing Resources Overview

2026-04-29T13:43:24Z

Weiler: /* The Phoenix Cluster */

== Doing Work and Computing ==

When doing research, running jobs and the like, please be careful of your resource consumption on the server you are on. Don't run too many threads or cores at once if such a thing overruns the RAM available or the disk IO available. If you are not sure of your potential RAM, CPU or disk impact, start small with one or two processes and work your way up from there. Also, before running your stuff, check what else is already happening on the server by using the 'top' command to see who else and what else is running and what kind of resources are already being consumed. If, after starting a process, you realize that the server slows down considerably or becomes unusable, kill
your processes and re-evaluate what you need to make things work. These servers are shared resources - be a good neighbor!

== Server Types and Management ==

After confirming your VPN software is working, you can ssh into one of the shared compute servers behind the VPN. The DNS suffix for all machines is ".prism". So, "mustard" would have a full DNS name of "mustard.prism":

{| class="wikitable" style="text-align:center;"
|- style="font-weight:bold; text-align:left;"
! Node Name
! Operating System 
! CPU Cores
! Memory
! Network Bandwidth
! Scratch Space
|-
| style="text-align:left;" | mustard
| style="text-align:left;" | Ubuntu 22.04
| 160
| 1.5 TB
| 10 Gb/s
| 9 TB
|-
| style="text-align:left;" | emerald
| style="text-align:left;" | Ubuntu 22.04
| 64
| 1 TB
| 10 Gb/s
| 690 GB
|-
| style="text-align:left;" | crimson
| style="text-align:left;" | Ubuntu 22.04
| 32
| 256 GB
| 10 Gb/s
| 5.5 TB
|-
| style="text-align:left;" | razzmatazz
| style="text-align:left;" | Ubuntu 22.04
| 32
| 256 GB
| 10 Gb/s
| 5.5 TB
|}

These ''shared'' servers are managed by the Genomics Institute Cluster Admin group. If you need software installed on any of these servers, please make your request by emailing cluster-admin@soe.ucsc.edu. The above mentioned scratch space is usually located in '''/data/scratch''' on each server.

== The Firewall ==

All servers are behind a firewall in this environment, and as such, you must connect to the VPN in order to access them. They will not be accessible from the greater Internet without VPN. Although you will be able to connect outbound from them to other servers on the internet to copy data in, sync git repos, stuff like that. It is only inbound connections that will be blocked. All machines behind the firewall have the private domain name suffix of "*.prism".

== The Phoenix Cluster ==

This is a cluster of 25 Ubuntu 22.04 nodes, some of which have GPUs in them. Each node generally has about 2TB RAM and 256 cores, although the cluster is heterogeneous and has multiple node types. You interact with the Phoenix Cluster via the Slurm Job Scheduler. You must specifically request access to use Slurm on the Phoenix Cluster, just email '''cluster-admin@soe.ucsc.edu''' for access.

This cluster is a fairly modern cluster comprised of:

{| class="wikitable" style="text-align:center;"
|- style="font-weight:bold;"
! Total CPU Cores
! Total Memory
! Local Scratch Storage
! Six GPU Nodes
! Primary Storage
! Secondary Storage
! Network Backbone
|-
| 7040
| 49.9 TB
| 16 TB (NVMe) per node, 400 TB total
| 8 GPUs per node (A100, A5500), for 48 GPUs total
| 1.7 PB of Clustered Ceph NVMe Solid State Storage
| 2.5 PB Spinning Disk Storage (Ceph + ZFS)
| 10G (nodes) / 100G+ (storage)
|}

{| class="wikitable" style="text-align:center;"
|- style="font-weight:bold;"
! Node Name
! Operating System 
! CPU Cores
! GPUs/Type
! Memory
! Network Bandwidth
! Scratch Space
|-
| style="text-align:left;" | phoenix-00
| style="text-align:left;" | Ubuntu 22.04
| 256
| 8 / Nvidia A100
| 1 TB
| 10 Gb/s
| 16 TB NVMe
|-
| style="text-align:left;" | phoenix-[01-05]
| style="text-align:left;" | Ubuntu 22.04
| 256
| 8 / Nvidia A5500
| 2 TB
| 10 Gb/s
| 16 TB NVMe
|-
| style="text-align:left;" | phoenix-[06-08]
| style="text-align:left;" | Ubuntu 22.04
| 256
| N/A
| 2 TB
| 10 Gb/s
| 16 TB NVMe
|-
| style="text-align:left;" | phoenix-[09-10]
| style="text-align:left;" | Ubuntu 22.04
| 384
| N/A
| 2.3 TB
| 10 Gb/s
| 16 TB NVMe
|-
| style="text-align:left;" | phoenix-[11-21]
| style="text-align:left;" | Ubuntu 22.04
| 256
| N/A
| 2 TB
| 10 Gb/s
| 16 TB NVMe
|-
| style="text-align:left;" | phoenix-[22-24]
| style="text-align:left;" | Ubuntu 22.04
| 384
| N/A
| 2.3 TB
| 10 Gb/s
| 16 TB NVMe
|}

The cluster head node is '''phoenix.prism'''. However, you cannot directly login to phoenix.prism in order to protect the scheduler from errant or runaway jobs there, so jobs can be submitted from any interactive compute server (mustard, emerald, razzmatazz or crimson). To learn more about how to use Slurm, refer to:

https://giwiki.gi.ucsc.edu/index.php/Genomics_Institute_Computing_Information#Slurm_at_the_Genomics_Institute

For scratch on the cluster, TMPDIR will be set to /data/tmp (which is local to each cluster node). That area is cleaned often so don't store any data there that isn't being used by your jobs.

==Graphical Diagram of the Firewalled Area==

This is a general representation of how things look:

[[File:Ucsc_gi_private_diagram.png|900px]]

Firewalled Computing Resources Overview

2026-04-29T13:41:51Z

Weiler: /* The Phoenix Cluster */

== Doing Work and Computing ==

When doing research, running jobs and the like, please be careful of your resource consumption on the server you are on. Don't run too many threads or cores at once if such a thing overruns the RAM available or the disk IO available. If you are not sure of your potential RAM, CPU or disk impact, start small with one or two processes and work your way up from there. Also, before running your stuff, check what else is already happening on the server by using the 'top' command to see who else and what else is running and what kind of resources are already being consumed. If, after starting a process, you realize that the server slows down considerably or becomes unusable, kill
your processes and re-evaluate what you need to make things work. These servers are shared resources - be a good neighbor!

== Server Types and Management ==

After confirming your VPN software is working, you can ssh into one of the shared compute servers behind the VPN. The DNS suffix for all machines is ".prism". So, "mustard" would have a full DNS name of "mustard.prism":

{| class="wikitable" style="text-align:center;"
|- style="font-weight:bold; text-align:left;"
! Node Name
! Operating System 
! CPU Cores
! Memory
! Network Bandwidth
! Scratch Space
|-
| style="text-align:left;" | mustard
| style="text-align:left;" | Ubuntu 22.04
| 160
| 1.5 TB
| 10 Gb/s
| 9 TB
|-
| style="text-align:left;" | emerald
| style="text-align:left;" | Ubuntu 22.04
| 64
| 1 TB
| 10 Gb/s
| 690 GB
|-
| style="text-align:left;" | crimson
| style="text-align:left;" | Ubuntu 22.04
| 32
| 256 GB
| 10 Gb/s
| 5.5 TB
|-
| style="text-align:left;" | razzmatazz
| style="text-align:left;" | Ubuntu 22.04
| 32
| 256 GB
| 10 Gb/s
| 5.5 TB
|}

These ''shared'' servers are managed by the Genomics Institute Cluster Admin group. If you need software installed on any of these servers, please make your request by emailing cluster-admin@soe.ucsc.edu. The above mentioned scratch space is usually located in '''/data/scratch''' on each server.

== The Firewall ==

All servers are behind a firewall in this environment, and as such, you must connect to the VPN in order to access them. They will not be accessible from the greater Internet without VPN. Although you will be able to connect outbound from them to other servers on the internet to copy data in, sync git repos, stuff like that. It is only inbound connections that will be blocked. All machines behind the firewall have the private domain name suffix of "*.prism".

== The Phoenix Cluster ==

This is a cluster of 25 Ubuntu 22.04 nodes, some of which have GPUs in them. Each node generally has about 2TB RAM and 256 cores, although the cluster is heterogeneous and has multiple node types. You interact with the Phoenix Cluster via the Slurm Job Scheduler. You must specifically request access to use Slurm on the Phoenix Cluster, just email '''cluster-admin@soe.ucsc.edu''' for access.

This cluster is a fairly modern cluster comprised of:

{| class="wikitable" style="text-align:center;"
|- style="font-weight:bold;"
! Total CPU Cores
! Total Memory
! Local Scratch Storage
! Six GPU Nodes
! Primary Storage
! Secondary Storage
! Network Backbone
|-
| 7040
| 49.9 TB
| 14 TB (NVMe) per node, 350 TB total
| 8 GPUs per node (A100, A5500), for 48 GPUs total
| 1.7 PB of Clustered Ceph NVMe Solid State Storage
| 2.5 PB Spinning Disk Storage (Ceph + ZFS)
| 10G/40G/100G
|}

{| class="wikitable" style="text-align:center;"
|- style="font-weight:bold;"
! Node Name
! Operating System 
! CPU Cores
! GPUs/Type
! Memory
! Network Bandwidth
! Scratch Space
|-
| style="text-align:left;" | phoenix-00
| style="text-align:left;" | Ubuntu 22.04
| 256
| 8 / Nvidia A100
| 1 TB
| 10 Gb/s
| 16 TB NVMe
|-
| style="text-align:left;" | phoenix-[01-05]
| style="text-align:left;" | Ubuntu 22.04
| 256
| 8 / Nvidia A5500
| 2 TB
| 10 Gb/s
| 16 TB NVMe
|-
| style="text-align:left;" | phoenix-[06-08]
| style="text-align:left;" | Ubuntu 22.04
| 256
| N/A
| 2 TB
| 10 Gb/s
| 16 TB NVMe
|-
| style="text-align:left;" | phoenix-[09-10]
| style="text-align:left;" | Ubuntu 22.04
| 384
| N/A
| 2.3 TB
| 10 Gb/s
| 16 TB NVMe
|-
| style="text-align:left;" | phoenix-[11-21]
| style="text-align:left;" | Ubuntu 22.04
| 256
| N/A
| 2 TB
| 10 Gb/s
| 16 TB NVMe
|-
| style="text-align:left;" | phoenix-[22-24]
| style="text-align:left;" | Ubuntu 22.04
| 384
| N/A
| 2.3 TB
| 10 Gb/s
| 16 TB NVMe
|}

The cluster head node is '''phoenix.prism'''. However, you cannot directly login to phoenix.prism in order to protect the scheduler from errant or runaway jobs there, so jobs can be submitted from any interactive compute server (mustard, emerald, razzmatazz or crimson). To learn more about how to use Slurm, refer to:

https://giwiki.gi.ucsc.edu/index.php/Genomics_Institute_Computing_Information#Slurm_at_the_Genomics_Institute

For scratch on the cluster, TMPDIR will be set to /data/tmp (which is local to each cluster node). That area is cleaned often so don't store any data there that isn't being used by your jobs.

==Graphical Diagram of the Firewalled Area==

This is a general representation of how things look:

[[File:Ucsc_gi_private_diagram.png|900px]]

Firewalled Computing Resources Overview

2026-04-29T13:41:15Z

Weiler: /* The Phoenix Cluster */

== Doing Work and Computing ==

When doing research, running jobs and the like, please be careful of your resource consumption on the server you are on. Don't run too many threads or cores at once if such a thing overruns the RAM available or the disk IO available. If you are not sure of your potential RAM, CPU or disk impact, start small with one or two processes and work your way up from there. Also, before running your stuff, check what else is already happening on the server by using the 'top' command to see who else and what else is running and what kind of resources are already being consumed. If, after starting a process, you realize that the server slows down considerably or becomes unusable, kill
your processes and re-evaluate what you need to make things work. These servers are shared resources - be a good neighbor!

== Server Types and Management ==

After confirming your VPN software is working, you can ssh into one of the shared compute servers behind the VPN. The DNS suffix for all machines is ".prism". So, "mustard" would have a full DNS name of "mustard.prism":

{| class="wikitable" style="text-align:center;"
|- style="font-weight:bold; text-align:left;"
! Node Name
! Operating System 
! CPU Cores
! Memory
! Network Bandwidth
! Scratch Space
|-
| style="text-align:left;" | mustard
| style="text-align:left;" | Ubuntu 22.04
| 160
| 1.5 TB
| 10 Gb/s
| 9 TB
|-
| style="text-align:left;" | emerald
| style="text-align:left;" | Ubuntu 22.04
| 64
| 1 TB
| 10 Gb/s
| 690 GB
|-
| style="text-align:left;" | crimson
| style="text-align:left;" | Ubuntu 22.04
| 32
| 256 GB
| 10 Gb/s
| 5.5 TB
|-
| style="text-align:left;" | razzmatazz
| style="text-align:left;" | Ubuntu 22.04
| 32
| 256 GB
| 10 Gb/s
| 5.5 TB
|}

These ''shared'' servers are managed by the Genomics Institute Cluster Admin group. If you need software installed on any of these servers, please make your request by emailing cluster-admin@soe.ucsc.edu. The above mentioned scratch space is usually located in '''/data/scratch''' on each server.

== The Firewall ==

All servers are behind a firewall in this environment, and as such, you must connect to the VPN in order to access them. They will not be accessible from the greater Internet without VPN. Although you will be able to connect outbound from them to other servers on the internet to copy data in, sync git repos, stuff like that. It is only inbound connections that will be blocked. All machines behind the firewall have the private domain name suffix of "*.prism".

== The Phoenix Cluster ==

This is a cluster of 25 Ubuntu 22.04 nodes, some of which have GPUs in them. Each node generally has about 2TB RAM and 256 cores, although the cluster is heterogeneous and has multiple node types. You interact with the Phoenix Cluster via the Slurm Job Scheduler. You must specifically request access to use Slurm on the Phoenix Cluster, just email '''cluster-admin@soe.ucsc.edu''' for access.

This cluster is a fairly modern cluster comprised of:

{| class="wikitable" style="text-align:center;"
|- style="font-weight:bold;"
! Total CPU Cores
! Total Memory
! Local Scratch Storage
! Six GPU Nodes
! Primary Storage
! Secondary Storage
! Network Backbone
|-
| style="text-align:left;" | phoenix-[22-24]
| style="text-align:left;" | Ubuntu 22.04
| 7040
| 49.9 TB
| 14 TB (NVMe) per node, 350 TB total
| 8 GPUs per node (A100, A5500), for 48 GPUs total
| 1.7 PB of Clustered Ceph NVMe Solid State Storage
| 2.5 PB Spinning Disk Storage (Ceph + ZFS)
| 10G/40G/100G
|}

{| class="wikitable" style="text-align:center;"
|- style="font-weight:bold;"
! Node Name
! Operating System 
! CPU Cores
! GPUs/Type
! Memory
! Network Bandwidth
! Scratch Space
|-
| style="text-align:left;" | phoenix-00
| style="text-align:left;" | Ubuntu 22.04
| 256
| 8 / Nvidia A100
| 1 TB
| 10 Gb/s
| 16 TB NVMe
|-
| style="text-align:left;" | phoenix-[01-05]
| style="text-align:left;" | Ubuntu 22.04
| 256
| 8 / Nvidia A5500
| 2 TB
| 10 Gb/s
| 16 TB NVMe
|-
| style="text-align:left;" | phoenix-[06-08]
| style="text-align:left;" | Ubuntu 22.04
| 256
| N/A
| 2 TB
| 10 Gb/s
| 16 TB NVMe
|-
| style="text-align:left;" | phoenix-[09-10]
| style="text-align:left;" | Ubuntu 22.04
| 384
| N/A
| 2.3 TB
| 10 Gb/s
| 16 TB NVMe
|-
| style="text-align:left;" | phoenix-[11-21]
| style="text-align:left;" | Ubuntu 22.04
| 256
| N/A
| 2 TB
| 10 Gb/s
| 16 TB NVMe
|-
| style="text-align:left;" | phoenix-[22-24]
| style="text-align:left;" | Ubuntu 22.04
| 384
| N/A
| 2.3 TB
| 10 Gb/s
| 16 TB NVMe
|}

The cluster head node is '''phoenix.prism'''. However, you cannot directly login to phoenix.prism in order to protect the scheduler from errant or runaway jobs there, so jobs can be submitted from any interactive compute server (mustard, emerald, razzmatazz or crimson). To learn more about how to use Slurm, refer to:

https://giwiki.gi.ucsc.edu/index.php/Genomics_Institute_Computing_Information#Slurm_at_the_Genomics_Institute

For scratch on the cluster, TMPDIR will be set to /data/tmp (which is local to each cluster node). That area is cleaned often so don't store any data there that isn't being used by your jobs.

==Graphical Diagram of the Firewalled Area==

This is a general representation of how things look:

[[File:Ucsc_gi_private_diagram.png|900px]]

Firewalled Computing Resources Overview

2026-04-29T13:37:12Z

Weiler: /* The Phoenix Cluster */

== Doing Work and Computing ==

When doing research, running jobs and the like, please be careful of your resource consumption on the server you are on. Don't run too many threads or cores at once if such a thing overruns the RAM available or the disk IO available. If you are not sure of your potential RAM, CPU or disk impact, start small with one or two processes and work your way up from there. Also, before running your stuff, check what else is already happening on the server by using the 'top' command to see who else and what else is running and what kind of resources are already being consumed. If, after starting a process, you realize that the server slows down considerably or becomes unusable, kill
your processes and re-evaluate what you need to make things work. These servers are shared resources - be a good neighbor!

== Server Types and Management ==

After confirming your VPN software is working, you can ssh into one of the shared compute servers behind the VPN. The DNS suffix for all machines is ".prism". So, "mustard" would have a full DNS name of "mustard.prism":

{| class="wikitable" style="text-align:center;"
|- style="font-weight:bold; text-align:left;"
! Node Name
! Operating System 
! CPU Cores
! Memory
! Network Bandwidth
! Scratch Space
|-
| style="text-align:left;" | mustard
| style="text-align:left;" | Ubuntu 22.04
| 160
| 1.5 TB
| 10 Gb/s
| 9 TB
|-
| style="text-align:left;" | emerald
| style="text-align:left;" | Ubuntu 22.04
| 64
| 1 TB
| 10 Gb/s
| 690 GB
|-
| style="text-align:left;" | crimson
| style="text-align:left;" | Ubuntu 22.04
| 32
| 256 GB
| 10 Gb/s
| 5.5 TB
|-
| style="text-align:left;" | razzmatazz
| style="text-align:left;" | Ubuntu 22.04
| 32
| 256 GB
| 10 Gb/s
| 5.5 TB
|}

These ''shared'' servers are managed by the Genomics Institute Cluster Admin group. If you need software installed on any of these servers, please make your request by emailing cluster-admin@soe.ucsc.edu. The above mentioned scratch space is usually located in '''/data/scratch''' on each server.

== The Firewall ==

All servers are behind a firewall in this environment, and as such, you must connect to the VPN in order to access them. They will not be accessible from the greater Internet without VPN. Although you will be able to connect outbound from them to other servers on the internet to copy data in, sync git repos, stuff like that. It is only inbound connections that will be blocked. All machines behind the firewall have the private domain name suffix of "*.prism".

== The Phoenix Cluster ==

This is a cluster of 25 Ubuntu 22.04 nodes, some of which have GPUs in them. Each node generally has about 2TB RAM and 256 cores, although the cluster is heterogeneous and has multiple node types. You interact with the Phoenix Cluster via the Slurm Job Scheduler. You must specifically request access to use Slurm on the Phoenix Cluster, just email '''cluster-admin@soe.ucsc.edu''' for access.

This cluster is a fairly modern cluster comprised of:

'''Total CPU Cores:''' 7040

'''Total Memory:''' 49.9 TB

'''Local Scratch Storage:''' 14 TB (NVMe) per node, 350 TB total

'''Six GPU Nodes:''' 8 GPUs per node (A100, A5500), for 48 GPUs total

'''Primary Storage''': 1.7 PB of Clustered Ceph NVMe Solid State Storage

'''Secondary Storage''': 2.5 PB Spinning Disk Storage (Ceph + ZFS)

{| class="wikitable" style="text-align:center;"
|- style="font-weight:bold;"
! Node Name
! Operating System 
! CPU Cores
! GPUs/Type
! Memory
! Network Bandwidth
! Scratch Space
|-
| style="text-align:left;" | phoenix-00
| style="text-align:left;" | Ubuntu 22.04
| 256
| 8 / Nvidia A100
| 1 TB
| 10 Gb/s
| 16 TB NVMe
|-
| style="text-align:left;" | phoenix-[01-05]
| style="text-align:left;" | Ubuntu 22.04
| 256
| 8 / Nvidia A5500
| 2 TB
| 10 Gb/s
| 16 TB NVMe
|-
| style="text-align:left;" | phoenix-[06-08]
| style="text-align:left;" | Ubuntu 22.04
| 256
| N/A
| 2 TB
| 10 Gb/s
| 16 TB NVMe
|-
| style="text-align:left;" | phoenix-[09-10]
| style="text-align:left;" | Ubuntu 22.04
| 384
| N/A
| 2.3 TB
| 10 Gb/s
| 16 TB NVMe
|-
| style="text-align:left;" | phoenix-[11-21]
| style="text-align:left;" | Ubuntu 22.04
| 256
| N/A
| 2 TB
| 10 Gb/s
| 16 TB NVMe
|-
| style="text-align:left;" | phoenix-[22-24]
| style="text-align:left;" | Ubuntu 22.04
| 384
| N/A
| 2.3 TB
| 10 Gb/s
| 16 TB NVMe
|}

The cluster head node is '''phoenix.prism'''. However, you cannot directly login to phoenix.prism in order to protect the scheduler from errant or runaway jobs there, so jobs can be submitted from any interactive compute server (mustard, emerald, razzmatazz or crimson). To learn more about how to use Slurm, refer to:

https://giwiki.gi.ucsc.edu/index.php/Genomics_Institute_Computing_Information#Slurm_at_the_Genomics_Institute

For scratch on the cluster, TMPDIR will be set to /data/tmp (which is local to each cluster node). That area is cleaned often so don't store any data there that isn't being used by your jobs.

==Graphical Diagram of the Firewalled Area==

This is a general representation of how things look:

[[File:Ucsc_gi_private_diagram.png|900px]]

Firewalled Computing Resources Overview

2026-04-29T13:36:54Z

Weiler: /* The Phoenix Cluster */

== Doing Work and Computing ==

When doing research, running jobs and the like, please be careful of your resource consumption on the server you are on. Don't run too many threads or cores at once if such a thing overruns the RAM available or the disk IO available. If you are not sure of your potential RAM, CPU or disk impact, start small with one or two processes and work your way up from there. Also, before running your stuff, check what else is already happening on the server by using the 'top' command to see who else and what else is running and what kind of resources are already being consumed. If, after starting a process, you realize that the server slows down considerably or becomes unusable, kill
your processes and re-evaluate what you need to make things work. These servers are shared resources - be a good neighbor!

== Server Types and Management ==

After confirming your VPN software is working, you can ssh into one of the shared compute servers behind the VPN. The DNS suffix for all machines is ".prism". So, "mustard" would have a full DNS name of "mustard.prism":

{| class="wikitable" style="text-align:center;"
|- style="font-weight:bold; text-align:left;"
! Node Name
! Operating System 
! CPU Cores
! Memory
! Network Bandwidth
! Scratch Space
|-
| style="text-align:left;" | mustard
| style="text-align:left;" | Ubuntu 22.04
| 160
| 1.5 TB
| 10 Gb/s
| 9 TB
|-
| style="text-align:left;" | emerald
| style="text-align:left;" | Ubuntu 22.04
| 64
| 1 TB
| 10 Gb/s
| 690 GB
|-
| style="text-align:left;" | crimson
| style="text-align:left;" | Ubuntu 22.04
| 32
| 256 GB
| 10 Gb/s
| 5.5 TB
|-
| style="text-align:left;" | razzmatazz
| style="text-align:left;" | Ubuntu 22.04
| 32
| 256 GB
| 10 Gb/s
| 5.5 TB
|}

These ''shared'' servers are managed by the Genomics Institute Cluster Admin group. If you need software installed on any of these servers, please make your request by emailing cluster-admin@soe.ucsc.edu. The above mentioned scratch space is usually located in '''/data/scratch''' on each server.

== The Firewall ==

All servers are behind a firewall in this environment, and as such, you must connect to the VPN in order to access them. They will not be accessible from the greater Internet without VPN. Although you will be able to connect outbound from them to other servers on the internet to copy data in, sync git repos, stuff like that. It is only inbound connections that will be blocked. All machines behind the firewall have the private domain name suffix of "*.prism".

== The Phoenix Cluster ==

This is a cluster of 25 Ubuntu 22.04 nodes, some of which have GPUs in them. Each node generally has about 2TB RAM and 256 cores, although the cluster is heterogeneous and has multiple node types. You interact with the Phoenix Cluster via the Slurm Job Scheduler. You must specifically request access to use Slurm on the Phoenix Cluster, just email '''cluster-admin@soe.ucsc.edu''' for access.

This cluster is a fairly modern cluster comprised of:

'''Total CPU Cores:''' 7040
'''Total Memory:''' 49.9 TB
'''Local Scratch Storage:''' 14 TB (NVMe) per node, 350 TB total
'''Six GPU Nodes:''' 8 GPUs per node (A100, A5500), for 48 GPUs total
'''Primary Storage''': 1.7 PB of Clustered Ceph NVMe Solid State Storage
'''Secondary Storage''': 2.5 PB Spinning Disk Storage (Ceph + ZFS)

{| class="wikitable" style="text-align:center;"
|- style="font-weight:bold;"
! Node Name
! Operating System 
! CPU Cores
! GPUs/Type
! Memory
! Network Bandwidth
! Scratch Space
|-
| style="text-align:left;" | phoenix-00
| style="text-align:left;" | Ubuntu 22.04
| 256
| 8 / Nvidia A100
| 1 TB
| 10 Gb/s
| 16 TB NVMe
|-
| style="text-align:left;" | phoenix-[01-05]
| style="text-align:left;" | Ubuntu 22.04
| 256
| 8 / Nvidia A5500
| 2 TB
| 10 Gb/s
| 16 TB NVMe
|-
| style="text-align:left;" | phoenix-[06-08]
| style="text-align:left;" | Ubuntu 22.04
| 256
| N/A
| 2 TB
| 10 Gb/s
| 16 TB NVMe
|-
| style="text-align:left;" | phoenix-[09-10]
| style="text-align:left;" | Ubuntu 22.04
| 384
| N/A
| 2.3 TB
| 10 Gb/s
| 16 TB NVMe
|-
| style="text-align:left;" | phoenix-[11-21]
| style="text-align:left;" | Ubuntu 22.04
| 256
| N/A
| 2 TB
| 10 Gb/s
| 16 TB NVMe
|-
| style="text-align:left;" | phoenix-[22-24]
| style="text-align:left;" | Ubuntu 22.04
| 384
| N/A
| 2.3 TB
| 10 Gb/s
| 16 TB NVMe
|}

The cluster head node is '''phoenix.prism'''. However, you cannot directly login to phoenix.prism in order to protect the scheduler from errant or runaway jobs there, so jobs can be submitted from any interactive compute server (mustard, emerald, razzmatazz or crimson). To learn more about how to use Slurm, refer to:

https://giwiki.gi.ucsc.edu/index.php/Genomics_Institute_Computing_Information#Slurm_at_the_Genomics_Institute

For scratch on the cluster, TMPDIR will be set to /data/tmp (which is local to each cluster node). That area is cleaned often so don't store any data there that isn't being used by your jobs.

==Graphical Diagram of the Firewalled Area==

This is a general representation of how things look:

[[File:Ucsc_gi_private_diagram.png|900px]]

Requirement for users to get GI VPN access

2026-04-17T00:32:39Z

Weiler:

Before you are allowed access to our firewalled/secure area ("Prism"), you have to complete 3 items and provide the completed certificates or forms. '''If you already have a VPN account with us and are just installing the VPN software on a new computer you do not need to do all these steps again'''. Simply continue on to installing the VPN software on your new computer. Otherwise, continue to follow these steps if you are getting a VPN account with us for the first time.

'''1''': You must take and complete the NIH Public Security Refresher Course online. You must complete the course in a single continuous sitting:

https://irtsectraining.nih.gov/publicUser

Click on the training session for '''"Information Security and Management Refresher"'''. We recommend using '''Google Chrome''' as your web browser when doing the training course, as other browsers have had issues displaying the certificates after the training in some cases. Complete the training on that page and retain the completion certificates:

At the end you will be able to save the completion certificate that should have your name on it.

'''2''': You need to sign the Genomics Institute VPN User Agreement (digital signature OK), located here for download:

[[Media:GI_VPN_Policy.pdf]]

'''3''': Please read and sign the last page of the NIH Genomic Data Sharing Policy agreement (digital signature OK), located here for download. By signing the document you agree that you have read and understand the policies described therein and that you agree to abide by those policies:

[[Media:NIH_GDS_Policy.pdf]]

When you have the three documents described above ready, please complete this form: https://app.smartsheet.com/b/form/a76dbd90ba0240ab9ea9d39b390586ce.

There are two parts in this process.

1. For the user, please fill in ALL required fields '''and attach''' all three required documents described above. The form then goes to your PI for approval - remind them to approve it, or it won't get sent to us for processing!

2. For the Sponsor/PI - you will receive an email from Smartsheets. Please fill in all required fields and submit.

We will receive your completed request and we will create your account, then you will receive a welcome email with instructions on how to configure your VPN client and gain access to our systems.

When using the VPN software off-campus, it will usually work unless the wireless network you are on has restrictions preventing it from functioning. Some other universities have such restrictions (notably UCSF), but most other wireless network and home wireless networks should work fine.

'''PLEASE NOTE:''' Because of the overhead required in setting up VPN access, please only request access if you have an immediate need to work on data that exists behind the firewall. We have had a decent number of people request access and go through the setup but then never use it. In other words, please do not request access because "one day you might need it", but because you '''do''' actually need it!

'''ALSO NOTE:''' VPN accounts typically expire after one year from the date of first gaining access. To renew for another year you will need your PI/sponsor to send us a note asking for renewal.