Firewalled Environment Storage Overview: Difference between revisions

From UCSC Genomics Institute Computing Infrastructure Information
mNo edit summary
 
(9 intermediate revisions by the same user not shown)
Line 36: Line 36:


Your home directory will be located as "/private/home/username" and has a 100GB quota.  Your home directory is meant for small scripts and login data, or a git repo.  Please do not try to store large data there or computer on large jobs using data in your home directory.
Your home directory will be located as "/private/home/username" and has a 100GB quota.  Your home directory is meant for small scripts and login data, or a git repo.  Please do not try to store large data there or computer on large jobs using data in your home directory.
This data is backed up once a week.
The home directories have a file in the root of them that contains current quota usage info. These files (.quota_info) are updated once an hour.  For example:
cat /private/home/weiler/.quota_info
==================================================
  ZFS Private Home Storage Quota Report: weiler
==================================================
Username:      weiler
Space Used:    2.4G
Total Quota:    101G
Available:      99G
Last Updated:  2026-07-18 17:21:12 PDT
==================================================


'''Groups Directories (/private/groups/labname)'''
'''Groups Directories (/private/groups/labname)'''
Line 41: Line 58:
The group storage directories are created per PI, and each group directory has a default 15TB hard quota.  For example, if David Haussler is the PI that you report to directly, then the directory would exist as /private/groups/hausslerlab.  Request access to that group directory and you will then be able to write to it.  Each of those group directories are shared by the lab it belongs to, so you must be wary of everyone's data usage and share the 15TB available per group accordingly.
The group storage directories are created per PI, and each group directory has a default 15TB hard quota.  For example, if David Haussler is the PI that you report to directly, then the directory would exist as /private/groups/hausslerlab.  Request access to that group directory and you will then be able to write to it.  Each of those group directories are shared by the lab it belongs to, so you must be wary of everyone's data usage and share the 15TB available per group accordingly.


'''Archive Directories (/private/warm-archive/labname)'''
This data is backed up once a month.


The archive storage directories are created per PI '''by request''', and each group directory has a default 50TB hard quota.  For example, if David Haussler is the PI that you report to directly, then the directory would exist as /private/warm-archive/hausslerlab.
The groups storage directories have a file in the root of them that contains current quota usage info. These files (.quota_info) are updated once an hour.  For example:


On the compute servers you can check your group's current '''/private/groups''' quota usage by using the 'getfattr' command.  You can only check the quota of a group you are part of (you would be a member of the UNIX group of the same name). If you wanted to check the quota usage of /private/groups/hausslerlab for example, you would do:
cat /private/groups/corbettlab/.quota_info
 
  $ getfattr -n ceph.dir.rbytes /private/groups/hausslerlab
  ==================================================
  CephFS Storage Quota Report: corbettlab
==================================================
Filesystem Tier: CephFS
  Space Used:     72T
Total Quota:    92T
  Available:      21T
   
   
  getfattr: Removing leading '/' from absolute path names
  Last Updated:   2026-05-30 09:22:31 PDT
# file: private/groups/hausslerlab
  ==================================================
  ceph.dir.rbytes="6522955553147"


That number is in bytes. So divide by 1,000,000,000,000 and you get '6.522 TB'.  That is how much data is currently being used.
Every lab directory has a '''.quota_info''' file in its root.  


To check the max quota limit, use this command:
'''Archive Directories (/private/warm-archive/labname)'''


$ getfattr -n ceph.quota.max_bytes /private/groups/hausslerlab
The archive storage directories are created per PI '''by request''', and each group directory has a default 50TB hard quota.  For example, if David Haussler is the PI that you report to directly, then the directory would exist as /private/warm-archive/hausslerlab.
  getfattr: Removing leading '/' from absolute path names
# file: private/groups/hausslerlab
ceph.quota.max_bytes="15000000000000"


And 15000000000000 divided by 1,000,000,000,000 is 15 TB.
This data is '''not''' backed up.


== Storage Quota Alerting ==
The archive storage directories have a file in the root of them that contains current quota usage info.  These files (.quota_info) are updated once an hour.  For example:


If you and/or folks in your lab would like an automated alert when the /private/groups/labname quota is getting to a certain percentage of fullness, we can set that up for you and others in your lab.  Just email '''cluster-admin@soe.ucsc.edu''' with the following information:
cat /private/warm-archive/corbettlab/.quota_info
 
  1: Which directory you would like to watch quotas on (i.e. /private/groups/somelab)
==================================================
  2: What % full you would like an email alert at
  ZFS Storage Quota Report: corbettlab
  3: What email addresses you want on the alert list
==================================================
  Project ID:     105
Space Used:    148G
  Total Quota:   50T
  Available:     50T
  Last Updated:  2026-05-30 08:45:57 PDT
==================================================


After setup, our alerting system will alert folks on that email list ''every 4 hours'' until the quota in question is reduced to an amount under the alerting % threshold you asked for.  So it is a bit noisy, but will force folks to delete data in order to stop the alerts.  When the system notices that the quota usage has decreased to under the alert threshold, you will receive one final email with an "OK" notification that things are OK now.
Every lab directory has a '''.quota_info''' file in its root.


== /data/scratch Space on the Servers ==
== /data/scratch Space on the Servers ==

Latest revision as of 15:02, 27 July 2026

Storage

Our servers mount two types of shared storage; home directories and group storage directories. These home directories will mount over the network to all shared compute servers and the phoenix cluster, so any server you login to will have these filesystems available:

Filesystem Specifications

Filesystem
 /private/home /private/groups /private/warm-archive
Default Quota 100 GB 15 TB 50 TB
Total Capacity 19 TB 1.7 PB 4.6 PB
Access Speed Very Fast (NVMe Flash Media) Very Fast (NVMe Flash Media, Distributed Filesystem) Slower (Spinning Disks)
Intended Use This space should be used for login scripts, small bits of code or software repos, etc. No large data should be stored here. This space should be used for large computational/shared (hot) data, large software installations and the like. Archival Use Only. Not meant for active cluster computation. Warm/Cold data only.

Home Directories (/private/home/username)

Your home directory will be located as "/private/home/username" and has a 100GB quota. Your home directory is meant for small scripts and login data, or a git repo. Please do not try to store large data there or computer on large jobs using data in your home directory.

This data is backed up once a week.

The home directories have a file in the root of them that contains current quota usage info. These files (.quota_info) are updated once an hour. For example: 

cat /private/home/weiler/.quota_info

==================================================
 ZFS Private Home Storage Quota Report: weiler
==================================================
Username:       weiler
Space Used:     2.4G
Total Quota:    101G
Available:      99G

Last Updated:   2026-07-18 17:21:12 PDT
==================================================

Groups Directories (/private/groups/labname)

The group storage directories are created per PI, and each group directory has a default 15TB hard quota. For example, if David Haussler is the PI that you report to directly, then the directory would exist as /private/groups/hausslerlab. Request access to that group directory and you will then be able to write to it. Each of those group directories are shared by the lab it belongs to, so you must be wary of everyone's data usage and share the 15TB available per group accordingly.

This data is backed up once a month.

The groups storage directories have a file in the root of them that contains current quota usage info. These files (.quota_info) are updated once an hour. For example: 

cat /private/groups/corbettlab/.quota_info

==================================================
 CephFS Storage Quota Report: corbettlab
==================================================
Filesystem Tier: CephFS
Space Used:      72T
Total Quota:     92T
Available:       21T

Last Updated:    2026-05-30 09:22:31 PDT
==================================================

Every lab directory has a .quota_info file in its root.

Archive Directories (/private/warm-archive/labname)

The archive storage directories are created per PI by request, and each group directory has a default 50TB hard quota. For example, if David Haussler is the PI that you report to directly, then the directory would exist as /private/warm-archive/hausslerlab.

This data is not backed up.

The archive storage directories have a file in the root of them that contains current quota usage info. These files (.quota_info) are updated once an hour. For example: 

cat /private/warm-archive/corbettlab/.quota_info

==================================================
 ZFS Storage Quota Report: corbettlab
==================================================
Project ID:     105
Space Used:     148G
Total Quota:    50T
Available:      50T

Last Updated:   2026-05-30 08:45:57 PDT
==================================================

Every lab directory has a .quota_info file in its root.

/data/scratch Space on the Servers

Each server will generally have a local /data/scratch filesystem that you can use to store temporary files. BE ADVISED that /data/scratch is not backed up, and the data there could disappear in the event of a disk failure or anything else. Do not store important data there. If it is important, it should be moved somewhere else very soon after creation.

Backups

/private/groups is backed up monthly on the first of the month (which usually takes a week to complete). Please note that the following directories in the tree WILL NOT be backed up: 

tmp/
temp/
TMP/
TEMP/
cache/
.cache/
scratch/
*.tmp/

So if you have data that you know isn't important and should be excluded from the backups, put them in a directory suffixed with ".tmp". Such as this example: 

/private/groups/clusteradmin/mybams.tmp/
Retrieved from "http://giwiki.gi.ucsc.edu/index.php?title=Firewalled_Environment_Storage_Overview&oldid=825"