UCSC Genomics Institute Computing Infrastructure Information - User contributions [en]

AWS S3 Lifecycle Management

2021-12-09T19:49:37Z

Righanse:

==AWS S3 Lifecycle Policy Overview==
AWS S3 buckets can be configured with lifecycle policies. These policies allow for automatically changing the storage class of objects based on the last time they were modified or accessed. AWS S3 objects are stored in the Standard storage class by default, which provides easy access, but has relatively high GB/month storage costs. Other storage classes, such as Infrequent Access and Glacier are more suitable for objects that are rarely accessed. These storage classes maintain a much lower GB/month cost as compared to the Standard S3 storage class, but also incur charges for access and retrieval.

It is recommended to utilize the appropriate storage classes for your data.
* If you have data that you do not expect to access more than once a '''month''', AWS Infrequent Access is a reasonable storage class to use.
* If you have data that you do not expect to access more than once a '''year''', AWS Glacier is a reasonable storage class to use.

==UCSC GI Automated Policy==
In order to reduce monthly S3 storage costs, the UCSC GI has implemented global S3 lifecycle policies that transition objects to AWS Intelligent-Tiering, which monitors S3 object access patterns and transitions objects to more efficient storage classes accordingly.
* Objects uploaded to S3 will remain in the Standard storage class for '''1''' day, at which point they will be transitioned to Intelligent-Tiering.
* Old and new S3 buckets will have this lifecycle policy automatically attached.

AWS Intelligent Tiering functionality:
* Intelligent-Tiering '''does not''' change object access patterns. This means you do not need to execute special API commands to access objects.
* Intelligent-Tiering '''does not''' incur charges for object retrieval from different tiers.
For more details on AWS Intelligent Tiering, see the [https://aws.amazon.com/s3/storage-classes/intelligent-tiering/ AWS Docs]

Genomics Institute Computing Information

2021-12-09T17:21:05Z

Righanse: /* Amazon Web Services Information */

Welcome to the Genomic Institute Computing Information Repository! Browse the below topics for help in the area you are curious about.

==GI Public Computing Environment==
*[[How to access the public servers]]

==GI Firewalled Computing Environment (PRISM)==
*[[Access to the Firewalled Compute Servers]]

==VPN Access==
*[[Requirement for users to get GI VPN access]]

== NIH dbGaP Access Requirements ==
*[[Requirements for dbGaP Access]]

== giCloud Openstack ==
*[[Overview of giCloud in the Genomics Institute]]
*[[Quick Start Instructions to Get Rolling with OpenStack]]

== Amazon Web Services Information ==
*[[Overview of Getting and Using an AWS IAM Account]]
*[[AWS Account List and Numbers]]
*[[AWS Shared Bucket Usage Graphs]]
*[[AWS Best Practices]]
*[[AWS S3 Lifecycle Management]]

==Kubernetes Information==

*[[Computational Genomics Kubernetes Installation]]
*[[Undiagnosed Disease Project Kubernetes Installation]]

== Problems or technical support ==

If you have any problems with the GI computing environment, please send an email to '''cluster-admin@soe.ucsc.edu'''

AWS S3 Lifecycle Management

2021-12-09T17:19:48Z

Righanse:

==AWS S3 Lifecycle Policy Overview==
AWS S3 buckets can be configured with lifecycle policies. These policies allow for automatically changing the storage class of objects based on the last time they were modified or accessed. AWS S3 objects are stored in the Standard storage class by default, which provides easy access, but has relatively high GB/month storage costs. Other storage classes, such as Infrequent Access and Glacier are more suitable for objects that are rarely accessed. These storage classes maintain a much lower GB/month cost as compared to the Standard S3 storage class, but also incur charges for access and retrieval.

It is recommended to utilize the appropriate storage classes for your data.
* If you have data that you do not expect to access more than once a '''month''', AWS Infrequent Access is a reasonable storage class to use.
* If you have data that you do not expect to access more than once a '''year''', AWS Glacier is a reasonable storage class to use.

==UCSC GI Automated Policy==
In order to reduce monthly S3 storage costs, the UCSC GI has implemented global S3 lifecycle policies that transition objects to AWS Intelligent Tiering, which monitors S3 object access patterns and transitions objects to more efficient storage classes accordingly.
* Objects uploaded to S3 will remain in the Standard storage class for '''1''' day, at which point they will be transitioned to Intelligent Tiering.
* Old and new S3 buckets will have this lifecycle policy automatically attached.

AWS Intelligent Tiering functionality:
* Intelligent Tiering '''does not''' change object access patterns. This means you do not need to execute special API commands to access objects.
* Intelligent Tiering '''does not''' incur charges for object retrieval from different tiers.
For more details on AWS Intelligent Tiering, see the [https://aws.amazon.com/s3/storage-classes/intelligent-tiering/ AWS Docs]

AWS S3 Lifecycle Management

2021-12-09T17:18:28Z

Righanse:

(This page is a work in progress. The policies defined below may still be adjusted, and are not actively deployed)

==AWS S3 Lifecycle Policy Overview==
AWS S3 buckets can be configured with lifecycle policies. These policies allow for automatically changing the storage class of objects based on the last time they were modified or accessed. AWS S3 objects are stored in the Standard storage class by default, which provides easy access, but has relatively high GB/month storage costs. Other storage classes, such as Infrequent Access and Glacier are more suitable for objects that are rarely accessed. These storage classes maintain a much lower GB/month cost as compared to the Standard S3 storage class, but also incur charges for access and retrieval.

It is recommended to utilize the appropriate storage classes for your data.
* If you have data that you do not expect to access more than once a '''month''', AWS Infrequent Access is a reasonable storage class to use.
* If you have data that you do not expect to access more than once a '''year''', AWS Glacier is a reasonable storage class to use.

==UCSC GI Automated Policy==
In order to reduce monthly S3 storage costs, the UCSC GI has implemented global S3 lifecycle policies that transition objects to AWS Intelligent Tiering, which monitors S3 object access patterns and transitions objects to more efficient storage classes accordingly.
* Objects uploaded to S3 will remain in the Standard storage class for '''1''' day, at which point they will be transitioned to Intelligent Tiering.
* Old and new S3 buckets will have this lifecycle policy automatically attached.

AWS Intelligent Tiering functionality:
* Intelligent Tiering '''does not''' change object access patterns. This means you do not need to execute special API commands to access objects.
* Intelligent Tiering '''does not''' incur charges for object retrieval from different tiers.
For more details on AWS Intelligent Tiering, see the [https://aws.amazon.com/s3/storage-classes/intelligent-tiering/ AWS Docs]

AWS S3 Lifecycle Management

2021-12-09T17:03:07Z

Righanse:

(This page is a work in progress. The policies defined below may still be adjusted, and are not actively deployed)

==AWS S3 Lifecycle Policy Overview==
AWS S3 buckets can be configured with lifecycle policies. These policies allow for automatically changing the storage class of objects based on the last time they were accessed. AWS S3 objects are typically stored in the Standard storage class, which provides easy access, but has relatively high GB/month storage costs. Other storage classes, such as Infrequent Access and Glacier are more suitable for objects that are rarely accessed. These storage classes maintain a much lower GB/month cost as compared to the Standard S3 storage class, but also incur charges for access and retrieval.

It is recommended to utilize the appropriate storage classes for your data.
* If you have data that you do not expect to access more than once a '''month''', AWS Infrequent Access is a reasonable storage class to use.
* If you have data that you do not expect to access more than once a '''year''', AWS Glacier is a reasonable storage class to use.

==UCSC GI Automated Policy==
In order to reduce monthly S3 storage costs, the UCSC GI has implemented global S3 lifecycle policies that transition objects to AWS Intelligent Tiering, which monitors S3 object access patterns and transitions objects accordingly.
* Objects uploaded to S3 will remain in the Standard storage class for '''1''' day, at which point they will be transitioned to Intelligent Tiering.
* Old and new S3 buckets will have this lifecycle policy automatically attached.

AWS Intelligent Tiering functionality:
* Intelligent Tiering '''does not''' change object access patterns. This means you do not need to execute special API commands to access objects.
* Intelligent Tiering '''does not''' incur charges for object retrieval from different tiers.
For more details on AWS Intelligent Tiering, see the [https://aws.amazon.com/s3/storage-classes/intelligent-tiering/ AWS Docs]

AWS S3 Lifecycle Management

2021-12-09T16:43:15Z

Righanse:

(This page is a work in progress. The policies defined below may still be adjusted, and are not actively deployed)

==AWS S3 Lifecycle Policy Overview==
AWS S3 buckets can be configured with lifecycle policies. These policies allow for automatically changing the storage class of objects based on the last time they were accessed. AWS S3 objects are typically stored in the Standard storage class, which provides easy access, but has relatively high GB/month storage costs. Other storage classes, such as Infrequent Access and Glacier are more suitable for objects that are rarely accessed. These storage classes maintain a much lower GB/month cost as compared to the Standard S3 storage class, but also incur charges for access and retrieval.

It is recommended to utilize the appropriate storage classes for your data.
* If you have data that you do not expect to access more than once a '''month''', AWS Infrequent Access is a reasonable storage class to use.
* If you have data that you do not expect to access more than once a '''year''', AWS Glacier is a reasonable storage class to use.

==UCSC GI Automated Policy==
In order to reduce monthly S3 storage costs, the UCSC GI has implemented global S3 lifecycle policies that transition objects to AWS Intelligent Tiering, which monitors S3 object access patterns and transitions objects accordingly.
* Objects uploaded to S3 will remain in the Standard storage class for '''1''' day, at which point they will be transitioned to Intelligent Tiering.
* Old and new S3 buckets will have this lifecycle policy automatically attached.

AWS Intelligent Tiering functionality:
* Intelligent Tiering '''does not''' change object access patterns. This means you can still treat the object as if it were in the Standard storage class.
* Intelligent Tiering '''does not''' incur charges for object retrieval from different tiers.
For more details on AWS Intelligent Tiering, see the [https://aws.amazon.com/s3/storage-classes/intelligent-tiering/ AWS Docs]

AWS S3 Lifecycle Management

2021-12-02T23:43:42Z

Righanse:

(This page is a work in progress. The policies defined below may still be adjusted, and are not actively deployed)

==AWS S3 Lifecycle Policy Overview==
AWS S3 buckets can be configured with lifecycle policies. These policies allow for automatically changing the storage class of objects based on the last time they were accessed. AWS S3 objects are typically stored in the Standard storage class, which provides easy access, but has relatively high GB/month storage costs. Other storage classes, such as Infrequent Access and Glacier are more suitable for objects that are rarely accessed. These storage class maintain a much lower GB/month cost as compared to the Standard S3 storage class, but also incur charges for access and retrieval.

In order to reduce monthly S3 storage costs, the UCSC GI has implemented global S3 lifecycle policies that transition objects to cheaper storage classes if they have not been accessed recently. Specifically, AWS S3 objects are transitioned as follows:
* After '''30''' days of inactivity, objects are transitioned to the '''Infrequent Access''' storage class.
* After '''180''' days of inactivity, objects are transitioned to the '''Glacier''' storage class.

==Object Recovery==
In case there are objects in S3 that have been transitioned out of the Standard storage class, they can be recovered. Due to the increased access charges of objects in Infrequent Access and Glacier, if an object is expected to be accessed frequently, it should be returned to the Standard storage class. The object recovery method depends on the storage class the object is in, with Glacier being more time consuming and challenging to restore from than Infrequent Access.

It is important to note that, in the event an object is recovered, the timer for transitioning the object back to Infrequent Access and Glacier is still running, and the objects will be moved again in the future if they meet the criteria.

===Infrequent Access===
===Glacier===

AWS S3 Lifecycle Management

2021-12-02T23:28:14Z

Righanse:

(This page is a work in progress. The policies defined below are still be adjusted, and are not actively deployed)

==AWS S3 Lifecycle Policy Overview==
AWS S3 buckets can be configured with lifecycle policies. These policies allow for automatically changing the storage class of objects based on the last time they were accessed. AWS S3 objects are typically stored in the Standard storage class, which provides easy access, but has relatively high GB/month storage costs. Other storage classes, such as Infrequent Access and Glacier are more suitable for objects that are rarely accessed. These storage class maintain a much lower GB/month cost as compared to the Standard S3 storage class.

In order to reduce monthly S3 storage costs, the UCSC GI has implemented global S3 lifecycle policies that transition objects to cheaper storage classes if they have not been accessed recently. Specifically, AWS S3 objects are transitioned as follows:
* After '''30''' days of inactivity, objects are transitioned to the '''Infrequent Access''' storage class.
* After '''180''' days of inactivity, objects are transitioned to the '''Glacier''' storage class.

AWS S3 Lifecycle Management

2021-12-02T22:55:43Z

Righanse: Created page with "Test page"

Test page