Gerstein Lab Computing in HPC - Revision history

Public: /* Navigating the cluster */

2024-12-31T21:13:32Z

Navigating the cluster

← Older revision		Revision as of 21:13, 31 December 2024
Line 22:		Line 22:
	There are several locations within McCleary where you can store data files and software packages, as described below. See https://docs.ycrc.yale.edu/data/hpc-storage/ for more information on these drives.		There are several locations within McCleary where you can store data files and software packages, as described below. See https://docs.ycrc.yale.edu/data/hpc-storage/ for more information on these drives.

	*Home directory: This is your main directory when logging into the cluster (also accessible at <code>~/</code> ). This is ideal for small files and software packages, and is limited to 125Gb and 500k files per person.		*'''Home''' directory: This is your main directory when logging into the cluster (also accessible at <code>~/</code> ). This is ideal for small files and software packages, and is limited to 125Gb and 500k files per person.

	*Project directory: This directory is accessible at <code>~/project</code>. It is useful for smaller data files and software files, and is the default install location for conda packages. As this is limited to 4Tb storage and 5M files for the entire lab, '''please do not store large data files in this directory'''.		*'''Project''' directory: This directory is accessible at <code>~/project</code>. It is useful for smaller data files and software files, and is the default install location for conda packages. As this is limited to 4Tb storage and 5M files for the entire lab, '''please do not store large data files in this directory'''.

	*Scratch directory: This directory is accessible at <code>~/palmer_scratch</code>. It is useful for large files and temporary files that do not need to be permanently saved on the cluster. It is limited to 10Tb storage and 15M files for the entire lab, and '''any files older than 60 days will be deleted'''.		*'''Scratch''' directory: This directory is accessible at <code>~/palmer_scratch</code>. It is useful for large files and temporary files that do not need to be permanently saved on the cluster. It is limited to 10Tb storage and 15M files for the entire lab, and '''any files older than 60 days will be deleted'''.

	*Gibbs storage directory: This is the main data directory for the Gerstein lab, located at <code>/gpfs/gibbs/pi/gerstein/</code>. In most cases, all of your main data and analysis files and folders should be kept here. If you are a new user, please create a directory here for yourself. As of December 2024, this drive contains approx. '''1.5PB of storage space''', limited to 110M files for the lab.		*'''Gibbs''' storage directory: This is the main data directory for the Gerstein lab, located at <code>/gpfs/gibbs/pi/gerstein/</code>. In most cases, all of your main data and analysis files and folders should be kept here. If you are a new user, please create a directory here for yourself. As of December 2024, this drive contains approx. '''1.5PB of storage space''', limited to 110M files for the lab.

	*Palmer storage directory: This is an auxiliary data directory for the Gerstein lab, located at <code>/vast/palmer/pi/gerstein/</code>. This directory is useful for any large files, especially if there is limited room on Gibbs. If you are a new user, please create a directory here for yourself. As of December 2024, this drive contains approx. '''140Tb of storage space''', limited to 50M files for the lab.		*'''Palmer''' storage directory: This is an auxiliary data directory for the Gerstein lab, located at <code>/vast/palmer/pi/gerstein/</code>. This directory is useful for any large files, especially if there is limited room on Gibbs. If you are a new user, please create a directory here for yourself. As of December 2024, this drive contains approx. '''140Tb of storage space''', limited to 50M files for the lab.

	*Slayman storage directory: This is a '''temporary, read-only''' directory currently located at <code>/gpfs/gibbs/pi/slayman/slayman-gerstein</code>, containing files previously stored on the now-defunct Slayman system. These files will be migrated to Gibbs in early 2025.		*'''Slayman''' storage directory: This is a '''temporary, read-only''' directory currently located at <code>/gpfs/gibbs/pi/slayman/slayman-gerstein</code>, containing files previously stored on the now-defunct Slayman system. These files will be migrated to Gibbs in early 2025.

	To see how much free space is available in each of these locations, use the <code>getquota</code> command on HPC.		To see how much free space is available in each of these locations, use the <code>getquota</code> command on HPC.


	== Controlling Data Access by ACL ==		== Controlling Data Access by ACL ==

Public: MJ: heavy revision/update of page with 2024 cluster data (this is a work in progress)

2024-12-31T19:06:58Z

MJ: heavy revision/update of page with 2024 cluster data (this is a work in progress)

Show changes

Public at 20:31, 15 January 2020

2020-01-15T20:31:09Z

← Older revision		Revision as of 20:31, 15 January 2020
Line 1:		Line 1:
	= General Usage Notes =		= General Usage Notes =

			== Getting Access ==

			Fill this form up
			http://research.computing.yale.edu/support/hpc/account-request

	== Training ==		== Training ==

Public: /* Controlling Access by ACL */

2019-09-04T20:46:14Z

Controlling Access by ACL

← Older revision		Revision as of 20:46, 4 September 2019
Line 12:		Line 12:
	Allow access to 'directory' by user 'netid':		Allow access to 'directory' by user 'netid':

	$ setfacl -m u:<netid>:rx <directory>		$ setfacl -m u:<netid>:rx <directory>

	Use getfacl to examine ACLs:		Use getfacl to examine ACLs:

	$ getfacl <directory>		$ getfacl <directory>

	= Current Hardware =		= Current Hardware =

Public: /* General Usage Notes */

2019-09-04T20:45:53Z

General Usage Notes

@@ Line 6: / Line 6: @@
 == Transfers ==
 https://docs.ycrc.yale.edu/clusters-at-yale/data/transfer/
 = Current Hardware =

Public at 20:05, 4 September 2019

2019-09-04T20:05:52Z

← Older revision		Revision as of 20:05, 4 September 2019
Line 1:		Line 1:
	= General Usage Notes =		= General Usage Notes =

			== Training ==
			https://research.computing.yale.edu/training/

	== Transfers ==		== Transfers ==

	https://docs.ycrc.yale.edu/clusters-at-yale/data/transfer/		https://docs.ycrc.yale.edu/clusters-at-yale/data/transfer/

Public at 19:25, 4 September 2019

2019-09-04T19:25:36Z

← Older revision		Revision as of 19:25, 4 September 2019
Line 1:		Line 1:
			= General Usage Notes =
			== Transfers ==

			https://docs.ycrc.yale.edu/clusters-at-yale/data/transfer/

	= Current Hardware =		= Current Hardware =
	== Compute ==		== Compute ==

Public at 19:23, 4 September 2019

2019-09-04T19:23:18Z

← Older revision		Revision as of 19:23, 4 September 2019
Line 23:		Line 23:
	=== [http://info.gersteinlab.org/Gerstein_Lab_Openstack_2017 OpenStack] ===		=== [http://info.gersteinlab.org/Gerstein_Lab_Openstack_2017 OpenStack] ===
	1 director node, 8 cores		1 director node, 8 cores

	~~=== [[General Usage Notes]] ==~~


	3 controllers nodes, 24 cores		3 controllers nodes, 24 cores

Public at 19:22, 4 September 2019

2019-09-04T19:22:43Z

← Older revision		Revision as of 19:22, 4 September 2019
Line 23:		Line 23:
	=== [http://info.gersteinlab.org/Gerstein_Lab_Openstack_2017 OpenStack] ===		=== [http://info.gersteinlab.org/Gerstein_Lab_Openstack_2017 OpenStack] ===
	1 director node, 8 cores		1 director node, 8 cores

			=== [[General Usage Notes]] ==


	3 controllers nodes, 24 cores		3 controllers nodes, 24 cores

Public: /* OpenStack */

2018-12-05T19:49:33Z

OpenStack

← Older revision		Revision as of 19:49, 5 December 2018
Line 26:		Line 26:
	3 controllers nodes, 24 cores		3 controllers nodes, 24 cores

	3 ~~Ceph~~ nodes, 24 cores		3 ceph nodes, 24 cores

	5 compute nodes, 40 cores		5 compute nodes, 40 cores