The best way to clean up your data is to run this command periodically:

delete_temp_s3_ebs

This command will go through all S3 buckets and EBS volumes and will remove all files older than the lifetime specified in the initialization script (cryoem-cloud-tools/aws_init.sh):

export EBS_LIFETIME=14

export S3_LIFETIME=12

  • These numbers are in days and will be referenced by delete_temp_s3_ebs when deciding if files should be deleted.

You can alter these numbers to limit how long data will live on AWS.

Include delete_temp_s3_ebs into nightly commands

The best solution would be to add this command to the nightly calls by your operating system, also called ‘chron jobs’. To this, read more here regarding how to include this command as a chron job.