Table of contents
Integration of Amazon Web Services into Relion-2.0
We have integrated AWS into Relion-2.0, which means that users interact with Relion-2.0 locally. This makes particle picking and other GUI-based tasks latency-free.
Once installed, users only need to do the following to launch jobs to AWS:
- Input & select job parameters as if you were doing it for a local job
- Submit to queue?
- Select queue:
- All job parameters are determined automatically, no need to decide or select
Run now!to submit job to the cloud
- All results are synced back in real time to local machine
- Users visualize & interact with outputs on local machine
- When job finishes, instance will be automatically terminated.
Cryo-EM structure determination on AWS using only a laptop
Setup & installation
Software required on local machine:
- Option #1: cryoem-cloud-tools comes with a pre-compiled 'light' version of Relion-2.0 for Mac OSX. This allows users to open the GUI for submitting jobs to AWS
- Option #2: Complete installation via Relion-2.0 website
Benefits of AWS/Relion-2.0 integration
- Only requires users to understand Relion-2.0 workflow. All AWS commands are integrated without user interaction.
- Real time downloading of output files
- Allows users to monitor output files on local machine as they are generated
- Provides appearance of running job locally
- Data storage management:
- Data are left on S3 and EBS volumes for user-defined periods of time (typically 2 - 3 weeks). This minimizes transfer time, allowing users to immediately start new analyses without waiting for files to upload (again).
- Terminate instance when job finishes
- Allows users to walk away / go to sleep after job is submitted
1. Select AWS queue in queue type (qsub_aws) and submit job through Relion-2.0 GUI from local machine
2. Execute Relion-2.0 command on instance (Automated back-end workflow, no user input required)
3. Sync results back to local machine every 10 second (Automated back-end workflow, no user input required)
-> Turn off instance when job finishes (Automated back-end workflow, no user input required)
Detailed description (expert)
AWS integration steps :
- Select queue type (qsub_aws) and submit job through Relion-2.0 GUI from local machine
- queue: qsub_aws
- Run now!
- Data upload to S3 bucket using rclone
- Multi-file uploads on 10G networking = ~500 MB/sec
- Located in temporary locations named:
- Start virtual machine on AWS for GPU-based computations
- Instance-type automatically determined based upon dataset size and job task
- Restrict IP address access to instance to local machine IP address
- Download data from S3 to EBS volume using rclone
- Execute Relion-2.0 command on instance
- Sync results back to local machine every 10 seconds
- When job finishes - instance is terminated, but S3 bucket and EBS volume remain on AWS