Here you will find all available command-line tools to interact with AWS. If you haven't done so, please install the required software to your workstations / laptop.

These tools will allow you to start virtual machines & process your data using Relion2 with GPU capabilities (p2 instances).

All of these commands are run from your local machine to monitor, start, stop, or manipulate the instances running on AWS.

If, at any point you forget the available commands, you can always find available commands by typing:

$ aws_help

Commands available: 

View all resources being used by a given user:

Tools for turning on / off instances:

Tools for creating / delete EBS volumes:

Other tools:


awsls

This command will list all resources tagged with the key pair name provided in your aws_init.sh setup script (This means that you will not be able to see other users resources, even in your own research group).

Commands:

awsls

  • List all resources (instances, volumes, snapshots)

awsls -i 

  • Only list instances

awsls -v

  • Only list EBS volumes

awsls -s

  • Only list EBS snapshots

Example output:

  • Note that it provides the command required to ssh into instances
  • When EBS volumes are attached to an instance, it will also display the instance ID for the given attachment

$ awsls


awslaunch

This command will launch an instance with cryo-EM software environments by default, with the option of automatically attaching an existing EBS volume to the instance after startup.

NOTE: EBS volumes can only be attached to a single instance at a time. Therefore, check the output from awsls to make sure the EBS volume you want to use is available.

NOTEThis script will script all incoming ssh traffic to the given IP address from the machine it was launched. This means that if you launch it from your work computer, you won't be able log in from your home laptop (even if you can see the login information from the awsls command).

For p2 instances, this will launch:

  • Relion-2.0 (stable release)
  • MotionCor2
  • MotionCorr
  • Gctf
  • CTFFIND-4.1
  • Gautomatch
  • Unblur
  • Summovie
  • ResMap

By default, for non-p2 instances, the software environment will have the following (To include all the software listed above that comes with Relion-2.0, you can override this by including the --relion2 input option):

  • Relion-1.3
  • EMAN-2.1
  • Sparx
  • EMAN1-1.9
  • Frealign-9.08
  • Xmipp-2.4
  • CTFFIND3 + CTFTILT
  • Signature
  • Spider-22.03
  • TiltPicker

Input options: 

  • --instance   Specify instance type (Include --instanceList to see full list)
  • --availZone   Indicate which availability zone into which your instance will be launched. This must match the region specified in your aws_init.sh script. Full listing of availability zones can be found here.
  • --spotPrice   If requesting a spot instances, indicate price here. NOTE: p2 instances are currently unavailable for spot instances as of March 2017. Click here to learn more about spot instances.
  • --volume   Provide volume ID from awsls command for an EBS volume to attach upon booting up instance. Volume will be attached at /data/ by default.
  • --relion2   If booting up a non-p2 instance type (non-GPU), by default the software environment will use relion-1.4. Provide this option to instead use relion2.
  • --AMI   To provide your own AMI, input the AMI ID here. This will override any other AMI that the script would have tried to use normally. This is where you would put the Rosetta AMI.
  • --noEBS   If you do not want to be prompted for EBS volume attachment, provide this option here. Otherwise, if no volume is provided, the program will ask you to confirm.
  • --instanceList   Provide this option to list all available instances and then exit.

Command help information: (run without any input options)

Example command: 

$ awslaunch --instance=p2.8xlarge --availZone=us-west-2a --volume=vol-0c05ffa99db516ef9

Automatic mounting & formatting of new EBS volumes:  After creating a new EBS volume, the launch script will automatically detect if it needs to format the file system (which is required for new EBS volumes). This removes the requirement for users to perform these cumbersome tasks.


awskill

To terminate a running instance, you can kill the instance with the awskill command.

To do so, obtain the instance ID from awsls and then copy it alongside the awskill command:

$ awskill i-04bb96515a0f8da5e

This will do the following:

  • Unmount EBS volume
  • Detach EBS volume
  • Terminate instance

If there are any running processes or other terminals logged into the machine, awskill cannot terminate the instance. You will have to log in and kill the remaining processes, which will be output to the terminal.


aws_ebs_create

EBS volumes are similar to external hard drives - you will put your data onto EBS volumes, attach it to an instance, run analysis, and then detach.

Before booting up an instance, you need to create a new EBS volume. Be careful! It is difficult to resize an existing EBS volume, so make sure to over-estimate generously the amount of space you think you will need.

To create a volume:

$ aws_ebs_create [Size in GB] [Availability zone]"Description in double quotes"

Example command:

$ aws_ebs_create 100 us-west-2a "My data set from Jan. 2017"


aws_ebs_attach

NOTE: You do not need to use this command, as it is automatically incorporated into awslaunch. Therefore, this is an expert user option and does not require regular use.

This command will attach an EBS volume to an already running instance:

$ aws_ebs_attach [instance ID] [volume ID]

ID numbers can be obtained from awsls.

Example command:

$ aws_ebs_attach i-0f3c4e1809a416176 vol-09d64fa31c27af83d


aws_ebs_detach

NOTE: You do not need to use this command, as it is automatically incorporated into awslaunch. Therefore, this is an expert user option and does not require regular use.

This command is required to disconnect an EBS volume from an instance. First, the EBS volume is unmounted from the instance using the command (on the AWS instance):

$ sudo umount /dev/xvdf

Then, you will be able to detach the instance with the following command (from local machine):

$ aws_ebs_detach [volume ID]

Example command:

$ aws_ebs_detach vol-09d64fa31c27af83d 


aws_ebs_delete

When you are finished using an EBS volume (and all of your data has been downloaded!!), you should delete the EBS volume, since you are paying for every hour that your data is being stored on AWS.

This cannot be undone! Be careful using this command!

To delete a volume:

$ aws_ebs_delete [volume ID]

Example command:

$ aws_ebs_delete vol-09d64fa31c27af83d


awslaunch_starcluster

For CPU-based tasks (e.g. Sparx, EMAN2), you can launch a cluster of instances pre-installed with Sun Grid Engine for job management.

Given that Relion-2.0 is GPU-accelerated, we no longer use CPU instances for routine cryo-EM structural analysis. However, we wanted to make sure this option remains available.

This requires starcluster to be installed on your local machine. (Installation instructions here)

Input help options: 

$ awslaunch_cluster

Example command: 

$ awslaunch_cluster  --instance=r3.8xlarge --num=4 --availZone=us-west-2a --volume=vol-09d64fa31c27af83d --spotPrice=0.5 --relion2 


aws_spot_price_history

For almost all instances on AWS (not p2 instances), you can bid on a lower price using the Spot Market.

To see all prices for a given instance type in a certain region:

$ aws_spot_price_history [instance type] [availability zone]

Example command & output: 

$ aws_spot_price_history m4.2xlarge us-west-2a