This space is home to learning materials and tutorials created for CyVerse products and services. To search the entire CyVerse wiki, use the box at the upper right.


LEARNING MATERIALS
 

 

 

 

Skip to end of metadata
Go to start of metadata

Jetstream Workflow That Creates Gene Expression Matrices (GEMs) from SRA/FASTQ NGS Files

Rationale and background

Pegasus-GEM is Pegasus Jetstream workflow that utilizes Jetstream resources to produce a Gene Expression Matrix (GEM) from DNA sequence files in FASTQ format. This is adapted from OSG-GEM workflow that currently runs on Open Science Grid (OSG)

William L. Poehlman, Mats Rynge, Chris Branton, D. Balamurugan and Frank A. Feltus. OSG-GEM: Gene Expression Matrix Construction Using the Open Science Grid. Bioinformatics and Biology Insights 2016:10 133–141 doi: 10.4137/BBI.S38193.

Introduction

This workflow processes both paired or single end FASTQ files to produce a matrix of normalized RNA molecule counts (FPKM). Pegasus-GEM also supports direct input downloads from NCBI SRA for processing. An indexed reference genome along with gene model annotation files must be obtained prior to configuring and running the workflow. The following tasks are directed by the Pegasus workflow manager:

  • Splitting input FASTQ files into files containing 20,000 sequences each.
  • Trimming raw sequences with Trimmomatic
  • Aligning reads to the reference genome using Hisat2 or Tophat2 or STAR
  • Merging BAM alignment files into a single sorted BAM file using Samtools merge
  • Quantifying RNA transcript levels using StringTie or Cufflinks

It is suggested that the user become familiar with the documentation associated with the following software packages:

This tutorial will take users through steps of:

  1. Running Pegasus-GEM on Jetstream cloud
  2. Running Pegasus-GEM on an example data

Considerations

Sounds great, what do I need to get started?

  1. XSEDE account
  2. XSEDE allocation. Users can request a startup XSEDE allocation. The startup XSEDE allocations are relatively easy to apply and get allotted. Please contact help@xsede.org for further information.
  3. Getting started with Jetstream
  4. Your data (or you can run example data)

What kind of data do I need?

  1. Mandatory requirements
    1. FASTQ files. Either single end or paired. You can also use SRR id's
    2. Select the type of reads. Single or Paired.
    3. Select the type of mapper (Tophat2 or Hisat2 or STAR)
    4. Select the type of assembler (Cufflinks or Stringtie)

What kind of resources will I need for my project?

  1. Enough SUs (Service Units) to run your computation
  2. One MASTER and several workers needed for running your computation
  3. Enough storage space on the WQ-MAKER Jetstream instance for both input and output files. Since most of the images have limited disk space, it is recommended to mount an external volume to the running Pegasus-GEM MASTER instance. 

Part 1: Connect to an instance of an Pegasus-GEM 2.0 Jetstream Image

Step 1. Go to https://use.jetstream-cloud.org/application and log in with your XSEDE credentials.

 

Step 2. Click on the "Create New Project"  in the Project tab on the top and enter the name of the project and a brief description 

Step 3. Launch an instance from the selected image and name it as MASTER 

After the project has been created and entered inside it, click the "New" button, select "Pegasus-GEM" image and then click Launch instance. In the next window (Basic Info),

  • name the instance as "MASTER" (don't worry if you forgot to name the instance at that point, as you can always modify the name of the instance later)
  • leave base image version as it is
  • leave the project name as it is or change to a different project if needed
  • select "Jetstream - Indiana University or Jetstream - TACC" as Provider (for this tutorial we will chose Jetstream - TACC) and click 'Continue'. Your choice of provider will depend on the resources you have available (SUs) and the needs of your instance
  • select "m1.medium" as Instance size and click "Continue". 
     
Icon

Make sure to check on the Projected Resources Usage to make sure you have enough resources to run the instance. If you need more SUs or CPUs to run instances, contact the Jetstream team at help@xsede.org. Review the details of the instance you are selecting to launch and click "Launch Instance"

Icon

As the instance is launched behind the scenes, you will get an update as it goes through each step.

Status updates of Instance launch (both MASTER and WORKER) include Build-requesting launch, Build-networking, Build-spawning, Active-networking, Active-deploying. Depending on the usage load on Jetstream, it can take anywhere from 2-5 mins for an instance to become active. You can force check updates by using the refresh button in the Instance launch page or the refresh button on your browser. Once the instance becomes active a virtual machine with the ip address provided will become available for you to connect to. This virtual machine will have all the necessary components to run WQ-MAKER and test files to run a MAKER demo. 

Step 4. Launch WORKER instance from  "Pegasus-GEM" image 

Launch two more instances from the  "Pegasus-GEM" image and name them as WORKER-1 and WORKER-2. Use the same configuration as the MASTER VM 

Step 5. Create and attach the volume to the MASTER instance

Since the m1 medium instance size (60GB disk space) selected for running MASTER instance of Pegasus-GEM may not be sufficient for most of the GEM runs, it is recommended to run it on volumes. Following are the three steps for mounting the external volume.

5.1: Create a volume

Click the "New" button in the project and select "Create Volume". Enter the name of the volume, volume size (GB) needed and the provider (TACC or Indiana) and finally click "Create Volume" 

5.2: Attach the created volume to the MASTER instance 

Click the checkbox by the side of Volume (1), then click the attach button (2), next chose the MASTER instance (3) and finally click the ATTACH VOLUME TO INSTANCE button (4)

Step 6. ssh into the MASTER

6.1 Add ssh keys on your computer

6.2 ssh key forwarding to the MASTER

Part 3: Set up a Pegasus-GEM 2.0 run using the Terminal window

Step 1. Get oriented. A workflow specific ssh key has to be created. This key is used for some of the data staging steps of the workflow.

Step 2. Copy the example data onto your home directory, change the permissions and navigate to that directory. 

List the contents using 'ls' command

List the contents in the Test_data folder

The example data contains an config file (osg-gem.conf) as well as input files from the 21st chromosome of Gencode Release 24 of the GRCh38 build of the human reference genome. Two small FASTQ files containing 200,000 sequences from NCBI dataset SRR1825962 lie within the Test_data directory of the example data folder

Step 3. Setting up the VM and password on the MASTER

Step 4. Setting up the VM and password on the WORKERS
4.1 Creat ansible.cfg file into your home directory which will help you to avoid host verification

4.2 Edit osg-hosts file and populate it with ip addresses of the WORKERS

4.2 Run Ansible playbook (worker-launch.yml) to add WORKERS to condor pool

4.3 Run condor_status command to see how many VMs are present in condor pool

Step 5.  Run Pegasus-GEM

5.1 Take a look at the config file first

5.2 Run default test data . The workflow, configured to run Hisat2 and StringTie, can then be launched by running:

Monitoring Workflow: Pegasus provides a set of commands to monitor workflow progress. The path to the workflow files as well as commands to monitor the workflow will print to screen upon submitting the workflow. For example:

Icon

$ pegasus-status -l /home/upendra/Pegasus-GEM_example_data/runs/osg-gem-1510267501/workflow/osg-gem-1510267501

STAT IN_STATE JOB
Run 04:21 gem-0 ( /home/upendra/Pegasus-GEM_example_data/runs/osg-gem-1510267501/workflow/osg-gem-1510267501 )
Run 03:05 ??subdax_level-2_ID0000004
Run 01:58 ??hisat2_ID0000003
Run 01:58 ??hisat2_ID0000007
Run 01:58 ??hisat2_ID0000006
Summary: 5 Condor jobs total (R:5)

UNRDY READY PRE IN_Q POST DONE FAIL %DONE STATE DAGNAME
6 0 0 3 0 12 0 57.1 Running 00/00/level-2/level-2.dag
2 0 0 1 0 4 0 57.1 Running *gem-0.dag
8 0 0 4 0 16 0 57.1 TOTALS (28 jobs)
Summary: 2 DAGs total (Running:2)

Make sure the run has finished by running the same command

You look at the statistics of the run using pegasus-stastics command

Output will be transferred at the base of this directory upon completion. For example this message tells you where the ouputs are

"An 'Output's directory will be created within the base of the workflow directory.

This directory, /home/upendra/Pegasus-GEM_example_data/runs/osg-gem-1510267501/outputs

Icon

From here, the user may follow our documentation to modify the software options as well as point to their own input datasets. Note that there are no test reference genome indices available for STAR, because they are too large to upload to github.

Pre-Workflow User Input

The user must provide indexed reference genome files as well as gene model annotation information prior to submitting the workflow. The user must select a reference prefix($REF_PREFIX) that will be recognized by Pegasus as well as by Hisat2 or Tophat2. In addition, information about splice sites or a reference transcriptome must be provided in order to guide accurate mapping of split input files. Once the user has downloaded a reference genome fasta file and gene annotation in GTF/GFF3 format, the following commands can be used to produce the necessary input files, using GRCh38 as an example $REF_PREFIX for Gencode Release 24 of the human reference genome:

Hisat2:

  • Index the reference genome

  • Generate Tab delimited list of splice sites using gene model GTF file as input (Python DefaultDictionary Module necessary)

Tophat2:

  • Index the reference genome

  • Generate and Index Reference Transcriptome


STAR:

  • Generate Tab delimited list of splice sites using gene model GTF file as input (Python DefaultDictionary Module necessary)

Note

Icon

Please contact upendra@cyverse.org if you need help determining the optimal amount of resources for running pegasus-GEM. 

  • No labels