This box searches only this space. The box at the upper right searches the entire iPlant wiki.

Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

 

Summary:

Team Star's solution is to integrate SequenceServer and WorkQueue to run workers on an HPC. To do this, we initialized an Atmosphere Virtual Machine Instance with CCTools to ensure that we could utilize WorkQueue. Team blastEasy provided a SeqInit.py script which starts SequenceServer and initiates the WorkQueue master file.  Once we adjusted the script to meet our needs, we were able to establish a connection between the Virtual Machine, which acts as the master for WorkQueue, and the HPC. From this, we were able to bring up the SequenceServer GUI in a browser using an IP address and port number. 

 

Description:

Team blastEasy's integration of WorkQueue limits the amount of possible blast searches occurring at the same time. Team Star has taken this a step further and improved scalability by using the power of an HPC.

In our solution, the  Team blastEasy sequenceserver2.0 Virtual Machine Instance is used as the Master. Simply, upon starting Sequenceserver, the Master is up and running, waiting for the workers to accept jobs.

At the same time, on an HPC, a fexible .pbs file allows the HPC to be populated with workers. This .pbs can be modified to scale depending on class size and required power. Once the .pbs is submitted and the workers are up and running, the Master VM will submit jobs to the Workers on the HPC.

The beauty of this method is that it does not have limitations: although made to assist blast searches for large classrooms, it can be scaled down as one may prefer.

Team Star is thankful for the work of team blastEasy, the sequenceserver team, the CCTools team, and all the students and instructors of the UArizona ACIC course.

 

For detailed instructions (requirements, installation, running) please visit our Github page.


Concept Map:

Presentation slides:

Team_star_midterm.pptx

 

Github:

https://github.com/Midterm-Team-Star

 

Benchmarking Data:

Benchmark was performed using the team blastEasy sequenserver2.0 VM image as the Master and the HPC as Workers.

Database used: Mouse protein database (46 Mb), custom.

The Benchmarks were performed with 4 different computers simultaneously running the same query search. Each computer would have 4 tabs of sequenceserver and each query was submitted at the same time (therefore 16 simultaneous blasts). Trial times were recorded as when Sequenceserver would display the results.

Notes: more benchmarking could be made - scaling the workers available resources (cpus, ram, nodes) would have resulted in better data.  

nodes = 1 , cpus = 6 , mem = 24gb
 time (trial 1)time (trial 2)time (trial 3)time (trial 4) time (trial 5)time (trial 6)
126.3415.8346.6731.19188.07

n/a

241.134.949.0525.67152.61n/a
343.637.350.7319.19137.19n/a
447.4239.3751.3713.83119.19n/a
511.812.3521.3532.37120.21n/a
616.0425.7133.8743.01133.48n/a
717.7940.9958.8843.88141.98n/a
844.943.2759.8149.77155.33n/a
9n/a7.0716.0322.13115.5n/a
10n/a12.9744.4523.01135.13n/a
11n/a 20.446.4840.24141.54n/a
1254.0124.5955.4944.34145.91n/a
139.6723.1532.4124.63n/a192.52
1441.2623.1950.0136.27n/a 235.4
1545.6424.496242.44n/a 255
1652.3732.7365.848.2n/a261.65
Average34.7646153826.14437546.52533.760625140.5116667236.1425
S.D.16.0434839411.0247521314.1600922811.2724770819.6373900531.14328004
# Runs16161616124
Average Sec / # Runsn/a1.6340234382.90781252.11003906311.7093055659.035625
Query size3,3363,33650,00050,000200,000500,000
Q Size3,3363.3365050200500


Milestones:

1- successful implementation of the master-worker setup within the same machine (tested with work_queue_example.py)
2- successful implementation of the master-worker setup between two virtual machines (tested with work_queue_example.py)
3- successful implementation of the master-worker setup between a virtual machine and the HPC  (tested with work_queue_example.py)
4- successfully ran blast search between master virtual machine and worker HPC
5- successful implementation of the master-worker setup between the blastEasy sequenceserver virtual machine and the HPC


Team member contributions:

Michele Cosi: Terminal, Documentation, PBS script

Emmanuel Gonzalez: Presentation, Terminal, PBS script

Anthony Dominguez: Coding, Terminal, PBS script

TJ Lippincott: Coding, Terminal, PBS script

Brandi Diesso: Presentation, Documentation


Test Sequence:

https://github.com/Midterm-Team-Star/SS_HPC_Integration_Scripts/blob/master/Test/Mouse_real_gene_3kb.fa


Post-Mortem Analysis:


The Good: pride in the final product; learning challenge.

The Bad: Communication. Need more communication between all parties.

The Ugly: Timing. Should have started earlier.

What to do different: Communicate. Listen. Start earlier.