Capstone Data Analysis Services: Micro-services Early ...

Capstone Data Analysis Services: Micro-services Early ...

Capstone Data Analysis Services: Micro-services Early Implementation (Vision and Architecture) Ramesh Baral Team: Marjani Peterson, Andre Guerrero Mentors: Eric Nienhouse, Seth McGinnis, Rich Loft 2/24/20 Vision Climate data used in research and planning Problems:

Complexity of climate data Requires expertise to work with Solution: Encapsulate expertise into re-usable modules Provision of an end-to-end server side climate data analytics system

Lowers time to solution 2 Example Use Case A city manager needs to know the number of days the temperature of Austin, TX gets above 100F, now and in the future. 3 Create workflow Set parameters

Run workflow Store workflow Non-experts Re-use workflow Fargo, ND; 2040-2070 Experts Austin, TX ; 2040-2070 Example Workflow Re-use 4 Architecture

5 Detailed Architecture 6 Workflow Patterns[1] Sequential Parallel A

B1 C Simple merge A1 A C A1 A2

B B C Structured loop [1] http://www.workflowpatterns.com 7 Workflow Spec and Engine

Problems: Workflow specs coupled to the engine (e.g.: GirderWorker, Pinball, Celery) Workflow engines o o o Client-side Non-generic Coupled with the execution scripts Solution: Simplicity o

UI to generate workflow spec Flexibility o o Make generic workflow spec Adaptor translates generic spec to selected engine 8 Example Workflow Spec { "inputs": [ {"type": "string, "name": myinputstring", "format": "text", "val": teststring"}, { "type" : "number", "name": myinputnumber", "format": "number", "val":30.25}

], "connections": [ {"input": " myinputstring ", "input_step": mystep1", "name:" step_inputstring }, {"input": " myinputnumber ", "input_step": mystep1","name:step_inputnumber} ] } 9 Workflow Adapter Purpose: Translate generic workflow spec to Workflow engine standard

Validate the workflow: Workflow inputs, outputs Workflow connections inputs Workflow connections steps 10 Microservices Independent functional units Individually accessible Separate process, and deployment

Different technology stack, and working team 11 Conclusion and Future work Microservice based climate data analytics Next: Deployment o o Automatic deployment Deployment on other cloud technologies (OpenStack) and other providers

Refine UI for non-expert users o drag and drop More microservices o o distributed across multiple VMs different technology stack (e.g.: Language) 12 References 1. 2.

3. 4. http://www.workflowpatterns.com http://flask-restful-cn.readthedocs.io http://girder-worker.readthedocs.io https://readthedocs.org/projects/celery 13 Q&A ? 14

Recently Viewed Presentations

  • Common Themes in the Development of the College

    Common Themes in the Development of the College

    AP Seminar and AP Research—and is designed . ... The AP Seminar course topics can be viewed through different disciplinary lenses which relate to courses in the AP Program. COMMON THEMES IN THE DEVELOPMENT OF COLLEGE BOARD'S COLLEGE READINESS SYSTEM...
  • 8.5.4 - Magnitude and Angles

    8.5.4 - Magnitude and Angles

    Sometimes, we will be able to discern the angle and magnitude just by looking at a particular equation. Other times, we may need to form a triangle using information about the vector, and use trig to help us . Recall...
  • Hot off the Press - Oregon Counseling

    Hot off the Press - Oregon Counseling

    Her mother. and grandmother (who is also deceased) suffered from . progressive Alzheimer's disease. My client revealed to me that . she administered a lethal dose of sleeping pills to her grandmother. during the final stages of the Alzheimer's, and...
  • Effect of Certain Agonists & Antagonists on Isolated Guinea ...

    Effect of Certain Agonists & Antagonists on Isolated Guinea ...

    Effect of Certain Agonists & their Antagonists on Isolated Guinea-pig Ileum Guinea pig ileum is a smooth muscle receive dual nerve supply from autonomic nervous system (sympathetic and parasympathetic) . Parasympathetic tone is dominant in ileum which Ach cause contraction...
  • AZA Application Using ZIMS to make it easier

    AZA Application Using ZIMS to make it easier

    The Inventory Report is one of the most commonly used reports in ZIMS. To answer this question you need to select Inventory Detail Report as it will include all animals you physically hold and own, hold but do not own,...
  • Formato Base dei Dati - UCLA

    Formato Base dei Dati - UCLA

    [Babcock' 02] B. Babcock, S. Babu, M. Datar, R. Motawani, and J. Widom. Models and issues in data stream systems. In PODS, 2002. [Bai' 06] Yijian Bai, Hetal Thakkar, Chang Luo, Haixun Wang, and Carlo Zaniolo. A data stream language...
  • Amazon.com's - California State University Channel Islands

    Amazon.com's - California State University Channel Islands

    e-book sales accounted for about 23% of publishers net. revenue for 2012. That is up from 17% in 2011. ... -Worry that e-books that go on sale the same day as regular. books will cut into their profit. -Worry that...
  • Dr Angela Kennedy Successes and struggles keeping trauma

    Dr Angela Kennedy Successes and struggles keeping trauma

    in addition, the trsut has a team to develp the pathways but used to give no suppport to implement. It was thought sufficient to write the document and then ask staff to follow it. this meeant that in reality the...