Capstone Data Analysis Services: Micro-services Early ...

Capstone Data Analysis Services: Micro-services Early Implementation (Vision and Architecture) Ramesh Baral Team: Marjani Peterson, Andre Guerrero Mentors: Eric Nienhouse, Seth McGinnis, Rich Loft 2/24/20 Vision Climate data used in research and planning Problems:

Complexity of climate data Requires expertise to work with Solution: Encapsulate expertise into re-usable modules Provision of an end-to-end server side climate data analytics system

Lowers time to solution 2 Example Use Case A city manager needs to know the number of days the temperature of Austin, TX gets above 100F, now and in the future. 3 Create workflow Set parameters

Run workflow Store workflow Non-experts Re-use workflow Fargo, ND; 2040-2070 Experts Austin, TX ; 2040-2070 Example Workflow Re-use 4 Architecture

5 Detailed Architecture 6 Workflow Patterns[1] Sequential Parallel A

B1 C Simple merge A1 A C A1 A2

B B C Structured loop [1] http://www.workflowpatterns.com 7 Workflow Spec and Engine

Problems: Workflow specs coupled to the engine (e.g.: GirderWorker, Pinball, Celery) Workflow engines o o o Client-side Non-generic Coupled with the execution scripts Solution: Simplicity o

UI to generate workflow spec Flexibility o o Make generic workflow spec Adaptor translates generic spec to selected engine 8 Example Workflow Spec { "inputs": [ {"type": "string, "name": myinputstring", "format": "text", "val": teststring"}, { "type" : "number", "name": myinputnumber", "format": "number", "val":30.25}

], "connections": [ {"input": " myinputstring ", "input_step": mystep1", "name:" step_inputstring }, {"input": " myinputnumber ", "input_step": mystep1","name:step_inputnumber} ] } 9 Workflow Adapter Purpose: Translate generic workflow spec to Workflow engine standard

Validate the workflow: Workflow inputs, outputs Workflow connections inputs Workflow connections steps 10 Microservices Independent functional units Individually accessible Separate process, and deployment

Different technology stack, and working team 11 Conclusion and Future work Microservice based climate data analytics Next: Deployment o o Automatic deployment Deployment on other cloud technologies (OpenStack) and other providers

Refine UI for non-expert users o drag and drop More microservices o o distributed across multiple VMs different technology stack (e.g.: Language) 12 References 1. 2.

3. 4. http://www.workflowpatterns.com http://flask-restful-cn.readthedocs.io http://girder-worker.readthedocs.io https://readthedocs.org/projects/celery 13 Q&A ? 14