Integrating Scale Out and Fault Tolerance in Stream Processing using Operator State Management with Raul Castro Fernandez* Matteo Migliavacca+ and Peter Pietzuch* * Imperial College London, +Kent Univerisity Peter R. Pietzuch [email protected] Big data in numbers: 2.5 billions on gigabytes of data every day LSST telescope, Chile 2016, 30 TB nightly (source IBM) come from everywhere: web feeds, social networking mobile devices, sensors, cameras scientific instruments online transactions (public and private sectors)
have value: Global Pulse forum for detecting human crises internationally real-time big data analytics in UK 25 billions 216 billions in 2012-17 recommendation applications (LinkedIn, Amazon) processing infrastructure for big data analysis 2 A black-box approach for big data analysis users issue analysis queries with real-time semantics streams of data updates, time-varying rates, generated in realtime streams of result data processing in near real-time Stream Processin g System time 3 Distributed Stream Processing System
queries consist of operators (join, map, select, ..., UDOs) operators form graphs operators process streams of tuples on-the-fly operators span nodes 4 Elastic DSPSs in the Cloud Real-time big data analysis challenge traditional DSPS: ? what about continuous workload surges? ? what about real-time resource allocation to workload variations? ? keeping the state correct forstateful operators? Massively scalable , cloud-based DSPSs [SIGMOD 2013] 1. gracefully handles stateful operators state 2. operator state management for combined scale out and fault tolerance 3. SEEP system and evaluation 4. related work 5. future research directions
5 Stream Processing in the Cloud clouds provide infinite pools of resources ? How do we build a stream processing platform in the Cloud Intra-query parallelism: Failure resilience: provisioning for workload peaks active fault-tolerance needs 2x resou unnecessarily conservative passive fault-tolerance leads to long recovery times dynamic scale out: increase resources when peaks appear hybrid fault-tolerance: low resource overhead with fast recovery Both mechanisms must support stateful operators 6 Stateless vs Stateful Operators operator state: a summary of past tuples processing
wit the h wit h counte r (with, ) counte counte rr stateless: failure recovery scale out (the, 2) !=12 10) (the, (with, 1) !=6 (with, 5)
stateful: failure recovery scale out 7 State Management processing state: (summary of past tuples processing) routing state: (routing of tuples) buffer state: (tuples) B A C operator state is an external entity managed by the DSPS primitives for state management mechanisms (scale out, failure recovery) on top of primitives dynamic reconfiguration of the dataflow graph 8 State Management Primitives checkpoint takes snapshot of state and makes it externally available
backup moves copy of state from one operator to another restore partition A A B splits state in a semantically correct fashion for parallel processing B 9 State Management Scale Out, Stateful Ops backup node already has state of operator to be parallelised backup periodically, stateful operators checkpoint and back up state to designated upstream
backup node, in memory checkpoint A A A A B le a sc A t ou A A restore A partition
ator r e p o w ne A upstream ops send unprocessed tuples to update checkpointed state A How do we partition stateful 10 Partitioning Stateful Operators 1. Processing state modeled as (key, value) dictionary 2. State partitioned according to key k of tuples 3. Tuples will be routed to correct operator as of k buffer state t=1, key=c, computer
t=3, key=c, cambridge t=1, computer t=2, laboratory splitter t=3, cambridge (a k), A routing (l z), A state t=2, key=l, laboratory counter processing state t=3, (c, computer:1, cambridge:1) A counter A t=2, (l, laboratory:1) 11
Passive Fault-Tolerance Model recreate operator state by replaying tuples after failure: upstream backup: sends acks upstream for tuples processed downstream ACKs A data a C B b D c d may result in long recovery times due to large buffers: system is reprocessing streams after failure inefficient 12 Recovering using State Management
(R+SM) Benefit from state management primitives: use periodically backed up state on upstream node to recover faster trim buffers at backup node same primitives as in scale out A A g it n u ch ro e te g a an st A n ce a t s n
i w ne A A A A state is restored and unprocessed tuples are replayed from buffer same primitives for parallel recovery 13 State Management in Action: SEEP 1) dynamic Scale Out: detect bottleneck , add new parallelised operat (2) failure Recovery: detect failure, replace with new operator queries query manager
EC2 VM pool stats scale out coordinator bottleneck detector scaling policy (1) faults deployment manager recovery coordinator fault detector (2) 14 Dynamic Scale Out: Detecting bottlenecks
85% CPU utilisation report 35% bottleneck detector 30% 35% 85% 30% logical infrastructure view 15 The VM Pool: Adding operators problem: allocating new VMs takes minutes... monitoring information bottleneck fault detector VM detector 2 virtual machine pool
VM 1 VM 2 add new VM to pool Cloud provider VM 3 (dynamic pool size) provision VM from cloud (order of mins) 16 Experimental Evaluation Goals: investigate effectiveness of scale out mechanism recovery time after failure using R+SM overhead of state management Scalable and Elastic Event Processing (SEEP): implemented in Java; Storm-like data flow model Sample queries + workload Linear Road Benchmark (LRB) to evaluate scale out
[VLDB04] provides an increasing stream workload over time query with 8 operators, 3 are stateful; SLA: results < 5 secs Windowed word count query (2 ops) to evaluate fault tolerance induce failure to observe performance impact Deployment on Amazon AWS EC2 sources and sinks on high-memory double extra large instances operators on small instances 17 Scale Out: LRB Workload scales to load factor L=350 with 50 VMs on Amazon EC2 (automated query parallelisation, scale out policy at 70%) L=512 highest result [VLDB12] (hand-crafted on cluster) scale out leadsquery to latency peaks
but remains within LRB SLA SEEP scales out to increasing workload in the Linear Road Benchmark 18 Conclusions Stream processing will grow in importance: handling the data deluge enables real-time response and decision making Integrated approach for scale out and failure recovery: operator state an independent entity primitives and mechanisms Efficient approach extensible for additional operators: effectively applied to Amazon EC2 running LRB parallel recovery 19
Interprocess Communication CSE 380 Lecture Note 8 Insup Lee Interprocess communication Shared Memory Message Passing Signals Shard Memory Shared Memory in Solaris Processes can share the same segment of memory directly when it is mapped into the address space of...
Misleading Graphs and Statistics PS.1.AC.5: Interpret and evaluate, with and without appropriate technology, graphical and tabular data displays for consistency with the data, appropriateness of type of graph or data display, scale, and overall message.
Chemical Reactions Glencoe Chapter 23 I. Chemical reactions change substances Substances undergo chemical changes to become new substances A. Signs of a chemical reaction 1. Production of gas or smoke 2. Change in color (not like painting a fence) 3....
Summary Physical attacks are becoming more prevalent DRM, software licensing, distributed computing, etc. Single-chip secure processors provide trusted execution environments with acceptable overhead Tamper-Evident environment, Private Tamper-Resistant environment Simulation results show 5-15% overhead for TE, 10-25% overhead for PTR processing...
to tanner and glove maker, John Shakespeare and his wife, Mary (Arden). His actual birthday is unknown but assumed to be April the 23 rd . His baptism was recorded in the Parish register of Holy Trinity Church on April...
Did Shakespeare write the 37 plays and 154 sonnets credited to him? The evidence above proves William existed but not that he was a playwright nor an actor nor a poet. In fact recently some academics who call themselves the...
Induction « Opéron Lactose » Synthèse des enzymes nécessaires au métabolisme du lactose Chez les procaryotes Répression « Opéron Tryptophane » Synthèse des enzymes impliquées dans la biosynthèse du tryptophane Structure de l'opéron « lac » L'opéron lactose d'Escherichia coli...
Ready to download the document? Go ahead and hit continue!