Characteristics of Big Data Applications and Scalable Software

Characteristics of Big Data Applications and Scalable Software

Characteristics of Big Data Applications and Scalable Software Systems Chesapeake Large-Scale Analytics Conference Loews Annapolis Hotel Annapolis October 16 2014 Geoffrey Fox [email protected] School of Informatics and Computing Digital Science Center Indiana University Bloomington HPC and Data Analytics

Identify/develop parallel large scale data analytics data analytics library SPIDAL (Scalable Parallel Interoperable Data Analytics Library ) of similar quality to PETSc and ScaLAPACK which have been very influential in success of HPC for simulations Analyze Big Data applications to identify analytics needed and generate benchmark applications and characteristics (Ogres with facets) Analyze existing analytics libraries (in practice limit to some application domains and some general libraries) catalog library members available and performance Apache Mahout low performance and not many entries; R largely sequential and missing key algorithms; Apache MLlib just starting Identify range of big data computer architectures Analyze Big Data Software and identify software model HPC-ABDS (HPC Apache Big Data Stack) to allow interoperability (Cloud/HPC) and high performance merging HPC and commodity cloud software Design or identify new or existing algorithms including and assuming parallel implementation Many more data scientists than computational scientists so HPC implications of data analytics could be influential on simulation software and hardware Analytics and the DIKW Pipeline Data goes through a pipeline Raw data Data Information Knowledge Wisdom Decisions Information

Data Analytics Knowledge Information More Analytics Each link enabled by a filter which is business logic or analytics We are interested in filters that involve sophisticated analytics which require non trivial parallel algorithms Improve state of art in both algorithm quality and (parallel) performance See Google Cloud Dataflow supporting pipelined analytics NIST Big Data Initiative Led by Chaitin Baru, Bob Marcus, Wo Chang NBD-PWG (NIST Big Data Public Working Group) Subgroups & Co-Chairs There were 5 Subgroups Note mainly industry Requirements and Use Cases Sub Group

Geoffrey Fox, Indiana U.; Joe Paiva, VA; Tsegereda Beyene, Cisco Definitions and Taxonomies SG Nancy Grady, SAIC; Natasha Balac, SDSC; Eugene Luster, R2AD Reference Architecture Sub Group Orit Levin, Microsoft; James Ketner, AT&T; Don Krapohl, Augmented Intelligence Security and Privacy Sub Group Arnab Roy, CSA/Fujitsu Nancy Landreville, U. MD Akhil Manchanda, GE Technology Roadmap Sub Group Carl Buffington, Vistronix; Dan McClary, Oracle; David Boyd, Data Tactics See And 5 Use Case Template 26 fields completed for 51 areas Government Operation: 4 Commercial: 8 Defense: 3

Healthcare and Life Sciences: 10 Deep Learning and Social Media: 6 The Ecosystem for Research: 4 Astronomy and Physics: 5 Earth, Environmental and Polar Science: 10 Energy: 1 6 51 Detailed Use Cases: Contributed July-September 2013 Covers goals, data features such as 3 Vs, software, hardware

26 Features for each use case (Section 5) Biased to science Government Operation(4): National Archives and Records Administration, Census Bureau Commercial(8): Finance in Cloud, Cloud Backup, Mendeley (Citations), Netflix, Web Search, Digital Materials, Cargo shipping (as in UPS) Defense(3): Sensors, Image surveillance, Situation Assessment Healthcare and Life Sciences(10): Medical records, Graph and Probabilistic analysis, Pathology, Bioimaging, Genomics, Epidemiology, People Activity models, Biodiversity Deep Learning and Social Media(6): Driving Car, Geolocate images/cameras, Twitter, Crowd Sourcing, Network Science, NIST benchmark datasets The Ecosystem for Research(4): Metadata, Collaboration, Language Translation, Light source experiments Astronomy and Physics(5): Sky Surveys including comparison to simulation, Large Hadron Collider at CERN, Belle Accelerator II in Japan Earth, Environmental and Polar Science(10): Radar Scattering in Atmosphere, Earthquake, Ocean, Earth Observation, Ice sheet Radar scattering, Earth radar mapping, Climate simulation datasets, Atmospheric turbulence identification, Subsurface Biogeochemistry (microbes to watersheds), AmeriFlux and FLUXNET gas sensors 7 Energy(1): Smart grid Application Example

Montage Table 4: Characteristics of 6 Distributed Applications Execution Unit Communication Coordination Execution Environment Multiple sequential and parallel executable Multiple concurrent parallel executables Multiple seq. and parallel executables Files Pub/sub Dataflow and events Climate Prediction (generation) Climate Prediction (analysis)

SCOOP Multiple seq. & parallel executables Files and messages Multiple seq. & parallel executables Files and messages MasterWorker, events Dataflow Coupled Fusion Multiple executable NEKTAR ReplicaExchange

Multiple Executable Stream based Files and messages Stream-based Dataflow (DAG) Dataflow Dataflow Dataflow Dynamic process creation, execution Co-scheduling, data streaming, async. I/O Decoupled coordination and messaging @Home (BOINC) Dynamics process

creation, workflow execution Preemptive scheduling, reservations Co-scheduling, data streaming, async I/O Part of Property Summary Table 8 Big Data Patterns the Ogres HPC Benchmark Classics Linpack or HPL: Parallel LU factorization for solution of linear equations NPB version 1: Mainly classic HPC solver kernels MG: Multigrid CG: Conjugate Gradient FT: Fast Fourier Transform IS: Integer sort EP: Embarrassingly Parallel BT: Block Tridiagonal SP: Scalar Pentadiagonal LU: Lower-Upper symmetric Gauss Seidel

13 Berkeley Dwarfs Dense Linear Algebra First 6 of these correspond to Sparse Linear Algebra Colellas original. Monte Carlo dropped. Spectral Methods N-body methods are a subset of N-Body Methods Particle in Colella. Structured Grids

Unstructured Grids Note a little inconsistent in that MapReduce is a programming MapReduce model and spectral method is a Combinational Logic numerical method. Graph Traversal Need multiple facets! Dynamic Programming Backtrack and Branch-and-Bound Graphical Models Finite State Machines 7 Computational Giants of NRC Massive Data Analysis Report 1) 2) 3) 4) 5) 6) 7) G1:

G2: G3: G4: G5: G6: G7: Basic Statistics (see MRStat later) Generalized N-Body Problems Graph-Theoretic Computations Linear Algebraic Computations Optimizations e.g. Linear Programming Integration e.g. LDA and other GML Alignment Problems e.g. BLAST 51 Use Cases: What is Parallelism Over? People: either the users (but see below) or subjects of application and often both Decision makers like researchers or doctors (users of application) Items such as Images, EMR, Sequences below; observations or contents of online store

Images or Electronic Information nuggets EMR: Electronic Medical Records (often similar to people parallelism) Protein or Gene Sequences; Material properties, Manufactured Object specifications, etc., in custom dataset Modelled entities like vehicles and people Sensors Internet of Things Events such as detected anomalies in telescope or credit card data or atmosphere (Complex) Nodes in RDF Graph Simple nodes as in a learning network Tweets, Blogs, Documents, Web Pages, etc. And characters/words in them

Files or data to be backed up, moved or assigned metadata Particles/cells/mesh points as in parallel simulations 13 Features of 51 Use Cases I PP (26) All Pleasingly Parallel or Map Only MR (18) Classic MapReduce MR (add MRStat below for full count) MRStat (7) Simple version of MR where key computations are simple reduction as found in statistical averages such as histograms and averages MRIter (23) Iterative MapReduce or MPI (Spark, Twister) Graph (9) Complex graph data structure needed in analysis Fusion (11) Integrate diverse data to aid discovery/decision making; could involve sophisticated algorithms or could just be a portal Streaming (41) Some data comes in incrementally and is processed this way Classify (30) Classification: divide data into categories S/Q (12) Index, Search and Query Features of 51 Use Cases II CF (4) Collaborative Filtering for recommender engines LML (36) Local Machine Learning (Independent for each parallel entity) application could have GML as well GML (23) Global Machine Learning: Deep Learning, Clustering, LDA,

PLSI, MDS, Large Scale Optimizations as in Variational Bayes, MCMC, Lifted Belief Propagation, Stochastic Gradient Descent, L-BFGS, Levenberg-Marquardt . Can call EGO or Exascale Global Optimization with scalable parallel algorithm Workflow (51) Universal GIS (16) Geotagged data and often displayed in ESRI, Microsoft Virtual Earth, Google Earth, GeoServer etc. HPC (5) Classic large-scale simulation of cosmos, materials, etc. generating (visualization) data Agent (2) Simulations of models of data-defined macroscopic entities represented as agents Global Machine Learning aka EGO Exascale Global Optimization Typically maximum likelihood or 2 with a sum over the N data items documents, sequences, items to be sold, images etc. and often links (point-pairs). Usually its a sum of positive numbers as in least squares Covering clustering/community detection, mixture models, topic determination, Multidimensional scaling, (Deep) Learning Networks PageRank is just parallel linear algebra Note many Mahout algorithms are sequential partly as MapReduce limited; partly because parallelism unclear

MLLib (Spark based) better SVM and Hidden Markov Models do not use large scale parallelization in practice? Some overlap/confusion with with graph analytics 13 Image-based Use Cases 13-15 Military Sensor Data Analysis/ Intelligence PP, LML, GIS, MR 7:Pathology Imaging/ Digital Pathology: PP, LML, MR for search becoming terabyte 3D images, Global Classification 18&35: Computational Bioimaging (Light Sources): PP, LML Also materials 26: Large-scale Deep Learning: GML Stanford ran 10 million images and 11 billion parameters on a 64 GPU HPC; vision (drive car), speech, and Natural Language Processing 27: Organizing large-scale, unstructured collections of photos: GML Fit position and camera direction to assemble 3D photo ensemble 36: Catalina Real-Time Transient Synoptic Sky Survey (CRTS): PP, LML followed by classification of events (GML) 43: Radar Data Analysis for CReSIS Remote Sensing of Ice Sheets: PP, LML to identify glacier beds; GML for full ice-sheet 44: UAVSAR Data Processing, Data Product Delivery, and Data Services: PP to find slippage from radar images 45, 46: Analysis of Simulation visualizations: PP LML ?GML find paths, classify orbits, classify patterns that signal earthquakes, instabilities, climate, turbulence

Internet of Things and Streaming Apps It is projected that there will be 24 (Mobile Industry Group) to 50 (Cisco) billion devices on the Internet by 2020. The cloud natural controller of and resource provider for the Internet of Things. Smart phones/watches, Wearable devices (Smart People), Intelligent River Smart Homes and Grid and Ubiquitous Cities, Robotics. Majority of use cases are streaming experimental science gathers data in a stream sometimes batched as in a field trip. Below is sample 10: Cargo Shipping Tracking as in UPS, Fedex PP GIS LML 13: Large Scale Geospatial Analysis and Visualization PP GIS LML 28: Truthy: Information diffusion research from Twitter Data PP MR for Search, GML for community determination 39: Particle Physics: Analysis of LHC Large Hadron Collider Data: Discovery of Higgs particle PP Local Processing Global statistics 50: DOE-BER AmeriFlux and FLUXNET Networks PP GIS LML 18 51: Consumption forecasting in Smart Grids PP GIS LML Big Data Ogres Facets I: These features (PP, MR, MRStat, MRIter, Graph, Fusion, Streaming, Classify, S/Q, CF, LML, GML, Workflow, GIS, HPC, Agents) plus some broad features familiar from past like BSP? (Bulk

Synchronous Processing), SPMD?, iterative?, irregular?, dynamic?, communication/compute, I-O/compute, Data abstraction (array, key-value) Facets II: Data source and access (see later) Kernels (generalized analytics): see later System Architecture 4 Forms of MapReduce (1) Map Only (2) Classic MapReduce Input Input (3) Iterative Map Reduce (4) Point to Point or or Map-Collective Map-Communication Input map

map map Iterations Local reduce reduce Output Graph PP MR MRStat BLAST Analysis Local Machine Learning Pleasingly Parallel High Energy Physics

(HEP) Histograms Distributed search Recommender Engines MRIter Expectation maximization Clustering e.g. K-means Linear Algebra, PageRank MapReduce and Iterative Extensions (Spark, Twister) Graph, HPC Classic MPI PDE Solvers and Particle Dynamics Graph Problems MPI, Giraph Integrated Systems such as Hadoop + Harp with Compute and Communication model separated Correspond to First 4 Big Data Architectures

Useful Set of Analytics Architectures Pleasingly Parallel: including local machine learning as in parallel over images and apply image processing to each image - Hadoop could be used but many other HTC, Many task tools Classic MapReduce including search, collaborative filtering and motif finding implemented using Hadoop etc. Map-Collective or Iterative MapReduce using Collective Communication (clustering) Hadoop with Harp, Spark .. Map-Communication or Iterative Giraph: (MapReduce) with pointto-point communication (most graph algorithms such as maximum clique, connected component, finding diameter, community detection) Vary in difficulty of finding partitioning (classic parallel load balancing) Large and Shared memory: thread-based (event driven) graph algorithms (shortest path, Betweenness centrality) and Large Ideas like workflow are orthogonal to this memory applications HPC-ABDS Integrating High Performance Computing with Apache Big Data Stack Shantenu Jha, Judy Qiu, Andre Luckow Kaleidoscope of (Apache) Big Data Stack (ABDS) and HPC Technologies October 10 2014

Cross-Cutting Functionalities 1) Message and Data Protocols: Avro, Thrift, Protobuf 2)Distributed Coordination: Zookeeper, Giraffe, JGroups 3)Security & Privacy: InCommon, OpenStack Keystone, LDAP, Sentry 4)Monitoring: Ambari, Ganglia, Nagios, Inca 17 layers ~200 Software Packages

17)Workflow-Orchestration: Oozie, ODE, ActiveBPEL, Airavata, OODT (Tools), Pegasus, Kepler, Swift, Taverna, Triana, Trident, BioKepler, Galaxy, IPython, Dryad, Naiad, Tez, Google FlumeJava, Crunch, Cascading, Scalding, eScience Central, 16)Application and Analytics: Mahout , MLlib , MLbase, DataFu, mlpy, scikit-learn, CompLearn, Caffe, R, Bioconductor, ImageJ, pbdR, Scalapack, PetSc, Azure Machine Learning, Google Prediction API, Google Translation API, Torch, Theano, H2O, Google Fusion Tables 15)High level Programming: Kite, Hive, HCatalog, Tajo, Pig, Phoenix, Shark, MRQL, Impala, Presto, Sawzall, Drill, Google BigQuery (Dremel), Google Cloud DataFlow, Summingbird, Google App Engine, Red Hat OpenShift 14A)Basic Programming model and runtime, SPMD, Streaming, MapReduce: Hadoop, Spark, Twister, Stratosphere, Reef, Hama, Giraph, Pregel, Pegasus 14B)Streaming: Storm, S4, Samza, Google MillWheel, Amazon Kinesis 13)Inter process communication Collectives, point-to-point, publish-subscribe: Harp, MPI, Netty, ZeroMQ, ActiveMQ, RabbitMQ, QPid, Kafka, Kestrel, JMS, AMQP, Stomp, MQTT Public Cloud: Amazon SNS, Google Pub Sub, Azure Queues 12)In-memory databases/caches: Gora (general object from NoSQL), Memcached, Redis (key value), Hazelcast, Ehcache 12)Object-relational mapping: Hibernate, OpenJPA, EclipseLink, DataNucleus and ODBC/JDBC 12)Extraction Tools: UIMA, Tika 11C)SQL: Oracle, DB2, SQL Server, SQLite, MySQL, PostgreSQL, SciDB, Apache Derby, Google Cloud SQL, Azure SQL, Amazon RDS 11B)NoSQL: HBase, Accumulo, Cassandra, Solandra, MongoDB, CouchDB, Lucene, Solr, Berkeley DB, Riak, Voldemort. Neo4J, Yarcdata, Jena, Sesame, AllegroGraph, RYA, Espresso Public Cloud: Azure Table, Amazon Dynamo, Google DataStore 11A)File management: iRODS, NetCDF, CDF, HDF, OPeNDAP, FITS, RCFile, ORC, Parquet 10)Data Transport: BitTorrent, HTTP, FTP, SSH, Globus Online (GridFTP), Flume, Sqoop 9)Cluster Resource Management: Mesos, Yarn, Helix, Llama, Celery, HTCondor, SGE, OpenPBS, Moab, Slurm, Torque,

Google Omega, Facebook Corona 8)File systems: HDFS, Swift, Cinder, Ceph, FUSE, Gluster, Lustre, GPFS, GFFS Public Cloud: Amazon S3, Azure Blob, Google Cloud Storage 7)Interoperability: Whirr, JClouds, OCCI, CDMI, Libcloud,, TOSCA, Libvirt 6)DevOps: Docker, Puppet, Chef, Ansible, Boto, Cobbler, Xcat, Razor, CloudMesh, Heat, Juju, Foreman, Rocks, Cisco Intelligent Automation for Cloud 5)IaaS Management from HPC to hypervisors: Xen, KVM, Hyper-V, VirtualBox, OpenVZ, LXC, Linux-Vserver, VMware ESXi, vSphere, OpenStack, OpenNebula, Eucalyptus, Nimbus, CloudStack, VMware vCloud, Amazon, Azure, Google and other public Clouds, Networking: Google Cloud DNS, Amazon Route 53 1) 2) 3) 4) 5) 6) 7) 8) 9) 10) 11) 12) 13) 14)

15) 16) 17) HPC-ABDS Layers Message Protocols Distributed Coordination: Security & Privacy: Monitoring: IaaS Management from HPC to hypervisors: DevOps: Here are 17 functionalities. Interoperability: File systems: 4 Cross cutting at top Cluster Resource Management: 13 in order of layered diagram starting Data Transport: at bottom SQL / NoSQL / File management: In-memory databases&caches / Object-relational mapping / Extraction Tools Inter process communication Collectives, point-to-point, publish-subscribe Basic Programming model and runtime, SPMD, Streaming, MapReduce, MPI: High level Programming:

Application and Analytics: Workflow-Orchestration: HPC ABDS SYSTEM (Middleware) 200 Software Projects System Abstraction/Standards Data Format and Storage HPC ABDS Hourglass HPC Yarn for Resource management Horizontally scalable parallel programming model Collective and Point to Point Communication Support for iteration (in memory processing) Application Abstractions/Standards Graphs, Networks, Images, Geospatial .. Scalable Parallel Interoperable Data Analytics Library (SPIDAL) High performance Mahout, R, Matlab .. High Performance Applications

Maybe a Big Data Initiative would include We dont need 200 software packages so can choose e.g. Workflow: Python or Kepler or Apache Crunch Data Analytics: Mahout, R, ImageJ, Scalapack High level Programming: Hive, Pig Parallel Programming model: Hadoop, Spark, Giraph (Twister4Azure, Harp), MPI; Storm, Kapfka or RabbitMQ (Sensors) In-memory: Memcached Data Management: Hbase, MongoDB, MySQL or Derby Distributed Coordination: Zookeeper Cluster Management: Yarn, Slurm File Systems: HDFS, Lustre DevOps: Cloudmesh, Chef, Puppet, Docker, Cobbler IaaS: Amazon, Azure, OpenStack, Libcloud

Monitoring: Inca, Ganglia, Nagios Harp Design Parallelism Model MapReduce Model M M M Map-Collective or MapCommunication Model Application M M Shuffle R Architecture M M

Optimal Communication Map-Collective or MapCommunication Applications MapReduce Applications M Harp Framework MapReduce V2 Resource Manager YARN R Features of Harp Hadoop Plugin

Hadoop Plugin (on Hadoop 1.2.1 and Hadoop 2.2.0) Hierarchical data abstraction on arrays, key-values and graphs for easy programming expressiveness. Collective communication model to support various communication operations on the data abstractions (will extend to Point to Point) Caching with buffer management for memory allocation required from computation and communication BSP style parallelism Fault tolerance with checkpointing WDA SMACOF MDS (Multidimensional Scaling) using Harp on IU Big Red 2 Parallel Efficiency: on 100-300K sequences Best available MDS (much better than that in R) Java 1.20 Parallel Efficiency

1.00 0.80 0.60 0.40 0.20 Cores =32 #nodes 0.00 0 20 40 60 80 100 120 140

Harp (Hadoop plugin) Number of Nodes 100K points 200K points 300K points Conjugate Gradient (dominant time) and Matrix Multiplication Increasing Communication Identical Computation 1000000 points 50000 centroids 10000000 points 5000 centroids 100000000 points 500 centroids

10000 1000 Time (in sec) 100 10

24 48 96 0.1

24 48 96 24 48 96 Number of Cores Hadoop MR Mahout Python Scripting Spark

Harp Mahout and Hadoop MR Slow due to MapReduce Python slow as Scripting; MPI fastest Spark Iterative MapReduce, non optimal communication Harp Hadoop plug in with ~MPI collectives MPI Effi ciency 1 1.0 Data Gathering, Storage, Use Data Source and Style Facet I (i) SQL or NoSQL: NoSQL includes Document, Column, Key-value, Graph, Triple store (ii) Other Enterprise data systems: e.g. Warehouses (iii) Set of Files: as managed in iRODS and extremely common in scientific research (iv) File, Object, Block and Data-parallel (HDFS) raw storage: Separated from computing?

(v) Internet of Things: 24 to 50 Billion devices on Internet by 2020 (vi) Streaming: Incremental update of datasets with new algorithms to achieve real-time response (G7) (vii) HPC simulations: generate major (visualization) output that often needs to be mined (viii) Involve GIS: Geographical Information Systems provide attractive access to geospatial data 2. Perform real time analytics on data source streams and notify users when specified events occur Specify filter Filter Identifying Events Streaming Data Streaming Data Streaming Data Post Selected Events Fetch streamed Data Posted Data

Identified Events Archive Repository Storm, Kafka, Hbase, Zookeeper 5. Perform interactive analytics on data in analyticsoptimized data system Mahout, R Hadoop, Spark, Giraph, Pig Data Storage: HDFS, Hbase Data, Streaming, Batch .. Data Source and Style Facet II Before data gets to compute system, there is often an initial data gathering phase which is characterized by a block size and timing. Block size varies from month (Remote Sensing, Seismic) to day (genomic) to seconds or lower (Real time control, streaming) There are storage/compute system styles: Shared, Dedicated, Permanent, Transient Other characteristics are needed for permanent auxiliary/comparison datasets and these could be

interdisciplinary, implying nontrivial data movement/replication 10 Data Access/Use Styles from Bob Marcus at NIST (you have seen his patterns 2 and 5 and my extension for science 5A follows) 5A. Perform interactive analytics on observational scientific data Science Analysis Code, Mahout, R Grid or Many Task Software, Hadoop, Spark, Giraph, Pig Data Storage: HDFS, Hbase, File Collection (Lustre) Direct Transfer Streaming Twitter data for Social Networking Record Scientific Data in field Transport batch of data to primary analysis data system Local Accumulate and initial

computing NIST Examples include LHC, Remote Sensing, Astronomy and Bioinformatics Analytics Facet (kernels) of the Ogres Map-Only Core Analytics I Pleasingly parallel - Local Machine Learning MapReduce: Search/Query/Index Summarizing statistics as in LHC Data analysis (histograms) (G1) Recommender Systems (Collaborative Filtering) Linear Classifiers (Bayes, Random Forests) Alignment and Streaming (G7) Genomic Alignment, Incremental Classifiers Global Analytics: Nonlinear Solvers (structure depends on

objective function) (G5,G6) Stochastic Gradient Descent SGD (L-)BFGS approximation to Newtons Method Levenberg-Marquardt solver Core Analytics II Global Analytics: Map-Collective (See Mahout, MLlib) (G2,G4,G6) Often use matrix-matrix,-vector operations, solvers (conjugate gradient) Clustering (many methods), Mixture Models, LDA (Latent Dirichlet Allocation), PLSI (Probabilistic Latent Semantic Indexing) SVM and Logistic Regression Outlier Detection (several approaches) PageRank, (find leading eigenvector of sparse matrix) SVD (Singular Value Decomposition) MDS (Multidimensional Scaling) Learning Neural Networks (Deep Learning) Hidden Markov Models Core Analytics III Global Analytics Map-Communication (targets for Giraph) (G3)

Graph Structure (Communities, subgraphs/motifs, diameter, maximal cliques, connected components) Network Dynamics - Graph simulation Algorithms (epidemiology) Global Analytics Asynchronous Shared Memory (may be distributed algorithms) Graph Structure (Betweenness centrality, shortest path) (G3) Linear/Quadratic Programming, Combinatorial Optimization, Branch and Bound (G5) Benchmark Suite in spirit of NAS Parallel Benchmarks or Berkeley Dwarfs Benchmarks across Facets Classic Database: TPC benchmarks NoSQL Data systems: store, index, query (e.g. on Tweets) Hard core commercial: Web Search, Collaborative Filtering (different structure and defer to Google!) Streaming: Gather in Pub-Sub(Kafka) + Process (Apache Storm) solution (e.g. gather tweets, Internet of Things) Pleasingly parallel (Local Analytics): as in initial steps of LHC, Astronomy, Pathology, Bioimaging (differ in type of data analysis)

Global Analytics: Deep Learning, SVM, Clustering, Multidimensional Scaling, Graph Community finding (~Clustering) to Shortest Path (? Shared memory) Workflow linking above Parallel Data Analytics Issues Remarks on Parallelism I Most use parallelism over items in data set Entities to cluster or map to Euclidean space Except deep learning (for image data sets)which has parallelism over pixel plane in neurons not over items in training set as need to look at small numbers of data items at a time in Stochastic Gradient Descent SGD Need experiments to really test SGD as no easy to use parallel implementations tests at scale NOT done Maybe got where they are as most work sequential Maximum Likelihood or 2 both lead to structure like Minimize sum items=1N (Positive nonlinear function of unknown parameters for item i) All solved iteratively with (clever) first or second order approximation to shift in objective function

Sometimes steepest descent direction; sometimes Newton 11 billion deep learning parameters; Newton impossible Have classic Expectation Maximization structure Steepest descent shift is sum over shift calculated from each point SGD take randomly a few hundred of items in data set and calculate shifts over these and move a tiny distance 45 Remarks on Parallelism II Need to cover non vector semimetric and vector spaces for clustering and dimension reduction (N points in space) MDS Minimizes Stress (X) = i~ 0.1 46 When is a Graph just a Sparse Matrix? Most systems are built of connected entities which can be considered a graph See multigrid meshes Particle dynamics PageRank is a graph algorithm or just sparse matrix multiplication to implement power method of finding leading eigenvector Force Diagrams for macromolecules and Facebook

Parallel Data Analytics Examples 446K sequences ~100 clusters clean sample of 446K O(N2) green-green and purplepurple interactions have value but green-purple are wasted O(N2) interactions between green and purple clusters should be able to represent by centroids as in Barnes-Hut. Hard as no Gauss theorem; no multipole expansion and points really in 1000 dimension space as clustered before 3D projection OctTree for 100K sample of Fungi We use OctTree for logarithmic

interpolation (streaming data) 52 Protein Universe Browser (map big data to 3D to use GIS) for COG Sequences with a few illustrative biologically identified clusters 53 Heatmap of biology distance (NeedlemanWunsch) vs 3D Euclidean Distances If d a distance, so is f(d) for any monotonic f. Optimize choice of f 54 Algorithm Challenges See NRC Massive Data Analysis report O(N) algorithms for O(N2) problems Parallelizing Stochastic Gradient Descent Streaming data algorithms balance and interplay between batch methods (most time consuming) and interpolative streaming methods

Graph algorithms Machine Learning Community uses parameter servers; Parallel Computing (MPI) would not recommend this? Is classic distributed model for parameter service better? Apply best of parallel computing communication and load balancing to Giraph/Hadoop/Spark Are data analytics sparse?; many cases are full matrices BTW Need Java Grande Some C++ but Java most popular in ABDS, with Python, Erlang, Go, Scala (compiles to JVM) .. Data Science at Indiana University 6 hours of Video describing 200 technologies from online class 5 hours of video on 51 use cases Online classes in Data Science Certificate /Masters Prettier as Google

Course Builder IU Data Science Masters Features Fully approved by University and State October 14 2014 Blended online and residential Department of Information and Library Science, Division of Informatics and Division of Computer Science in the Department of Informatics and Computer Science, School of Informatics and Computing and the Department of Statistics, College of Arts and Science, IUB 30 credits (10 conventional courses) Basic (general) Masters degree plus tracks Currently only track is Computational and Analytic Data Science Other tracks expected A purely online 4-course Certificate in Data Science has been running since January 2014 (Technical and Decision Maker paths) A Ph.D. Minor in Data Science has been proposed. McKinsey Institute on Big Data Jobs Decision maker and Technical paths

There will be a shortage of talent necessary for organizations to take advantage of big data. By 2018, the United States alone could face a shortage of 140,000 to 190,000 people with deep analytical skills as well as 1.5 million managers and analysts with the know-how to use the analysis of big data to make effective decisions. At [email protected], Informatics/ILS aimed at 1.5 million jobs. Computer Science covers the 140,000 to 190,000 60 Lessons / Insights Proposed classification of Big Data applications with features and kernels for analytics Data intensive algorithms do not have the well developed high performance libraries familiar from HPC Global Machine Learning or (Exascale Global Optimization) particularly challenging Develop SPIDAL (Scalable Parallel Interoperable Data Analytics Library) New algorithms and new high performance parallel implementations Integrate (dont compete) HPC with Commodity Big data (Google to Amazon to Enterprise Data Analytics) i.e. improve Mahout; dont compete with it

Use Hadoop plug-ins rather than replacing Hadoop Enhanced Apache Big Data Stack HPC-ABDS has ~200 members with HPC opportunities at Resource management, Storage/Data, Streaming, Programming, monitoring, workflow layers. Thank you NSF 3 yr. XPS: FULL: DSD: Collaborative Research: Rapid Prototyping HPC Environment for Deep Learning IU, Tennessee (Dongarra), Stanford (Ng) Rapid Python Deep Learning Infrastructure (RaPyDLI) Builds optimized Multicore/GPU/Xeon Phi kernels (best exascale dataflow) with Python front end for general deep learning problems with ImageNet exemplar. Leverage Caffe from UCB. 5 yr. Datanet: CIF21 DIBBs: Middleware and High Performance Analytics Libraries for Scalable Data Science IU, Rutgers (Jha), Virginia Tech (Marathe), Kansas (CReSIS), Emory (Wang), Arizona(Cheatham), Utah(Beckstein) HPC-ABDS: Cloud-HPC interoperable software performance of HPC (High Performance Computing) and the rich functionality of the commodity Apache Big Data Stack. SPIDAL (Scalable Parallel Interoperable Data Analytics Library): Scalable Analytics for Biomolecular Simulations, Network and Computational Social Science, Epidemiology, Computer Vision, Spatial Geographical Information Systems, Remote Sensing for Polar Science and Pathology Informatics.

Machine Learning in Network Science, Imaging in Computer Vision, Pathology, Polar Science, Biomolecular Simulations Algorithm Applications Features Status Parallelism Graph Analytics Community detection Social networks, webgraph P-DM GML-GrC Subgraph/motif finding Webgraph, biological/social networks P-DM GML-GrB Finding diameter

Social networks, webgraph P-DM GML-GrB Clustering coefficient Social networks Page rank Webgraph P-DM GML-GrC Maximal cliques Social networks, webgraph P-DM GML-GrB Connected component Social networks, webgraph

P-DM GML-GrB Betweenness centrality Social networks Shortest path Social networks, webgraph Graph . Graph, static P-DM GML-GrC Non-metric, P-Shm GML-GRA P-Shm Spatial Queries and Analytics Spatial queries relationship

Distance based queries based P-DM PP GIS/social networks/pathology informatics Geometric P-DM PP Spatial clustering Seq GML Spatial modeling Seq PP

GML Global (parallel) ML GrA Static GrB Runtime partitioning 63 Some specialized data analytics in SPIDAL Algorithm aa Applications Features Parallelism P-DM PP P-DM PP

P-DM PP Seq PP Todo PP Todo PP P-DM GML Core Image Processing Image preprocessing Object detection & segmentation

Image/object feature computation Status Computer vision/pathology informatics Metric Space Point Sets, Neighborhood sets & Image features 3D image registration Object matching Geometric 3D feature extraction Deep Learning Learning Network, Stochastic Gradient Descent

Image Understanding, Language Translation, Voice Recognition, Car driving PP Pleasingly Parallel (Local ML) Seq Sequential Available GRA Good distributed algorithm needed Connections in artificial neural net Todo No prototype Available P-DM Distributed memory Available P-Shm Shared memory Available 64 Some Core Machine Learning Building Blocks Algorithm Applications Features Status //ism

DA Vector Clustering DA Non metric Clustering Kmeans; Basic, Fuzzy and Elkan Levenberg-Marquardt Optimization Accurate Clusters Accurate Clusters, Biology, Web Fast Clustering Non-linear Gauss-Newton, use in MDS Vectors Non metric, O(N2) Vectors P-DM P-DM P-DM GML GML GML

Least Squares P-DM GML SMACOF Dimension Reduction DA- MDS with general weights Squares, P-DM GML Vector Dimension Reduction DA-GTM and Others Find nearest neighbors document corpus Least O(N2) Vectors P-DM

GML TFIDF Search in P-DM Bag of words Find pairs of documents with (image features) TFIDF distance below a threshold Todo PP Support Vector Machine SVM Random Forest Gibbs sampling (MCMC) Latent Dirichlet Allocation LDA with Gibbs sampling or Var. Bayes Singular Value Decomposition SVD

Learn and Classify Vectors Learn and Classify Vectors Solve global inference problems Graph Seq P-DM Todo GML PP GML Topic models (Latent factors) Bag of words P-DM GML Dimension Reduction and PCA Vectors

Seq GML Hidden Markov Models (HMM) Global inference on sequence Vectors models Seq 65 GML All-pairs similarity search GML PP &

Recently Viewed Presentations

  • Slide Title Goes Here

    Slide Title Goes Here

    CMS M&O at FNAL Jim Freeman US CMS M&O Manager May 17, 2006 M&O Management Organization Management Tools MOUs Change Requests PMG DOE/NSF Review, MEG M&O WBS Organization 11 EMU (L2 = Giorgio Apollinari, FNAL) 12 Hcal 13 Trigger 14...
  • TSX-V: CAD Colonial Coal International Corporation Western Canadas

    TSX-V: CAD Colonial Coal International Corporation Western Canadas

    The scientific and technical information relating to the Huguenot property has been derived from the Huguenot Technical Report dated September 23, 2013, a copy of which will be filed on under Colonial Coal International Corp., copies of the technical...


    Los verbos transitivos son… aquellos verbos que pueden ir acompañados de un complemento directo. Ejemplos. comprar → Pedro compra dulces. recoger → recoge su cuarto . leer → Lee un libro. mirar → Marta miró el atardecer. saltar → El...
  • Data Warehousing - University of Oklahoma

    Data Warehousing - University of Oklahoma

    Purpose of a data warehouse Provides an architecture for the flow of data from operational systems to decision support systems DW involves a many record analysis, during which all data has to be locked Used to discover trends and patterns...
  • Gibbs Free Energy - Mesa Community College

    Gibbs Free Energy - Mesa Community College

    Gibbs Free Energy, G Multiply through by -T -T∆Suniv = ∆Hsys - T∆Ssys -T∆Suniv = change in Gibbs free energy for the system = ∆Gsystem Under standard conditions — ∆Gosys = ∆Hosys - T∆Sosys ∆Go = ∆Ho - T∆So Gibbs...
  • Aurasium: Practical Policy Enforcement for Android Applications

    Aurasium: Practical Policy Enforcement for Android Applications

    Aurasium: Practical Policy Enforcement for Android Applications. R. Xu, H. Saidi and R. Anderson. ... TaintDroid(OSDI 10) CRePE(ISC 10) AppFence(CCS 11) ... Practical Policy Enforcement for Android Applications
  • Chronic Myeloid Leukemia

    Chronic Myeloid Leukemia

    Chronic Myeloid Leukemia Leukemia ALL, AML, CLL Chronic Myelogenous Leukemia Cancer of the granulocytes or monocytes, compared to leukocytes in lymphocytic leukemias Comprises about 14% of all adult leukemias Males slightly higher than females One of the first cancers to...
  • Inmunoterapia: Presente y futuro. Cáncer de vejiga

    Inmunoterapia: Presente y futuro. Cáncer de vejiga

    Disclosures. Honoraria for ad boards and/or lectures: Pfizer, BMS, IPSEN, Roche, Eisai, Eusa Pharma, MSD, Sanofi-Genzyme, Adacap, Novartis, Pierre Fabre, Lexicon, Celgene