Exnay-Javaway In the Clouday (an experience report)

Exnay-Javaway In the Clouday (an experience report)

Big Data, Data Mining, Tools N = ALL CORRELATION vs CAUSATION Data Sources...

Data Creation, Storage, Costs Infrastructure NoSQL Flavors https://www.youtube.com/watch?v=qI_g07C_Q5I NoSQL

Not Only SQL (sort of) Greater scalability Designed with distributed computing and commodity (not cheap) hardware.

Variety of flavors https://www.youtube.com/watch?v=qI_g07C_Q5I Topic: Algorithms Tools Speaking of the Cloud

High Level Flow Example Hadoop MapReduce HDFS Distributed file system.

Write-once/read many Fault tolerance / Redundance

Processing logic close to data http://www.ibm.com/developerworks/library/wa-introhdfs/ 2.

1/ m ap re d _t ut

or ial .h tm l# S Tr

a ou d i nt tio ha n do i op

n a .a l pa J ch a w

e. or v o g/ a do r cs

d /r 1. ou rc e +C

od e Hive CREATE TABLE docs (line STRING); CREATE TABLE word_counts AS SELECT word, count(1) as count FROM

(SELECT explode(split(line, ' ')) AS word FROM docs) w GROUP BY word ORDER BY word; Hive with Some Structure Data 123 456

789 111 222 333 444 555 F M

M M M F F M create table if not exists p_genders ( p_id string,

gender string ) ROW FORMAT DELIMITED FIELDS TERMINATED BY '\t'; SELECT * from p_genders; Pig Latin A = load 'S3://pmb4bucket/input/bleakhouse/bleakhouse.txt'; B = foreach A generate flatten(TOKENIZE((chararray)$0)) as word; C = group B by word;

D = foreach C generate COUNT(B), group; store D into 's3://pmb4hadoop/output/bleakhouse'; Complex Event Processing Tools Data Scientist

Not just a bean counter - its about modeling General skill set: o Math (linear algebra, statistics, calculus, discrete math) o Business sense o Programming skills

o Communication o etc, etc, etc https://www.youtube.com/watch?v=ceeiUAm bfZk Our Schedule

Setting the goals for a data mining project. Setting up KNime Gathering and preparing data. Visualization Machine Learning

Nave Bayes Clustering and Classification Dimension reduction But first

Recently Viewed Presentations

  • Reliability Issues of 3-Phase Locomotives 14-15 Feb, 2019

    Reliability Issues of 3-Phase Locomotives 14-15 Feb, 2019

    If driver close DJ within 5 seconds before frequency coming down below 35 Hz, auxiliary converter over current message appears and immediately DJ trips. CGPSIL has to study the issue and propose the solution by 20.02.2019. 5. MCB-100 trip message...
  • Dias nummer 1 - sundhed.dk

    Dias nummer 1 - sundhed.dk

    Penny Thomas Charles. Ussing Joachim. Øvlisen Mette. Koldill Ionka Dimitrova. Falkenberg Pedersbæk Mikkel. Bojesen Paula Johanne Savstru. Pedersen Hans_Christian. Banke-Hansen Pia. Arredondo Bruun Lars Patrick. Tholl Inger. Stefanova Stine Diana Maria. Iversen Jette Gottlieb. Windelin Gerda ...
  • C# GENERICS - Western Washington University

    C# GENERICS - Western Washington University

    C# Generics. Definition. Pros & Cons. In Depth . Greg. Class activity: create a little game where someone has to come up and demonstrate the basic concept of generics for us. It would involve them having to use several different...
  • Titles - NLI

    Titles - NLI

    Definitions (RDA 5.1.2) The term workrefers to a distinct intellectual or artistic creation (i.e., the intellectual or artistic content).. The term . expression. refers to the intellectual or artistic realization of a work in the form of alpha-numeric, musical or...
  • PowerPoint 演示文稿

    PowerPoint 演示文稿

    Possible answers to the question. 1. It may be that the predominance of intrafirm trade between Mexico and the United States reduces the incentives toward competition and innovation that would arise from trade and foreign investment reforms.
  • Taking A Math Test I love math !

    Taking A Math Test I love math !

    I thought my teacher would be happy, but she was not. She told me that when I yell out the answers, I may make other students feel sad that they are not as good as math as me. I learned...
  • Presentation to: name of organisation

    Presentation to: name of organisation

    David Relph. Director. Lisa Wheatley. Programme Manager. Zoe Trinder-Widdess. Communications Manager. Trish Harding. Evaluation Officer. David Evans. PPI Lead. What is a HIT? Involve multiple organisations. Cross disciplines and professional boundaries. Research into practice.
  • Chapter 2

    Chapter 2

    exploratory interview. is a short, informal talk with someone who works in a career that interests you. That could include family, friends, neighbors, teachers, etc. An exploratory interview can give you insight into a career that you may not learn...