MIS 5208 Week 11: Processing and Analyzing Data

MIS 5208 Week 11: Processing and Analyzing Data

MIS 5208 Week 11: Processing and Analyzing Data Ed Ferrara, MSIA, CISSP [email protected] P l e a s e s i g n u p fo r t h i s t ra i n i n g ! Temple, as a member school of Internet2, is entitled to free

training and certification exams for Splunk Power Users: http://www.internet2.edu/blogs/detail/10079 If you register and take the courses described in the blog posting yourself, you will have access to the teaching materials in PDF form as part of the elearning course. Reminder Fox School of Busines Re v i e w S e a rc h

There are ____ components to the Search and Reporting interface? 5 2 7 1

7 Fox School of Busines Re v i e w S e a rc h Fox School of Busines Re v i e w S e a rc h

What is the most efficient way to filter events in Splunk? By Time By Host With the admin user In app By Time

Fox School of Busines Re v i e w S e a rc h When a search is run, events are returned in _________?

Chronological order Pdf Alphabetical order Reverse chronological order Reverse chronological order Fox School of Busines Re v i e w S e a rc h

Commands that create statistics or visualizations are called _________? Transforming commands Machine learning commands Math

Data science Transforming Commands Fox School of Busines Re v i e w S e a rc h The search & reporting App has how many search modes _________?

2 5 3 4 3

Fox School of Busines Re v i e w S e a rc h Which character acts as a wildcard in Splunk __? ~

% * ! * Fox School of Busines Re v i e w S e a rc h What are Boolean operators in Splunk __?

AND NOT AFTER OR IF

AND NOT OR Fox School of Busines D ata S o u rc e s

A number of system applications and network devices such as routers switches relay events over network ports using the TCP or UDP protocols. Some applications make use of the SNMP standard to send events over UDP. Syslog, which is a standard for computer data logging is another set of sources where there is a wealth of information that could be captured at a network port level. Splunk can be enabled to accept input from a TCP or UDP port. Use the Splunk Web user interface and configure a network input source where you specify:

Host Port Sourcetype

Once you save the configuration, Splunk will start indexing the data coming out of the specified network port. This kind of network input can be used to capture syslog information that gets generated on remote machines and the data does not reside locally to a Splunk instance. Splunk forwarders can also be used to gather data on remote hosts. Fox School of Busines W i n d o w s D ata

The Windows operating system churns out a number of log files that have information about: Windows events Registry Active Directory WMI Performance Other data.

Splunk recognizes Windows log streams as a source type and allows adding one more of these log streams to be indexed as input for further processing. Although Windows sources such as Active Directory or others can be individually configured, Splunk provides a better and easy way of dealing with these Windows logs or events by using: Splunk App for Windows Splunk Technology Add-on for Windows Fox School of Busines W i n d o w s Te c h n o l o g y A d d - O n O n L i n u x

Note: Windows Technology add-on can be installed on Splunk running on Windows. If you are running Splunk on Linux then the Windows TA can be installed on a forwarder running on a

Windows machine. Forwarders are explained later in this chapter and in Chapter 15 of the text. Fox School of Busines

Splunk also provides similar Technology Add-ons for Linux and Unix known as *Nix. This Add-on makes use of both log files and scripting to get different sets of event and log data available in Linux or Unix into Splunk. You can install *Nix technology on Linux systems Fox School of Busines

O t h e r A p ps Fox School of Busines G e tti n g to Kn o w C o m b i n e d A c c e s s L o g D ata

In the real world, enterprises have numerous applications and most of them will be running on a heterogeneous infrastructure, which includes all sorts of hardware, databases, middleware, and application programs. It will not be possible to have Splunk running locally or near to each of the applications or infrastructure, meaning the data will not be local to Splunk. What we have seen in this chapter is how we can get data into Splunk which is local to it. The use cases assumed that Splunk will be able to access files or directories, which could be on local or file systems that have remote data, but they are attached to the machine where Splunk is running.

A Splunk forwarder is the same as a standard Splunk instance but with only the essential components that are required to forward data to receivers, which could be the main Splunk instance or indexer. Fox School of Busines Processing and Analyzing Data P ro c e s s i n g a n d A n a l y z i n g D ata

Requirement Understand the data set that you want to process and analyze. Get intimately acquainted with the data you will work with first. Review

Log files are generated by almost all kinds of applications and servers: End-user applications Web servers Complex middleware platforms Operating systems and firmware also generate huge amounts of raw data into log files.

The challenge lies in understanding, analyzing, and mining the raw data in the log files and making sense out of it. Fox School of Busines P re v i e w D ata 127.0.0.1 - JohnDoe [10/Oct/2000:13:55:36 -0700] "GET /apache_pb.gif HTTP/1.0" 200 2326 "http://www.google.com" "Opera/9.20 (Windows NT 6.0; U; en)" 127.0.0.1 This is the IP address of the client (the machine, host, or proxy server)

that was making an HTTP request to access either a web application or an individual web page. The value in the field could be represented as hostname. - This field is used to identify the client making the HTTP request. The contents of this field are highly unreliable, a hyphen is typically used, which indicates the information is not available. JohnDoe

This is the user id of the user who is requesting the web page or an application. 10/Jan/2013:10:32:55 -0800 The timestamp of when the server finished processing the request. The format can be controlled using web server settings. GET /apache_pb.gif HTTP/1.0 This is the request line that is received from the client. It shows the method information, in this example GET, the resource that the client

was requesting, in this case /apache_pb.gif, and the protocol used, in this case HTTP/1.0 200 This is the status code that the server sends back to the client. Status codes are very important information as they tell whether the request from the client was successfully fulfilled or failed, in which case some action needs to be taken. 200 in this case indicates that the request has been successful. 2326

This number indicates the size of the data returned to the client. In this case 2326 bytes were sent back to the client. If no content was returned to the client, this value will be a hyphen. Fox School of Busines P re v i e w D ata 127.0.0.1 - JohnDoe [10/Oct/2000:13:55:36 -0700] "GET /apache_pb.gif HTTP/1.0" 200 2326 "http://www.google.com" "Opera/9.20 (Windows NT 6.0; U; en)" http://www.google.com

This field is known as a referrer field and shows from where the request has been referred. You could be seeing web site URLs like http://www.google.com, http://www.yahoo.com, or http://www.bing.com as the values in the referrer field. Referrer information helps web sites or online applications to see how the users are coming in to the web site and this information could be used to determine where the online advertisement dollars should be spent. As you may notice that referrer has an extra r. That is intentional and originated from the original proposal submitted in the HTTP specification. In browsers like Chrome where users can use incognito mode, or have referrers disabled, the values in the field will not be accurate. In HTML5 the user agent that is reporting this

information can be instructed not to send the referrer information. Opera/9.20 (Windows NT 6.0; U; en) This is the user-agent field, and it has the information that the client browser reports about itself. You will see values like Opera/9.20 (Windows NT 6.0; U; en), which means that the request is coming from an Opera browser running on a Windows NT (actually Windows Vista or Windows Server 2008) operating system. User-agent information helps to optimize web sites and web applications and cater for requests coming from smaller form factor devices such as the iPad and mobile phones.

Fox School of Busines L o o k at s o m e o f t h e d ata Fox School of Busines Lab 4 L o a d t h e D ata Fox School of Busines

L o a d t h e D ata Fox School of Busines L o a d t h e D ata Fox School of Busines L o a d t h e D ata Fox School of Busines

L o a d t h e D ata Fox School of Busines L o a d t h e D ata Fox School of Busines L o a d t h e D ata Fox School of Busines

L o a d t h e D ata Fox School of Busines L o a d t h e D ata Fox School of Busines L o a d t h e D ata Fox School of Busines

L o a d t h e D ata Fox School of Busines S e a rc h t h e D ata Fox School of Busines Chapter 3 L i st A l l F i e l d s

Fox School of Busines L i st A l l F i e l d s Fox School of Busines L i st A l l F i e l d s Fox School of Busines L i st A l l F i e l d s

Fox School of Busines L i st A l l F i e l d s - C at e go r y I D Fox School of Busines L i st A l l F i e l d s d ate _ h o u r Fox School of Busines L i st A l l F i e l d s T i m e S e l e c ti o n

Fox School of Busines L i st A l l F i e l d s Ave ra g e O ve r T i m e Fox School of Busines Thank you

Recently Viewed Presentations

  • Apply Modern Biotechnology  To better understand and manage

    Apply Modern Biotechnology To better understand and manage

    Unmodified fish of these species produce growth hormone, but the GMOs produce the hormone at novel times and levels and in novel tissues. In salmon and trout, the fish's growth rate is de-linked from the normal environmental cue of dropping...
  • the Juxtaposing Autism Spectrum genes On Neurons project

    the Juxtaposing Autism Spectrum genes On Neurons project

    For example, mutants with cerebellar disorders can be grouped and examined together: Reln, Rora, Kcnj6, Grid2. 1. Identify the expression pattern of a specific gene - on ctdtb homepage, enter gene (Rora) in Gene Name & Keyword Search to search.
  • Discretionary Grant Application Process 16/17 January 2016 Purpose

    Discretionary Grant Application Process 16/17 January 2016 Purpose

    Discretionary Grant Funding windows opened annually parallel with Mandatory Grant from 01 January to 30 April. Process aligned to DHET reporting requirements. Projects implementation within each respective financial year. No 15/16 funding awards will be implemented in the 16/17 financial...
  • Exams & Assessment - rutlish.merton.sch.uk

    Exams & Assessment - rutlish.merton.sch.uk

    at Rutlish school. ... - parents can usually view homework on Fronter 19. Encourage your child to face difficult tasks and not give up but also to ask for help when in difficulty. Attend all parent's evenings. Ensure that your...
  • End-To-End Residential Broadband Architecture

    End-To-End Residential Broadband Architecture

    U.S. Residential High-Speed Data Service Connections By Technology (M) (IDC 10/98, Jupiter 8/98, Dataquest 10/98) Microsoft Residential Broadband Strategy Windows Support For Broadband Networking Residential Broadband Service Model Residential Broadband Service Requirements Residential Broadband Service Requirements DSL Technologies What Is...
  • Fyzika na scéně - exploratorium pro žáky základních a ...

    Fyzika na scéně - exploratorium pro žáky základních a ...

    Název úlohy: 8.6 Polarizace světla Fyzikální princip Polarizované světlo je příčné elektromagnetické vlnění, jehož vektor E kmitá stále v jedné rovině.
  • Current Home Canning Practices in the U.S. E.

    Current Home Canning Practices in the U.S. E.

    4. National Center for Home Food Preservation. 1999. Information collected from the Cooperative Extension System, cookbooks and internet. Unpublished data. Athens, GA: The University of Georgia. This material is based upon work supported by the Cooperative State Research, Education, and
  • And Then There Were None - PBworks

    And Then There Were None - PBworks

    This will be a paragraph of work. Illustrator: Your job is to draw a picture or diagram related to the novel so far. It can be a sketch, cartoon, diagram, flow chart, or stick figure scene. Connector: Your job is...