SWE 637: Here! Test this!

SWE 637: Here! Test this!

Introduction to Software Testing (2nd edition) Chapter 1 Why Do We Test Software? Paul Ammann & Jeff Offutt http://www.cs.gmu.edu/~offutt/soft waretest/ Updated September 2015 First version, 28 August 2011 Testing in the 21st Century Software defines behavior network routers, finance, switching networks, other infrastructure Todays software market : Industry is going is much bigger is more competitive has more users through a revolution in what testing means to the success of software Embedded control applications products

airplanes, air traffic control spaceships PDAs watches memory seats ovens DVD players remote controllers garage door openers cell phones Agile processes put increased pressure on testers Programmers must unit test with no training or education! Tests are key to functional requirements but who builds those Introduction to Software Testing, Edition 2 (Ch 1) Ammann & Offutt 2 Software is a Skin that Surrounds Our Civilization Quote due to Dr. Mark Harman Introduction to Software Testing, Edition 2 (Ch 1)

Ammann & Offutt 3 Software Faults, Errors & Failures Software Fault : A static defect in the software Software Failure : External, incorrect behavior with respect to the requirements or other description of the expected behavior Software Error : An incorrect internal state that is the manifestation of some fault Faults in software are equivalent to design mistakes in hardware. Software does not degrade. Introduction to Software Testing, Edition 2 (Ch 1) Ammann & Offutt 4 Fault and Failure Example

A patient gives a doctor a list of symptoms Failures The doctor tries to diagnose the root cause, the ailment Fault The doctor may look for anomalous internal conditions (high blood pressure, irregular heartbeat, bacteria in the blood stream) Errors Most medical problems result from external attacks (bacteria, viruses) or physical degradation as we age. Software faults were there at the beginning and do not appear when a part wears out. Introduction to Software Testing, Edition 2 (Ch 1) Ammann & Offutt 5 A Concrete Example Fault: Should start searching at 0, not 1

public static int numZero (int [ ] arr) Test 1 { // Effects: If arr is null throw NullPointerException [ 2, 7, 0 ] // else return the number of occurrences of 0 in arr Expected: int count = 0; 1 for (int i = 1; i < arr.length; i++) Actual: 1 Error: i is 1, not 0, { Test 2 on the first if (arr [ i ] == 0) [ 0, 2, 7 ] iteration { Expected: Failure: none count++; 1 } Error: i is 1, not 0 Actual: 0 } Error propagates to the variable return count; count

} Failure: count is 0 at the return statement Introduction to Software Testing, Edition 2 (Ch 1) Ammann & Offutt 6 The Term Bug Bug is used informally Sometimes speakers mean fault, sometimes error, sometimes failure often the speaker doesnt know what it means ! This class will try to use words that have precise, an analyzing process defined, and unambiguous meanings must equally have been BUG It has been just so in all of my inventions. The first step is an intuition, and comes with a burst, then difficulties arisethis thing gives out and [it is] then that 'Bugs'as such little faults and difficulties are calledshow themselves and months of intense watching, study and labor are requisite. . . Thomas Edison

Introduction to Software Testing, Edition 2 (Ch 1) Ammann & Offutt performed in order to furnish the Analytical Engine with the necessary operative data; and that herein may also lie a possible source of error. Granted that the actual mechanism is unerring in its processes, the cards may give it wrong orders. Ada, Countess Lovelace (notes on Babbages Analytical Engine) 7 Spectacular Software Failures NASAs Mars lander: September 1999, THERAC-25 design crashed due to a units integration fault Mars Polar Lander crash site?

THERAC-25 radiation machine : Poor testing of safety-critical software can cost lives : 3 patients were killed Ariane 5 explosion : Millions of $$ Intels Pentium FDIV fault : Public relations nightmare We need our software to be dependable Testing is one way to assess Introduction to Software Testing, Edition 2 (Ch 1) Ammann & Offutt Ariane 5: exception-handling bug : forced self destruct on maiden flight (64-bit to 16-bit conversion: about 370 million $ lost) 8 Northeast Blackout of 2003 508 generating units and 256 power plants shut down

Affected 10 million people in Ontario, Canada Affected 40 million people in 8 US states Financial losses of $6 Billion USD The alarm system in the energy management system failed due to a software error and operators were not informed of the power overload in the system Introduction to Software Testing, Edition 2 (Ch 1) Ammann & Offutt 9 Costly Software Failures NIST report, The Economic Impacts of Inadequate Infrastructure for Software Testing (2002) Inadequate software testing costs the US alone between $22 and $59 billion annually Better approaches could cut this amount in half Huge losses due to web application failures Financial services : $6.5 million per hour (just in USA!)

Credit card sales applications : $2.4 million per hour (in USA) In Dec 2006, amazon.coms BOGO offer turned into a double discount World-wide 2007 : Symantec says that security monetary loss duemost to poor software is vulnerabilities arestaggering due to faulty software Introduction to Software Testing, Edition 2 (Ch 1) Ammann & Offutt 10 Testing in the 21st Century

More safety critical, real-time software Embedded software is ubiquitous check your pockets Enterprise applications means bigger programs, more users Paradoxically, free software increases our expectations ! Security is now all about software faults Secure software is reliable software The web offers a new deployment platform Very competitive and very available to more users Web apps are distributed Industry desperately needs our Web apps must be highly reliable Introduction to Software Testing, Edition 2 (Ch 1) inventions ! Ammann & Offutt 11 What Does This Mean?

Software testing is getting more important What are we trying to do when we test ? What are our goals ? Introduction to Software Testing, Edition 2 (Ch 1) Ammann & Offutt 12 Validation & Verification (IEEE) Validation : The process of evaluating software at the end of software development to ensure compliance with intended usage Verification : The process of determining whether the products of a given phase of the software development process fulfill the requirements established during the previous phase IV&V stands for independent verification and Introduction to Software Testing, Edition 2 (Ch 1) Ammann & Offutt

13 Testing Goals Based on Test Process Maturity Level 0 : Theres no difference between testing and debugging Level 1 : The purpose of testing is to show correctness Level 2 : The purpose of testing is to show that the software doesnt work Level 3 : The purpose of testing is not to prove anything specific, but to reduce the risk of using the software Level 4 : Testing is a mental discipline that helps all IT professionals develop higher quality software Introduction to Software Testing, Edition 2 (Ch 1) Ammann & Offutt 14 Level 0 Thinking Testing is the same as debugging Does not distinguish between incorrect behavior and mistakes in the program

Does not help develop software that is reliable or safe This is what we teach undergraduate CS majors Introduction to Software Testing, Edition 2 (Ch 1) Ammann & Offutt 15 Level 1 Thinking Purpose is to show correctness Correctness is impossible to achieve What do we know if no failures? Good software or bad tests? Test engineers have no: Strict goal Real stopping rule Formal test technique Test managers are powerless This is what hardware engineers often expect

Introduction to Software Testing, Edition 2 (Ch 1) Ammann & Offutt 16 Level 2 Thinking Purpose is to show failures Looking for failures is a negative activity Puts testers and developers into an adversarial relationship What if there are no failures? This describes most software companies. How can we move to a team approach ?? Introduction to Software Testing, Edition 2 (Ch 1) Ammann & Offutt

17 Level 3 Thinking Testing failures can only show the presence of Whenever some risk we use software, we incur Risk may be small and consequences unimportant Risk may be great and consequences This describes a few enlightened software catastrophic companies Introduction to Software Testing, Edition 2 (Ch 1) Ammann & Offutt 18

Level 4 Thinking A mental discipline that increases quality Testing is only one way to increase quality Test engineers can become technical leaders of the project Primary responsibility to measure and improve software quality This is the way traditional engineering Their expertise should help the developers works Introduction to Software Testing, Edition 2 (Ch 1) Ammann & Offutt 19 Where Are You?

Are you at level 0, 1, or 2 ? Is your organization at work at level 0, 1, or 2 ? Or 3? We hope to teach you to become change agents in your workplace Advocates for level 4 thinking Introduction to Software Testing, Edition 2 (Ch 1) Ammann & Offutt 20 Tactical Goals : Why Each Test ? If you dont know why youre conducting each test, it wont be very helpful Written test objectives and requirements must be documented What are your planned coverage levels? How much testing is enough? Common objective spend the budget test until the ship-date Sometimes called the date criterion Introduction to Software Testing, Edition 2 (Ch 1) Ammann & Offutt

21 Cost of Not Testing Poor Program Managers might say: Testing is too expensive. Testing is the most time consuming and expensive part of software development Not testing is even more expensive If we have too little testing effort early, the cost of testing increases Planning for testing after development is prohibitively expensive Introduction to Software Testing, Edition 2 (Ch 1) Ammann & Offutt 22 Cost of Late Testing 60 50 Assume $1000 unit cost, per fault, 100 faults $3 6 40 30

20 0 R m e r i u $2 0K $1 3K $6 K 10 eq (several large programs) $10 0 0 $25

K 0K Fault origin (%) Fault detection (%) K Unit cost (X) t n e s e D gn i s Pr og /U t i n

t s Te e t In gr i t a on t s Te s y S m e t Po t s Te

ep -D t s m y lo t n e Software Engineering Institute; Carnegie Mellon University; Handbook CMU/ SEI-96-HB-002 Introduction to Software Testing, Edition 2 (Ch 1) Ammann & Offutt 23 Software Testing Foundations (2.1) Testing can only show the presence of failures Not their absence Introduction to Software Testing, Edition 2 (Ch 2) Ammann & Offutt 24 Testing & Debugging

Testing : Evaluating software by observing its execution Test Failure : Execution of a test that results in a software failure Debugging : The process of finding a fault given a failure Not all inputs will trigger a fault into causing a failure Introduction to Software Testing, Edition 2 (Ch 2) Ammann & Offutt 25 Fault & Failure Model (RIPR) Four conditions necessary for a failure to be observed 1. Reachability : The location or locations in the program that contain the fault must be reached 2. Infection : The state of the program must be incorrect 3. Propagation : The infected state must cause

some output or final state of the program to be incorrect 4. Reveal : The tester must observe part of the incorrect portion of the program state Introduction to Software Testing, Edition 2 (Ch 2) Ammann & Offutt 26 RIPR Model Reachabilit y Infection Propagation Revealabilit y Introduction to Software Testing, Edition 2 (Ch 2) Test Final Program State Observed Final Program Observed FinalState Program State Incorrect Final

State Reache s Fault Infect s Incorre ct Progra m State Propagat es Reveal s Test Oracle s Ammann & Offutt 27 Software Testing Activities (2.2) Test Engineer : An IT professional who is in charge of one or more technical test activities

Designing test inputs Producing test values Running test scripts Analyzing results Reporting results to developers and managers Test Manager : In charge of one or more test engineers Sets test policies and processes Interacts with other managers on the project Otherwise supports the engineers Introduction to Software Testing, Edition 2 (Ch 2) Ammann & Offutt 28 Traditional Testing Levels (2.3) main Class P Class A

Class B method mA1() method mB1() method mA2() method mB2() This view obscures underlying similarities Introduction to Software Testing, Edition 2 (Ch 2) Ammann & Offutt Acceptance testing : Is the software acceptable to the user? System testing : Test the overall functionality of the system Integration testing : Test how modules interact

with each other Module testing (developer testing) : Test each class, file, module, Unit testing component (developer testing) : Test each unit (method) 29 Object-Oriented Testing Levels Class A Inter-class testing : Test multiple classes together Class B

method mA1() method mB1() method mA2() method mB2() Intra-class testing : Test an entire class as sequences of calls Inter-method testing : Test pairs of methods in the same class Intra-method testing : Test each method individually Introduction to Software Testing, Edition 2 (Ch 2) Ammann & Offutt

30 Coverage Criteria (2.4) Even small programs have too many inputs to fully test them all private static double computeAverage (int A, int B, int C) On a 32-bit machine, each variable has over 4 billion possible values Over 80 octillion possible tests!! Input space might as well be infinite Testers search a huge input space Trying to find the fewest inputs that will find the most problems Coverage criteria give structured, practical ways to search the input space Search the input space thoroughly Introduction to Software Testing, Edition 2 (Ch 2) Ammann & Offutt

31 Summary: Why Do We Test Software ? A testers goal is to eliminate faults as early as possible Improve quality Reduce cost Preserve customer satisfaction Introduction to Software Testing, Edition 2 (Ch 1) Ammann & Offutt 32

Recently Viewed Presentations

  • Satire Unit - MS. GILPIN

    Satire Unit - MS. GILPIN

    Satirical Tone. Ironic/Sarcastic. Always opposed to pretense and hypocrisy. More than a little bit prone to references to things society finds taboo or disgusting (bodily functions, sexuality, etc.) ... Satire Unit Last modified by: Gilpin, Courtney SHS - Staff
  • Benchmarking Pathology and the Demand on its Service

    Benchmarking Pathology and the Demand on its Service

    Established in late 2005 and formally launched in Spring 2006. Funded originally by DH Pathology Modernisation. Now funded by GM Commissioners. A model for building engagement, trust and respect and facilitating change
  • Blue and Grey - Ms Tolliday&#x27;s Classroom Website

    Blue and Grey - Ms Tolliday's Classroom Website

    Balfour report led to the Statute of Westminster Statute of Westminster 1931: Statute of Westminster Recognized in law the Balfour report that had made Canada an autonomous nation at the Imperial Conference. Canada recognized as a commonwealth country completely free...
  • Lecture 16: Caches and Memory

    Lecture 16: Caches and Memory

    Must be correct address and register! Non-Binding prefetch: Load into cache. Can be incorrect. Frees HW/SW to guess! Issuing prefetch instructions takes time Is cost of prefetch issues < savings in reduced misses? Higher superscalar reduces difficulty of issue bandwidth...
  • A Quilt Of A Country Anna Quindlen - English 9

    A Quilt Of A Country Anna Quindlen - English 9

    A Quilt Of A CountryAnna Quindlen. Hope Schmidt. Audrye Lehmann. Jonathan Schneider . 4th Hour. She wrote the text after 9/11. 2. She won the Pulitzer Prize of commentary in 1992. 3. Anna was in a mixed marriage. ... A...
  • LOCKE VS HOBBES ON THE ROLE OF GOVERNMENT

    LOCKE VS HOBBES ON THE ROLE OF GOVERNMENT

    LOCKE VS HOBBES - ON THE ROLE OF GOVERNMENT John Locke 1632 - 1704 England Thomas Hobbes 1588 - 1678 England LOCKE VS HOBBES B OOK Two Treatises on Government (1690) Leviathan (1651) LOCKE VS HOBBES MOTIV ATI ON The...
  • Similes! - Mrs. Phillips&#x27; and Mrs. Pittman&#x27;s Fourth Grade Team

    Similes! - Mrs. Phillips' and Mrs. Pittman's Fourth Grade Team

    The baby's smile was bright like the morning sunshine. Comparing. to. ... Title your Writing journal: "Similes" then pick the 3 sentences on the next slide that contain similes and copy them in your journal. I like orange juice the...
  • Media Literacy - Weebly

    Media Literacy - Weebly

    The photograph does not clearly show the faces of either person involved in this embrace and several people have claimed to be the subjects. The photograph was shot just south of 45th Street looking north from a location where ....