2013 REU Program at ECU Software Testing - Foundations, Tools, and Applications Lecture 2 May 21, 2013 Software Testing Research Dr. Sergiy Vilkomir Exhaustive testing is impossible A restricted number of test cases should be selected There are different ways (testing methods/strategies/criteria/approaches) to select test cases There are no method that guarantee that software is 100% correct Testing can show that defects are present, but cannot prove that there are no defects. 2/38 Pair-Wise Pair-Wise: A value of each level for each factor must be combined with a value from every levels for each other factor. In other words, we must cover all possible pairs of values. Pair-wise testing provides a small set of test cases. Why? One test case covers several pairs Test case A B C

green red blue green red It is acovers practical alternative to testing all combinations. green blue pairs: red blue Pair-Wise Pair-Wise: A value of each level for each factor must be combined with a value from every levels for each other factor. A B C red red red red green blue green red blue blue

red blue blue blue green green green green blue red green blue green red red blue green Exercise: Choose one pair and find a test case which covers this pair Pairwise designs: Example Suppose that a program to be tested requires 3 inputs, one corresponding to each input variable. Each variable can take only one of two distinct values. Considering each input variable as a factor, the total number of factor combinations is 23. Let X, Y, and Z denote the three input variables and {X1, X2}, {Y1, Y2}, {Z1, Z2} their respective sets of values. All possible combinations of these three factors follow. Pairwise designs: Reducing the combinations Now suppose we want to generate tests such that each pair appears in at least one test. There are 12 such pairs: (X1, Y1), (X1, Y2), (X1, Z1), (X1, Z2), (X2, Y1), (X2, Y2), (X2, Z1), (X2, Z2),

(Y1, Z1), (Y1, Z2), (Y2, Z1), and (Y2, Z2). The following four combinations cover all pairs: The above design is also known as a pairwise design. It is a balanced design because each value occurs exactly the same number of times. There are several sets of four combinations that cover all 12 pairs. Pairwise combinatorial testing Pairwise combination (instead of exhaustive) Generate combinations that efficiently cover all pairs Rationale: most failures are triggered by single values or combinations of a few values. Covering pairs reduces the number of test cases, but reveals most faults Example: Display Control The total number of combinations 432 (3x4x3x4x3) test cases Display Mode Language Fonts Color full-graphics English Minimal Monochrome Hand-held text-only

French Standard Color-map Laptop limitedbandwidth Spanish Documentloaded 16-bit Full-size Portuguese True-color Screen size Pairwise combinations: 17 test cases Language Color Display Mode Fonts Screen Size English Monochrome

Full-graphics Minimal Hand-held English Color-map Text-only Standard Full-size English 16-bit Limited-bandwidth - Full-size English True-color Text-only Document-loaded Laptop French

Monochrome Limited-bandwidth Standard Laptop French Color-map Full-graphics Document-loaded Full-size French 16-bit Text-only Minimal - French True-color - - Hand-held Spanish

Monochrome - Document-loaded Full-size Spanish Color-map Limited-bandwidth Minimal Hand-held Spanish 16-bit Full-graphics Standard Laptop Spanish True-color Text-only - Hand-held

Portuguese - - Monochrome Text-only Portuguese Color-map - Minimal Laptop Portuguese 16-bit Limited-bandwidth Document-loaded Hand-held Portuguese True-color Full-graphics Minimal Full-size

Portuguese True-color Limited-bandwidth Standard Hand-held Exercise: Choose one pair and find a test case which covers this pair Number of test cases Example 1 Example 2 Example3 Number of factors 2 3 5 Number of all combinations 8 27 432 Number of pairwise test cases 4

9 17 Number of factors Number of all combinations Number of pairwise test cases System 1 System 2 61 75 29 2 levels 17 3 levels 15 4 levels 35 2 levels 39 3 levels 1 4 levels 7.4 x 1025 5.5 x 1029 41 28 In-class exercise Consider software with input parameters: Dest: London, Paris, Chicago, NY M: 1000, 2000, 3000

Direct: Y, N Create test cases according to pair-wise approach. Number of all possible combinations? Number of pairs? 4 x 3 x 2 = 24 Minimal number of test cases? Start with factors with max numbers of levels: Dest - M: 4 x 3 = 12 We need min 12 test cases Dest - M: 4 x 3 = 12 Dest Direct: 4 x 2 = 8 M Direct: 3 x 2 = 6 Total: 26 In-class exercise Dest: London, Paris, Chicago, NY M: 1000, 2000, 3000 Direct: Y, N Dest t1 L t2 L t3 L t4

P t5 P t6 P t7 C t8 C t9 C t10 N t11 N t12 N M Direct

In-class exercise Dest: London, Paris, Chicago, NY M: 1000, 2000, 3000 Direct: Y, N Dest M t1 L 1 t2 L 2 t3 L 3 t4 P 1 t5 P 2 t6

P 3 t7 C 1 t8 C 2 t9 C 3 t10 N 1 t11 N 2 t12 N

3 Direct In-class exercise Dest: London, Paris, Chicago, NY M: 1000, 2000, 3000 Direct: Y, N Dest M Direct t1 L 1 Y t2 L 2 N t3 L 3 - t4

P 1 - t5 P 2 Y t6 P 3 N t7 C 1 N t8 C 2 -

t9 C 3 Y t10 N 1 Y t11 N 2 N t12 N 3 - t-Wise t-Wise: A value of each level for each group of t factors must be combined. Sometimes: t-way (3-way, 4-way, etc)

A natural extension is to require combinations of t values instead of 2 t-wise is expensive and benefits are not clear Example: 4-way 30 parameters 5 values each 3,800 test cases (could be too many) Combinatorial Methods in Software Testing National Institute of Standards and Technology (NIST) http://csrc.nist.gov/groups/SNS/acts/index.html Presentation by Rick Kuhn, (NIST) at ECU, March 22, 2012 http://core.ecu.edu/ST RG/seminars.html 16/18 Software Testing Study Software Testing is a part of ECU MSc SE program SENG 6265 Foundations of Software Testing SENG 6270 Software Verification and Validation Software Testing Research Research - ?

New Scientific approach: investigation, analysis, comparison, justification, etc. Publications http://core.ecu.edu/STRG/ Project: Testing of Mobile Applications Android Development Tools Monkeyrunner tools from Android SDK Case study (testing) Robotium Calabash-Android Comparison, analysis Pair-wise approach for mobile testing Cloud Testing of Mobile Systems (CTOMS) framework