Experience Report - Applied Physics Laboratory

Experience Report - Applied Physics Laboratory

Model-based Testing of a Software Bus applied on Core Flight Executive Dharmalingam Ganesan, Mikael Lindvall Dave McComas (NASA GSFC) 2014 Fraunhofer USA, Inc. Center for Experimental Software Engineering 1 Traditional Test Automation Only test execution is automated Test cases are manually constructed E.g. Junit, CuTest, etc. Effort intensive Some issues with traditional testing:

Insufficient coverage of off-nominal behaviors Tests are too-detailed with low-level details Not easy to test multi-tasking architecture More on this later. 2014 Fraunhofer USA, Inc. Center for Experimental Software Engineering 2 Model-based Testing (MBT) The tester develops a model (a.k.a. testing models) The model becomes the test oracle Test cases are auto-generated from the model

instead of writing suite of test cases based on requirements, API documentations Key benefits: Tester works at a high-level of abstraction Innumerable number of test cases derived from the model Triggers many, if not all, off-nominal behaviors Precise specification 2014 Fraunhofer USA, Inc. Center for Experimental Software Engineering 3 System under test (SUT) The Core Flight Software System (CFS) a mission-independent, platform-independent, Flight

Software (FSW) environment integrating a reusable core flight executive (cFE) The CFS is a product-line developed by the NASA Goddard Space Flight Center (GSFC) CFS is implemented in C cFE layer of the CFS is open source 2014 Fraunhofer USA, Inc. Center for Experimental Software Engineering 4 cFE/CFS Context Diagram Housekeeping EDAC Memory Memory Checksum Scrubber Manager Memory Dwell Self Test

Mass Storage GN&C Instrument System ApplicationsManager (4) Data Storage Software Scheduler File Manager Stored Commanding Inter-task Message Router (SW Bus) Health & Safety Manager Limit Checker Time Table Executive Event 1553 Bus Telemetry CommandSoftware Services Services Services Services

Bus Support Output Ingest Commands cFE core AppSummit Chip CFS Applications Mission Apps Local Storag CFDP File e Transfer Comm Cards Transponders Real-time Telemetry (UDP) 2014 Fraunhofer USA, Inc. File downlink Center for Experimental Software (CFDP) Engineering 5

MBT of Software Bus cFE has a software bus (SB) SB has unit-tests (developed by NASA GSFC) Good coverage but not taking multi-tasking into consideration Goals: find defects related to multi-tasking (difficult!) Apps communicate indirectly using the SB Publish-Subscribe architectural style Generate test cases for SB

Generate the bubbles (the apps on previous slide) Demonstrate the applicability of MBT Developed an approach of testing SB Allows testing of multi-tasking architectures 2014 Fraunhofer USA, Inc. Center for Experimental Software Engineering 6 Scope of the current model Modeled the following behaviors Create Pipes (to hold messages) Delete Pipes Subscribe to Messages Send Messages Receive Messages

Multiple Apps (dynamically instantiated) Innumerable test cases (in C) auto-generated Model based on Microsofts Spec Explorer Tool Will get back to this 2014 Fraunhofer USA, Inc. Center for Experimental Software Engineering 7 Challenges of testing a SB To test a SB we need apps that publishsubscribe Apps are runtime tasks that communicate using the SB

Each app cannot decide on its own the correctness Apps publish/subscribe to messages Correctness depends on the global state of the system E.g: subscribe(msg), RecvMsg() may not work if no other task is publishing any message The order of execution of tasks also matters 2014 Fraunhofer USA, Inc. Center for Experimental Software Engineering Need a test architecture! 8 Test Architecture Key Ideas

Parent/Child architecture for testing Each test case is a parent Each test case runs as an app At run-time, one or more child tasks are spanned by the parent Model controls the behavior of the parent All test assertions are decided by the parent All child tasks share the codebase How the parent and children communicate? Why not just use the software bus itself? 2014 Fraunhofer USA, Inc. Center for Experimental Software Engineering 9 Test Architecture

Command Pipe e p i P t l u s Re Pare nt Child 1 Child 2 Child n CFE All child tasks share the codebase 2014 Fraunhofer USA, Inc. Center for Experimental Software Engineering

10 Test Architecture Key Ideas Each child task subscribes to all commands, such as Create Pipe, Delete Pipe, Subscribe, etc. Parent broadcast commands to all child tasks Task id of the child is also part of the message struct Only the target child can perform a certain command Child tasks perform the command and send the return code back to the parent Communication uses CFE infrastructure

Child tasks send out a result msg Parent task subscribes to the result msg The parent verifies test assertion Asserts are generated from the model 2014 Fraunhofer USA, Inc. Center for Experimental Software Engineering 11 Spec Explorer Brief Background Tester develops a model (a.k.a. model program) Spec Explorer runs as a plug-in to MS Visual Studio Model programs are written in C# like syntax The model program is a simplified version of the SUT Spec Explorer generates state machines from models

Also checks whether model satisfies invariants Helps in validating the model Test cases are automatically derived from state machines SUTs behavior is automatically compared with model Tests failure: Deviation between model and SUT Tests success: model and SUT are consistent 2014 Fraunhofer USA, Inc. Center for Experimental Software Engineering 12 Abstractions for the model program

Model program is another implementation of the SUT But we do not want to create two implementations No one wants to maintain two implementations No one wants to develop a system two times How did we create a simplified version of the SUT? Key idea: Apply abstractions Model of GPM, which uses CFE 2014 Fraunhofer USA, Inc. Center for Experimental Software Engineering 13 Sample abstractions for a model

Model is agnostic to multi-tasking complexity of the SB Model has a very simple message structure Message is modeled as an int (not C structures) Message queues/pipes are also abstracted Finite depth Message queues are modeled as simple sequences instead of using shared memory No pointers, threads, semaphores at the model level Very simple data structure using very basic data

2014 Fraunhofer USA, Inc. Center for Experimental Software types Engineering 14 Structure of the model program E.g.: Number of apps, pipes Represent the state of the bus Used for excluding the uninteresting states Which states are good for terminating tests Models the actual logic of the software bus Preconditions for enabling the rule methods to fire Generates values for rule metho Utilties for rule methods and guards 2014 Fraunhofer USA, Inc. Center for Experimental Software Engineering 15 State Data 2014 Fraunhofer USA, Inc.

Center for Experimental Software Engineering 16 Fragments of the model program Rules are enabled only if Condition.IsTrue returns true 2014 Fraunhofer USA, Inc. Center for Experimental Software Engineering 17 Fragments of the guards 2014 Fraunhofer USA, Inc. Center for Experimental Software Engineering 18 Slicing the model for specific tests 2014 Fraunhofer USA, Inc. Center for Experimental Software Engineering 19

Generated from the model program We generate the model! In regular MBT you have to manually create the model. 2014 Fraunhofer USA, Inc. Center for Experimental Software Engineering 20 Generated test sequences - sample Each chain is a test case 2014 Fraunhofer USA, Inc. Center for Experimental Software Engineering 21 SUT Adapter

Adapter wraps the SUT Converts data/commands from the model into SUTs syntax Adapter simplifies modeling complexity Methods of the model should map to the adapter Our adapter is in C# We print test code from our adapter in C Converts C# tests into C tests Recall that CFEs SB interface is in C 2014 Fraunhofer USA, Inc. Center for Experimental Software Engineering 22 Our abstracted interface for Testing int32 InitApp_w(int32 appName);

int32 CreatePipe_w(int32 appName, int32 pipeName, int32 pipeDepth); int32 DeletePipe_w(int32 appName, int32 pipeName); int32 Subscribe_w(int32 appName, int32 msgId, int32 pipeName); int32 UnSubscribe_w(int32 appName, int32 msgId, int32 pipeName); int32 RcvMsg_w(int32 appName, int32 pipeName, int32* actualMsgId); 2014 Fraunhofer USA, Inc. Center for Experimental Software Engineering int32 SendMsg_w(int32 appName, int32 msgId); 23 Sample generated test case void Parent_TestAppMain( void ) { int32 status; uint32 RunStatus = CFE_ES_APP_RUN; Parent_TestAppInit(); status = InitApp_w(APP_0); assert(status == CFE_SUCCESS); status = CreatePipe_w(APP_0, PIPE_0, 1); assert(status == CFE_SUCCESS); status = Subscribe_w(APP_0, MSG_0, PIPE_0); assert(status == CFE_SUCCESS); status = Subscribe_w(APP_0, MSG_1, PIPE_0);

assert(status == CFE_SUCCESS); status = UnSubscribe_w(APP_0, MSG_1, PIPE_0); assert(status == CFE_SUCCESS); } 2014 Fraunhofer USA, Inc. CFE_ES_ExitApp(RunStatus); Center for Experimental Software Engineering 24 Advantages of using Model-Based Testing The model focuses on the domain (easier to understand) We automatically generate an endless number of executable test cases (high coverage) Instead of manually writing individual test cases

The information is in one place: in the model, easy to maintain Instead of being source code oriented (harder to understand) Instead of being spread out (hard to maintain) The test cases can easily be run over and over again 2014 Fraunhofer USA, Inc. Center for Experimental Software Engineering 25 Advantages of using Spec Explorer Generated tests are pretty readable Data parameters are well handled

E.g., Model can be configured to test multiple apps Models are programs This is due to the ability to slice models into smaller models Ideal for programmers (who prefer coding) Models can be formally verified Invariants encoded in the model help to validate the model 2014 Fraunhofer USA, Inc. Center for Experimental Software Engineering 26 Challenges with Spec Explorer

Modeling errors can lead to infinite state machine Syntax for slicing the model is powerful but not that easy Easy to misuse some of (algebraic) operators for slicing Completeness of our slices Need to be careful even for small models (e.g., int parameters) Did we miss any combination of behaviors during slicing? Model debugging. For example:

Why a new state was generated? Where/Why the invariants are violated? 2014 Fraunhofer USA, Inc. Center for Experimental Software Engineering 27 Applicability to other flight software The same approach is applicable to other types of sw Requirements are that The software has an interface (e.g. API, GUI) Through which commands (stimuli) can be sent Through which results (responses) can be received Need (some) specification Optional: Sample test cases, API usage examples 2014 Fraunhofer USA, Inc.

Center for Experimental Software Engineering 28 Conclusion MBT works well for testing of multi-tasking architecture Parent/Child test architecture facilitates testing Individual tasks cannot decide correctness of their own Parent coordinates with children and asserts correctness Models and generated state machines: a good spec! Innumerable number of test cases from the model

In this case of a software bus Test cases are agnostic to cFE syntax but still executable Need to be careful in managing the models complexity Abstraction is important Otherwise the model will be as complex as the system under test 2014 Fraunhofer USA, Inc. Center for Experimental Software Engineering 29 Acknowledgement Jan-Philip Quirmbach (Fraunhofer Intern) Alan Cudmore (NASA GSFC) OSMA SARP:

cFE is open source not an issue for foreign interns Martha Wetherholt (NASA HQ) Ken Rehm (NASA IV&V) Ricky A. Forquer (NASA IV&V) This work was partly funded by SARP 2014 Fraunhofer USA, Inc. Center for Experimental Software Engineering 30 Questions Dharma Ganesan ([email protected]) 2014 Fraunhofer USA, Inc. Center for Experimental Software Engineering

31

Recently Viewed Presentations

  • History of Engineering - University of Alabama

    History of Engineering - University of Alabama

    ME 383 Modern Manufacturing Practices Lecture Note #3 Stress-Strain & Yield Criteria Dr. Y.B. Guo Mechanical Engineering The University of Alabama
  • The Forecast Process

    The Forecast Process

    How has the circulation evolved? Why did past forecasts go wrong or right? Step 3: The Forecast Funnel. Start with the synoptic scale and then downscale to the meso and local scales. Major steps: I. Synoptic Model Evaluation Which synoptic...
  • Exploring Angles and Angle Relationships

    Exploring Angles and Angle Relationships

    Vertical angles- two nonadjacent angles formed by two intersecting lines. Vertical angles are congruent (equal). Linear pair- adjacent angles whose . noncommon. sides are opposite rays (add to make 180°). Supplementary- add to make 180. Complementary- add to make 90.
  • https://www.youtube.com/watch?v=yHfLyMAHrQE Making of the Modern World  THE EMPIRE

    https://www.youtube.com/watch?v=yHfLyMAHrQE Making of the Modern World THE EMPIRE

    What is post-colonialism? Not chronological, but a conceptual term . A post-colonial perspective looks at phenomena in the world . in the wake of colonialism
  • Chapter 11: Heat Chapter 12: Thermodynamics

    Chapter 11: Heat Chapter 12: Thermodynamics

    Review. Chapter 11. Heat. heat - a form of energy in transit. SI unit is the joule (J) common nonstandard units are the kilocalorie (kcal) and the British thermal unit (BTU) mechanical equivalent of heat- relates joules to kilocalories.
  • Gas Chemistry - Columbia College

    Gas Chemistry - Columbia College

    Gases have much lower densities than liquids and solids. 5.1 Physical Characteristics of Gases Units of Pressure 1 pascal (Pa) = 1 N/m2 1 atm = 760 mmHg = 760 torr 1 atm = 101,325 Pa 5.2 Barometer Pressure =...
  • Unit - III Problem 6 Posterior mediastinum

    Unit - III Problem 6 Posterior mediastinum

    Thoracic splanchnic nerves. Introduction Esophagus It is a muscular tube passing between the pharynx in the neck and the stomach in the abdomen (from C6 - T11). Descends on the anterior aspect of the bodies of the vertebrae, generally in...
  • LECTURES 25 - 26 The Euro In 1961

    LECTURES 25 - 26 The Euro In 1961

    The plan was administered by a European High Authority, which encouraged collective action by member countries to solve common problems. Many cooperative arrangements were instituted, including a smooth international payments system called the European Payments Union, or EPU, in 1950.