$wgGroupPermissions['user']['read'] = true; $wgGroupPermissions['*']['read'] = true; Testing in CamiTK - Forge Wiki
SCM

Testing in CamiTK

From Forge Wiki

Jump to: navigation, search

CamiTK provides some tools to test your code at different granularity levels. In this wiki page you will find information on our testing procedure, what we test and how you can use our already defined tests.

Contents

Introduction

Testing a software code is always a good practice to detect bugs and errors. Several types of tests exist depending on the granularity level the tests are based on. In CamiTK we categorize our tests in three categories: unit testing, functionnal testing and integration testing.

Note that, our software testing method is only a part of many different ways of testing a software. For more information about software testing, you may read the wikipedia article about it.

For each type, tests are built following the same principles. First of all, in the next section we will present the steps followed to create test in CamiTK. Next, the different testing level section talks more about what tests are already implemented in CamiTK (and what are our plans to improve them). Finally, the last section deals more about presenting how to write new tests in CamiTK.

CamiTK testing steps

To create any type of test (unit, functionnal ...), two basic steps are always followed:

  • implement the C++ code that will test your C++ class method / application ...
  • configure your project to create some cross-platform tests that would call your C++ code in order to run your test on any platform.

C++ code to test your class / program

Assuming we have some C++ code to test gathered in a library / extension, we then basically create a C++ application in order to:

  • load the library containing the C++ classes to test.
  • do some tests on the class' methods and output the test result.

For this last part, for unit testing you might use an available testing framework such as Boost or Qt.
In CamiTK, we did not have chosen one specific framework yet in our unit testing procedure.

Cross-platform independent tests definition and configuration

Here the idea is to configure some tests to run the C++ application previously built and analyse its results in some specific context. Those results would help appreciate whether the test passes or not.

As you know, in order to be cross-platform, CamiTK uses some of Kitware tools, such as CMake for its configuration. In the same philosophy, our testing procedure is done once for all platforms.
More specifically:

  • the tests definition and configuration are also done using Cmake, which allows defining and configuring tests once for all platforms.
  • the tests execution is then done using CTest , which allows running each test on any platform.
  • the tests results are displayed on our dashboard, built using CDash.

CMake macros for testing

To define and configure tests using CMake, we use two specific CMake macros:

CamiTK CMake macros for testing

To faster the configuration of our tests, we created two CMake macros:

  • camitk_init_test(): to initialize all the common CMake configuration for all the test of the current CMake project.
  • camitk_add_test(): add a new test for the current CamiTK project. Encapsulate the default add_test CMake macros.

Therefore, to configure any CamiTK test, in the CMake configuration of our CamiTK extension (i.e. in the CMakeLists.txt file) we call:

  • once the camitk_init_test() macro to initialize our tests for our extension.
  • once for each test the camitk_add_test() macro.

Those macros are either called:

  • directly in the CMakeLists.txt file. It allows you to manually add your tests to your CamiTK extension.
  • automatically by using the camitk_extension CamiTK CMake macro with the specific options to enable some automatic testing.

Once tests have been configured, they can be either, run on your own platform or integrated in our continuous integration system

The different tests done in CamiTK

Unit testing

Unit testing in CamiTK consider testing the lowest parts of your code, such as the different methods of your class.
At the moment, we did not have defined a precise process of unit testing in CamiTK, but we are working on a specific one.

Functionnal testing

Functionnal testing involves testing your software at a higher granularity level than unit testing.
In CamiTK, we develop some functionnal tests on the component and action extensions. Those tests are automatically configured using the CamiTK CMake macros and are run on our continuous integration system.

These functionnal tests are categorized in several levels

Automatically testing of actions and components

Automatic tests on actions and components are already implemented in the CamiTK framework. Those tests are configured in our CMake macros and the results are visible on our dashboard. For those tests we followed the two basic steps to build tests.

  • We created two C++ applications testactions and testcomponents, runned by the corresponding tests.
  • We configured many test using the camitk_init_test and the camitk_add_test CamiTK CMake macros. Those macros are automatically called within the camitk_extension CamiTK CMake macro.

Actions

To test each action provided in the CamiTK framework, we insure that calling it on the proper components it has been defined on, works correctly. To do so, to test one specific action, we uses the testaction application

  • We load all the CamiTK components the action can work on. Each component has a set of files (it can open), located in its testdata/ctestdata directory.
  • For each file in each component's testdata directory, we open it and call the action on it (i.e. running the action's apply() method on it).
  • The test passes if the action return Action::ApplyStatus::SUCCESS, fails otherwise.

At the moment, those tests have a known limitation: if the action cannot be applied to this type of component, the test passes.

Components

To test each component in the CamiTK framework, we insure that opening file associated with this component and saving it (when available) both work. To do so, to test one specific component, we uses the testcomponents application.

  • We open each file the component has within its testdata/ctestdata folder.
  • [If available] we save the file using a different name
  • [if possible] we make a diff between the input and the saved file.

For each step and according the component's testing level configuration a test is created and run.

Integration testing

In the CamiTK framework, we consider integration testing our highest granularity level of testing. It consists in integrating CamiTK extensions together and running them in a specific context. At the moment we do not do integration testing but we work on using the Action state machine application to run some integration tests.

How to create your tests

At the moment you have two ways of creating tests in CamiTK:

  • define and create manually your tests following the common steps
  • configure your action / component to run some of our automatic tests

Manually create your tests

In this section you will find information on how to create manually your tests for your CMake project.

Initialize testing

The first thing is to tell CMake that for this project you want to enable your test. In the top level CMakeLists.txt of your project (if you're working on a CamiTK Extenstion Project, open the top level CMakeLists.txt file, where the camitk_extension_project(...) macro is called) and write down the following line:


enable_testing()


This line is needed before all calls to the add_test() or camitk_add_test() CMake macros, else no tests would be configured.

Create an application for testing

You have one or more libraries containing C++ classes with methods that you would want to test. The basic idea to test those methods is to create a simple C++ application that will be in charge of loading the libraries you want to test and call your classes' methods to test. Depending on the results of your test, the application will return different values at runtime. Two different values are already defined in the stdlib: EXIT_SUCCESS and EXIT_FAILURE.

In other words, if your C++ application returns:

  • EXIT_SUCCESS the test will be considered as passed by CTest.
  • EXIT_FAILURE the test will be considered as failed.


To get more information than a simple PASSED or FAILED when running your test, your application may also print some information on the standard output (std::cout). All the output printed on it will be displayed when running your tests, especially on our dashboard. This is a good practice to get more information on the reason why a test passes or fails.

Also, if you want to call a same application to run several tests, you will have to call it with different input arguments in order to distinguish which test the application would have to run. To do so, simply switch the input argument of your C++ applications and consider each input argument separately.

For some example of such applications, take a look at the CamiTK applications testactions, testcomponents, pmltest or lmltest located in the CamiTK SDK and Modeling CEP

Configure your test using CMake

Once you implemented your C++ application which is in charge of calling your test, you will have to configure each test separately using CMake macros. You may use two types of macros, the basic CMake one add_test() or our CamiTK CMake macros camitk_init_test() and camitk_add_test().

We recommend you to use the CamiTK CMake macros as they encapsulate the original CMake macro add_test() and add some useful functionalities.

In the CMakeLists.txt file of your extension / library you would like to test :

  • call once your the camitk_init_test() macro.
  • for each test, call once camitk_add_test() macro. Check out this macro signature to call it with the good arguments. You will have to configure the test by telling it to run your C++ application with some specified input arguments. Here you will define which arguments will be send to your C++ application.

Automatically create tests for your CamiTK actions / components

The second and faster way of doing test in CamiTK, is to use our already implemented and configured functionnal tests on actions and components. Here is the procedure to automatically test your action / component:

  • Create a testdata/ctestdata directory in the source code of your extension.
  • Add all the file handled by your component you wish to test in this folder.
  • Find out which LEVEL of automatic testing suits you best.
  • In the camitk_extension() CamiTK CMake macro call of your extension's CMakeLists.txt file add the following option as an argument of this macro.

for an action:

TEST_APPLICATION ${APPLICATION_TEST_ACTION}

for a component:

TEST_APPLICATION ${APPLICATION_TEST_COMPONENT}


This will automatically configure tests to the highest level possible (1 for actions and 3 for components). To downgrade the level of your test, specifically on components add the following option:

  • DISABLE_TESTLEVEL3: to downgrade to tests of level 2 (which would create one test level 1 and one test level 2 per file to test + one test of level 0).
  • DISABLE_TESTLEVEL2: to downgrade to tests of level 1 (which would create one test level 1 per file to test + one test of level 0).
  • DISABLE_TESTLEVEL1: to downgrade to tests of level 0 (which would create one test level 0 only).

Run your tests

CMake automatically creates a target to run the test you have configured.

  • On Linux system, the target's name is test so simply run
make test
  • On Windows sysem, running a Visual Studio instance, simply build the project RUN_TESTS
Powered By FusionForge