Testing in CamiTK

From Forge Wiki

Jump to: navigation, search

CamiTK provides some tools to test your code At different granularity levels. In this wiki page you will find information on our testing procedure, what we test and how you can use our already defined tests.



Testing a software code is always a good practice to detect bugs and errors. Several types of tests exist depending on the granularity level the tests are based on. There are classically three categories of tests: unit testing, functionnal testing and integration testing.

Note that, our software testing methodology is only of of the many different ways of testing a software. For more information about software testing, please refer to the wikipedia article.

For each type of test, tests are built following the same principles. First of all, In the next section we will present a (very limited and simple) testing crash course. Next, we explain how testing is done In CamiTK and describe the automatic test framework provided by CamiTK (and what are the plans to improve it). Finally, the last section explains the basic of adding new tests In CamiTK.

Basic testing crash course

To create any type of test (unit, functional ...), three basic steps are always followed:

  • Implement the C++ code that will test your C++ class method / application ...
  • Configure your project to create some cross-platform tests that would call your C++ code In order to run your test on any platform.
  • Check the test results.

C++ code to test your class / program

Assuming we have some C++ code to test gathered In a library / extension, we then basically create a C++ application In order to:

  • Load the library containing the C++ classes to test.
  • Do some tests on the class' methods and output the test result.

A number of unit testing framework exists such As Boost, QTest...
They can help you automate to design, setup and continous integration test steps.

Cross-platform independent tests definition and configuration

Once the test C++ application is built, it should Be run each time a modification is done on the tested code. Using more than one plateform (I.e., OS + compiler) generally makes a more stable code, therefore testing should Be a "natural" stage after configuration and build.

CMake comes with a specific module to add, configure, and run test. Using CMake makes it possible to define and configure tests once for all platforms. To define and configure tests using CMake, two specific CMake macros should Be used:

  • enable_testing() to enable the testing functionality on our project
  • add_test(...) to create a new test.

More specifically:

  • The test executions are done using CTest, which manage which test are run
  • The test results can Be sent to a dashboard using CDash

Check the test results

Once the tests are run, the results should Be sent to dashboard where the programmer can analyse the results (I.e., check which test pass, which fail). This is crucial to identify problems and potential bug. Another step is to check the test coverage, I.e., how much of the actual code lines are checked during the tests. This is crucial to identify where your test code can Be improved In order to check more things.

Overview of the CamiTK testing framework

CamiTK comes with a specific test framework built on top of the CMake/CTest framework. This section describes the main level of tests that are used In the CamiTK open-source project.

Note that the automatic test defined for the CamiTK open-source can Be used In any other CEP independently.

Unit testing

Unit testing consider testing the lowest parts of the code, such As the different methods of your class.

This is a big initiative that we are currently working on. A unit test framework should Be used for that.

Functional testing

Functional testing involves testing the main functionality of a software, I.e., test the code At a higher granularity level than unit testing.

In CamiTK, we develop some automatic functional tests for component and action extensions. Those tests are automatically configured using the CamiTK CMake macros and are run on our continuous integration system. They are called "automatic" tests are they do not focus on a specific code, but take the advantage of the CamiTK API to automatically check the main expected behaviour of component and action extensions.

In the near future, we plan to add more consequent and relevant functional testing by using the Action state machine application to design test more specific to components and actions.

Integration testing

In the CamiTK framework, we consider that integration testing mainly consists of:

  • Checking that CamiTK extensions can Be loaded, can work together and can Be used In a specific context. At the moment, this is based on the camitk-config application (using a bash script)
  • Checking that the code generator can automatically creates valid C++ code, that can Be configured, built and integrated In a specific context. This is based on camitk-cepgenerator application (using a bash script).

In the near future, we plan to add more consequent and relevant functional testing by using the Action state machine application to design more specific tests that takes the specific functionalities of components and actions into account.

Note that a part of the integration tests is also done during the automatic functional test for component and action extensions (e.g. check if a given extension is loaded properly In a CamiTK application).

CamiTK test results and test coverage dashboard

All the test are run each time a modification is detected on the git source code management system. All test are reported on the CamiTK dashboard.

Not only the test results are visible, but CamiTK also offers a way to check the test coverage.

CamiTK testing framework

The CamiTK testing framework is defined As a layer on top of CMake/CTest, and provides:

  • Automatic test for action and component extensions,
  • Specific CMake macros that help defining test and test coverage
  • A method to test the integration of action and component extensions,
  • A method to test correct CamiTK installation As well As C++ code generation, configuration, and integration.

The following sections describe these four different parts.

Automatic test for action and component extensions As well As test coverage procedures can Be used In any CamiTK Extension Project.

Please visit this wiki page for more information about how to use the testing framework for your CEP.

Automatic test of action and component extensions

As CamiTK defines an API for action and component extensions, this API can Be used to check the default/expected basic behaviour of component and action extensions. In order to do so, the CamiTK testing framework provide two extra C++ applications: testactions and testcomponents:

  • The testactions application provide (At the time of writing) one level of automatic test for a given action extension.
  • The testcomponent application offers (At the time of writing) three different levels of automatic test for a given component extension.

These tests can Be configured and run for any CEP thanks to the camitk_extension CMake macros (see below). They are used In the CamiTK open-source multi CEP and the results are visible on our dashboard, but they can also Be used In any other CEP. Moreover it is possible to test other CEP component exensions with installed action that were not aware of these component extensions when they were initially built and tested (this is called the "additional component extension automatic tests").

Detail description of the automatic extension test framework is available In this wiki page.

Testing action extensions

To test each action provided In the CamiTK framework, we insure that calling it on the proper components it has been defined on, works correctly. The testaction application has three main steps:

  1. Load all the available CamiTK components the action can work on,
  2. Load the action extension
  3. Open a component file and set it As the action input component,
  4. Call the action (I.e., run the action'S apply() method),
  5. The test passes if the apply() methods returns Action::ApplyStatus::SUCCESS or Action::ApplyStatus::ABORT (In this case please check that the ABORT status is what is expected) and fails otherwise.

Testing component extensions

To test each component In the CamiTK framework, we insure that opening file associated with this component and (when possible) saving it both work. To do so, to test one specific component, we uses the testcomponents application that for each available (or specified) files of the component'S testdata subdirectory:

  1. Load the component extension
  2. Open the given file (test level 1)
  3. Save the file using a different name (optional test level 2)
  4. Check that the saved file is identical to the original file (optional test level 3)

The optional test define a possible test level (see this wiki page for more information).

Testing extensions In CamiTK SDK

The list of automatic extension tests done In the CamiTK SDK is available In this wiki page.

CamiTK CMake macros for testing

CamiTK testing framework provides three specific CMake macros to configure test more efficiently inside the CamiTK environment:

  • Camitk_init_test(...) helps to initialize all the common CMake configuration for all the test of the current CMake project.
  • Camitk_add_test(...) adds a new test for the current CamiTK project. This macro encapsulates the default add_test(...) macro provided by CMake.
  • Setup_target_for_coverage(...), entirely based on modules/blob/master/CodeCoverage.cmake Lars Bilke macro, simplifies the generation of test coverage report. More information about enabling test coverage for your CEP can Be found In this wiki page

These macros can Be called directly In the CMakeLists.txt file. It allows you to manually add your tests to your CamiTK extension, library or application.

In the CamiTK open-source project each extension are tested using camitk_init_test(...) and camitk_add_test(...) macros As these macros are used for generating automatic tests In the camitk_extension(...) macro (see this wiki page for more information). In the CamiTK open-source project, most of the applications are also tested using camitk_init_test(...) and camitk_add_test(...) As the latter facilitates the declaration of test outcome expectancy (what should Be expected from the test outcome).

Once tests have been configured, they can Be either run on your own platform or integrated In our continuous integration system.

Testing the extension integration

Once a component or action extension is built, it is important to check that it can Be loaded and used In a CamiTK application. CamiTK provides the "config" (camitk-config) application. For non-installed extension, it should Be run from the root directory or an extension repository (e.g, the build dir of your extension). camitk-config provides all the information you need to check that an extension can Be loaded (I.e., all the shared library required by the extension can Be loaded).

The CamiTK open-source project tests if the number of expected action and component extensions are found. This test is defined In a bash script, but you can use camitk-config In any CEP to check that the number of extensions provided by your project is correct As well.

Testing the CamiTK development framework

Once installed, CamiTK should Be ready to Be used In a CEP.

The CamiTK open-source project tests test that the CamiTK development framework is ready to use doing the following steps (all In bash script):

  1. Test that the C++ code generation from a cep core schema XML is possible: this tests the availability of the camitk-cepgenerator application
  2. Test that the generated code can Be configure: this tests that the CamiTK is installed correctly with all the provided CMake environment and CamiTK macros
  3. Test that the generated code can Be compiled: this tests that the core library and other dependencies are available with all the C++ compilation tool chain
  4. Test that the generated code can Be integrated: this uses camitk-config to test that the newly compiled extensions are immediately available In a CamiTK application

Note that many different cep core schema XML examples are tested. All the XML examples are included In the bash script.

You can also create your own example from a tweaked core schema XML if your CEP provide a component or action extension than can Be used itself by other developers In external CEP.

Using the CamiTK testing framework In your CEP

At the moment there are 4 different ways you can improve your CEP by enabling tests thansk to the CamiTK testing framework:

  • Configure your action / component to run the automatic tests → see this page,
  • Configure your component to run automatic test on an already installed action → see this page,
  • Enable test coverage report → see this page,
  • Define and create your own tests using the CamiTK testing CMake macros → see this page.
Powered By FusionForge