# Using the testing framework
 PIO uses a custom testing framework (`<PIO_SOURCE_DIR>/tests/general/util/*`) to generate tests in the `<PIO_SOURCE_DIR>/tests/general` directory. The testing framework provides,

* Faster creation of tests : Easier creation of tests from simple stub source
* Improved testcase code readability : Most of the boiler plate code is generated by the framework
* Better error checking and reporting : User friendly parallel asserts
* More testing with less code : Templated tests, auto generation of boiler plate PIO code
* Easy integration with CTest

# Writing a test

## Testcase format
 All testcases recognized by the framework need to be subroutines with the following tags to indicate the start and end of the procedure,

* `PIO_TF_AUTO_TEST_SUB_BEGIN` : This tag is used in the first line of the subroutine definition followed by the subroutine name. Consider this tag as the replacement of the `SUBROUTINE` keyword in Fortran
* `PIO_TF_AUTO_TEST_SUB_END` : This tag marks the end of the subroutine, like the `"END SUBRTOUINE"` keyword in Fortran. This tag is followed by the subroutine name.

 A simple hello world test program is shown below,

    PIO_TF_AUTO_TEST_SUB_BEGIN hellow

      print *, "Hello world"

    PIO_TF_AUTO_TEST_SUB_END hellow

 The testing framework converts this stub into a Fortran subroutine. The testing framework also adds a test driver that includes necessary code to initialize (and finalize) MPI, PIO and includes calls to all testcases tagged with the `PIO_TF_AUTO_TEST_SUB_BEGIN/PIO_TF_AUTO_TEST_SUB_END` tags.

 Fortran code in the stub source file that don't have `PIO_TF_AUTO_TEST_SUB_BEGIN/PIO_TF_AUTO_TEST_SUB_END` tags are not modified by the framework. A test function cannot call other test functions but can call other Fortran utility functions and subroutines. It is however recommended that these utility functions return error codes that can be interpreted by a test function using a parallel assert provided by the framework (to signal a failure in these utility functions).

## Runtime environment
 The framework also adds code to read command line arguments to set,

* PIO Stride : Set via "--pio-tf-stride" command line option
* Number of PIO I/O tasks : Set via "--pio-tf-num-io-tasks" command line option
* Number of PIO aggregators : Set via "--pio-tf-num-aggregators" command line option
* PIO logging level : Set via "--pio-tf-log-level" command line option

 The above options allow us to run the same test with different PIO parameters.

## Testcase utils

 The following utilities are available from the framework,

*  PIO_TF_CHECK_VAL : Check if two variables are equal. This macro allows checking between two real/integer/double values or arrays. It also allows checking if all the values in an array is equal to a scalar value.

        PIO_TF_CHECK_VAL(scalar_var1, scalar_var2)
        PIO_TF_CHECK_VAL(array1, array2)
        PIO_TF_CHECK_VAL(array1, scalar_var1)

* `PIO_TF_CHECK_ERR` : Macro used to check the return code from a PIO function. The format for the macro is shown below,

        PIO_TF_CHECK_ERR(pio_return_code, `<ERROR MSG>`)

* `PIO_TF_PASSERT` : A parallel assert macro. The format is shown below,

        PIO_TF_PASSERT(`<LOGICAL COND>`, `<ERROR MSG>`)

* `PIO_TF_ASSERT` : An assert (not parallel, local to each process) macro. The format is shown below,

        PIO_TF_ASSERT(`<LOGICAL COND>`, `<ERROR MSG>`)

* `PIO_TF_LOG` : Allows logging info into stdout

        PIO_TF_LOG(*,*)(`<LOG MSG>`)

* `PIO_TF_ERROR` : Used to flag fatal errors

        PIO_TF_ERROR(`<ERROR MSG>`)

* `pio_tf_world_sz_, pio_tf_world_rank_` : Variables that provide size and rank of each process.

## Templated testcases

 Testcases can be templated for the different types supported by PIO. The framework provides a mapped collection of recognized fortran types and the corresponding PIO types. Templated testcases precede with a `PIO_TF_TEMPLATE<>` tag as show below,

    PIO_TF_TEMPLATE<PIO_TF_PREDEF_TYPENAME PIO_TF_DATA_TYPE, PIO_TF_PREDEF_TYPENAME PIO_TF_FC_DATA_TYPE>
    PIO_TF_AUTO_TEST_SUB_BEGIN templated_hellow

     PIO_TF_FC_DATA_TYPE, dimension(10) :: buf
     ...

     call PIO_initdecomp(pio_tf_iosystem_, PIO_TF_DATA_TYPE, dims, compdof, iodesc)

     ...

    PIO_TF_AUTO_tEST_SUB_END templated_hellow

 For the code shown above the framework generates multiple Fortran subroutines by replacing `PIO_TF_FC_DATA_TYPE` and `PIO_TF_DATA_TYPE` with matching Fortran and PIO types respectively.

# Building a testcase
## Generating the testcase

 Testcases are generated by the '<PIO_SOURCE_DIR>/tests/general/util/pio_tf_f90gen.pl' utility. It parses the stub source code provided as a command line argument and generates Fortran test cases (with all the boiler plate code necessary). The utility can be run as shown below,


    ./util/pio_tf_f90gen.pl --annotate-source --out=hellow.F90 hellow.F90.in


 In the above command hellow.F90.in is the stub test case created by the developer, and the pio_tf_f90gen.pl utility creates hellow.F90, the test case. The newly generated testcase (hellow.F90) is a Fortran program that depends only on '<PIO_SOURCE_DIR>/tests/general/util/pio_tutil.F90' to build and contains all the code required to run the tests in it.

## Building the generated testcase
 The test case generated is a Fortran program that depends only on '<PIO_SOURCE_DIR>/tests/general/util/pio_tutil.F90" to build. The test case can be built using any MPI compiler wrapper (mpif90).

 The previous two steps are incorporated into CMakeLists.txt to add a new test to the testsuite.

# Running the testcase

## Running the test suite on your laptop
 The test suite can be run by invoking the "ctest" command from the PIO build directory

## Running the test suite on Supercomputers/Clusters

 TODO

## Running a single test on your laptop
 A single test can be run by passing the test executable to an MPI job launcher (mpiexec), similar to running any MPI program.

    mpiexec -n 4 ./hellow

# Adding a test into the test suite
 The build and running steps need to be incorporated into CMakeLists.txt to add a new testcase into the test suite. Please modify `<PIO_SOURCE_DIR>/tests/general/CMakeLists.txt` as suggested below to add a new test to the test suite,

* Add the testcase stub source into `GENERATED_SRCS`
* Add a CMake build step : To generate the test executable

        add_executable (pio_init_finalize EXCLUDE_FROM_ALL
            pio_init_finalize.F90
            ${CMAKE_CURRENT_SOURCE_DIR}/util/pio_tutil.F90)
        target_link_libraries (pio_init_finalize piof)
        add_dependencies (tests pio_init_finalize)

* Use CMake user defined macro add_mpi_test (or add_test() for mpi-serial) to run the test

        add_mpi_test(init_finialize_1_proc
                EXECUTABLE ${CMAKE_CURRENT_BINARY_DIR}/pio_init_finalize
                NUMPROCS 1
                TIMEOUT ${DEFAULT_TEST_TIMEOUT})

# Debugging failed tests

* Run the test suite using ctest with the "--verbose" option. This outputs a lot of information from the testing framework (to stdout), including the various tests run and the reason for failure. All information in the output from the PIO testing framework is prepended with the `PIO_TF: ` tag. To run a single failing test use the "-R" option available with ctest (e.g. To only run "pio_iosystem_tests3", ctest --verbose -R pio_iosystem_tests3).
* Run the failing test manually (using the MPI job launcher) and debug it. Also consider running the test with PIO log level > 0 (mpiexec -n 4 ./tests/general/pio_iosystem_tests3 --pio-tf-log-level=6)
