Unit testing with MPI, googletest, and cmake
Introduction
In this article we take a short detour into the problem of continuous unit testing
of code that contains MPI calls and use either mpich
or Open MPI. I have recently moved from
CTest-based testing to
a combination of CMake and
googletest. The reason for the shift
is the convenience googletest
provides. However, the googletest
primer
and advanced manual
do not contain many examples and can be cryptic at times. I hope this
article will provide pointers to those who run into a roadblock when testing MPI
applications with googletest
.
Installing googletest
I typically install googletest
as a submodule
in my git repository. For example,
git submodule add git@github.com:google/googletest.git ./googletest
There are several caveats when using git submodules. For our purposes, we only have to remember that when we clone the repository we have to use
git clone --recursive
Other tips can be found in a nicely condensed form in Sentheon’s blog.
Making sure cmake finds and compiles googletest
To make sure that googletest
can be found and is built during the compile process,
I add the following to by root CMakeLists.txt
file:
if (USE_CLANG)
set(CMAKE_CXX_COMPILER "/usr/local/bin/clang++")
endif ()
set(gtest_force_shared_crt ON CACHE BOOL "" FORCE)
add_subdirectory(googletest)
The first line is needed because I have not been successful in passing the clang
compiler
name to googletest
without specifying the full path. I’m sure one can use a more general
approach, but I haven’t felt the need to spend the time trying to figure out a better way.
The add_subdirectory
command is all that is needed for cmake
to compile googletest
and
produce static libraries.
Adding local unit tests
Next I add a UnitTests
directory in my directory of interest and modify the local
CMakeLists.txt
to be able to find the UnitTests
directory. For example,
SET(SPH_SRCS
${CMAKE_CURRENT_SOURCE_DIR}/SmoothParticleHydro.cpp
)
SET(ELLIP3D_SRCS
${ELLIP3D_SRCS}
${SPH_SRCS}
PARENT_SCOPE
)
add_subdirectory(UnitTests)
In this case I am going to test my Smoothed Particle Hydrodynamics code.
The CMakeLists.txt file in UnitTests
Now we are finally really to add our unit tests to the build chain. The CMakeLists.txt
file in the UnitTests
directory has two sections (which can be simplified if you
are so inclined).
In the first section, we find the location of the googletest
headers and libraries:
set(GTEST_INCLUDE_DIR "${CMAKE_SOURCE_DIR}/googletest/googletest/include")
set(GTEST_LIB gtest_main gtest)
include_directories(${GTEST_INCLUDE_DIR})
Note that the two googletest
libraries that we use are gtest_main
and gtest
.
In the second section, we add the actual test that requires MPI
. Once again, these details
can be abstracted into a cmake
function if you so desire.
add_executable(testSPHParticleScatter testSPHParticleScatter.cpp)
target_link_libraries(testSPHParticleScatter
${GTEST_LIB}
ellip3D_lib
${MPI_LIBRARY}
....
)
set(UNIT_TEST testSPHParticleScatter)
set(MPI_COMMAND mpirun -np 2 ${UNIT_TEST})
add_custom_command(
TARGET ${UNIT_TEST}
POST_BUILD
COMMAND ${MPI_COMMAND}
The add_executable
line identifies the unit test program testSPHParticleScatter.cpp
which tests the scatter operation between two MPI processes.
The target_link_libraries
lists the libraries that are needed: the googletest
libraries (GTEST_LIB
) and our coupled DEM-SPH code library (ellip3D_lib
)
Then we set up a command to run (MPI_COMMAND
) and make it use mpirun
. You
can generalize this if you want.
Finally we, add the add_custom_command
line that tells cmake
to run the MPI_COMMAND
after the build is complete (POST_BUILD
).
The actual test C++ code
Now that the build system has been configured, we just write our unit test
testSPHParticleScatter.cpp
.
First, we include the required headers:
#include <SmoothParticleHydro/SmoothParticleHydro.h>
#include <gtest/gtest.h>
#include <boost/mpi.hpp>
Note that we are using the Boost MPI wrappers.
The MPI test environment class
But we cannot use the boost::mpi::environment
call to set up MPI
(because the environment
object is deleted before googletest
s are run). Instead, we have to set up a custom environment
to run our MPI
tests by creating a MPIEnvironment
class that extends the ::testing::Environment
class provided by googletest
.
class MPIEnvironment : public ::testing::Environment
{
public:
virtual void SetUp() {
char** argv;
int argc = 0;
int mpiError = MPI_Init(&argc, &argv);
ASSERT_FALSE(mpiError);
}
virtual void TearDown() {
int mpiError = MPI_Finalize();
ASSERT_FALSE(mpiError);
}
virtual ~MPIEnvironment() {}
};
Here, the SetUp
function calls MPI_Init
and sets up the environment while the TearDown
function
calls ‘MPI_Finalize`. All tests are performed when the environment is active.
The main
test function
The main
function in typically not needed in standard googletest
tests and is generate
by some internal magic. However, we do need a main
function in testSPHParticleScatter.cpp
because we are using mpirun
to run the test.
int main(int argc, char* argv[])
{
::testing::InitGoogleTest(&argc, argv);
::testing::AddGlobalTestEnvironment(new MPIEnvironment);
return RUN_ALL_TESTS();
}
The MPI specific environment is created by the AddGlobalTestEnvironment
function to which we
pass a MPIEnvironment
object. The object is deleted internally by googletest
.
The tests in testSPHParticleScatter.cpp
are then run using RUN_ALL_TESTS()
. Note that this
function returns from main
and, therefore, and non-googletest environment objects created in
main (e.g., with boost::mpi::environment) are deleted before the tests are run.
The actual test
Finally, we add an actual test to testSPHParticleScatter.cpp
as follows:
TEST(SPHParticleScatterTest, scatter)
{
// Set up communicator
boost::mpi::communicator boostWorld;
// Set up SPH object
SmoothParticleHydro sph;
sph.setCommunicator(boostWorld);
// ... Some code to create particles
// ....
sph.setParticles(particles);
sph.scatterSPHParticle(domain, ghostWidth, domainBuffer);
if (boostWorld.rank() == 0) {
EXPECT_EQ(sph.getSPHParticleVec().size(), 100);
} else {
EXPECT_EQ(sph.getSPHParticleVec().size(), 200);
}
}
The test can be written in the standard way and numerous examples can be found on the web. Of course, we have to be careful about keeping in mind that the test will be run on two processes in this particular case.
Caveat
The Boost
MPI environment created by
boost::mpi::environment
allows MPI calls to
fail without throwing an exception. For example, in my Patch
code discussed in
an earlier article, I have
MPI_Cart_rank(cartComm, neighborCoords.data(), &neighborRank);
I deliberately send invalid neighborCoords
to this function to find out if a patch
is a boundary patch. This does not create a problem when I use the boost::mpi::environment
set up. But when setting up the environment explicitly for the unit tests, I have
to add
MPI_Errhandler_set(cartComm, MPI_ERRORS_RETURN);
before I call MPI_Cart_rank
to make sure I don’t get errors when I use invalid
values of neighborCoords
.
The output from make
Now, if we run make
(after setting up the makefiles with cmake, of course),
we not only compile the code but also run the unit test! In this particular case,
here’s what the output looks like:
[ 69%] Built target ellip3D_lib
[ 71%] Built target paraEllip3D
[ 74%] Built target gtest
[ 76%] Built target gtest_main
.......
Scanning dependencies of target testSPHParticleScatter
[ 90%] Building CXX object SmoothParticleHydro/UnitTests/CMakeFiles/testSPHParticleScatter.dir/testSPHParticleScatter.cpp.o
[ 91%] Linking CXX executable testSPHParticleScatter
[==========] Running 1 test from 1 test case.
[----------] Global test environment set-up.
[==========] Running 1 test from 1 test case.
[----------] Global test environment set-up.
Set up environment
Set up environment
[----------] 1 test from SPHParticleScatterTest
[----------] 1 test from SPHParticleScatterTest
[ RUN ] SPHParticleScatterTest.scatter
[ RUN ] SPHParticleScatterTest.scatter
[ OK ] SPHParticleScatterTest.scatter (5 ms)
[----------] 1 test from SPHParticleScatterTest (5 ms total)
[----------] Global test environment tear-down
[ OK ] SPHParticleScatterTest.scatter (6 ms)
[----------] 1 test from SPHParticleScatterTest (6 ms total)
[----------] Global test environment tear-down
Tore down environment
[==========] 1 test from 1 test case ran. (289 ms total)
[ PASSED ] 1 test.
Tore down environment
[==========] 1 test from 1 test case ran. (289 ms total)
[ PASSED ] 1 test.
[ 91%] Built target testSPHParticleScatter
[ 95%] Built target gmock
[100%] Built target gmock_main
Remarks
I hope this article has been of use to you. Our series on communication between patches will continue when I get some free time.