Note
Looking for help? We’ve got you covered! ❤️
The goal of this tutorial is to demonstrate how to use Mull with ‘non-standard’ test suites, such as when the test suite is a separate program. The best example is integration tests written in interpreted languages.
Consider the following (absolutely synthetic) program under test:
extern int printf(const char *, ...);
extern int strcmp(const char *, const char *);
int test1(int a, int b) {
return a + b;
}
int test2(int a, int b) {
return a * b;
}
int main(int argc, char **argv) {
if (argc == 1) {
printf("NOT ENOUGH ARGUMENTS\n");
return 1;
}
if (strcmp(argv[1], "first test") == 0) {
if (test1(2, 5) == 7) {
printf("first test passed\n");
return 0;
} else {
printf("first test failed\n");
return 1;
}
} else if (strcmp(argv[1], "second test") == 0) {
if (test2(2, 5) == 10) {
printf("second test passed\n");
return 0;
} else {
printf("second test failed\n");
return 1;
}
} else {
printf("INCORRECT TEST NAME %s\n", argv[1]);
return 1;
}
return 0;
}
The program accepts a command-line argument, and depending on the value of the argument it either runs one of the tests or exists with an error. Here is an example:
$ clang main.c -o test
$ ./test
NOT ENOUGH ARGUMENTS
$ ./test "first test"
first test passed
$ ./test "second test"
second test passed
$ ./test "third test"
INCORRECT TEST NAME third test
Running these tests manually is a tedious and error-prone process, so we create a separate test runner:
import sys
import subprocess
test_executable = sys.argv[1]
subprocess.run([test_executable, "first test"], check=True)
subprocess.run([test_executable, "second test"], check=True)
The script takes the program under test as its argument and runs the tests against that program.
$ clang main.c -o test
$ python3 test.py ./test
first test passed
second test passed
Usage of Mull in this case is very similar to a “typical” use-case (see Hello World tutorial).
Create config file mull.yml
:
mutators:
- cxx_add_to_sub
- cxx_mul_to_div
Generate mutated executable
$ clang-12 -fexperimental-new-pass-manager \
-fpass-plugin=/usr/local/lib/mull-ir-frontend-12 \
-g -grecord-command-line \
main.c -o test.exe
Run analysis using mull-runner:
$ mull-runner-12 ./test.exe -ide-reporter-show-killed \
-test-program=python3 -- test.py ./test.exe
[info] Using config /tmp/sc-kGN35Gr1f/mull.yml
[info] Warm up run (threads: 1)
[################################] 1/1. Finished in 347ms
[info] Filter mutants (threads: 1)
[################################] 1/1. Finished in 0ms
[info] Baseline run (threads: 1)
[################################] 1/1. Finished in 76ms
[info] Running mutants (threads: 2)
[################################] 2/2. Finished in 81ms
[info] Killed mutants (2/2):
/tmp/sc-kGN35Gr1f/main.c:5:12: warning: Killed: Replaced + with - [cxx_add_to_sub]
return a + b;
^
/tmp/sc-kGN35Gr1f/main.c:9:12: warning: Killed: Replaced * with / [cxx_mul_to_div]
return a * b;
^
[info] All mutations have been killed
[info] Mutation score: 100%
[info] Total execution time: 509ms
Note, test.exe appears twice in the arguments list: the first appearance
is required for mull-runner to extract the mutants generated at the second step.
The second appearance is passed as an argument to the test program test.py
.