-
Notifications
You must be signed in to change notification settings - Fork 1
Injecting Faults and Testing the Framework
In this chapter of our report we will discuss the design and implementation of 5 faults into the moodle source code, describe the development of a script that will automate the fault injection process, and analyze the results of the testing framework we have developed over the course of this project when these faults have been injected into the moodle source code.
When designing the faults to test our framework we decided to inject one fault into each of the methods we had written test cases for. The faults are defined in the ./project/faults directory where modified lines are indicated by comments in the format “// FAULT:” and the file that the fault definition should replace is specified on line 1 with a comment in the format “#Destination: ./path/from/moodle”. The details for the 5 faults we have defined will be provided below.
fault1.php
./moodle/lib/html2text/Html2Text.php
Line 144 changed so that the class improperly handles html with
elements.
“'/<(br)[^>]>[ ]/i',” to “'/<(brk)[^>]>[ ]/i',”
This fault causes the test case with the following id to fail: 8
fault2.php
./moodle/lib/maxmind/GeoIp2/Util.php
Line 23 changed so that the loop executes 1 time less then necessary.
“for ($i = 0; $i < \strlen($ipBytes) && $curPrefix > 0; $i++) {“
to
“for ($i = 0; $i < \strlen($ipBytes)-1 && $curPrefix > 0; $i++) {“
This fault causes the test case with the following id to fail: 12
fault3.php
./moodle/lib/htmlpurifier/HTMLPurifier/UnitConverter.php
Line 194 changed so that negative numbers are improperly handled.
“$n = ltrim($n, '0+-');” to “$n = ltrim($n, '0+');”
This fault causes the test case with the following id to fail: 19
fault4-5.php
./moodle/lib/evalmath/evalmath.class.php
Line 436 changed so that expressions with subtraction throw an error.
“} elseif (in_array($token, array('+', '-', '', '/', '^', '>', '<', '==', '<=', '>='), true)) {“
to
“} elseif (in_array($token, array('+', '+', '', '/', '^', '>', '<', '==', '<=', '>='), true)) {“
This fault causes the test case with the following id to fail: 2
Line 586 changed so that mod is calculated incorrectly.
“return $op1 % $op2;” to “return $op2 % $op1;”
This fault causes the test cases with the following ids to fail: 22, 23, 24
To automate the injection of faults into the moodle source code two routines, or actions, were added to the moodleMod script: inject and reset. The inject routine finds all fault definitions in ../project/faults/ and copies them to their destinations in the moodle source code, which are specified in a comment in the first line of each fault definition file. The reset routine cleans the local moodle repository, resetting it to the original state. The routines are executed by running the moodleMod script from the TestAutomation directory as follows:
./scripts/moodleMod.sh inject
./scripts/moodleMod.sh reset
In addition to managing faults, the moodleMod script also implements two other actions to manage the cloned moodle repository: clone and delete. The clone action will clone the appropriate version of moodle from the official moodle repository into ./project/moodle and the delete action will delete the repository.
To test our framework we performed the following procedure:
Clone the New-Leaf team repository
Move into ../New-Leaf/TestAutomation/
Clone moodle by executing: ./scripts/moodleMod.sh clone
Execute: ./scripts/runAllTests.py
Record results without faults
Inject faults by executing: ./scripts/moodleMod.sh inject
Execute: ./scripts/runAllTests.py
Record results with faults
Remove faults by executing: ./scripts/moodleMod.sh reset
Initial Results (without fault injection):
Final Results (with fault injection):
As expected, all tests passed prior to fault injection, and after fault injection the expected test cases failed.