diff --git a/notebooks/Alhazen.ipynb b/notebooks/Alhazen.ipynb index 5a8c7702..04820b24 100644 --- a/notebooks/Alhazen.ipynb +++ b/notebooks/Alhazen.ipynb @@ -69,21 +69,7 @@ "source": [ "## Synopsis\n", "\n", - "\n", - "\n", - "\n", - "\n", - "_For those only interested in using the code in this chapter (without wanting to know how it works), give an example. This will be copied to the beginning of the chapter (before the first section) as text with rendered input and output._\n", - "\n", - "You can use `int_fuzzer()` as:\n", - "\n", - "```python\n", - "print(int_fuzzer())\n", - "```\n", - "```python\n", - "=> 76.5\n", - "\n", - "```\n" + "\n" ] }, { @@ -194,7 +180,8 @@ "* [Step 4: Generating New Samples](#Step-4:-Generating-New-Samples)\n", "* [Step 5: Executing New Inputs](#Step-5:-Generating-New-Inputs)\n", "\n", - "After this is done, we can compose all these into a single `Alhazen` class and [run it on a sample input](#A-Sample-Run)." + "After this is done, we can compose all these into a single `Alhazen` class and [run it on a sample input](#A-Sample-Run).\n", + "If you want to see Alhazen in action first (before going into all the details, check out the sample run.)" ] }, { @@ -214,8 +201,10 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "**Info:** We use the functionality provided by [The Fuzzing Book](https://www.fuzzingbook.org).\n", - "For a more detailed description of Grammars, have a look at the chapter [\"Fuzzing with Grammars\"](https://www.fuzzingbook.org/html/Grammars.html)" + "Alhazen heavily builds on _grammars_ as a means to decompose inputs into individual elements, such that it can reason about these elements, and also generate new ones automatically.\n", + "\n", + "To work with grammars, we use the framework provided by [The Fuzzing Book](https://www.fuzzingbook.org).\n", + "For a more detailed description of Grammars and how to use them for production, have a look at the chapter [\"Fuzzing with Grammars\"](https://www.fuzzingbook.org/html/Grammars.html)" ] }, { @@ -238,15 +227,63 @@ "from fuzzingbook.Parser import EarleyParser" ] }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Let us build a simple grammar for a calculator.\n", + "The calculator code is listed below." + ] + }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ - "import pandas\n", - "import numpy\n", - "import matplotlib" + "\"\"\"\n", + "This file contains the code under test for the example bug.\n", + "The sqrt() method fails on x <= 0.\n", + "\"\"\"\n", + "from math import tan as rtan\n", + "from math import cos as rcos\n", + "from math import sin as rsin\n", + "\n", + "\n", + "def task_sqrt(x):\n", + " \"\"\"Computes the square root of x, using the Newton-Raphson method\"\"\"\n", + " if x <= -12 and x >= -42:\n", + " x = 0 # Guess where the bug is :-)\n", + " else:\n", + " x = 1\n", + " x = max(x, 0)\n", + " approx = None\n", + " guess = x / 2\n", + " while approx != guess:\n", + " approx = guess\n", + " guess = (approx + x / approx) / 2\n", + " return approx\n", + "\n", + "\n", + "def task_tan(x):\n", + " return rtan(x)\n", + "\n", + "\n", + "def task_cos(x):\n", + " return rcos(x)\n", + "\n", + "\n", + "def task_sin(x):\n", + " return rsin(x)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "\n", + "\n", + "The language consists of functions (``) that are being invoked on a numerical value (``)." ] }, { @@ -293,7 +330,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "Now, let us load two initial input samples:\n", + "Let us load two initial input samples:\n", "- `sqrt(-16)`\n", "- `sqrt(4)`" ] @@ -312,7 +349,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "Now, let's execute our two input samples and observe the calculator's behavior.\n", + "Let's execute our two input samples and observe the calculator's behavior.\n", "We implement the function `sample_runner(sample)` that lets us execute the calculator for a single sample. `sample_runner(sample)` returns an `OracleResult` for the sample." ] }, @@ -340,48 +377,6 @@ " return self.value" ] }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "\"\"\"\n", - "This file contains the code under test for the example bug.\n", - "The sqrt() method fails on x <= 0.\n", - "\"\"\"\n", - "from math import tan as rtan\n", - "from math import cos as rcos\n", - "from math import sin as rsin\n", - "\n", - "\n", - "def task_sqrt(x):\n", - " \"\"\"Computes the square root of x, using the Newton-Raphson method\"\"\"\n", - " if x <= -12 and x >= -42:\n", - " x = 0 # Guess where the bug is :-)\n", - " else:\n", - " x = 1\n", - " x = max(x, 0)\n", - " approx = None\n", - " guess = x / 2\n", - " while approx != guess:\n", - " approx = guess\n", - " guess = (approx + x / approx) / 2\n", - " return approx\n", - "\n", - "\n", - "def task_tan(x):\n", - " return rtan(x)\n", - "\n", - "\n", - "def task_cos(x):\n", - " return rcos(x)\n", - "\n", - "\n", - "def task_sin(x):\n", - " return rsin(x)" - ] - }, { "cell_type": "code", "execution_count": null, @@ -396,6 +391,7 @@ " testcode = sample\n", "\n", " try:\n", + " # Simply execute the calculator code, with the functions replaced\n", " exec(testcode, {\"sqrt\": task_sqrt, \"tan\": task_tan, \"sin\": task_sin, \"cos\": task_cos}, {})\n", " return OracleResult.NO_BUG\n", " except ZeroDivisionError:\n", @@ -444,7 +440,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "What happens if we parse inputs to calculator, that do not conform to its input format?" + "What happens if we parse inputs to calculator that do not conform to its input format?" ] }, { @@ -467,7 +463,19 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "The finally we provide the function `execute_samples(sample_list)` that obtains the oracle/label for a list of samples." + "Finally, we provide the function `execute_samples(sample_list)` that obtains the oracle/label for a list of samples.\n", + "We use the `pandas` module to place these in a data frame." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "import pandas\n", + "import numpy\n", + "import matplotlib" ] }, { @@ -487,34 +495,52 @@ " return pandas.DataFrame.from_records(data)" ] }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Let us define a bigger list of samples to execute..." + ] + }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ - "# let us define a list of samples to execute\n", "sample_list = [\"sqrt(-20)\", \"cos(2)\", \"sqrt(-100)\", \"undef_function(foo)\"]" ] }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "... and obtain the execution outcome" + ] + }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ - "# we obtain the execution outcome\n", "labels = execute_samples(sample_list)\n", "display(labels)" ] }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "We can combine these with the `sample_list`:" + ] + }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ - "# combine with the sample_list\n", "for i, row in enumerate(labels['oracle']): print(sample_list[i].ljust(30) + str(row))" ] }, @@ -522,7 +548,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "To remove the undefined input samples, you could invoke something similar to this:" + "We can remove the undefined input samples like this:" ] }, { @@ -531,21 +557,40 @@ "metadata": {}, "outputs": [], "source": [ - "# clean up data\n", "clean_data = labels.drop(labels[labels.oracle.astype(str) == \"UNDEF\"].index)\n", "display(clean_data)" ] }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "We can combine sample and labels by iterating over the obtained oracle:" + ] + }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ - "# Load function execute samples\n", - "# execute_samples(List[str])\n", "oracle = execute_samples(sample_list)\n", - "oracle" + "for i, row in enumerate(oracle['oracle']):\n", + " print(sample_list[i].ljust(30) + str(row))" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "We observe that the sample `sqrt(-16)` triggers a bug in the calculator, whereas the sample `sqrt(4)` does not show unusual behavior. Of course, we want to know why the sample fails the program. In a typical use case, the developers of the calculator program would now try other input samples and evaluate if similar inputs also trigger the program's failure. Let's try some more input samples; maybe we can refine our understanding of why the calculator crashes:" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Our guesses - maybe the failure is also in the `cos()` or `tan()` function?" ] }, { @@ -554,16 +599,30 @@ "metadata": {}, "outputs": [], "source": [ - "# Combined sample and labels by iterating over the obtained oracle\n", - "for i, row in enumerate(oracle['oracle']):\n", - " print(sample_list[i].ljust(30) + str(row))" + "guess_samples = ['cos(-16)', 'tan(-16)', 'sqrt(-100)', 'sqrt(-20.23412431234123)']" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ - "We observe that the sample `sqrt(-16)` triggers a bug in the calculator, whereas the sample `sqrt(4)` does not show unusual behavior. Of course, we want to know why the sample fails the program. In a typical use case, the developers of the calculator program would now try other input samples and evaluate if similar inputs also trigger the program's failure. Let's try some more input samples; maybe we can refine our understanding of why the calculator crashes:" + "Let's obtain the execution outcome for each of our guesses:\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "guess_oracle = execute_samples(guess_samples)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Here come the results:" ] }, { @@ -572,13 +631,6 @@ "metadata": {}, "outputs": [], "source": [ - "# Our guesses (maybe the failure is also in the cos or tan function?)\n", - "guess_samples = ['cos(-16)', 'tan(-16)', 'sqrt(-100)', 'sqrt(-20.23412431234123)']\n", - "\n", - "# lets obtain the execution outcome for each of our guess\n", - "guess_oracle = execute_samples(guess_samples)\n", - "\n", - "# lets show the results\n", "for i, row in enumerate(guess_oracle['oracle']):\n", " print(guess_samples[i].ljust(30) + str(row))" ] @@ -587,14 +639,18 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "It looks like the failure only occurs in the `sqrt` function, however, only for specific `x` values. We could now try other values for `x` and repeat the process. However, this would be highly time-consuming and not an efficient debugging technique for a larger and more complex test subject." + "It looks like the failure only occurs in the `sqrt()` function, however, only for specific `x` values.\n", + "We could now try other values for `x` and repeat the process.\n", + "However, this would be highly time-consuming and not an efficient debugging technique for a larger and more complex test subject." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ - "Wouldn't it be great if there was a tool that automatically does this for us? And this is exactly what _Alhazen_ is used for. It helps us explain why specific input files fail a program. " + "Wouldn't it be great if there was a tool that automatically does this for us? \n", + "And this is exactly what _Alhazen_ is there for.\n", + "It helps us explain why specific input features cause a program to fail." ] }, { @@ -611,16 +667,16 @@ "In this section, we are concerned with the problem of extracting semantic features from inputs. In particular, Alhazen defines various features based on the input grammar, such as *existence* and *numeric interpretation*. These features are then extracted from the parse trees of the inputs (see Section 3 of \\cite{Kampmann2020} for more details).\n", "\n", "The implementation of the feature extraction module consists of the following three tasks:\n", - "1. Implementation of individual feature classes, whose instances allow deriving specific feature values from inputs\n", - "2. Extraction of features from the grammar through instantiation of the aforementioned feature classes\n", - "3. Computation of feature vectors from a set of inputs, which will then be used as input for the decision tree" + "1. Implementation of individual _feature classes_, whose instances allow deriving specific feature values from inputs\n", + "2. _Extraction of features from the grammar_ through instantiation of the aforementioned feature classes\n", + "3. Computation of _feature vectors_ from a set of inputs, which will then be used as input for the decision tree" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ - "### Feature Classes" + "### Implementing Feature Classes" ] }, { @@ -893,6 +949,13 @@ " + extract_numeric_features(grammar))" ] }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Here are all the features from our calculator grammar:" + ] + }, { "cell_type": "code", "execution_count": null, @@ -902,6 +965,13 @@ "extract_all_features(CALC_GRAMMAR)" ] }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "The `friendly` format is a bit more concise and more readable:" + ] + }, { "cell_type": "code", "execution_count": null, @@ -922,7 +992,8 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "**Note**: This is a rather slow implementation, for many grammars with many syntactically features, the feature collection can be optimized" + "This is a rather slow implementation.\n", + "For many grammars with many syntactically features, the feature collection can be optimized." ] }, { @@ -1015,6 +1086,13 @@ "### Excursion: Transforming Grammars" ] }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Alhazen requires grammars to be transformed, such that for each non-terminal symbol in the grammar, the word derived by this symbol in the input is added as an alternative to the symbol (as written in \\cite{Kampmann2020}). Here, we iterate through the derivation tree of the input and add the derived word of each nonterminal as alternatives to the grammar." + ] + }, { "cell_type": "code", "execution_count": null, @@ -1036,14 +1114,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "For the grammar transformation, we perform a *rewrite step* that for each non-terminal symbol in the grammar, determines the word derived by this symbol in the input and adds it as an alternative to the symbol (as written in the Alhazen-paper). Here, we iterate through the derivation tree of the input and add the derived word of each non-terminal as alternatives to the grammar." - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Let us write a function `transform_grammar` that given a sample input and a grammar, transforms it according to Kampmann et al." + "Let us write a function `transform_grammar()` that given a sample input and a grammar, transforms it according to Kampmann et al." ] }, { @@ -1062,17 +1133,16 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "**INPUT**:\n", - "the function requires the following input parameter:\n", - "- sample: an input sample \n", - "- grammar: the grammar that should be transformed/extended" + "The function requires the following input parameter:\n", + "- `sample`: an input sample \n", + "- `grammar`: the grammar that should be transformed/extended" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ - "**OUTPUT**: the function should return the transformed and extended grammar." + "The function returns the transformed and extended grammar." ] }, { @@ -1081,7 +1151,7 @@ "metadata": {}, "outputs": [], "source": [ - "# Then, recursively iterate through the derivation tree and for each non-terminal,\n", + "# Recursively iterate through the derivation tree and for each non-terminal,\n", "# add the derived word to the grammar\n", "\n", "def extend_grammar(derivation_tree, grammar):\n", @@ -1120,14 +1190,19 @@ " return transformed_grammar" ] }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Here is an example of a transformed grammar:" + ] + }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ - "# TODO Add better test case for correct validation\n", - "\n", "transformed_grammar = transform_grammar(\"1 + 2\", EXPR_GRAMMAR)\n", "for rule in transformed_grammar:\n", " print(rule.ljust(10), transformed_grammar[rule])" @@ -1144,7 +1219,18 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "## Step 2: Train Classification Model" + "## Step 2: Train Classification Model\n", + "\n", + "Now that we have all the input features and the test outcomes, we can start training a machine learner from these.\n", + "Although other machine learning models have much higher accuracy, we use _decision trees_ as machine learning models because they are easy to interpret by humans.\n", + "This is crucial as it will be these very same humans that have to fix the code." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Before we start with our actual implementation, let us first illustrate how training such a classifier works, again using our calculator as an example." ] }, { @@ -1158,7 +1244,8 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "We will also use `scikit-learn` as the machine learning library. We will use the `DecisionTreeClassifier` to learn the syntactical input features that are responsible for the bug-triggering behavior of our Calculator." + "We will use `scikit-learn` as the machine learning library.\n", + "The `DecisionTreeClassifier` can then learn the syntactical input features that are responsible for the bug-triggering behavior of our Calculator." ] }, { @@ -1177,7 +1264,8 @@ "metadata": {}, "source": [ "First, we transform the individual input features (represented as Python dictionaries) into a NumPy array.\n", - "For this example, we use the following four features (`function-sqrt`, `function-cos`, `function-sin`, `number`) to describe an input feature. (Please note that this is an extremely reduced example; this is not the complete list of features that should be extracted from the `CALCULATOR` Grammar.)" + "For this example, we use the following four features (`function-sqrt`, `function-cos`, `function-sin`, `number`) to describe an input feature.\n", + "(Please note that this is an extremely reduced example; this is not the complete list of features that should be extracted from the `CALC_GRAMMAR` Grammar.)" ] }, { @@ -1192,7 +1280,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "For each (x), the `number` feature describes which value was used for `x`. For instance, the first input `sqrt(-900)` corresponds to 'function-sqrt': 1 and 'number': -900." + "For each `(x)`, the `number` feature describes which value was used for `x`. For instance, the first input `sqrt(-900)` corresponds to 'function-sqrt': 1 and 'number': -900." ] }, { @@ -1302,6 +1390,13 @@ "show_decision_tree(clf, vec.get_feature_names_out())" ] }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Here is a much reduced textual variant, still retaining the essential features:" + ] + }, { "cell_type": "code", "execution_count": null, @@ -1370,8 +1465,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "For _Alhazen's_ second step (Train Classification Model), we write a function `train_tree(data)` that trains a decision tree on a given data frame.\n", - "`train_tree(data)` should return the learned decision tree." + "For _Alhazen's_ second step (Train Classification Model), we write a function `train_tree(data)` that trains a decision tree on a given data frame:" ] }, { @@ -1389,9 +1483,11 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "**INPUT**:\n", - "the function requires the following parameter:\n", - "- data: a pandas dataframe containing the parsed and extracted features and the outcome of the executed input sample (oracle). For instance, the data frame may look similar to this:" + "The function requires the following parameter:\n", + "\n", + "- data: a `pandas` data frame containing the parsed and extracted features and the outcome of the executed input sample (oracle).\n", + "\n", + "For instance, the data frame may look similar to this:" ] }, { @@ -1408,7 +1504,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "**Note:** Each row of data['oracle'] is of type `OracleResult`.\n", + "**Note:** Each row of `data['oracle']` is of type `OracleResult`.\n", "However, sci-kit learn requires an array of strings.\n", "We have to convert them to learn the decision tree." ] @@ -1439,7 +1535,8 @@ " str(\"NO_BUG\"):\n", " (1.0/(sample_count - sample_bug_count))})\n", " clf = clf.fit(data.drop('oracle', axis=1), data['oracle'].astype(str))\n", - " # self.__tree = treetools.remove_infeasible(clf, features) # MARTIN: This is optional, but is a nice extesion that results in nicer decision trees\n", + " # MARTIN: This is optional, but is a nice extesion that results in nicer decision trees\n", + " # clf = treetools.remove_infeasible(clf, features)\n", " return clf" ] }, @@ -1454,7 +1551,8 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "In this section, we will extract the learned features from the decision tree" + "In this section, we will extract the learned features from the decision tree.\n", + "Again, let us first test this manually on our calculator example." ] }, { @@ -1500,24 +1598,18 @@ "clf = DecisionTreeClassifier(random_state=10)\n", "\n", "# Train with DictVectorizer\n", + "# **Note:** The sklearn `DictVectorizer` uses an internal sort function as default. This will result in different feature_name indices. If you want to use the `Dictvectorizer` please ensure that you only access the feature_names with the function `vec.get_feature_names_out()`.\n", + "# We recommend that you use the `pandas` data frame, since this is also the format used in the feedback loop.\n", "# clf = clf.fit(X_vec, oracle)\n", "\n", "# Train with Pandas Dataframe\n", "clf = clf.fit(X_data, oracle)\n", "\n", - "dot_data = sklearn.tree.export_graphviz(clf, out_file=None, \n", + "dot_data = sklearn.tree.export_graphviz(clf, out_file=None,\n", " feature_names=feature_names,\n", - " class_names=[\"BUG\", \"NO BUG\"], \n", - " filled=True, rounded=True) \n", - "graph = graphviz.Source(dot_data) " - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "**Note:** The sklearn `DictVectorizer` uses an internal sort function as default. This will result in different feature_name indices. If you want to use the `Dictvectorizer` please ensure that you only access the feature_names with the function `vec.get_feature_names_out()`.\n", - "We recommend that you use the pandas Dataframe, since this is also the format used in the feedback loop." + " class_names=[\"BUG\", \"NO BUG\"],\n", + " filled=True, rounded=True)\n", + "graph = graphviz.Source(dot_data)" ] }, { @@ -1545,14 +1637,19 @@ "### Excursion: Tree helper functions" ] }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "We bundle several functions that are helpful when working with decision trees." + ] + }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ - "\"\"\"This file bundles several functions which are helpful when working with decision trees.\"\"\"\n", - "\n", "def all_path(clf, node=0):\n", " \"\"\"Iterate over all path in a decision tree. Path will be represented as\n", " a list of integers, each integer is the index of a node in the clf.tree_ structure.\"\"\"\n", @@ -1798,6 +1895,13 @@ "### Excursion: Converting Trees to Paths" ] }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "We bundle a number of helper functions that extract paths from trees." + ] + }, { "cell_type": "code", "execution_count": null, @@ -1961,13 +2065,6 @@ " return paths" ] }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### End of Excursion" - ] - }, { "cell_type": "code", "execution_count": null, @@ -1978,6 +2075,13 @@ "all_paths = tree_to_paths(clf, feature_names)" ] }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Here is an example:" + ] + }, { "cell_type": "code", "execution_count": null, @@ -1991,6 +2095,13 @@ " print(f\"Path {count}: {string_path}, is_bug: {path.is_bug()}\")" ] }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### End of Excursion" + ] + }, { "cell_type": "markdown", "metadata": {}, @@ -2002,14 +2113,15 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "### Negating Requirements" + "The next step is to generate new samples.\n", + "For this purpose, we _negate_ the requirements on a path to refine and refute the decision tree." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ - "The next step is to negate the requirements on a path to refine and refute the decision tree" + "### Negating Requirements" ] }, { @@ -2058,13 +2170,6 @@ " print(f\"Path {count}: {negated_string_path}, is_bug: {path.is_bug()}\")" ] }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Let's verify if we can negate a whole path." - ] - }, { "cell_type": "markdown", "metadata": {}, @@ -2076,7 +2181,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "We will use the Decision tree and extract new input specifications to refine or refute our hypothesis (See Section 4.1 - Extracting Prediction Paths in \\cite{Kampmann2020}).\n", + "We will use the Decision tree and extract new input specifications to refine or refute our hypothesis (See Section 4.1 \"Extracting Prediction Paths\" in \\cite{Kampmann2020}).\n", "These input specifications will be parsed to the input generator that tries to generate new inputs that fulfill the defined input specifications." ] }, @@ -2087,12 +2192,11 @@ "outputs": [], "source": [ "def extracting_prediction_paths(clf, feature_names, data):\n", - " \n", " # determine the bounds\n", " bounds = pandas.DataFrame([{'feature': c, 'min': data[c].min(), 'max': data[c].max()}\n", " for c in feature_names],\n", " columns=['feature', 'min', 'max']).set_index(['feature']).transpose()\n", - " \n", + "\n", " # go through tree leaf by leaf\n", " all_reqs = set()\n", " for path in tree_to_paths(clf, feature_names):\n", @@ -2154,6 +2258,13 @@ "### Input Specification Parser" ] }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Once we have input specifications, we must again extract them from the decision tree so we can interpret them." + ] + }, { "cell_type": "code", "execution_count": null, @@ -2246,7 +2357,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "Let's also try with some real requirement specifications" + "Let's also try with some real requirement specifications:" ] }, { @@ -2277,7 +2388,9 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "### Retrieving New input Specifications" + "### Retrieving New input Specifications\n", + "\n", + "The following classes represent requirements for the test cases to be generated." ] }, { @@ -2495,8 +2608,8 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "We implement a _Grammar-Based Input Generator_ that generates new input samples from a List of `Input Specifications`.\n", - "The Input Specifications are extracted from the decision tree boundaries in the previous Step 3: `RequirementExtraction`.\n", + "We implement a _Grammar-Based Input Generator_ that generates new input samples from a List of `InputSpecifications`.\n", + "The input specifications are extracted from the decision tree boundaries in the previous Step 3: `RequirementExtraction`.\n", "\n", "An `InputSpecification` consists of **1 to n** many predicates or requirements (e.g. ` >= value`, or `num() <= 13`).\n", "We generate a new input for each `InputSpecification`.\n", @@ -2507,25 +2620,24 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "**Info:** For further details, please refer to Section 4.4 and 4.5 of \\cite{Kampmann2020} and the Chapter [Efficient Grammar Fuzzing](https://www.fuzzingbook.org/html/GrammarFuzzer.html) in the Fuzzing Book." + "For further details, please refer to Section 4.4 and 4.5 of \\cite{Kampmann2020} and the Chapter [Efficient Grammar Fuzzing](https://www.fuzzingbook.org/html/GrammarFuzzer.html) in the Fuzzing Book." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ - "**INPUT**:\n", - "the function requires the following input parameter:\n", - "- grammar: the grammar this is used to produce new inputs (e.g. the CALCULATOR-Grammar)\n", - "- new_input_specification: a List of new inputs specifications (`List\\[InputSpecification\\]`)\n", - "- timeout: a max time budget. Return the generated inputs when the time budget is exceeded." + "We define a function `generate_samples()` with the following input parameters:\n", + "- `grammar`: the grammar used to produce new inputs (e.g. the CALCULATOR-Grammar)\n", + "- `new_input_specification`: a List of new inputs specifications (type `List[InputSpecification]`)\n", + "- `timeout`: a max time budget. Return the generated inputs when the time budget is exceeded." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ - "**OUTPUT**: the function should return a list of new inputs that are specified by the given input specifications." + "The function returns a list of new inputs that are specified by the given input specifications." ] }, { @@ -2758,7 +2870,16 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "### The Alhazen Class" + "We are almost done! All that is left is putting the above pieces together and create a _loop_ around them." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### The Alhazen Class\n", + "\n", + "We implement a class `Alhazen` that serves as main entry point for our approach." ] }, { @@ -2998,7 +3119,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "We initialize Alhazen with the previously used sample_list:" + "We initialize Alhazen with the previously used `initial_sample_list`:" ] }, { @@ -3121,7 +3242,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "This chapter provides an implementation of the _Alhazen_ approach [[KHSZ20](https://publications.cispa.saarland/3107/7/fse2020-alhazen.pdf)], which trains machine learning _classifiers_ from input features.\n", + "This chapter provides an implementation of the _Alhazen_ approach \\cite{Kampmann2020}, which trains machine learning _classifiers_ from input features.\n", "Given a test function, a grammar, and a set of inputs, the `Alhazen` class produces a decision tree that _characterizes failure circumstances_:" ] }, diff --git a/notebooks/PICS/Alhazen-synopsis-1.png b/notebooks/PICS/Alhazen-synopsis-1.png index 1698d4b8..8dcbc76f 100644 Binary files a/notebooks/PICS/Alhazen-synopsis-1.png and b/notebooks/PICS/Alhazen-synopsis-1.png differ diff --git a/notebooks/PICS/Alhazen-synopsis-1.svg b/notebooks/PICS/Alhazen-synopsis-1.svg index bf90b9c1..0aaea204 100644 --- a/notebooks/PICS/Alhazen-synopsis-1.svg +++ b/notebooks/PICS/Alhazen-synopsis-1.svg @@ -4,269 +4,114 @@ - - + + Tree - + 0 - -<lead-digit> <= 3.5 -gini = 0.5 -samples = 1626 -value = [1.0, 1.0] -class = NO_BUG + +<term> <= -11.5 +gini = 0.5 +samples = 997 +value = [1.0, 1.0] +class = BUG 1 - -<function> == 'sqrt' <= 0.5 -gini = 0.346 -samples = 485 -value = [0.952, 0.272] -class = BUG + +<term> <= -42.4 +gini = 0.38 +samples = 371 +value = [1.0, 0.342] +class = BUG 0->1 - - -True + + +True - - -14 - -<lead-digit> <= 4.5 -gini = 0.115 -samples = 1141 -value = [0.048, 0.728] -class = NO_BUG + + +8 + +gini = -0.0 +samples = 626 +value = [0.0, 0.658] +class = NO_BUG - - -0->14 - - -False + + +0->8 + + +False 2 - -gini = 0.0 -samples = 260 -value = [0.0, 0.166] -class = NO_BUG + +gini = 0.0 +samples = 276 +value = [0.0, 0.29] +class = NO_BUG 1->2 - - + + 5 - -<term> <= -11.75 -gini = 0.18 -samples = 225 -value = [0.952, 0.106] -class = BUG + +<function> == 'sqrt' <= 0.5 +gini = 0.095 +samples = 95 +value = [1.0, 0.053] +class = BUG 1->5 - - + + 6 - -<value> <= 69.5 -gini = 0.065 -samples = 112 -value = [0.952, 0.033] -class = BUG + +gini = -0.0 +samples = 50 +value = [0.0, 0.053] +class = NO_BUG 5->6 - - - - - -11 - -gini = 0.0 -samples = 113 -value = [0.0, 0.072] -class = NO_BUG - - - -5->11 - - + + 7 - -gini = -0.0 -samples = 60 -value = [0.952, 0.0] -class = BUG + +gini = -0.0 +samples = 45 +value = [1.0, 0.0] +class = BUG - + -6->7 - - - - - -8 - -gini = 0.0 -samples = 52 -value = [0.0, 0.033] -class = NO_BUG - - - -6->8 - - - - - -15 - -<value> <= 41.505 -gini = 0.475 -samples = 120 -value = [0.048, 0.075] -class = NO_BUG - - - -14->15 - - - - - -24 - -gini = 0.0 -samples = 1021 -value = [0.0, 0.653] -class = NO_BUG - - - -14->24 - - - - - -16 - -<function> == 'sqrt' <= 0.5 -gini = 0.175 -samples = 11 -value = [0.048, 0.005] -class = BUG - - - -15->16 - - - - - -21 - -gini = 0.0 -samples = 109 -value = [0.0, 0.07] -class = NO_BUG - - - -15->21 - - - - - -17 - -gini = 0.0 -samples = 5 -value = [0.0, 0.003] -class = NO_BUG - - - -16->17 - - - - - -18 - -<term> == '<value>' <= 0.5 -gini = 0.074 -samples = 6 -value = [0.048, 0.002] -class = BUG - - - -16->18 - - - - - -19 - -gini = 0.0 -samples = 3 -value = [0.048, 0.0] -class = BUG - - - -18->19 - - - - - -20 - -gini = -0.0 -samples = 3 -value = [0.0, 0.002] -class = NO_BUG - - - -18->20 - - +5->7 + +