diff --git a/notebooks/Alhazen.ipynb b/notebooks/Alhazen.ipynb index 35adb5d0..8dc8d48e 100644 --- a/notebooks/Alhazen.ipynb +++ b/notebooks/Alhazen.ipynb @@ -111,17 +111,38 @@ } }, "source": [ - "## Alhazen in a Nutshell\n", + "## Machine Learning for Automated Debugging\n", "\n", - "when diagnosing why a program fails, the first step is to determine the circumstances under which the program failed. Kampmann et al. [[KHSZ20](https://publications.cispa.saarland/3107/7/fse2020-alhazen.pdf)] presented an approach to automatically discover the circumstances of program behavior.\n", - "Their approach associates the program’s failure with the syntactical features of the input data, allowing them to learn and extract the properties that result in the specific behavior.\n", + "When diagnosing why a program fails, the first step is to determine the circumstances under which the program fails.\n", + "In past chapters, we have examined approaches that [correlate execution features with failures](StatisticalDebugger.ipynb)\n", + "as well as tools that systematically _generate inputs_ to [reduce failure-inducing inputs](DeltaDebugger.ipynb) or [generalize failure circumstances](DDSetDebugger.ipynb).\n", + "In this chapter, we will go one step further and make use of full-fledged _machine learning_ to identify failure circumstances (and causes)." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### The Alhazen Approach" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "In 2020, Kampmann et al. [[KHSZ20](https://publications.cispa.saarland/3107/7/fse2020-alhazen.pdf)] presented one of the first approaches to automatically learn circumstances of (failing) program behavior.\n", + "Their approach associates the program’s failure with the _syntactical features_ of the input data, allowing them to learn and extract the properties that result in the specific behavior.\n", "\n", "Their reference implementation _Alhazen_ can generate a diagnosis and explain why, for instance, a particular bug occurs.\n", - "More formally, Alhazen forms a hypothetical model based on the observed inputs.\n", - "Additional test inputs are generated and executed to refine or refute the hypothesis, eventually obtaining a prediction model of the circumstances of why the behavior in question takes place.\n", - "Alhazen use a _Decision Tree classifier_ to learn the association between the program behavior and the input features.\n", - "\n", - "![title](PICS/Alhazen.png)" + "Alhazen forms a hypothetical model based on the observed inputs.\n", + "Additional test inputs are generated and executed to refine or refute the hypothesis, eventually obtaining a prediction model of the circumstances of why the behavior in question takes place." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "" ] }, { @@ -134,6 +155,69 @@ "one of the inventors of the _scientific method_, the key process in the Alhazen tool." ] }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "\n", + "![title](PICS/Alhazen.png)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "In a nutshell, this is how Alhazen works.\n", + "Alhazen is given an _input grammar_ and a number of _input files_ (whose format is given by the grammar),\n", + "and produces a _decision tree_ – a machine learning model that explains under which circumstances the program fails.\n", + "\n", + "Alhazen determines and refines these decision trees in five steps, illustrated above.\n", + "\n", + "1. For each input file, Alhazen extracts a number of _input features_ that apply.\n", + " These input features are predicates over the individual elements of the input grammar, such as ` > 0` (an `` element is larger than zero) or `exists()` (the input contains a minus sign).\n", + "2. The test outcomes of the input files _label_ these input files as _buggy_ or _non-buggy_.\n", + " From the respective input features and the labels, Alhazen trains a _decision tree_ that associates these features with the labels - that is, the decision tree explains which features lead to _buggy_ or _non-buggy_.\n", + "3. As it is typically trained on few samples only, the initial classification model may be imprecise.\n", + " Hence, Alhazen extracts further _requirements_ for additional test cases that may help in increasing precision, such as ` == '6'` (we need more inputs in which the `` field has a value of `6`.)\n", + "4. Satisfying these requirements, Alhazen then generates additional inputs...\n", + "5. ...which it executes, thus again labeling them as _buggy_ or _non-buggy_.\n", + " From the new inputs, we can again extract the features, and repeat the cycle.\n", + "\n", + "The whole process keeps on refining decision trees with more and more inputs.\n", + "Eventually, the decision trees are supposed to be precise enough that they can become _theory_ - that is, an explanation of why the program fails with high predictive power for future inputs." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "The Alhazen process thus automates the _scientific method of debugging_:\n", + "\n", + "* making initial _observations_ (Steps 1 and 2),\n", + "* coming up with _hypotheses_ that explain the observations (Step 3),\n", + "* designing _experiments_ to further _support_ or _refute_ the hypotheses (Steps 4 and 5),\n", + "* and repeating the entire process until we have a _predicting theory_ on why the program fails." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Structure of this Chapter\n", + "\n", + "In the remainder of this chapter, we will first introduce [grammars](#Inputs-and-Grammars).\n", + "\n", + "We then explore and implement the individual steps of Alhazen:\n", + "\n", + "* [Step 1: Extracting Features](#Step-1:-Extracting-Features)\n", + "* [Step 2: Train Classification Model](#Step-2:-Train-Classification-Model)\n", + "* [Step 3: Extract Feature Requirements](#Step-3:-Extract-Feature-Requirements)\n", + "* [Step 4: Generating New Samples](#Step-4:-Generating-New-Samples)\n", + "* [Step 5: Executing New Inputs](#Step-5:-Generating-New-Inputs)\n", + "\n", + "After this is done, we can compose all these into a single `Alhazen` class and [run it on a sample input](#A-Sample-Run)." + ] + }, { "cell_type": "markdown", "metadata": { @@ -144,7 +228,7 @@ } }, "source": [ - "## Motivation" + "## Inputs and Grammars" ] }, { @@ -534,20 +618,6 @@ "Wouldn't it be great if there was a tool that automatically does this for us? And this is exactly what _Alhazen_ is used for. It helps us explain why specific input files fail a program. " ] }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## The Alhazen Algorithm" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "_Alhazen_ is a tool that automatically learns the circumstances of program failure by associating syntactical features of sample inputs with the execution outcome. The produced explanations (in the form of a decision tree) help developers focus on the input space's relevant aspects." - ] - }, { "cell_type": "markdown", "metadata": {}, @@ -2852,7 +2922,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "### A Sample Run" + "## A Sample Run" ] }, {