Skip to content

Commit

Permalink
New: more intro text
Browse files Browse the repository at this point in the history
  • Loading branch information
@andreas-zeller
andreas-zeller committed Jan 6, 2025
1 parent 9fdb6f2 commit 50819e7
Showing 1 changed file with 94 additions and 24 deletions.
118 changes: 94 additions & 24 deletions notebooks/Alhazen.ipynb
View file
Original file line number Diff line number Diff line change
Expand Up @@ -111,17 +111,38 @@
}
},
"source": [
"## Alhazen in a Nutshell\n",
"## Machine Learning for Automated Debugging\n",
"\n",
"when diagnosing why a program fails, the first step is to determine the circumstances under which the program failed. Kampmann et al. [[KHSZ20](https://publications.cispa.saarland/3107/7/fse2020-alhazen.pdf)] presented an approach to automatically discover the circumstances of program behavior.\n",
"Their approach associates the program’s failure with the syntactical features of the input data, allowing them to learn and extract the properties that result in the specific behavior.\n",
"When diagnosing why a program fails, the first step is to determine the circumstances under which the program fails.\n",
"In past chapters, we have examined approaches that [correlate execution features with failures](StatisticalDebugger.ipynb)\n",
"as well as tools that systematically _generate inputs_ to [reduce failure-inducing inputs](DeltaDebugger.ipynb) or [generalize failure circumstances](DDSetDebugger.ipynb).\n",
"In this chapter, we will go one step further and make use of full-fledged _machine learning_ to identify failure circumstances (and causes)."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### The Alhazen Approach"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"In 2020, Kampmann et al. [[KHSZ20](https://publications.cispa.saarland/3107/7/fse2020-alhazen.pdf)] presented one of the first approaches to automatically learn circumstances of (failing) program behavior.\n",
"Their approach associates the program’s failure with the _syntactical features_ of the input data, allowing them to learn and extract the properties that result in the specific behavior.\n",
"\n",
"Their reference implementation _Alhazen_ can generate a diagnosis and explain why, for instance, a particular bug occurs.\n",
"More formally, Alhazen forms a hypothetical model based on the observed inputs.\n",
"Additional test inputs are generated and executed to refine or refute the hypothesis, eventually obtaining a prediction model of the circumstances of why the behavior in question takes place.\n",
"Alhazen use a _Decision Tree classifier_ to learn the association between the program behavior and the input features.\n",
"\n",
"![title](PICS/Alhazen.png)"
"Alhazen forms a hypothetical model based on the observed inputs.\n",
"Additional test inputs are generated and executed to refine or refute the hypothesis, eventually obtaining a prediction model of the circumstances of why the behavior in question takes place."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<!-- _Alhazen_ is a tool that automatically learns the circumstances of program failure by associating syntactical features of sample inputs with the execution outcome. The produced explanations (in the form of a decision tree) help developers focus on the input space's relevant aspects. -->"
]
},
{
Expand All @@ -134,6 +155,69 @@
"one of the inventors of the _scientific method_, the key process in the Alhazen tool."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"![title](PICS/Alhazen.png)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"In a nutshell, this is how Alhazen works.\n",
"Alhazen is given an _input grammar_ and a number of _input files_ (whose format is given by the grammar),\n",
"and produces a _decision tree_ – a machine learning model that explains under which circumstances the program fails.\n",
"\n",
"Alhazen determines and refines these decision trees in five steps, illustrated above.\n",
"\n",
"1. For each input file, Alhazen extracts a number of _input features_ that apply.\n",
" These input features are predicates over the individual elements of the input grammar, such as `<expr> > 0` (an `<expr>` element is larger than zero) or `exists(<minus-sign>)` (the input contains a minus sign).\n",
"2. The test outcomes of the input files _label_ these input files as _buggy_ or _non-buggy_.\n",
" From the respective input features and the labels, Alhazen trains a _decision tree_ that associates these features with the labels - that is, the decision tree explains which features lead to _buggy_ or _non-buggy_.\n",
"3. As it is typically trained on few samples only, the initial classification model may be imprecise.\n",
" Hence, Alhazen extracts further _requirements_ for additional test cases that may help in increasing precision, such as `<digit> == '6'` (we need more inputs in which the `<digit>` field has a value of `6`.)\n",
"4. Satisfying these requirements, Alhazen then generates additional inputs...\n",
"5. ...which it executes, thus again labeling them as _buggy_ or _non-buggy_.\n",
" From the new inputs, we can again extract the features, and repeat the cycle.\n",
"\n",
"The whole process keeps on refining decision trees with more and more inputs.\n",
"Eventually, the decision trees are supposed to be precise enough that they can become _theory_ - that is, an explanation of why the program fails with high predictive power for future inputs."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The Alhazen process thus automates the _scientific method of debugging_:\n",
"\n",
"* making initial _observations_ (Steps 1 and 2),\n",
"* coming up with _hypotheses_ that explain the observations (Step 3),\n",
"* designing _experiments_ to further _support_ or _refute_ the hypotheses (Steps 4 and 5),\n",
"* and repeating the entire process until we have a _predicting theory_ on why the program fails."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Structure of this Chapter\n",
"\n",
"In the remainder of this chapter, we will first introduce [grammars](#Inputs-and-Grammars).\n",
"\n",
"We then explore and implement the individual steps of Alhazen:\n",
"\n",
"* [Step 1: Extracting Features](#Step-1:-Extracting-Features)\n",
"* [Step 2: Train Classification Model](#Step-2:-Train-Classification-Model)\n",
"* [Step 3: Extract Feature Requirements](#Step-3:-Extract-Feature-Requirements)\n",
"* [Step 4: Generating New Samples](#Step-4:-Generating-New-Samples)\n",
"* [Step 5: Executing New Inputs](#Step-5:-Generating-New-Inputs)\n",
"\n",
"After this is done, we can compose all these into a single `Alhazen` class and [run it on a sample input](#A-Sample-Run)."
]
},
{
"cell_type": "markdown",
"metadata": {
Expand All @@ -144,7 +228,7 @@
}
},
"source": [
"## Motivation"
"## Inputs and Grammars"
]
},
{
Expand Down Expand Up @@ -534,20 +618,6 @@
"Wouldn't it be great if there was a tool that automatically does this for us? And this is exactly what _Alhazen_ is used for. It helps us explain why specific input files fail a program. "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## The Alhazen Algorithm"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"_Alhazen_ is a tool that automatically learns the circumstances of program failure by associating syntactical features of sample inputs with the execution outcome. The produced explanations (in the form of a decision tree) help developers focus on the input space's relevant aspects."
]
},
{
"cell_type": "markdown",
"metadata": {},
Expand Down Expand Up @@ -2852,7 +2922,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"### A Sample Run"
"## A Sample Run"
]
},
{
Expand Down

0 comments on commit 50819e7

Please sign in to comment.