Verify numbers produced in pipeline verification (#6)

* Better verification scripts

* Allow old versions

* Update number checking

* Update

* Remove uneeded CI files

* bump

* Add binutils

* Update the README

.github/workflows/python-app.yml

@@ -28,11 +28,11 @@ jobs:
     - name: Install dependencies
       run: |
         pip install --upgrade pip
-        sudo apt-get update && sudo apt-get install unzip openjdk-19-jdk graphviz-dev -y
+        sudo apt-get update && sudo apt-get install gcc make binutils unzip openjdk-19-jdk graphviz-dev -y
         pip3 install -e .
-        pip3 install angr
+        pip3 install angr pytest
     - name: Run verification tests
       run: |
         # a hack to allow for docker containers in github actions and ARM
         export JAVA_OPTS="-Djdk.lang.Process.launchMechanism=vfork" 
-        ./scripts/ci_verify_pipeline.sh
+        pytest tests/test_pipeline.py

README.md

@@ -2,7 +2,7 @@
 The SAILR evaluation pipeline, `sailreval`, is a tool for measuring various aspects of decompilation quality.
 This evaluation pipeline was originally developed for the USENIX 2024 paper ["Ahoy SAILR! There is No Need to DREAM of C:
 A Compiler-Aware Structuring Algorithm for Binary Decompilation"](https://www.zionbasque.com/files/publications/sailr_usenix24.pdf). It supports 26 different C packages from Debian,
-for compiling, decompiling and measuring. Currently, angr, Hex-Rays (IDA Pro), and Ghidra are supported as decompilers.
+for compiling, decompiling, and measuring. Currently, angr, Hex-Rays (IDA Pro), and Ghidra are supported as decompilers.
 
 If you are only looking to use the SAILR version of angr, then jump to the [using SAILR on angr](#using-sailr-on-angr-decompiler) section.
 
@@ -23,6 +23,7 @@ If you are only looking to use the SAILR version of angr, then jump to the [usin
 
 
 ## Overview:
+This repo contains the `sailreval` Python package and information about the SAILR paper artifacts.
 `sailreval` is the Python package that contains all the code for running the evaluation pipeline.
 `sailreval` evaluates the quality of decompilation by comparing it to the original source code.
 This evaluation is done in four phases:
@@ -35,34 +36,49 @@ Each phase requires the phase directly before it to have run, however, you can s
 required files. For example, you can skip compilation phase if you already have the object files and preprocessed source.
 
 ## Installation
-`sailreval` can be used in two ways: locally or in a docker container. If you plan on reproducing the results from the paper,
-or using pre-set decompilers, then you should use the docker container.
-Run the setup script to install the dependencies:
+The `sailreval` package can be used in two ways: locally or in a docker container.
+If you plan on reproducing the results of the SAILR paper, or using some pre-packaged decompiler like Ghidra, than you
+will need both. Below are two methods for installing, one is heavy (docker and local) and one is light (only local).
+Make sure you have Docker installed on your system. 
+
+### Install Script (Recommended)
+On Linux and MacOS:
 ```bash
 ./setup.sh
 ```
 
-This will install the Python package locally and build the docker container. If you know you don't want to use the docker
-container, then you can directly install the Python package with `pip3 insatll .`. Note: you need `graphviz` on your system.
+This will build the Docker container, install system dependencies, and install the Python package locally.
+
+### Only Python Package
+If you want to use only local decompilers, and you have the build dependencies installed for your compiled project, you
+can install the Python package without the Docker container. For an example of this use case, see 
+our [CI runner](./.github/workflows/python-app.yml).
+```bash
+pip3 insatll -e .
+```
 
+Note: you will need to install the system dependencies for the Python project yourself, listed [here]([CI runner](./.github/workflows/python-app.yml).
+The package is also available on PyPi, so remote installation works as well. 
+
+### Install Verification
 Verify the installation by running:
 ```bash
 ./scripts/verify_pipeline.sh
 ```
 
-If your installation is correct, you should see some final output like:
-```
+This will use both the Docker container and your local install to run the Pipeline. 
+If you installed correctly, you should see some final output like:
+```md
 # Evaluation Data
 ## Stats
 Layout: ('sum', 'mean', 'median')
 ### O2
-Metric     | source      | angr_sailr  | angr_dream 
+Metric     | source      | angr_sailr  | angr_dream
 ---------- | ----------- | ----------- | -----------
-cfged      | 0/0/0.0     | 14/1.75/2.0 | 34/4.25/2.0
+gotos      | 1/0.12/0.0  | 1/0.12/0.0     | 0/0/0.0
 ...
 ```
 
-
 ## Usage
 After installation, if you used the script normally (i.e. the docker install), than you can use the `docker-eval.sh` script
 which is a proxy to the `eval.py` script, but inside the container. 

sailreval/__init__.py

@@ -1,4 +1,4 @@
-__version__ = "1.1.0"
+__version__ = "1.2.0"
 
 # create loggers
 import logging

sailreval/decompilers/angr_dec.py

@@ -279,11 +279,17 @@ def generate_linemaps(dec, codegen, base_addr=0x400000):
         return
 
     base_addr = dec.project.loader.main_object.image_base_delta
-
+    if hasattr(dec, "unmodified_clinic_graph"):
+        nodes = dec.unmodified_clinic_graph.nodes
+    else:
+        l.warning(f"You are likely using an older version of angr that has no unmodified_clinic_graph."
+                  f" Using clinic_graph instead, results will be less accurate...")
+        nodes = dec.clinic.cc_graph.nodes
+
     # get the mapping of the original AIL graph
     mapping = defaultdict(set)
     ail_node_addr_map = {
-        node.addr: node for node in dec.unmodified_clinic_graph.nodes
+        node.addr: node for node in nodes
     }
     for addr, ail_block in ail_node_addr_map.items():
         # get instructions of this block

scripts/verify_pipeline.sh

@@ -2,7 +2,6 @@
 
 SCRIPT_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )"
 PROJECT_DIR="$SCRIPT_DIR/.."
-EXAMPLE_PROJECT_DIR="$SCRIPT_DIR/../tests/example_project"
 RESULTS_DIR="$SCRIPT_DIR/../results/O2"
 CORES="$(nproc --all)"
 
@@ -28,5 +27,5 @@ echo "[+] Running measurement pipeline (gotos, bools, calls, cfged)..." && \
 echo "[+] Running aggregation pipeline..." && \
 ./eval.py --summarize-targets example_project --use-dec source angr_sailr angr_dream --use-metric gotos cfged bools func_calls --show-stats && \
 # cleanup
-#rm -rf "$RESULTS_DIR/example_project" && \
+rm -rf "$RESULTS_DIR/example_project" && \
 echo "[+] The pipeline has successfully finished!" || (echo "[!] Pipeline failed, check the last stage it was in to figure out where!" && exit 1)