This is the accompanying code to the paper "Disparate Vulnerability to Membership Inference Attacks" which appears in PETS 2022.
The code enables to reproduce all the paper experiments with the corresponding plots and tables.
System Requirements. You need Python 3.8 and poetry 1.1.8. You can install poetry, e.g., as follows:
pip install --user poetry==1.1.8
To reproduce the plots exactly, you also need a LaTeX distribution with certain packages. On a Debian-based system, these can be installed as follows:
sudo apt install texlive-latex-extra cm-super texlive-science dvipng
Environment. Use poetry to set up the Python environment:
poetry install
Data. We use ADULT and Texas Hospital Discharge datasets. The ADULT dataset is checked into the repository; to install the Texas Hospital Discharge data use the following command:
make data
Notebooks sync. We use jupytext to automatically convert Jupyter notebooks to Python scripts and keep them in sync. To generate the notebooks initially:
make sync
Alternatively, we provide a docker image that can be used as:
docker build -t dv .
docker run -it --rm dv <command such as 'make tests'>
The docker image already includes all the steps in the manual setup.
To test that the setup works as expected, you can use:
make tests
This runs the same scripts that are used for the paper experiments but with fewer models (3 vs 200 models needed for the full reproduction). The tests can take several minutes to run. The tests fail if the command terminates unsuccessfully. Warnings are OK.
The repo contains the following relevant modules and directories:
- mia.py - Implementations of Membership Inference Attacks.
- model_zoo.py - Definitions of target ML models used in experiments.
- plot_params.py - Setup of the plot style parameters.
- plotting.py - Plotting utilities.
- utils.py - Misc. utilities.
- data/ - The directory in which
make datastores the data files. - loaders/ - The directory that contains modules for loading the datasets.
- results/ - The directory where the experiment data are saved.
- images/ - The directory where the plots are saved.
Jupyter notebooks (committed as Python scripts, see Notebooks Sync above):
- simulations.py - All experiments on our synthetic dataset (Sections 5.2, 5.3, 6.1, 6.2)
- adult.py - Experiments on the ADULT dataset (Section 7)
- texas.py - Experiments on the TEXAS-50K dataset (Section 7)
- reproducing_chang.py - Experiments on the synthetic dataset from "On the Privacy Risks of Algorithmic Fairness" (Section 5.2)
To reproduce the paper results, you need to launch the Jupyter server:
poetry run jupyter-notebook .
The command will output instructions on accessing the server.
If using the Docker container, you can launch the Jupyter server like so:
docker run -it --rm dv -p <port>:<port> jupyter-notebook --ip=0.0.0.0 --port=<port>
This will run the container on a given port (e.g., 8888) on your host machine.
To reproduce all the experimental data, tables, and plots, you need to execute each of the notebooks in the root folder using the launched Jupyter server. If you have not used Jupyter before, you can check this tutorial to learn how to do this.
Executing the notebooks will re-run the experiments and might take from 20 minutes to about 6 hours depending on the experiment and the hardware.
The reproduction will not be exact, as scikit-learn is not deterministic. We include the experimental data used in the paper in the results/ folder---results of the experiments. Using this data, you can reproduce the tables and plots from the paper without actually re-running the experiments.
To do so, you need to run the relevant notebook (see Modules and scripts)
using the Jupyter server, with a modification: set RESTORE_SAVED_DATA to True (if this flag is used
in the notebook). You will see the plots and tables displayed inline of the respective cells.