On the reproducibility of ”Fairness Guarantees under Demographic Shift”

This repository contains the code used for the paper On the reproducibility of ”Fairness Guarantees under Demographic Shift”. It is largely similar to the original codebase of the ICLR 2022 paper Fairness Guarantees under Demographic Shift, but contains the following contributions:

• An env.yaml file containing the correct requirements for reproducing the experiments
• Adjustments made to the code to run it without errors on all machines (Linux, Windows, and Mac)
• Code to aggregate and save results to json, calculation of statistics and generation of latex tables
• Additional experiments on different bound sizes and optimization methods

Requirements

After cloning this repository, simply run:

conda env create -f Python/env.yaml
conda activate fact

The datasets are already contained in this repository, but can also be downloaded at Adult and Brazil.

Reproduction

Reproducing results of original paper

The experiments from the original paper, which were also reproduced in our implementation, can be executed by running the provided file from the Python directory, as follows:

# on Linux/Mac:
./experiments/scripts/iclr_ds_experiments.sh

# on Windows:
experiments/scripts/iclr_ds_experiments.bat

Hyperparameters and models are specified in this file. After running these experiments, their results can be extracted to json by running:

python experiments/scripts/results_to_json.py

The figures can be created by running

python -m experiments.scripts.iclr_figures_adult
python -m experiments.scripts.iclr_figures_adult --unknown_ds
python -m experiments.scripts.iclr_figures_brazil
python -m experiments.scripts.iclr_figures_brazil --unknown_ds

The new figures will be saved to Python/figures/* by default. The tables, together with the statistical analysis, can be generated by running

python experiments/scripts/json-stats_to_latex.py

This will save the results as reported in the paper to tables/reproduction.txt

Reproducing results beyond original paper

Bound size of unknown demographic shift

This experiment can be reproduced by running:

# on Linux/Mac
./experiments/scripts/bounds.sh

# on Windows
experiments/scripts/bounds.bat

The results can be extracted to json format by running

python experiments/scripts/results_to_json.py --directory results_bounds --bounds True

and then, the plot shown in our paper can be created by running

python experiments/scripts/plot_bounds.py

This will save the plot to figures/Bounds.png.

Comparing different optimizers with Shifty

This experiment can be reproduced by running:

# on Linux/Mac
./experiments/scripts/optimizers.sh

# on Windows
experiments/scripts/optimizers.bat

---

NOTE

If the experiments of the original paper have been executed and the results are saved, the commands in the optimizer script with the --optimizer cmaes argument may be removed.

---

Again, the results can be extracted to json format by running

python experiments/scripts/results_to_json.py

and then, the tables shown in our paper can be created by running

python experiments/scripts/json-optimizer_to_latex.py

This will save the results as reported in the paper to tables/optimizers.txt

License

This repository is released under the MIT license.