This repository contains the code for the paper Adversarial Inverse Reinforcement Learning for Market Making (2024) by Juraj Zelman (Richfox Capital & ETH Zürich), Martin Stefanik (Richfox Capital & ETH Zürich), Moritz Weiß (ETH Zürich) and Prof. Dr. Josef Teichmann (ETH Zürich). The paper was published and presented at the 5th ACM International Conference on AI in Finance (ICAIF ’24) in New York, USA. The links of both affiliations: Richfox Capital and ETH Zürich (Dept. of Mathematics).

The full training pipeline can be found in main.ipynb. Beforehand, see the Installation section below.

We hope you will learn something new and that this project sparks your curiosity to explore reinforcement learning methods, as well as other related fields!

Abstract

In this paper, we propose a novel application of the Adversarial Inverse Reinforcement Learning (AIRL) algorithm combining the framework of generative adversarial networks and inverse reinforcement learning for automated reward acquisition in the market making problem. We demonstrate that this algorithm can be used to learn a pure market making strategy. An advantage of such an approach is the possibility for further training in the reinforcement learning framework while potentially preserving its explainability and already acquired domain knowledge incorporated in an expert's decision process.

Installation

In order to run the code, we recommend you to install pyenv (for managing Python versions) and poetry (for managing dependencies) or alternative management tools. The project can be installed as follows:

1. Clone and cd into the repository:

   git clone [email protected]:JurajZelman/airl-market-making.git
   cd airl-market-making

2. Set the Python version to 3.11.9:

   pyenv install 3.11.9
   pyenv local 3.11.9

3. Install the dependencies:

   poetry install

   We recommend you to install the dependencies this way, as the repository uses two modified packages located in the .packages directory for better monitoring of the training process and a minor bug fix (see the Package modifications section below).

4. Run the main.ipynb notebook. Note that in order to query the pricing and trades data, you will need an active data provider subscription as described in Section 1 of the notebook.

Package modifications

The repository uses two modified packages stable-baselines3 and imitation. The modified versions are located in the .packages directory (all of our modifications are marked with # MODIFIED comments). The modifications are as follows:

• common.py: In this file we implement multiple enhancements that seemed to empirically improve the training process. Firstly, we update perform the discriminator optimization step for every batch of data instead of as an aggregate. Further, we scale the rewards for the PPO (generator) training and implement balancing of expert samples in the training process to improve the stability of the training process. Lastly, we enhance the monitoring of the training statistics.
• utils.py: Here we add a minor fix to avoid overflow warnings that might sometimes appear during the training process of the PPO algorithm (generator).
• logger.py: In this file we suppress the logging printouts during the training process.
• bc.py: Lastly, this enhancement of training process monitoring for the behavioral cloning algorithm is an artifact of the initial research and is not used in the final implementation.
• setup.py: Here we comment the use_scm_version setting to avoid installation issues.

Contributing

In case you find any bugs, have any suggestions for improvements or have any questions, feel free to open an issue or a pull request. I am happy to help and discuss any potential improvements.

Citation

For referencing this work, please use the following citation:

@inproceedings{
    10.1145/3677052.3698641,
    author = {Zelman, Juraj and Stefanik, Martin and Weiss, Moritz and Teichmann, Josef},
    title = {Adversarial Inverse Reinforcement Learning for Market Making},
    year = {2024},
    isbn = {9798400710810},
    publisher = {Association for Computing Machinery},
    address = {New York, NY, USA},
    url = {https://doi.org/10.1145/3677052.3698641},
    doi = {10.1145/3677052.3698641},
    booktitle = {Proceedings of the 5th ACM International Conference on AI in Finance},
    pages = {81–89},
    numpages = {9},
    location = {Brooklyn, NY, USA},
    series = {ICAIF '24}
}

Disclaimer

This repository was prepared for informational purposes, in part, by members, in their personal capacity, of the Research Department of Richfox Capital Investment Management AG hereinafter "Richfox" and is not a product of Richfox. The information is not intended to be relied upon as investment advise or as a recommendation to buy, sell, or hold any securities or financial instruments. The contents of this repository should not be construed as an offer, solicitation, or recommendation to engage in any investment activity. The members have taken reasonable care in preparing this repository, but do not warrant or guarantee the accuracy, completeness, or timeliness of the information contained herein. The repository may include information obtained from third-party sources, which we believe to be reliable. However, Richfox does not endorse or make any representation or warranty regarding the accuracy or completeness of such third-party information. Any reliance on the information contained is at the reader’s own risk. By accessing or using this repository, you acknowledge and agree to the terms of this disclaimer. If you do not agree with these terms, please do not rely on the information contained herein.