Skip to content

Latest commit

 

History

History
112 lines (71 loc) · 4.25 KB

README.md

File metadata and controls

112 lines (71 loc) · 4.25 KB
Raw

Note: The dev branch of this repository has been migrated to a separate repository Agents. This legacy branch contains reference code for IFAC Safeproccess 2018 paper: Comparison of Model Predictive and Reinforcement Learning Methods for Fault Tolerant Control by Ibrahim Ahmed, Gautam Biswas, and Hamed Khorasgani.

QLearning

A Reinforcement Learning library.

QLearning implements:

  • Model-based,
  • Temporal,
  • Online/Offline,
  • Tabular/Functional

Reinforcement Learning with a focus non-stationary environments.

RL on a topology with faults Visualization of a fuel tank system modelled as a circuit
RL on a topology with faults. Visualization of a fuel tank system modelled as a circuit.

Features include:

  • A modelling framework based on Netlist syntax and a circuit simulator. An environment can be programmatically/statically represented as an electrical circuit. Or the framework can be used to define custom elements and manipulate a modular system to represent changes in the environment.

  • Modular structure. A custom environment model can be easily used. All learner classes can be subclassed and still maintain compatibility.

  • Learning in descrete and continuous state spaces. Spaces represented by a finite combination of flags (e.g multidimensional grids) can be encoded into integers or decoded into vectors for tabular representation and functional approximations.

  • Flexible value function learning. Learner classes provide errors which can be used to train a custom approximation to the value function.

  • A built-in testbench to provide continuous/descrete environments to test and visualize learning.

The default learning algorithm is n-step tree back-up with variable step sizes.

Installation

Qlearning requires:

  • Python 3
  • Dependencies:
    • flask
    • numpy
    • ahkab
    • matplotlib
  1. Install Python 3.
  2. Download/clone this repository.
  3. Open terminal/console and navigate to this cloned repository. Then use the package manager pip to install dependencies:
cd Qlearning
pip install -r requirements.txt

Demo

The models/ directory comes with a 6-tank model for a C-130 plane's fuel tanks.

SixTankModel

The code ships with a simple model of 6 fuel tanks in a plane. The state space is 12 dimensional (6 tank levels + 6 valve states). The action space is 6 dimensional (6 valve states).

    T1  T2  TLA | TRA T3  T4

Each tank is controlled by a valve connected to a shared conduit for fuel transfer between tanks. Fuel pumps remove fuel to the engines according to pre-defined logic.

The state space is the fuel level in tanks (0-100) and the state of valves (on/off).

    [T1  T2  TLA TRA T3  T4 V1  V2  VLA VRA V3  V4]

The action space is the state of valves.

[V1  V2  VLA VRA V3  V4]

The SixTanksModel class is a programmatic model of the tanks. The class has a run() method which simulates the next state, given current tank levels, valve states, and action. For more details, see models/fuel_tanks.py.

from models import SixTankModel

tanks = SixTankModel()

next_state = tanks.run(state=(10, 20, 10, 30, 14, 50, 0, 0, 1, 1, 0, 1),
                       action=(1, 1, 1, 0, 0, 0))

Visualizing demo

There are several demos illustrating various control approaches to the problem of fuel transfer. The process can be visualized in a browser. The three scripts tankssimple.py, tanksnetlist.py and tankscustom.py use different approaches to modelling the system:

  • tankssimple.py: Models tanks as a circuit of capacitors and current sources. Creates a netlist at runtime to simulate.
  • tankssimple.py: Models tanks as a circuit of capacitors and resistors. Loads a netlist file models/tanks.netlist.
  • tankscustom.py: Models tanks as a set of formulae written in python. Uses the SixTankModel class defined in models/fuel_tanks.py.
# to see help message about arguments
python tankscustom.py -h

# visualize tank model behavior
# python tankscustom.py -x

The visualization can be viewed at http://localhost:5000