Skip to content

Commit

Permalink
Update Readme
Browse files Browse the repository at this point in the history
  • Loading branch information
StefanUlbrich committed Aug 21, 2023
1 parent f088cad commit 316963d
Showing 1 changed file with 54 additions and 33 deletions.
87 changes: 54 additions & 33 deletions Readme.md
Original file line number Diff line number Diff line change
Expand Up @@ -46,50 +46,71 @@ efficiency and simplicity (both in terms of implementation and
comprehensibility). Plus, it's my favorite and I'm doing active research with
it.

* Numerical python extensions written in Rust
* Seamless integration via [PyO3](https://pyo3.rs/v0.16.1/) and [rust-numpy](https://docs.rs/numpy/0.7.0/numpy/)
* High-performance via [ndarray](https://github.com/rust-ndarray/ndarray)/[rayon](https://docs.rs/ndarray/0.13.1/ndarray/parallel/index.html), [tch-rs](https://github.com/LaurentMazare/tch-rs) ([PyTorch](https://pytorch.org/) bindings)
* Dependency management / publishing with [Poetry](https://python-poetry.org/docs/) and [Maturin](https://github.com/PyO3/maturin)
* [Monorepo](https://en.wikipedia.org/wiki/Monorepo) with [Cargo workspaces][(](<https://doc.rust-lang.org/book/ch14-03-cargo-workspaces.html>))
* Technical documentation / GitHub page with [Sphinx](https://www.sphinx-doc.org/en/master/) and [MyST](https://myst-parser.readthedocs.io/en/latest/sphinx/intro.html)
* Eventually, distributed computation (e.g., [actor model](https://en.wikipedia.org/wiki/Actor_model) or [timely
dataflow](https://timelydataflow.github.io/timely-dataflow/))
* Hopefully, GUI application using [Tauri](https://tauri.studio/), [Angular](https://angular.io/) and [ThreeJS](https://threejs.org/)

Feel free use as a basis for your own projects (MIT licensed).
The development explores:

## Development

For the monorepo, there is a top-level python project defined using poetry. This project is
mainly for dependency locking, integration testing and the overall project's documentation.
The python module
build with Maturin is in a nested folder and it's project file is also created with poetry.
This is necessary as the top-level project can only add local packages that are either
created with Poetry or contain a `setup.py` file.
* Numerical python extensions written in Rust: Seamless integration via [PyO3](https://pyo3.rs/v0.16.1/) and [rust-numpy](https://docs.rs/numpy/0.7.0/numpy/). Dependency management / publishing with [Poetry](https://python-poetry.org/docs/) and [Maturin](https://github.com/PyO3/maturin)
* High-performance via [ndarray](https://github.com/rust-ndarray/ndarray)/[rayon](https://docs.rs/ndarray/0.13.1/ndarray/parallel/index.html).

While this setup supports a monorepo setup (and should support integration
testing on the imported local packages), there is another caveat. Building the python extension with
`pip` creates a temp directory which does not copy the local rust dependencies. Newer versions of `pip`
build within the tree, so this limitation can be avoided easily. Use the following commands to
setup the environment.
## The cerebral and potpourri Python packages

> :warning: We want to avoid creating a virtual environment for the nested packages. Work in
> a top-level shell instead.
The`potpourri` and `citrate` sub folders contain the Python bindings for
the two crates in `potpourri-rs` and `citrate-rs`. You can build them in a
similar fashion.

```sh
cd self-organization
pyenv shell 3.10.2
poetry env use 3.10.2
poetry run pip install -U pip # only required until pip>=22.0 becomes the default
cd citrate
# skip the next line if you are not using pyenv
pyenv shell 3.10.2 # replace with desired/installed version
# skip if you only have one version of python installed
poetry env use 3.10.2 # replace with desired/installed version
# Install dependencies
poetry install
# to debug / develop the extension
# Or optionally, with jupyter
poetry install --with jupyter
# activate the virtual environment
poetry shell
cd pysom
# install into the current environment
maturin develop
# with optimizations (recommended)
maturin develop --release
# build an installable package
maturin build --release
```

To install the virtual environment as a kernel for jupyter:

```sh
python -m ipykernel install --user --name py310_selforganization --display-name "Python3.10 (self-organization)"
poetry add ipykernel
# Adjust names as suitable
python -m ipykernel install --user --name py320_citrate --display-name "Python3.10 (Citrate)"
```

## Development

* Bulding documentation. We need extra headers to have formulas in the rustdoc

```sh
RUSTDOCFLAGS="--html-in-header ./static/header.html" cargo doc -p cerebral --no-deps
RUSTDOCFLAGS="--html-in-header ./static/header.html" cargo doc -F ndarray -p potpourri --no-deps
# Open (e.g. with Firefox)
firefox target/doc/citrate/index.html
firefox target/doc/potpourri/index.html
```

* Benchmarks (currently only in Potpourri)

```sh
cd potpourri-rs
cargo bench -F ndarray
```

* Running individual tests

```sh
cd potpourri-rs
cargo test -F ndarray -p potpourri test_multi_pass
```

## License

Published under the MIT license.

0 comments on commit 316963d

Please sign in to comment.