Skip to content

Latest commit

 

History

History
77 lines (53 loc) · 3.67 KB

README.md

File metadata and controls

77 lines (53 loc) · 3.67 KB
Raw

multi-core-python

Enabling CPython multi-core parallelism via subinterpreters.

This repo is for tracking the effort and as a place to keep any tooling.

wiki | projects

Project Summary

From my python-ideas post (June 2015):

Python's multi-core story is murky at best.  Not only can we be more
clear on the matter, we can improve Python's support.  The result of any
effort must make multi-core (and concurrency) support in Python obvious,
unmistakable, and undeniable (and keep it Pythonic).

The goal of this project is to do just that, by using the existing subinterpreter C-API.

Solution

The minimal multi-core solution will involve:

  • resolving existing bugs and blockers
  • exposing the existing support (from C-API) in a stdlib module
  • adding a mechanism to safely "share" basic immutable objects between interpreters
  • no longer sharing the GIL between interpreters

Once a minimal solution is in place we can expand from there.

At a high level, we're doing the following concrete tasks:

Constraints / Requirements

  1. no significant impact on single-threaded performance
  2. maintain backward compatibility (C-API, etc.)
  3. (pseudo-)compatibility with multiprocessing/threading/concurrent.futures APIs
  4. a multi-core concurrency model/approach that fits our brains
  5. Python APIs
  6. supportable on other Python implementations

Status

  • slow progress
  • high level of community excitement
  • increasing number of collaborators
  • phase 1 ("minimal solution") completion getting closer
    • still hopeful for Python 3.9
    • PEP 554 low-level implementation mostly complete
    • currently working on moving globals to runtime/interpreter state (or removing them)
      • will take a while (started with ~1500 global variables)
    • next up is making memory allocators per-interpreter (and then the GIL itself)

Key Collaborators

This project wouldn't be what it is without the help of these great people:

  • @ncoghlan - inspired the solution; PEP 432
  • @encukou - better subinterpreter support in extension modules
  • @vstinner - all sorts of runtime stuff
  • @emilyemorehouse - reviews & sanity checks
  • @nanjekyejoannah - C-API; PEP 554 high-level module
  • @eduardo-elizondo (Facebook/Instagram) - extension modules; globals
  • @vsajip - globals
  • many others

Contributing

There are many ways to contribute to this project. Aside from the main technical work in the CPython runtime, there are also a number of jobs that need to be done that are less expert-related and even (somewhat) non-technical. All of it is important and anyone interested in helping is welcome!

Keep in mind that this project/repo is actually just a tool to organize the effort. The actual work is done on bugs.python.org, github.com/python/cpython, and the python-dev mailing list.

Also note that contacting @ericsnowcurrently directly is fine, but you might get a faster response through those other channels or through the issue tracker here. :)

For more information see the wiki page.