Process-kit

What?

process-kit is an alternative implementation of the python multiprocessing.Process standard library class.

Why

Although multiprocessing.Process code base is pretty robust, it felt like not being explicit nor obvious enough. Moreover, the whole design of the standard library class looks like it has been made with the whole multiprocessing library in mind rather than a clean interface to unix processes.

So we've decided to try to clean it a bit, simplify it, and make it more obvious; by getting rid of the global variables, private classes, and pretty complicated abstractions imbrications we had found in there.

Additional resources for the curious

• Process-kit presentation: https://speakerdeck.com/botify/fixing-the-process
• Blog post about the what, the why, and the how: https://labs.botify.com/blog/process-kit-why-and-how-we-decided-to-rewrite-python-s-process-class/

Installation

Easy as:

pip install process-kit

Usage

The Process class

How to

Just like the multiprocessing.Process class, there are two different ways to run a task into a unix process using process-kit: - By providing a callable target to it at construction. - By overriding it's run method.

Providing a target callable

from pkit.process import Process

def mytarget(*args, **kwargs):
    do_something()

proc = Process(target=mytarget, args=("abc 123",))
proc.start()
proc.join()

Overriding Process.run method

from pkit.process import Process

class MyProcessObj(Process):
    def run(self, *args, **args):
        do_something()
        
my_process_obj = MyProcessObj()
my_process_obj.start()
my_process_obj.join()

Augmented features

Reusability

Processes are cleaned up once their execution is over. They are hulls for your executions pieces. It means that once a process is collected, it's execution context attributes are reset and you can reuse it for other purpose.

import time

from pkit.process import Process

process = pkit.Process(target=lambda: time.sleep(1))
pid = process.start(wait=True)
assert pid is not None

exitcode = process.join()

assert process.pid is None
assert exitcode == 0

Blocking start

Passing the wait option to Process.start will ensure you the method call will block until the underlying process is ready to start it's execution.

import os
import time
import signal
from pkit.process import Process

process = Process(target=lambda: time.sleep(10))
process.start(wait=True)
os.kill(process.pid, signal.SIGTERM)
assert process.is_alive() is False

Blocking terminate

Passing the wait option to Process.terminate will ensure you the method call will block until the underlying process is actually stopped. As terminate() method uses a SIGTERM signal under the hood, it ensures you the call will block until the SIGTERM has been processed.

import time
from pkit.process import Process

process = Process(target=lambda: time.sleep(10))
process.start()
process.terminate(wait=True)
assert process.is_alive() is False

Exit callbacks

pkit.Process class exposes an on_exit callback option. Pass it a callable taking a single proc argument, and it will be executed as soon as the process terminates

import time

from pkit.process import Process


EXITED_PID = None


def exit_callback(proc):
	EXITED_PID = proc.pid


process = Process(target=lambda: time.sleep(1), on_exit=exit_callback)
pid = process.start()
process.terminate()

assert EXITED_PID == pid

Restartable

Processes are restartable following a provided policy: forced shutdown or graceful termination.

import time

from pkit.process import Process


EXITED_PID = None


def exit_callback(proc):
	EXITED_PID = proc.pid


process = Process(target=lambda: time.sleep(1), on_exit=exit_callback)
pid = process.start()
process.terminate()

assert EXITED_PID == pid