Skip to content

Commit

Permalink
Avoid using internal functions/object with multiprocessing
Browse files Browse the repository at this point in the history
A Multiprocessing start method is the technique used
by the Multiprocessing library to start child processes in Python.

There are three start methods:

* spawn: start a new Python process.
* fork: copy a Python process from an existing process.
* forkserver: new process from which future forked processes will be copied.

Each OS has a list of supported start methods:
* Windows (win32): spawn
* MacOS (darwin): spawn (default), fork, forkserver.
* Linux (unix): spawn, fork (default), forkserver.

Spawn not only is the most supported, but as per recent investigations
of the devs of the Multiprocessing library, is the most secure on all
the OS.

Spawn method makes use of the Pickle library for starting the new
subprocesses and Pickle is unable to process internal objects and
functions.

Get rid an internal function (handler inside the make_logging_handler
method of the Command class) and use a callable object instead.

Consider substituting internal objects and functions with Pickle
compliant options all around the code and eventually enforce the usage
of the spawn method.

Signed-off-by: Giulio Calacoci <[email protected]>
  • Loading branch information
gcalacoci committed Sep 12, 2023
1 parent cb42b10 commit 0197300
Showing 1 changed file with 18 additions and 9 deletions.
27 changes: 18 additions & 9 deletions barman/command_wrappers.py
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,22 @@
_logger = logging.getLogger(__name__)


class Handler:
def __init__(self, logger, level, prefix=None):
self.class_logger = logger
self.level = level
self.prefix = prefix

def run(self, line):
if line:
if self.prefix:
self.class_logger.log(self.level, "%s%s", self.prefix, line)
else:
self.class_logger.log(self.level, "%s", line)

__call__ = run


class StreamLineProcessor(object):
"""
Class deputed to reading lines from a file object, using a buffered read.
Expand Down Expand Up @@ -526,7 +542,7 @@ def make_logging_handler(cls, level, prefix=None):
"""
Build a handler function that logs every line it receives.
The resulting function logs its input at the specified level
The resulting callable object logs its input at the specified level
with an optional prefix.
:param level: The log level to use
Expand All @@ -535,14 +551,7 @@ def make_logging_handler(cls, level, prefix=None):
"""
class_logger = logging.getLogger(cls.__name__)

def handler(line):
if line:
if prefix:
class_logger.log(level, "%s%s", prefix, line)
else:
class_logger.log(level, "%s", line)

return handler
return Handler(class_logger, level, prefix)

@staticmethod
def make_output_handler(prefix=None):
Expand Down

0 comments on commit 0197300

Please sign in to comment.