cannot pickle '_thread.RLock' object error

[Huge]

Hi, I was eager to try it out and hack on it, got everything installed via
. .venv/bin/activate && python3 -m pip install -r requirements.txt
but when running python3 -m moggie -y I get:

Traceback (most recent call last):
  File "/opt/homebrew/Caskroom/miniforge/base/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/opt/homebrew/Caskroom/miniforge/base/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/Users/admin/prog/moggie-mail-reader/moggie/__main__.py", line 8, in <module>
    Main(sys.argv[1:])
  File "/Users/admin/prog/moggie-mail-reader/moggie/app/__init__.py", line 186, in Main
    _run(command)
  File "/Users/admin/prog/moggie-mail-reader/moggie/app/__init__.py", line 179, in _run
    result = command(wd, args)
  File "/Users/admin/prog/moggie-mail-reader/moggie/app/__init__.py", line 137, in CommandMuttalike
    return CommandTUI(wd, tui_args, draft)
  File "/Users/admin/prog/moggie-mail-reader/moggie/app/__init__.py", line 60, in CommandTUI
    return tui.Main(wd, tui_args, draft)
  File "/Users/admin/prog/moggie-mail-reader/moggie/app/tui/__init__.py", line 142, in Main
    app_worker = AppWorker(workdir).connect()
  File "/Users/admin/prog/moggie-mail-reader/moggie/workers/app.py", line 283, in connect
    conn = super().connect(*args, **kwargs)
  File "/Users/admin/prog/moggie-mail-reader/moggie/workers/base.py", line 387, in connect
    self.start()
  File "/opt/homebrew/Caskroom/miniforge/base/lib/python3.10/multiprocessing/process.py", line 121, in start
    self._popen = self._Popen(self)
  File "/opt/homebrew/Caskroom/miniforge/base/lib/python3.10/multiprocessing/context.py", line 224, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
  File "/opt/homebrew/Caskroom/miniforge/base/lib/python3.10/multiprocessing/context.py", line 288, in _Popen
    return Popen(process_obj)
  File "/opt/homebrew/Caskroom/miniforge/base/lib/python3.10/multiprocessing/popen_spawn_posix.py", line 32, in __init__
    super().__init__(process_obj)
  File "/opt/homebrew/Caskroom/miniforge/base/lib/python3.10/multiprocessing/popen_fork.py", line 19, in __init__
    self._launch(process_obj)
  File "/opt/homebrew/Caskroom/miniforge/base/lib/python3.10/multiprocessing/popen_spawn_posix.py", line 47, in _launch
    reduction.dump(process_obj, fp)
  File "/opt/homebrew/Caskroom/miniforge/base/lib/python3.10/multiprocessing/reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)
TypeError: cannot pickle '_thread.RLock' object

Most likely problem with the miniforge distro of Python, not sure...

[Huge]

I have tried removing the .venv an go with python3.11, but seems the same trouble..:

python3.11 -m moggie -y
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/Users/admin/prog/moggie-mail-reader/moggie/__main__.py", line 8, in <module>
    Main(sys.argv[1:])
  File "/Users/admin/prog/moggie-mail-reader/moggie/app/__init__.py", line 186, in Main
    _run(command)
  File "/Users/admin/prog/moggie-mail-reader/moggie/app/__init__.py", line 179, in _run
    result = command(wd, args)
             ^^^^^^^^^^^^^^^^^
  File "/Users/admin/prog/moggie-mail-reader/moggie/app/__init__.py", line 137, in CommandMuttalike
    return CommandTUI(wd, tui_args, draft)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/admin/prog/moggie-mail-reader/moggie/app/__init__.py", line 60, in CommandTUI
    return tui.Main(wd, tui_args, draft)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/admin/prog/moggie-mail-reader/moggie/app/tui/__init__.py", line 142, in Main
    app_worker = AppWorker(workdir).connect()
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/admin/prog/moggie-mail-reader/moggie/workers/app.py", line 283, in connect
    conn = super().connect(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/admin/prog/moggie-mail-reader/moggie/workers/base.py", line 387, in connect
    self.start()
  File "/opt/homebrew/Cellar/[email protected]/3.11.3/Frameworks/Python.framework/Versions/3.11/lib/python3.11/multiprocessing/process.py", line 121, in start
    self._popen = self._Popen(self)
                  ^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/[email protected]/3.11.3/Frameworks/Python.framework/Versions/3.11/lib/python3.11/multiprocessing/context.py", line 224, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/[email protected]/3.11.3/Frameworks/Python.framework/Versions/3.11/lib/python3.11/multiprocessing/context.py", line 288, in _Popen
    return Popen(process_obj)
           ^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/[email protected]/3.11.3/Frameworks/Python.framework/Versions/3.11/lib/python3.11/multiprocessing/popen_spawn_posix.py", line 32, in __init__
    super().__init__(process_obj)
  File "/opt/homebrew/Cellar/[email protected]/3.11.3/Frameworks/Python.framework/Versions/3.11/lib/python3.11/multiprocessing/popen_fork.py", line 19, in __init__
    self._launch(process_obj)
  File "/opt/homebrew/Cellar/[email protected]/3.11.3/Frameworks/Python.framework/Versions/3.11/lib/python3.11/multiprocessing/popen_spawn_posix.py", line 47, in _launch
    reduction.dump(process_obj, fp)
  File "/opt/homebrew/Cellar/[email protected]/3.11.3/Frameworks/Python.framework/Versions/3.11/lib/python3.11/multiprocessing/reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)
TypeError: cannot pickle '_thread.RLock' object

[BjarniRunar]

Thank you for reporting this.

That isn't crashing in my code - it looks like just having a threading.RLock in a multiprocessing object causes things to explode? It's definitely platform specific, I'm on Python 3.10.12 on Ubuntu and I am not seeing this issue.

It's possible you could work around this by moving the creation of the RLock out of the constructor and into moggie.workers.base.run() , so doesn't have to cross the process boundary. If that works for you, let me know!

[ahknight]

Running into this on macOS as well. I can test moving the lock, however there are now several locks...

$ grep -r RLock moggie
moggie/crypto/gpgi.py:        self._lock = threading.RLock()
moggie/config/__init__.py:        self.lock = threading.RLock()
moggie/search/engine.py:        self.lock = threading.RLock()
moggie/workers/base.py:        self.lock = threading.RLock()

Which one are you referring to and where should it go?

[BjarniRunar]

I am pretty sure the underlying issue here is that I am using multiprocessing to spawn workers, and the default multiprocessing on MacOS uses pickling to transfer state across process boundaries. I probably need to rethink this a bit.