Can't unpickle objects defined in __main__

[crusaderky]

(https://bugs.python.org/issue37292)

ORIGINAL POST:

As of CPython 3.8.0b1, main branch (please let me know if there's a different branch I should use):

If one pickles an object that is defined in the __main__ module, sends it to a subinterpreter as bytes, and then tries unpickling it there, it fails saying that __main__ doesn't define it.

import _xxsubinterpreters as interpreters
import pickle


class C:
    pass


c = C()

interp_id = interpreters.create()
c_bytes = pickle.dumps(c)
interpreters.run_string(
    interp_id,
    "import pickle; pickle.loads(c_bytes)",
    shared={"c_bytes": c_bytes},
)

If the above is executed directly with the python command-line, it fails.
If it's imported from another module, it works.

I'm unsure if that's working as intended or not; I was expecting behaviour compatible with sub-processes spawned with the spawn method, where the__main__ of the parent process is visilble to the subprocess too.

Workarounds:
1 - define everything that must be pickled in an imported module
2 - use cloudPickle, which implements a hardcoded special case that makes it pickle the whole code of any object defined in __main__.

Possible future solutions:

• opt-in support for re-running __main__ in a subinterpreter
• a helper for re-running __main__ in a subinterpreter
• support in PEP 499 for updating __module__ for all objects in __main__
• opt-in support for mirroring a subinterpreter's __main__ as a named module (in sys.modules)
  • this is similar to PEP 499
  • also update __module__ for all objects

[ericsnowcurrently]

This definitely sounds like a bug. :( Thanks for finding that! Please open a new issue on bugs.python.org and feel free to nosy me.

[ericsnowcurrently]

https://bugs.python.org/issue37292

@crusaderky, Thanks!

[ericsnowcurrently]

I'm going to track this here along with other subinterpreter-related bugs that need short-term attention.