Submodules not showing up for (native) extension modules #319

robamler · 2021-03-02T00:57:18Z

When running pdoc on an extension modules (aka, a native "C" extensions), the extension module's submodules don't show up in the documentation even though <TAB>-autocomplete in a Python REPL can find the submodules. This seems to be because pdoc searches for submodules by inspecting the source directory, which isn't available for extension modules.

I've proposed PR #318 to fix this issue. The proposed solution works but I'm not sure if it is safe enough to remove the old "source directory traversal" method. I'd appreciate guidance on completing the PR.

Expected Behavior

Running pdoc on a native extension module should generate documentation for the entire extension module, including its submodules.

Actual Behavior

With current master: only items in the top-level module appear in the documentation. Submodules don't show up in the documentation.
With the proposed fix in Recurse through submodules in extension modules #318: works as expected.

Steps to Reproduce

The following steps generate a minimalistic native extension module in Rust that exhibits the problem. The language shouldn't matter though.

Install a rust toolchain, see https://rustup.rs
Create the following directory structure:

pyext/
├── Cargo.toml
└── src/
    └── lib.rs

with the following file contents:

Cargo.toml:

[package]
authors = ["Name <[email protected]>"]
edition = "2018"
name = "pyext"
version = "0.1.0"

[lib]
crate-type = ["cdylib"]

[dependencies]
pyo3 = {version = "0.13.2", features = ["extension-module"]}

src/lib.rs:

use pyo3::{prelude::*, wrap_pymodule};

/// Docstring of main module.
#[pymodule(pyext)]
fn init_main_module(_py: Python<'_>, module: &PyModule) -> PyResult<()> {
    module.add_wrapped(wrap_pymodule!(submodule))?;
    Ok(())
}

/// Docstring of submodule
#[pymodule(submodule)]
fn init_submodule(_py: Python<'_>, submodule: &PyModule) -> PyResult<()> {
    submodule.add("variable", 42)?;
    Ok(())
}

Compile the extension module: cargo build
Create a properly named symlink to the object file:
- on Linux: ln -s target/debug/libpyext.so pyext.so
- on Mac: ln -s target/debug/libpyext.dylib pyext.so
- on Windows: rename target\debug\libpyext.dll to pyext.pyd
Start a Python REPL from the directory containing the pyext.so file and verify that the submodule exists and can be found by tab completion:

$ python
Python 3.6.10 (default, May 22 2020, 17:59:48) 
[GCC 9.2.1 20191008] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pyext
>>> pyext.<TAB>  --> autocompletes to "pyext.submodule", proving that the submodule can be found
>>> pyext.submodule.variable
42

Run pdoc --html pyext from the same directory.
- With the version at current master, the generated documentation leaves out the submodule.
- With the version proposed in Recurse through submodules in extension modules #318, the generated documentation includes the submodule.

Additional info

pdoc version: master and 0.9.2 don't work, the one from Recurse through submodules in extension modules #318 works.
tested on linux

The text was updated successfully, but these errors were encountered:

kernc · 2021-03-05T13:01:21Z

Thanks for an exemplary bug report!

Just to clarify: Step 5, when we import pyext, could we just as well have done:

>>> import pyext.submodule

# or

>>> from pyext.submodule import variable

Does this run?

robamler · 2021-03-07T19:40:49Z

Just tested it:

import pyext.submodule doesn't work;
from pyext.submodule import variable doesn't work;
however, from pyext import submodule works.

$ python
Python 3.8.2 (default, Mar  2 2021, 23:57:34) 
[GCC 9.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pyext.submodule
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'pyext.submodule'; 'pyext' is not a package
>>> from pyext.submodule import variable
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'pyext.submodule'; 'pyext' is not a package
>>> from pyext import submodule
>>> submodule.variable
42

I think this is because the extension module pyext is compiled into a single binary file that cannot be loaded "in parts" (unlike regular modules, whose implementation is typically scattered across several source files). The python interpreter doesn't know about the submodules until it actively loads the pyext module, which (I think) it only does when you explicitly say either import pyext or from pyext import xxx.

In other words, I think from A import B actually loads A (but only brings A.B into scope, as B), so from pyext.submodule import variable would try to load pyext.submodule, which doesn't exist in the file system because it only gets generated "in memory" when you load pyext.

kernc · 2021-03-07T20:53:17Z

That's exactly why I asked because I remembered resolving to wontfix about a similar issue just recently. See my thoughts in #252 (comment). The simple fact is:

ModuleNotFoundError: No module named 'pyext.submodule'; 'pyext' is not a package

pyext.submodule is not a module to have stuff imported from, so I'm hesitant to make pdoc list it as such.

Can you investigate if you can set .__package__ and .__path__ attributes (or whatever is necessary to interpret Python module as a package) upon the relevant package/module objects and if maybe that automatically does something?

robamler · 2021-03-07T21:51:43Z

Thank you for the explanation! Unfortunately, setting .__package__ and .__path__ in the extension module doesn't help.

I respect your decision if you don't want to address this. I'd just like to raise two counter arguments for your consideration. First, I think this issue will probably affect a lot of people (probably all authors of native extension modules that don't find some sort of workaround). Second, I am interpreting the ModuleNotFoundError in a different way. In fact, I get the same error message when I try to import, e.g., pdoc:

>>> import pdoc
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'pdoc'

The reason is that I'm in a python environment where pdoc isn't installed, and that's why the python interpreter can't find it and thus throws a ModuleNotFoundError. So, even though pdoc definitely is a module (it's even a package), it just can't be found at the moment. But as soon as you bring it in your sys.path, it can be found. I'd argue that the situation for pyext.submodule is quite similar: it is a module, it just can't be found at the moment. But as soon as you import pyext (which is the package on which I want to run pdoc anyway), then pyext.submodule can be found (and is recognized as a module):

>>> import pyext
>>> type(pyext.submodule)
<class 'module'>

I agree that pyext.submodule is not a package (e.g., it doesn't have a .__path__ set), but I think that shouldn't make a difference.

kernc · 2021-03-08T02:06:31Z

it just can't be found at the moment

That's correct. That's why Python has sys.path_hooks (to maybe provide a suitable finder/loader for a given package) and sys.meta_path, which is a list of already registered default finders.

Following related upstream issues:

ModuleNotFoundError: No module named 'supermodule.submodule'; 'supermodule' is not a package PyO3/pyo3#759,
Crates with multiple modules / submodules PyO3/maturin#266,

I think PyO3 might wish to provide a finder akin to the one removed in PyO3/pyo3@8d14568 (briefly discussed in PyO3/pyo3#1269 (comment)), and add it to sys.meta_path upon loading the top-level extension module.
This way, both pdoc pyext as well as >>> import pyext.submodule would work flawlessly, and it'll be justified to call pyext.submodule a package and its submodule (instead of merely a variable pointing to a module object such as with >>> import re as my_re).

I'd just hate to have pdoc's deviate from the Python's interpretation of stuff.

Then again, we do check in #318 that the object is present in __all__, so the intent is visible, and there's little utility in documenting modules containing further objects as mere variables. The end-user will be confused that they can't:

from your_pyext.submodule.nested import Something

But that's not really our problem ... 🤔

kernc · 2021-03-28T14:56:37Z

There's apparently a workaround described in PyO3/pyo3#1517 (comment).

kernc linked a pull request Mar 3, 2021 that will close this issue

Recurse through submodules in extension modules #318

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Submodules not showing up for (native) extension modules #319

Submodules not showing up for (native) extension modules #319

robamler commented Mar 2, 2021 •

edited

Loading

kernc commented Mar 5, 2021

robamler commented Mar 7, 2021 •

edited

Loading

kernc commented Mar 7, 2021 •

edited

Loading

robamler commented Mar 7, 2021 •

edited

Loading

kernc commented Mar 8, 2021

kernc commented Mar 28, 2021

Submodules not showing up for (native) extension modules #319

Submodules not showing up for (native) extension modules #319

Comments

robamler commented Mar 2, 2021 • edited Loading

Expected Behavior

Actual Behavior

Steps to Reproduce

Additional info

kernc commented Mar 5, 2021

robamler commented Mar 7, 2021 • edited Loading

kernc commented Mar 7, 2021 • edited Loading

robamler commented Mar 7, 2021 • edited Loading

kernc commented Mar 8, 2021

kernc commented Mar 28, 2021

robamler commented Mar 2, 2021 •

edited

Loading

robamler commented Mar 7, 2021 •

edited

Loading

kernc commented Mar 7, 2021 •

edited

Loading

robamler commented Mar 7, 2021 •

edited

Loading