Merge branch 'master' into 295-184-reparenting-rework

.github/workflows/static.yaml

@@ -16,7 +16,7 @@ jobs:
     - name: Set up Python
       uses: actions/setup-python@v2
       with:
-        python-version: '3.8'
+        python-version: '3.12'
 
     - name: Install tox
       run: |

.github/workflows/unit.yaml

@@ -14,7 +14,7 @@ jobs:
 
     strategy:
       matrix:
-        python-version: [pypy-3.7, 3.7, 3.8, 3.9, '3.10', 3.11, '3.12.0-rc.3']
+        python-version: [pypy-3.7, 3.7, 3.8, 3.9, '3.10', 3.11, '3.12', '3.13-dev']
         os: [ubuntu-20.04]
         include:
           - os: windows-latest

README.rst

@@ -78,11 +78,17 @@ in development
 
 This is the last major release to support Python 3.7.
 
-* Drop support for Python 3.6
-* Add support for Python 3.12
+* Drop support for Python 3.6.
+* Add support for Python 3.12 and Python 3.13.
+* Astor is no longer a requirement starting at Python 3.9.
 * `ExtRegistrar.register_post_processor()` now supports a `priority` argument that is an int.
   Highest priority callables will be called first during post-processing.
 * Fix too noisy ``--verbose`` mode (suppres some ambiguous annotations warnings).
+* Fix type processing inside restructuredtext consolidated fields.
+* Add options ``--cls-member-order`` and ``--mod-member-order`` to customize the presentation
+  order of class members and module/package members, the supported values are "alphabetical" or "source".
+  The default behavior is to sort all members alphabetically.
+* Make sure the line number coming from ast analysis has precedence over the line of a ``ivar`` field.
 
 pydoctor 23.9.1
 ^^^^^^^^^^^^^^^

docs/source/contrib.rst

@@ -17,23 +17,65 @@ If you like the project and think you could help with making it better, there ar
 Any contribution would be of great help and I will highly appreciate it! If you have any questions, please create a new issue.
 
 
-Pre-commit checks
------------------
+Development process
+-------------------
 
-Make sure all the tests pass and the code pass the coding standard checks::
+Create a fork of the git repository and checkout a new branch from ``master`` branch. 
+The branch name may start with an associated issue number so that we can easily 
+cross-reference them. For example, use ``1234-some-brach-name`` as the name of the branch working to fix issue ``1234``.
+Once you're ready to run a full batterie of tests to your changes, open a pull request.
 
-    tox -p all
+Don't forget to sync your fork once in while to work from the latest revision.
 
-That should be the minimum check to run on your local system.
+Pre-commit checks
+-----------------
+
+Make sure all the unit tests pass and the code pass the coding standard checks.
+
+We use `tox <https://tox.wiki/en/stable/>`_ for running our checks, but you can roughly do the same thing from your python environment. 
+
+.. list-table:: Pre-commit checks
+   :widths: 10 45 45
+   :header-rows: 1
+
+   * - \
+     - Using `tox`
+     - Using your environment
+   * - Run unit tests
+     - ``tox -e test``
+     - ``pip install '.[test]' && pytest pydoctor``
+   * - Run pyflakes
+     - ``tox -e pyflakes``
+     - ``pip install pyflakes && find pydoctor/ -name \*.py ! -path '*/testpackages/*' ! -path '*/sre_parse36.py' ! -path '*/sre_constants36.py' | xargs pyflakes``
+   * - Run mypy
+     - ``tox -e mypy``
+     - ``pip install '.[mypy]' && mypy pydoctor``
+   * - Run pydoctor on it's own source
+     - ``tox -e apidocs``
+     - ``pip install . && pydoctor --privacy "HIDDEN:pydoctor.test" -q -W pydoctor``
+
+These should be the minimum check to run on your local system.
 A pull request will trigger more tests and most probably there is a tox
 environment dedicated to that extra test.
 
+Other things hapenning when a PR is open
+----------------------------------------
+
+- System tests: these tests checks if pydoctor can generate the documentation for a few
+  specific packages that have been considered as problematic in the past.
+- Pydoctor primer: this is to pydoctor what ``mypy_primer`` is to ``mypy``. 
+  It runs pydoctor on a corpus of open source code and compares the output of the application before and after a modification in the code.
+  Then it reports in comments the result for a PR. The source code of this tool is here: https://github.com/twisted/pydoctor_primer.
+- Readthedocs build: For every PR, the sphinx documentation is built and available at ``https://pydoctor--{pr-number}.org.readthedocs.build/en/``.
 
 Review process and requirements
 -------------------------------
 
 - Code changes and code added should have tests: untested code is buggy code. Except special cases, overall test coverage should be increased.
 - If your pull request is a work in progress, please mark it as draft such that reviewers do not loose time on a PR that is not ready yet.
+- There is no strict coding style standard. Since pydoctor is more than 20 years old and we have vendored some code from 
+  other packages as well (namely epydoc and sre_parse), so we can’t really enforce the same style everywhere. It's up to the reviewers
+  to request refactors when the code is too ugly.
 - All code changes must be reviewed by at least one person who is not an author of the code being added. 
   This helps prevent bugs from slipping through the net and gives another source for improvements.
   If the author of the PR is one of the core developers of pydoctor* and no one has reviewed their PR after 9 calendar days, they can review the code changes themselves and proceed with next steps. 

docs/source/quickstart.rst

@@ -62,6 +62,7 @@ Pydoctor is a pretty verbose tool by default. It’s quite unlikely that you get
 But don’t worry, pydoctor should have produced useful HTML pages no matter your project design or docstrings. 
 
 Exit codes includes:
+
 - ``0``: All docstrings are well formatted (warnings may be printed).
 - ``1``: Pydoctor crashed with traceback (default Python behaviour).
 - ``2``: Some docstrings are mal formatted.

pydoctor/astbuilder.py

@@ -15,12 +15,11 @@
 )
 
 import attr
-import astor
 from pydoctor import epydoc2stan, model, node2stan, extensions, linker
 from pydoctor.epydoc.markup._pyval_repr import colorize_inline_pyval
 from pydoctor.astutils import (is_none_literal, is_typing_annotation, is_using_annotations, is_using_typing_final, node2dottedname, node2fullname, 
                                is__name__equals__main__, unstring_annotation, iterassign, extract_docstring_linenum, infer_type, get_parents,
-                               get_docstring_node, NodeVisitor, Parentage, Str)
+                               get_docstring_node, unparse, NodeVisitor, Parentage, Str)
 
 def parseFile(path: Path) -> ast.Module:
     """Parse the contents of a Python source file."""
@@ -439,8 +438,8 @@ def visit_ClassDef(self, node: ast.ClassDef) -> None:
                 name_node = base_node.value
 
             str_base = '.'.join(node2dottedname(name_node) or \
-                # Fallback on astor if the expression is unknown by node2dottedname().
-                [astor.to_source(base_node).strip()]) 
+                # Fallback on unparse() if the expression is unknown by node2dottedname().
+                [unparse(base_node).strip()]) 
 
             # Store the base as string and as ast.expr in rawbases list.
             rawbases += [(str_base, base_node)]
@@ -584,7 +583,6 @@ def _importNames(self, modname: str, names: Iterable[ast.alias], linenumber:int)
             orgname, asname = al.name, al.asname
             if asname is None:
                 asname = orgname
-
             # If we're importing from a package, make sure imported modules
             # are processed (getProcessedModule() ignores non-modules).
             if isinstance(mod, model.Package):

pydoctor/astutils.py

@@ -7,15 +7,28 @@
 import platform
 import sys
 from numbers import Number
-from typing import Any, Iterator, Optional, List, Iterable, Sequence, TYPE_CHECKING, Tuple, Union, cast
+from typing import Any, Callable, Collection, Iterator, Optional, List, Iterable, Sequence, TYPE_CHECKING, Tuple, Union, cast
 from inspect import BoundArguments, Signature
 import ast
 
+if sys.version_info >= (3, 9):
+    from ast import unparse as _unparse
+else:
+    from astor import to_source as _unparse
+
 from pydoctor import visitor
 
 if TYPE_CHECKING:
     from pydoctor import model
 
+def unparse(node:ast.AST) -> str:
+    """
+    This function convert a node tree back into python sourcecode.
+
+    Uses L{ast.unparse} or C{astor.to_source} for python versions before 3.9.
+    """
+    return _unparse(node)
+
 # AST visitors
 
 def iter_values(node: ast.AST) -> Iterator[ast.AST]:
@@ -250,7 +263,7 @@ def visit_Subscript(self, node: ast.Subscript) -> ast.Subscript:
         else:
             # Other subscript; unstring the slice.
             slice = self.visit(node.slice)
-        return ast.copy_location(ast.Subscript(value, slice, node.ctx), node)
+        return ast.copy_location(ast.Subscript(value=value, slice=slice, ctx=node.ctx), node)
 
     # For Python >= 3.8:
 
@@ -488,13 +501,14 @@ def _annotation_for_value(value: object) -> Optional[ast.expr]:
             if ann_value is None:
                 ann_elem = None
             elif ann_elem is not None:
-                ann_elem = ast.Tuple(elts=[ann_elem, ann_value])
+                ann_elem = ast.Tuple(elts=[ann_elem, ann_value], ctx=ast.Load())
         if ann_elem is not None:
             if name == 'tuple':
-                ann_elem = ast.Tuple(elts=[ann_elem, ast.Constant(value=...)])
-            return ast.Subscript(value=ast.Name(id=name),
-                                 slice=ast.Index(value=ann_elem))
-    return ast.Name(id=name)
+                ann_elem = ast.Tuple(elts=[ann_elem, ast.Constant(value=...)], ctx=ast.Load())
+            return ast.Subscript(value=ast.Name(id=name, ctx=ast.Load()),
+                                 slice=ann_elem,
+                                 ctx=ast.Load())
+    return ast.Name(id=name, ctx=ast.Load())
 
 def _annotation_for_elements(sequence: Iterable[object]) -> Optional[ast.expr]:
     names = set()
@@ -507,7 +521,7 @@ def _annotation_for_elements(sequence: Iterable[object]) -> Optional[ast.expr]:
             return None
     if len(names) == 1:
         name = names.pop()
-        return ast.Name(id=name)
+        return ast.Name(id=name, ctx=ast.Load())
     else:
         # Empty sequence or no uniform type.
         return None
@@ -540,3 +554,118 @@ def _yield_parents(n:Optional[ast.AST]) -> Iterator[ast.AST]:
             yield from _yield_parents(p)
     yield from _yield_parents(getattr(node, 'parent', None))
 
+#Part of the astor library for Python AST manipulation.
+#License: 3-clause BSD
+#Copyright (c) 2015 Patrick Maupin
+_op_data = """
+    GeneratorExp                1
+
+          Assign                1
+       AnnAssign                1
+       AugAssign                0
+            Expr                0
+           Yield                1
+       YieldFrom                0
+              If                1
+             For                0
+        AsyncFor                0
+           While                0
+          Return                1
+
+           Slice                1
+       Subscript                0
+           Index                1
+        ExtSlice                1
+    comprehension_target        1
+           Tuple                0
+  FormattedValue                0
+
+           Comma                1
+       NamedExpr                1
+          Assert                0
+           Raise                0
+    call_one_arg                1
+
+          Lambda                1
+           IfExp                0
+
+   comprehension                1
+              Or   or           1
+             And   and          1
+             Not   not          1
+
+              Eq   ==           1
+              Gt   >            0
+             GtE   >=           0
+              In   in           0
+              Is   is           0
+           NotEq   !=           0
+              Lt   <            0
+             LtE   <=           0
+           NotIn   not in       0
+           IsNot   is not       0
+
+           BitOr   |            1
+          BitXor   ^            1
+          BitAnd   &            1
+          LShift   <<           1
+          RShift   >>           0
+             Add   +            1
+             Sub   -            0
+            Mult   *            1
+             Div   /            0
+             Mod   %            0
+        FloorDiv   //           0
+         MatMult   @            0
+          PowRHS                1
+          Invert   ~            1
+            UAdd   +            0
+            USub   -            0
+             Pow   **           1
+           Await                1
+             Num                1
+        Constant                1
+"""
+
+_op_data = [x.split() for x in _op_data.splitlines()] # type:ignore
+_op_data = [[x[0], ' '.join(x[1:-1]), int(x[-1])] for x in _op_data if x] # type:ignore
+for _index in range(1, len(_op_data)):
+    _op_data[_index][2] *= 2 # type:ignore
+    _op_data[_index][2] += _op_data[_index - 1][2] # type:ignore
+
+_deprecated: Collection[str] = ()
+if sys.version_info >= (3, 12):
+    _deprecated = ('Num', 'Str', 'Bytes', 'Ellipsis', 'NameConstant')
+_precedence_data = dict((getattr(ast, x, None), z) for x, y, z in _op_data if x not in _deprecated) # type:ignore
+_symbol_data = dict((getattr(ast, x, None), y) for x, y, z in _op_data if x not in _deprecated) # type:ignore
+
+class op_util:
+    """
+    This class provides data and functions for mapping
+    AST nodes to symbols and precedences.
+    """
+    @classmethod
+    def get_op_symbol(cls, obj:ast.operator|ast.boolop|ast.cmpop|ast.unaryop, 
+                      fmt:str='%s', 
+                      symbol_data:dict[type[ast.AST]|None, str]=_symbol_data, 
+                      type:Callable[[object], type[Any]]=type) -> str:
+        """Given an AST node object, returns a string containing the symbol.
+        """
+        return fmt % symbol_data[type(obj)]
+    @classmethod
+    def get_op_precedence(cls, obj:ast.operator|ast.boolop|ast.cmpop|ast.unaryop, 
+                          precedence_data:dict[type[ast.AST]|None, int]=_precedence_data, 
+                          type:Callable[[object], type[Any]]=type) -> int:
+        """Given an AST node object, returns the precedence.
+        """
+        return precedence_data[type(obj)]
+
+    if not TYPE_CHECKING:
+        class Precedence(object):
+            vars().update((cast(str, x), z) for x, _, z in _op_data)
+            highest = max(cast(int, z) for _, _, z in _op_data) + 2
+    else:
+        Precedence: Any
+
+del _op_data, _index, _precedence_data, _symbol_data, _deprecated
+# This was part of the astor library for Python AST manipulation.