Merge branch 'main' into adaptive

Doc/conf.py

@@ -78,6 +78,13 @@
 .. |python_version_literal| replace:: ``Python {version}``
 .. |python_x_dot_y_literal| replace:: ``python{version}``
 .. |usr_local_bin_python_x_dot_y_literal| replace:: ``/usr/local/bin/python{version}``
+
+.. Apparently this how you hack together a formatted link:
+   (https://www.docutils.org/docs/ref/rst/directives.html#replacement-text)
+.. |FORCE_COLOR| replace:: ``FORCE_COLOR``
+.. _FORCE_COLOR: https://force-color.org/
+.. |NO_COLOR| replace:: ``NO_COLOR``
+.. _NO_COLOR: https://no-color.org/
 """
 
 # There are two options for replacing |today|. Either, you set today to some

Doc/howto/gdb_helpers.rst

@@ -180,7 +180,7 @@ regular machine-level integer::
     (gdb) p some_python_integer
     $4 = 42
 
-The internal structure can be revealed with a cast to :c:expr:`PyLongObject *`:
+The internal structure can be revealed with a cast to :c:expr:`PyLongObject *`::
 
     (gdb) p *(PyLongObject*)some_python_integer
     $5 = {ob_base = {ob_base = {ob_refcnt = 8, ob_type = 0x3dad39f5e0}, ob_size = 1},

Doc/library/doctest.rst

@@ -136,6 +136,10 @@ examples of doctests in the standard Python test suite and libraries.
 Especially useful examples can be found in the standard test file
 :file:`Lib/test/test_doctest/test_doctest.py`.
 
+.. versionadded:: 3.13
+   Output is colorized by default and can be
+   :ref:`controlled using environment variables <using-on-controlling-color>`.
+
 
 .. _doctest-simple-testmod:
 

Doc/library/pathlib.rst

@@ -1289,35 +1289,6 @@ Reading directories
    raised.
 
 
-.. method:: Path.scandir()
-
-   When the path points to a directory, return an iterator of
-   :class:`os.DirEntry` objects corresponding to entries in the directory. The
-   returned iterator supports the :term:`context manager` protocol. It is
-   implemented using :func:`os.scandir` and gives the same guarantees.
-
-   Using :meth:`~Path.scandir` instead of :meth:`~Path.iterdir` can
-   significantly increase the performance of code that also needs file type or
-   file attribute information, because :class:`os.DirEntry` objects expose
-   this information if the operating system provides it when scanning a
-   directory.
-
-   The following example displays the names of subdirectories. The
-   ``entry.is_dir()`` check will generally not make an additional system call::
-
-      >>> p = Path('docs')
-      >>> with p.scandir() as entries:
-      ...     for entry in entries:
-      ...         if entry.is_dir():
-      ...             entry.name
-      ...
-      '_templates'
-      '_build'
-      '_static'
-
-   .. versionadded:: 3.14
-
-
 .. method:: Path.glob(pattern, *, case_sensitive=None, recurse_symlinks=False)
 
    Glob the given relative *pattern* in the directory represented by this path,

Doc/library/traceback.rst

@@ -44,6 +44,10 @@ The module's API can be divided into two parts:
   necessary for later formatting without holding references to actual exception
   and traceback objects.
 
+.. versionadded:: 3.13
+   Output is colorized by default and can be
+   :ref:`controlled using environment variables <using-on-controlling-color>`.
+
 
 Module-Level Functions
 ----------------------

Doc/library/unittest.rst

@@ -46,7 +46,6 @@ test runner
    a textual interface, or return a special value to indicate the results of
    executing the tests.
 
-
 .. seealso::
 
    Module :mod:`doctest`
@@ -198,6 +197,9 @@ For a list of all the command-line options::
    In earlier versions it was only possible to run individual test methods and
    not modules or classes.
 
+.. versionadded:: 3.14
+   Output is colorized by default and can be
+   :ref:`controlled using environment variables <using-on-controlling-color>`.
 
 Command-line options
 ~~~~~~~~~~~~~~~~~~~~

Doc/tutorial/datastructures.rst

@@ -142,8 +142,8 @@ Using Lists as Stacks
 
 The list methods make it very easy to use a list as a stack, where the last
 element added is the first element retrieved ("last-in, first-out").  To add an
-item to the top of the stack, use :meth:`!~list.append`.  To retrieve an item from the
-top of the stack, use :meth:`!~list.pop` without an explicit index.  For example::
+item to the top of the stack, use :meth:`!append`.  To retrieve an item from the
+top of the stack, use :meth:`!pop` without an explicit index.  For example::
 
    >>> stack = [3, 4, 5]
    >>> stack.append(6)
@@ -340,7 +340,7 @@ The :keyword:`!del` statement
 =============================
 
 There is a way to remove an item from a list given its index instead of its
-value: the :keyword:`del` statement.  This differs from the :meth:`!~list.pop` method
+value: the :keyword:`del` statement.  This differs from the :meth:`!pop` method
 which returns a value.  The :keyword:`!del` statement can also be used to remove
 slices from a list or clear the entire list (which we did earlier by assignment
 of an empty list to the slice).  For example::
@@ -500,8 +500,8 @@ any immutable type; strings and numbers can always be keys.  Tuples can be used
 as keys if they contain only strings, numbers, or tuples; if a tuple contains
 any mutable object either directly or indirectly, it cannot be used as a key.
 You can't use lists as keys, since lists can be modified in place using index
-assignments, slice assignments, or methods like :meth:`!~list.append` and
-:meth:`!~list.extend`.
+assignments, slice assignments, or methods like :meth:`!append` and
+:meth:`!extend`.
 
 It is best to think of a dictionary as a set of *key: value* pairs,
 with the requirement that the keys are unique (within one dictionary). A pair of

Doc/using/cmdline.rst

@@ -663,14 +663,6 @@ output. To control the color output only in the Python interpreter, the
 precedence over ``NO_COLOR``, which in turn takes precedence over
 ``FORCE_COLOR``.
 
-.. Apparently this how you hack together a formatted link:
-
-.. |FORCE_COLOR| replace:: ``FORCE_COLOR``
-.. _FORCE_COLOR: https://force-color.org/
-
-.. |NO_COLOR| replace:: ``NO_COLOR``
-.. _NO_COLOR: https://no-color.org/
-
 Options you shouldn't use
 ~~~~~~~~~~~~~~~~~~~~~~~~~
 

Doc/whatsnew/3.13.rst

@@ -252,15 +252,6 @@ Improved error messages
   the canonical |NO_COLOR|_ and |FORCE_COLOR|_ environment variables.
   (Contributed by Pablo Galindo Salgado in :gh:`112730`.)
 
-.. Apparently this how you hack together a formatted link:
-   (https://www.docutils.org/docs/ref/rst/directives.html#replacement-text)
-
-.. |FORCE_COLOR| replace:: ``FORCE_COLOR``
-.. _FORCE_COLOR: https://force-color.org/
-
-.. |NO_COLOR| replace:: ``NO_COLOR``
-.. _NO_COLOR: https://no-color.org/
-
 * A common mistake is to write a script with the same name as a
   standard library module. When this results in errors, we now
   display a more helpful error message:

Doc/whatsnew/3.14.rst

@@ -532,12 +532,6 @@ pathlib
 
   (Contributed by Barney Gale in :gh:`73991`.)
 
-* Add :meth:`pathlib.Path.scandir` to scan a directory and return an iterator
-  of :class:`os.DirEntry` objects. This is exactly equivalent to calling
-  :func:`os.scandir` on a path object.
-
-  (Contributed by Barney Gale in :gh:`125413`.)
-
 
 pdb
 ---
@@ -616,6 +610,13 @@ unicodedata
 unittest
 --------
 
+* :mod:`unittest` output is now colored by default.
+  This can be controlled via the :envvar:`PYTHON_COLORS` environment
+  variable as well as the canonical |NO_COLOR|_
+  and |FORCE_COLOR|_ environment variables.
+  See also :ref:`using-on-controlling-color`.
+  (Contributed by Hugo van Kemenade in :gh:`127221`.)
+
 * unittest discovery supports :term:`namespace package` as start
   directory again. It was removed in Python 3.11.
   (Contributed by Jacob Walls in :gh:`80958`.)

Include/internal/pycore_object.h

@@ -697,8 +697,52 @@ _PyObject_SetMaybeWeakref(PyObject *op)
     }
 }
 
+extern int _PyObject_ResurrectEndSlow(PyObject *op);
 #endif
 
+// Temporarily resurrects an object during deallocation. The refcount is set
+// to one.
+static inline void
+_PyObject_ResurrectStart(PyObject *op)
+{
+    assert(Py_REFCNT(op) == 0);
+#ifdef Py_REF_DEBUG
+    _Py_IncRefTotal(_PyThreadState_GET());
+#endif
+#ifdef Py_GIL_DISABLED
+    _Py_atomic_store_uintptr_relaxed(&op->ob_tid, _Py_ThreadId());
+    _Py_atomic_store_uint32_relaxed(&op->ob_ref_local, 1);
+    _Py_atomic_store_ssize_relaxed(&op->ob_ref_shared, 0);
+#else
+    Py_SET_REFCNT(op, 1);
+#endif
+}
+
+// Undoes an object resurrection by decrementing the refcount without calling
+// _Py_Dealloc(). Returns 0 if the object is dead (the normal case), and
+// deallocation should continue. Returns 1 if the object is still alive.
+static inline int
+_PyObject_ResurrectEnd(PyObject *op)
+{
+#ifdef Py_REF_DEBUG
+    _Py_DecRefTotal(_PyThreadState_GET());
+#endif
+#ifndef Py_GIL_DISABLED
+    Py_SET_REFCNT(op, Py_REFCNT(op) - 1);
+    return Py_REFCNT(op) != 0;
+#else
+    uint32_t local = _Py_atomic_load_uint32_relaxed(&op->ob_ref_local);
+    Py_ssize_t shared = _Py_atomic_load_ssize_acquire(&op->ob_ref_shared);
+    if (_Py_IsOwnedByCurrentThread(op) && local == 1 && shared == 0) {
+        // Fast-path: object has a single refcount and is owned by this thread
+        _Py_atomic_store_uint32_relaxed(&op->ob_ref_local, 0);
+        return 0;
+    }
+    // Slow-path: object has a shared refcount or is not owned by this thread
+    return _PyObject_ResurrectEndSlow(op);
+#endif
+}
+
 /* Tries to incref op and returns 1 if successful or 0 otherwise. */
 static inline int
 _Py_TryIncref(PyObject *op)

Include/internal/pycore_pystate.h

@@ -190,10 +190,18 @@ static inline void
 _Py_EnsureFuncTstateNotNULL(const char *func, PyThreadState *tstate)
 {
     if (tstate == NULL) {
+#ifndef Py_GIL_DISABLED
         _Py_FatalErrorFunc(func,
             "the function must be called with the GIL held, "
             "after Python initialization and before Python finalization, "
             "but the GIL is released (the current Python thread state is NULL)");
+#else
+        _Py_FatalErrorFunc(func,
+            "the function must be called with an active thread state, "
+            "after Python initialization and before Python finalization, "
+            "but it was called without an active thread state. "
+            "Are you trying to call the C API inside of a Py_BEGIN_ALLOW_THREADS block?");
+#endif
     }
 }
 

InternalDocs/garbage_collector.md

@@ -199,22 +199,22 @@ unreachable:
 
 ```pycon
 >>> import gc
->>> 
+>>>
 >>> class Link:
 ...    def __init__(self, next_link=None):
 ...        self.next_link = next_link
-...  
+...
 >>> link_3 = Link()
 >>> link_2 = Link(link_3)
 >>> link_1 = Link(link_2)
 >>> link_3.next_link = link_1
 >>> A = link_1
 >>> del link_1, link_2, link_3
->>> 
+>>>
 >>> link_4 = Link()
 >>> link_4.next_link = link_4
 >>> del link_4
->>> 
+>>>
 >>> # Collect the unreachable Link object (and its .__dict__ dict).
 >>> gc.collect()
 2
@@ -459,11 +459,11 @@ specifically in a generation by calling `gc.collect(generation=NUM)`.
 >>> # Create a reference cycle.
 >>> x = MyObj()
 >>> x.self = x
->>> 
+>>>
 >>> # Initially the object is in the young generation.
 >>> gc.get_objects(generation=0)
 [..., <__main__.MyObj object at 0x7fbcc12a3400>, ...]
->>> 
+>>>
 >>> # After a collection of the youngest generation the object
 >>> # moves to the old generation.
 >>> gc.collect(generation=0)
@@ -515,6 +515,44 @@ increment. All objects directly referred to from those stack frames are
 added to the working set.
 Then the above algorithm is repeated, starting from step 2.
 
+Determining how much work to do
+-------------------------------
+
+We need to do a certain amount of work to enusre that garbage is collected,
+but doing too much work slows down execution.
+
+To work out how much work we need to do, consider a heap with `L` live objects
+and `G0` garbage objects at the start of a full scavenge and `G1` garbage objects
+at the end of the scavenge. We don't want the amount of garbage to grow, `G1 ≤ G0`, and
+we don't want too much garbage (say 1/3 of the heap maximum), `G0 ≤ L/2`.
+For each full scavenge we must visit all objects, `T == L + G0 + G1`, during which
+`G1` garbage objects are created.
+
+The number of new objects created `N` must be at least the new garbage created, `N ≥ G1`,
+assuming that the number of live objects remains roughly constant.
+If we set `T == 4*N` we get `T > 4*G1` and `T = L + G0 + G1` => `L + G0 > 3G1`
+For a steady state heap (`G0 == G1`) we get `L > 2G0` and the desired garbage ratio.
+
+In other words, to keep the garbage fraction to 1/3 or less we need to visit
+4 times as many objects as are newly created.
+
+We can do better than this though. Not all new objects will be garbage.
+Consider the heap at the end of the scavenge with `L1` live objects and `G1`
+garbage. Also, note that `T == M + I` where `M` is the number of objects marked
+as reachable and `I` is the number of objects visited in increments.
+Everything in `M` is live, so `I ≥ G0` and in practice `I` is closer to `G0 + G1`.
+
+If we choose the amount of work done such that `2*M + I == 6N` then we can do
+less work in most cases, but are still guaranteed to keep up.
+Since `I ≳ G0 + G1` (not strictly true, but close enough)
+`T == M + I == (6N + I)/2` and `(6N + I)/2 ≳ 4G`, so we can keep up.
+
+The reason that this improves performance is that `M` is usually much larger
+than `I`. If `M == 10I`, then `T ≅ 3N`.
+
+Finally, instead of using a fixed multiple of 8, we gradually increase it as the
+heap grows. This avoids wasting work for small heaps and during startup.
+
 
 Optimization: reusing fields to save memory
 ===========================================

Lib/asyncio/selector_events.py

@@ -1175,6 +1175,7 @@ def writelines(self, list_of_data):
         # If the entire buffer couldn't be written, register a write handler
         if self._buffer:
             self._loop._add_writer(self._sock_fd, self._write_ready)
+            self._maybe_pause_protocol()
 
     def can_write_eof(self):
         return True