Merge: 'upstream/main' - bring fork up to date with upstream ready fo…

…r PR4099

.github/pull_request_template.md

@@ -8,6 +8,6 @@ Please consider adding the following to your pull request:
  - docs to all new functions and / or detail in the guide
  - tests for all new or changed functions
 
-PyO3's CI pipeline will check your pull request. To run its tests
+PyO3's CI pipeline will check your pull request, thus make sure you have checked the `Contributing.md` guidelines. To run most of its tests
 locally, you can run ```nox```. See ```nox --list-sessions```
 for a list of supported actions.

.github/workflows/gh-pages.yml

@@ -46,7 +46,7 @@ jobs:
 
       - name: Deploy docs and the guide
         if: ${{ github.event_name == 'release' }}
-        uses: peaceiris/actions-gh-pages@v3
+        uses: peaceiris/actions-gh-pages@v4
         with:
           github_token: ${{ secrets.GITHUB_TOKEN }}
           publish_dir: ./target/guide/

CHANGELOG.md

@@ -10,6 +10,42 @@ To see unreleased changes, please see the [CHANGELOG on the main branch guide](h
 
 <!-- towncrier release notes start -->
 
+## [0.21.2] - 2024-04-16
+
+### Changed
+
+- Deprecate the `PySet::empty()` gil-ref constructor. [#4082](https://github.com/PyO3/pyo3/pull/4082)
+
+### Fixed
+
+- Fix compile error for `async fn` in `#[pymethods]` with a `&self` receiver and more than one additional argument. [#4035](https://github.com/PyO3/pyo3/pull/4035)
+- Improve error message for wrong receiver type in `__traverse__`. [#4045](https://github.com/PyO3/pyo3/pull/4045)
+- Fix compile error when exporting a `#[pyclass]` living in a different Rust module using the `experimental-declarative-modules` feature. [#4054](https://github.com/PyO3/pyo3/pull/4054)
+- Fix `missing_docs` lint triggering on documented `#[pymodule]` functions. [#4067](https://github.com/PyO3/pyo3/pull/4067)
+- Fix undefined symbol errors for extension modules on AIX (by linking `libpython`). [#4073](https://github.com/PyO3/pyo3/pull/4073)
+
+## [0.21.1] - 2024-04-01
+
+### Added
+
+- Implement `Send` and `Sync` for `PyBackedStr` and `PyBackedBytes`. [#4007](https://github.com/PyO3/pyo3/pull/4007)
+- Implement `Clone`, `Debug`, `PartialEq`, `Eq`, `PartialOrd`, `Ord` and `Hash` implementation for `PyBackedBytes` and `PyBackedStr`, and `Display` for `PyBackedStr`. [#4020](https://github.com/PyO3/pyo3/pull/4020)
+- Add `import_exception_bound!` macro to import exception types without generating GIL Ref functionality for them. [#4027](https://github.com/PyO3/pyo3/pull/4027)
+
+### Changed
+
+- Emit deprecation warning for uses of GIL Refs as `#[setter]` function arguments. [#3998](https://github.com/PyO3/pyo3/pull/3998)
+- Add `#[inline]` hints on many `Bound` and `Borrowed` methods. [#4024](https://github.com/PyO3/pyo3/pull/4024)
+
+### Fixed
+
+- Handle `#[pyo3(from_py_with = "")]` in `#[setter]` methods [#3995](https://github.com/PyO3/pyo3/pull/3995)
+- Allow extraction of `&Bound` in `#[setter]` methods. [#3998](https://github.com/PyO3/pyo3/pull/3998)
+- Fix some uncovered code blocks emitted by `#[pymodule]`, `#[pyfunction]` and `#[pyclass]` macros. [#4009](https://github.com/PyO3/pyo3/pull/4009)
+- Fix typo in the panic message when a class referenced in `pyo3::import_exception!` does not exist. [#4012](https://github.com/PyO3/pyo3/pull/4012)
+- Fix compile error when using an async `#[pymethod]` with a receiver and additional arguments. [#4015](https://github.com/PyO3/pyo3/pull/4015)
+
+
 ## [0.21.0] - 2024-03-25
 
 ### Added
@@ -1709,6 +1745,9 @@ Yanked
 
 - Initial release
 
+[Unreleased]: https://github.com/pyo3/pyo3/compare/v0.21.2...HEAD
+[0.21.2]: https://github.com/pyo3/pyo3/compare/v0.21.1...v0.21.2
+[0.21.1]: https://github.com/pyo3/pyo3/compare/v0.21.0...v0.21.1
 [0.21.0]: https://github.com/pyo3/pyo3/compare/v0.20.3...v0.21.0
 [0.21.0-beta.0]: https://github.com/pyo3/pyo3/compare/v0.20.3...v0.21.0-beta.0
 [0.20.3]: https://github.com/pyo3/pyo3/compare/v0.20.2...v0.20.3

Cargo.toml

@@ -1,6 +1,6 @@
 [package]
 name = "pyo3"
-version = "0.21.0"
+version = "0.21.2"
 description = "Bindings to Python interpreter"
 authors = ["PyO3 Project and Contributors <https://github.com/PyO3>"]
 readme = "README.md"
@@ -22,10 +22,10 @@ memoffset = "0.9"
 portable-atomic = "1.0"
 
 # ffi bindings to the python interpreter, split into a separate crate so they can be used independently
-pyo3-ffi = { path = "pyo3-ffi", version = "=0.21.0" }
+pyo3-ffi = { path = "pyo3-ffi", version = "=0.21.2" }
 
 # support crates for macros feature
-pyo3-macros = { path = "pyo3-macros", version = "=0.21.0", optional = true }
+pyo3-macros = { path = "pyo3-macros", version = "=0.21.2", optional = true }
 indoc = { version = "2.0.1", optional = true }
 unindent = { version = "0.2.1", optional = true }
 
@@ -38,7 +38,7 @@ pyo3-testing = { path = "pyo3-testing", version = "=0.1.0", optional = true}
 # crate integrations that can be added using the eponymous features
 anyhow = { version = "1.0", optional = true }
 chrono = { version = "0.4.25", default-features = false, optional = true }
-chrono-tz = { version = ">= 0.6, < 0.9", default-features = false, optional = true }
+chrono-tz = { version = ">= 0.6, < 0.10", default-features = false, optional = true }
 either = { version = "1.9", optional = true }
 eyre = { version = ">= 0.4, < 0.7", optional = true }
 hashbrown = { version = ">= 0.9, < 0.15", optional = true }
@@ -52,7 +52,7 @@ smallvec = { version = "1.0", optional = true }
 [dev-dependencies]
 assert_approx_eq = "1.1.0"
 chrono = "0.4.25"
-chrono-tz = ">= 0.6, < 0.9"
+chrono-tz = ">= 0.6, < 0.10"
 # Required for "and $N others" normalization
 trybuild = ">=1.0.70"
 proptest = { version = "1.0", default-features = false, features = ["std"] }
@@ -63,7 +63,7 @@ rayon = "1.6.1"
 futures = "0.3.28"
 
 [build-dependencies]
-pyo3-build-config = { path = "pyo3-build-config", version = "=0.21.0", features = ["resolve-config"] }
+pyo3-build-config = { path = "pyo3-build-config", version = "=0.21.2", features = ["resolve-config"] }
 
 [features]
 default = ["macros", "testing"]

Contributing.md

@@ -92,9 +92,7 @@ Here are a few things to note when you are writing PRs.
 
 ### Continuous Integration
 
-The PyO3 repo uses GitHub Actions. PRs are blocked from merging if CI is not successful.
-
-Formatting, linting and tests are checked for all Rust and Python code. In addition, all warnings in Rust code are disallowed (using `RUSTFLAGS="-D warnings"`).
+The PyO3 repo uses GitHub Actions. PRs are blocked from merging if CI is not successful. Formatting, linting and tests are checked for all Rust and Python code. In addition, all warnings in Rust code are disallowed (using `RUSTFLAGS="-D warnings"`).
 
 Tests run with all supported Python versions with the latest stable Rust compiler, as well as for Python 3.9 with the minimum supported Rust version.
 
@@ -103,6 +101,24 @@ If you are adding a new feature, you should add it to the `full` feature in our
 You can run these tests yourself with
 `nox`. Use  `nox -l` to list the full set of subcommands you can run.
 
+#### Linting Python code
+`nox -s ruff`
+
+#### Linting Rust code
+`nox -s rustfmt`
+
+#### Semver checks
+`cargo semver-checks check-release`
+
+#### Clippy
+`nox -s clippy-all`
+
+#### Tests
+`cargo test --features full`
+
+#### Check all conditional compilation
+`nox -s check-feature-powerset`
+
 #### UI Tests
 
 PyO3 uses [`trybuild`][trybuild] to develop UI tests to capture error messages from the Rust compiler for some of the macro functionality.
@@ -190,7 +206,7 @@ Second, there is a Python-based benchmark contained in the `pytests` subdirector
 
 You can view what code is and isn't covered by PyO3's tests. We aim to have 100% coverage - please check coverage and add tests if you notice a lack of coverage!
 
-- First, ensure the llmv-cov cargo plugin is installed. You may need to run the plugin through cargo once before using it with `nox`.
+- First, ensure the llvm-cov cargo plugin is installed. You may need to run the plugin through cargo once before using it with `nox`.
 ```shell
 cargo install cargo-llvm-cov
 cargo llvm-cov

README.md

@@ -68,7 +68,7 @@ name = "string_sum"
 crate-type = ["cdylib"]
 
 [dependencies]
-pyo3 = { version = "0.21.0", features = ["extension-module"] }
+pyo3 = { version = "0.21.2", features = ["extension-module"] }
 ```
 
 **`src/lib.rs`**
@@ -137,7 +137,7 @@ Start a new project with `cargo new` and add  `pyo3` to the `Cargo.toml` like th
 
 ```toml
 [dependencies.pyo3]
-version = "0.21.0"
+version = "0.21.2"
 features = ["auto-initialize"]
 ```
 

examples/decorator/.template/pre-script.rhai

@@ -1,4 +1,4 @@
-variable::set("PYO3_VERSION", "0.21.0");
+variable::set("PYO3_VERSION", "0.21.2");
 file::rename(".template/Cargo.toml", "Cargo.toml");
 file::rename(".template/pyproject.toml", "pyproject.toml");
 file::delete(".template");

examples/getitem/src/lib.rs

@@ -2,7 +2,6 @@
 use pyo3::exceptions::PyTypeError;
 use pyo3::prelude::*;
 use pyo3::types::PySlice;
-use std::os::raw::c_long;
 
 #[derive(FromPyObject)]
 enum IntOrSlice<'py> {
@@ -29,7 +28,7 @@ impl ExampleContainer {
         } else if let Ok(slice) = key.downcast::<PySlice>() {
             // METHOD 1 - the use PySliceIndices to help with bounds checking and for cases when only start or end are provided
             // in this case the start/stop/step all filled in to give valid values based on the max_length given
-            let index = slice.indices(self.max_length as c_long).unwrap();
+            let index = slice.indices(self.max_length as isize).unwrap();
             let _delta = index.stop - index.start;
 
             // METHOD 2 - Do the getattr manually really only needed if you have some special cases for stop/_step not being present
@@ -62,7 +61,7 @@ impl ExampleContainer {
     fn __setitem__(&self, idx: IntOrSlice, value: u32) -> PyResult<()> {
         match idx {
             IntOrSlice::Slice(slice) => {
-                let index = slice.indices(self.max_length as c_long).unwrap();
+                let index = slice.indices(self.max_length as isize).unwrap();
                 println!(
                     "Got a slice! {}-{}, step: {}, value: {}",
                     index.start, index.stop, index.step, value

examples/maturin-starter/.template/pre-script.rhai

@@ -1,4 +1,4 @@
-variable::set("PYO3_VERSION", "0.21.0");
+variable::set("PYO3_VERSION", "0.21.2");
 file::rename(".template/Cargo.toml", "Cargo.toml");
 file::rename(".template/pyproject.toml", "pyproject.toml");
 file::delete(".template");

examples/plugin/.template/pre-script.rhai

@@ -1,4 +1,4 @@
-variable::set("PYO3_VERSION", "0.21.0");
+variable::set("PYO3_VERSION", "0.21.2");
 file::rename(".template/Cargo.toml", "Cargo.toml");
 file::rename(".template/plugin_api/Cargo.toml", "plugin_api/Cargo.toml");
 file::delete(".template");

examples/setuptools-rust-starter/.template/pre-script.rhai

@@ -1,4 +1,4 @@
-variable::set("PYO3_VERSION", "0.21.0");
+variable::set("PYO3_VERSION", "0.21.2");
 file::rename(".template/Cargo.toml", "Cargo.toml");
 file::rename(".template/setup.cfg", "setup.cfg");
 file::delete(".template");

examples/word-count/.template/pre-script.rhai

@@ -1,4 +1,4 @@
-variable::set("PYO3_VERSION", "0.21.0");
+variable::set("PYO3_VERSION", "0.21.2");
 file::rename(".template/Cargo.toml", "Cargo.toml");
 file::rename(".template/pyproject.toml", "pyproject.toml");
 file::delete(".template");

guide/src/class.md

@@ -998,6 +998,44 @@ impl MyClass {
 Note that `text_signature` on `#[new]` is not compatible with compilation in
 `abi3` mode until Python 3.10 or greater.
 
+### Method receivers and lifetime elision
+
+PyO3 supports writing instance methods using the normal method receivers for shared `&self` and unique `&mut self` references. This interacts with [lifetime elision][lifetime-elision] insofar as the lifetime of a such a receiver is assigned to all elided output lifetime parameters.
+
+This is a good default for general Rust code where return values are more likely to borrow from the receiver than from the other arguments, if they contain any lifetimes at all. However, when returning bound references `Bound<'py, T>` in PyO3-based code, the GIL lifetime `'py` should usually be derived from a GIL token `py: Python<'py>` passed as an argument instead of the receiver.
+
+Specifically, signatures like
+
+```rust,ignore
+fn frobnicate(&self, py: Python) -> Bound<Foo>;
+```
+
+will not work as they are inferred as
+
+```rust,ignore
+fn frobnicate<'a, 'py>(&'a self, py: Python<'py>) -> Bound<'a, Foo>;
+```
+
+instead of the intended
+
+```rust,ignore
+fn frobnicate<'a, 'py>(&'a self, py: Python<'py>) -> Bound<'py, Foo>;
+```
+
+and should usually be written as
+
+```rust,ignore
+fn frobnicate<'py>(&self, py: Python<'py>) -> Bound<'py, Foo>;
+```
+
+The same problem does not exist for `#[pyfunction]`s as the special case for receiver lifetimes does not apply and indeed a signature like
+
+```rust,ignore
+fn frobnicate(bar: &Bar, py: Python) -> Bound<Foo>;
+```
+
+will yield compiler error [E0106 "missing lifetime specifier"][compiler-error-e0106].
+
 ## `#[pyclass]` enums
 
 Enum support in PyO3 comes in two flavors, depending on what kind of variants the enum has: simple and complex.
@@ -1329,3 +1367,6 @@ impl pyo3::impl_::pyclass::PyClassImpl for MyClass {
 [classattr]: https://docs.python.org/3/tutorial/classes.html#class-and-instance-variables
 
 [`multiple-pymethods`]: features.md#multiple-pymethods
+
+[lifetime-elision]: https://doc.rust-lang.org/reference/lifetime-elision.html
+[compiler-error-e0106]: https://doc.rust-lang.org/error_codes/E0106.html

guide/src/ecosystem/async-await.md

@@ -120,7 +120,7 @@ Export an async function that makes use of `async-std`:
 use pyo3::{prelude::*, wrap_pyfunction};
 
 #[pyfunction]
-fn rust_sleep(py: Python<'_>) -> PyResult<&Bound<'_, PyAny>>> {
+fn rust_sleep(py: Python<'_>) -> PyResult<&Bound<'_, PyAny>> {
     pyo3_asyncio::async_std::future_into_py(py, async {
         async_std::task::sleep(std::time::Duration::from_secs(1)).await;
         Ok(Python::with_gil(|py| py.None()))

guide/src/migration.md

@@ -246,14 +246,26 @@ Because the new `Bound<T>` API brings ownership out of the PyO3 framework and in
 - `Bound<PyTuple>::iter_borrowed` is slightly more efficient than `Bound<PyTuple>::iter`. The default iteration of `Bound<PyTuple>` cannot return borrowed references because Rust does not (yet) have "lending iterators". Similarly `Bound<PyTuple>::get_borrowed_item` is more efficient than `Bound<PyTuple>::get_item` for the same reason.
 - `&Bound<T>` does not implement `FromPyObject` (although it might be possible to do this in the future once the GIL Refs API is completely removed). Use `bound_any.downcast::<T>()` instead of `bound_any.extract::<&Bound<T>>()`.
 - `Bound<PyString>::to_str` now borrows from the `Bound<PyString>` rather than from the `'py` lifetime, so code will need to store the smart pointer as a value in some cases where previously `&PyString` was just used as a temporary. (There are some more details relating to this in [the section below](#deactivating-the-gil-refs-feature).)
+- `.extract::<&str>()` now borrows from the source Python object. The simplest way to update is to change to `.extract::<PyBackedStr>()`, which retains ownership of the Python reference. See more information [in the section on deactivating the `gil-refs` feature](#deactivating-the-gil-refs-feature).
 
-To convert between `&PyAny` and `&Bound<PyAny>` you can use the `as_borrowed()` method:
+To convert between `&PyAny` and `&Bound<PyAny>` use the `as_borrowed()` method:
 
 ```rust,ignore
 let gil_ref: &PyAny = ...;
 let bound: &Bound<PyAny> = &gil_ref.as_borrowed();
 ```
 
+To convert between `Py<T>` and `Bound<T>` use the `bind()` / `into_bound()` methods, and `as_unbound()` / `unbind()` to go back from `Bound<T>` to `Py<T>`.
+
+```rust,ignore
+let obj: Py<PyList> = ...;
+let bound: &Bound<'py, PyList> = obj.bind(py);
+let bound: Bound<'py, PyList> = obj.into_bound(py);
+
+let obj: &Py<PyList> = bound.as_unbound();
+let obj: Py<PyList> = bound.unbind();
+```
+
 <div class="warning">
 
 ⚠️ Warning: dangling pointer trap 💣
@@ -325,6 +337,8 @@ There is just one case of code that changes upon disabling these features: `From
 
 To make PyO3's core functionality continue to work while the GIL Refs API is in the process of being removed, disabling the `gil-refs` feature moves the implementations of `FromPyObject` for `&str`, `Cow<'_, str>`, `&[u8]`, `Cow<'_, u8>` to a new temporary trait `FromPyObjectBound`. This trait is the expected future form of `FromPyObject` and has an additional lifetime `'a` to enable these types to borrow data from Python objects.
 
+PyO3 0.21 has introduced the [`PyBackedStr`]({{#PYO3_DOCS_URL}}/pyo3/pybacked/struct.PyBackedStr.html) and [`PyBackedBytes`]({{#PYO3_DOCS_URL}}/pyo3/pybacked/struct.PyBackedBytes.html) types to help with this case. The easiest way to avoid lifetime challenges from extracting `&str` is to use these. For more complex types like `Vec<&str>`, is now impossible to extract directly from a Python object and `Vec<PyBackedStr>` is the recommended upgrade path.
+
 A key thing to note here is because extracting to these types now ties them to the input lifetime, some extremely common patterns may need to be split into multiple Rust lines. For example, the following snippet of calling `.extract::<&str>()` directly on the result of `.getattr()` needs to be adjusted when deactivating the `gil-refs-migration` feature.
 
 Before:

guide/src/types.md

@@ -160,6 +160,45 @@ for i in 0..=2 {
 # Python::with_gil(example).unwrap();
 ```
 
+### Casting between smart pointer types
+
+To convert between `Py<T>` and `Bound<'py, T>` use the `bind()` / `into_bound()` methods. Use the `as_unbound()` / `unbind()` methods to go back from `Bound<'py, T>` to `Py<T>`.
+
+```rust,ignore
+let obj: Py<PyAny> = ...;
+let bound: &Bound<'py, PyAny> = obj.bind(py);
+let bound: Bound<'py, PyAny> = obj.into_bound(py);
+
+let obj: &Py<PyAny> = bound.as_unbound();
+let obj: Py<PyAny> = bound.unbind();
+```
+
+To convert between `Bound<'py, T>` and `Borrowed<'a, 'py, T>` use the `as_borrowed()` method. `Borrowed<'a, 'py, T>` has a deref coercion to `Bound<'py, T>`. Use the `to_owned()` method to increment the Python reference count and to create a new `Bound<'py, T>` from the `Borrowed<'a, 'py, T>`.
+
+```rust,ignore
+let bound: Bound<'py, PyAny> = ...;
+let borrowed: Borrowed<'_, 'py, PyAny> = bound.as_borrowed();
+
+// deref coercion
+let bound: &Bound<'py, PyAny> = &borrowed;
+
+// create a new Bound by increase the Python reference count
+let bound: Bound<'py, PyAny> = borrowed.to_owned();
+```
+
+To convert between `Py<T>` and `Borrowed<'a, 'py, T>` use the `bind_borrowed()` method. Use either `as_unbound()` or `.to_owned().unbind()` to go back to `Py<T>` from `Borrowed<'a, 'py, T>`, via `Bound<'py, T>`.
+
+```rust,ignore
+let obj: Py<PyAny> = ...;
+let borrowed: Borrowed<'_, 'py, PyAny> = bound.as_borrowed();
+
+// via deref coercion to Bound and then using Bound::as_unbound
+let obj: &Py<PyAny> = borrowed.as_unbound();
+
+// via a new Bound by increasing the Python reference count, and unbind it
+let obj: Py<PyAny> = borrowed.to_owned().unbind().
+```
+
 ## Concrete Python types
 
 In all of `Py<T>`, `Bound<'py, T>`, and `Borrowed<'a, 'py, T>`, the type parameter `T` denotes the type of the Python object referred to by the smart pointer.

newsfragments/3761.changed.md

@@ -0,0 +1 @@
+Change the type of `PySliceIndices::slicelength` and the `length` parameter of `PySlice::indices()`.

newsfragments/3966.packaging.md

@@ -0,0 +1 @@
+Update `heck` dependency to 0.5.