Merge tag 'v1.104.0' into develop

- Fix regression when using OIDC provider. Introduced in v1.104.0rc1. ([\#17031](element-hq/synapse#17031))

- Add an OIDC config to specify extra parameters for the authorization grant URL. IT can be useful to pass an ACR value for example. ([\#16971](element-hq/synapse#16971))
- Add support for OIDC provider returning JWT. ([\#16972](element-hq/synapse#16972), [\#17031](element-hq/synapse#17031))

- Fix a bug which meant that, under certain circumstances, we might never retry sending events or to-device messages over federation after a failure. ([\#16925](element-hq/synapse#16925))
- Fix various long-standing bugs which could cause incorrect state to be returned from `/sync` in certain situations. ([\#16949](element-hq/synapse#16949))
- Fix case in which `m.fully_read` marker would not get updated. Contributed by @SpiritCroc. ([\#16990](element-hq/synapse#16990))
- Fix bug which did not retract a user's pending knocks at rooms when their account was deactivated. Contributed by @hanadi92. ([\#17010](element-hq/synapse#17010))

- Updated `start.py` to generate config using the correct user ID when running as root (fixes [\#16824](element-hq/synapse#16824), [\matrix-org#15202](element-hq/synapse#15202)). ([\#16978](element-hq/synapse#16978))

- Add a query to force a refresh of a remote user's device list to the "Useful SQL for Admins" documentation page. ([\#16892](element-hq/synapse#16892))
- Minor grammatical corrections to the upgrade documentation. ([\#16965](element-hq/synapse#16965))
- Fix the sort order for the documentation version picker, so that newer releases appear above older ones. ([\#16966](element-hq/synapse#16966))
- Remove recommendation for a specific poetry version from contributing guide. ([\#17002](element-hq/synapse#17002))

- Improve lock performance when a lot of locks are all waiting for a single lock to be released. ([\#16840](element-hq/synapse#16840))
- Update power level default for public rooms. ([\#16907](element-hq/synapse#16907))
- Improve event validation. ([\#16908](element-hq/synapse#16908))
- Multi-worker-docker-container: disable log buffering. ([\#16919](element-hq/synapse#16919))
- Refactor state delta calculation in `/sync` handler. ([\#16929](element-hq/synapse#16929))
- Clarify docs for some room state functions. ([\#16950](element-hq/synapse#16950))
- Specify IP subnets in canonical form. ([\#16953](element-hq/synapse#16953))
- As done for SAML mapping provider, let's pass the module API to the OIDC one so the mapper can do more logic in its code. ([\#16974](element-hq/synapse#16974))
- Allow containers building on top of Synapse's Complement container is use the included PostgreSQL cluster. ([\#16985](element-hq/synapse#16985))
- Raise poetry-core version cap to 1.9.0. ([\#16986](element-hq/synapse#16986))
- Patch the db conn pool sooner in tests. ([\#17017](element-hq/synapse#17017))

* Bump anyhow from 1.0.80 to 1.0.81. ([\#17009](element-hq/synapse#17009))
* Bump black from 23.10.1 to 24.2.0. ([\#16936](element-hq/synapse#16936))
* Bump cryptography from 41.0.7 to 42.0.5. ([\#16958](element-hq/synapse#16958))
* Bump dawidd6/action-download-artifact from 3.1.1 to 3.1.2. ([\#16960](element-hq/synapse#16960))
* Bump dawidd6/action-download-artifact from 3.1.2 to 3.1.4. ([\#17008](element-hq/synapse#17008))
* Bump jinja2 from 3.1.2 to 3.1.3. ([\#17005](element-hq/synapse#17005))
* Bump log from 0.4.20 to 0.4.21. ([\#16977](element-hq/synapse#16977))
* Bump mypy from 1.5.1 to 1.8.0. ([\#16901](element-hq/synapse#16901))
* Bump netaddr from 0.9.0 to 1.2.1. ([\#17006](element-hq/synapse#17006))
* Bump pydantic from 2.6.0 to 2.6.4. ([\#17004](element-hq/synapse#17004))
* Bump pyo3 from 0.20.2 to 0.20.3. ([\#16962](element-hq/synapse#16962))
* Bump ruff from 0.1.14 to 0.3.2. ([\#16994](element-hq/synapse#16994))
* Bump serde from 1.0.196 to 1.0.197. ([\#16963](element-hq/synapse#16963))
* Bump serde_json from 1.0.113 to 1.0.114. ([\#16961](element-hq/synapse#16961))
* Bump types-jsonschema from 4.21.0.20240118 to 4.21.0.20240311. ([\#17007](element-hq/synapse#17007))
* Bump types-psycopg2 from 2.9.21.16 to 2.9.21.20240311. ([\#16995](element-hq/synapse#16995))
* Bump types-pyopenssl from 23.3.0.0 to 24.0.0.20240311. ([\#17003](element-hq/synapse#17003))

CHANGES.md

@@ -1,3 +1,145 @@
+# Synapse 1.104.0 (2024-04-02)
+
+### Bugfixes
+
+- Fix regression when using OIDC provider. Introduced in v1.104.0rc1. ([\#17031](https://github.com/element-hq/synapse/issues/17031))
+
+
+# Synapse 1.104.0rc1 (2024-03-26)
+
+### Features
+
+- Add an OIDC config to specify extra parameters for the authorization grant URL. IT can be useful to pass an ACR value for example. ([\#16971](https://github.com/element-hq/synapse/issues/16971))
+- Add support for OIDC provider returning JWT. ([\#16972](https://github.com/element-hq/synapse/issues/16972), [\#17031](https://github.com/element-hq/synapse/issues/17031))
+
+### Bugfixes
+
+- Fix a bug which meant that, under certain circumstances, we might never retry sending events or to-device messages over federation after a failure. ([\#16925](https://github.com/element-hq/synapse/issues/16925))
+- Fix various long-standing bugs which could cause incorrect state to be returned from `/sync` in certain situations. ([\#16949](https://github.com/element-hq/synapse/issues/16949))
+- Fix case in which `m.fully_read` marker would not get updated. Contributed by @SpiritCroc. ([\#16990](https://github.com/element-hq/synapse/issues/16990))
+- Fix bug which did not retract a user's pending knocks at rooms when their account was deactivated. Contributed by @hanadi92. ([\#17010](https://github.com/element-hq/synapse/issues/17010))
+
+### Updates to the Docker image
+
+- Updated `start.py` to generate config using the correct user ID when running as root (fixes [\#16824](https://github.com/element-hq/synapse/issues/16824), [\#15202](https://github.com/element-hq/synapse/issues/15202)). ([\#16978](https://github.com/element-hq/synapse/issues/16978))
+
+### Improved Documentation
+
+- Add a query to force a refresh of a remote user's device list to the "Useful SQL for Admins" documentation page. ([\#16892](https://github.com/element-hq/synapse/issues/16892))
+- Minor grammatical corrections to the upgrade documentation. ([\#16965](https://github.com/element-hq/synapse/issues/16965))
+- Fix the sort order for the documentation version picker, so that newer releases appear above older ones. ([\#16966](https://github.com/element-hq/synapse/issues/16966))
+- Remove recommendation for a specific poetry version from contributing guide. ([\#17002](https://github.com/element-hq/synapse/issues/17002))
+
+### Internal Changes
+
+- Improve lock performance when a lot of locks are all waiting for a single lock to be released. ([\#16840](https://github.com/element-hq/synapse/issues/16840))
+- Update power level default for public rooms. ([\#16907](https://github.com/element-hq/synapse/issues/16907))
+- Improve event validation. ([\#16908](https://github.com/element-hq/synapse/issues/16908))
+- Multi-worker-docker-container: disable log buffering. ([\#16919](https://github.com/element-hq/synapse/issues/16919))
+- Refactor state delta calculation in `/sync` handler. ([\#16929](https://github.com/element-hq/synapse/issues/16929))
+- Clarify docs for some room state functions. ([\#16950](https://github.com/element-hq/synapse/issues/16950))
+- Specify IP subnets in canonical form. ([\#16953](https://github.com/element-hq/synapse/issues/16953))
+- As done for SAML mapping provider, let's pass the module API to the OIDC one so the mapper can do more logic in its code. ([\#16974](https://github.com/element-hq/synapse/issues/16974))
+- Allow containers building on top of Synapse's Complement container is use the included PostgreSQL cluster. ([\#16985](https://github.com/element-hq/synapse/issues/16985))
+- Raise poetry-core version cap to 1.9.0. ([\#16986](https://github.com/element-hq/synapse/issues/16986))
+- Patch the db conn pool sooner in tests. ([\#17017](https://github.com/element-hq/synapse/issues/17017))
+
+
+
+### Updates to locked dependencies
+
+* Bump anyhow from 1.0.80 to 1.0.81. ([\#17009](https://github.com/element-hq/synapse/issues/17009))
+* Bump black from 23.10.1 to 24.2.0. ([\#16936](https://github.com/element-hq/synapse/issues/16936))
+* Bump cryptography from 41.0.7 to 42.0.5. ([\#16958](https://github.com/element-hq/synapse/issues/16958))
+* Bump dawidd6/action-download-artifact from 3.1.1 to 3.1.2. ([\#16960](https://github.com/element-hq/synapse/issues/16960))
+* Bump dawidd6/action-download-artifact from 3.1.2 to 3.1.4. ([\#17008](https://github.com/element-hq/synapse/issues/17008))
+* Bump jinja2 from 3.1.2 to 3.1.3. ([\#17005](https://github.com/element-hq/synapse/issues/17005))
+* Bump log from 0.4.20 to 0.4.21. ([\#16977](https://github.com/element-hq/synapse/issues/16977))
+* Bump mypy from 1.5.1 to 1.8.0. ([\#16901](https://github.com/element-hq/synapse/issues/16901))
+* Bump netaddr from 0.9.0 to 1.2.1. ([\#17006](https://github.com/element-hq/synapse/issues/17006))
+* Bump pydantic from 2.6.0 to 2.6.4. ([\#17004](https://github.com/element-hq/synapse/issues/17004))
+* Bump pyo3 from 0.20.2 to 0.20.3. ([\#16962](https://github.com/element-hq/synapse/issues/16962))
+* Bump ruff from 0.1.14 to 0.3.2. ([\#16994](https://github.com/element-hq/synapse/issues/16994))
+* Bump serde from 1.0.196 to 1.0.197. ([\#16963](https://github.com/element-hq/synapse/issues/16963))
+* Bump serde_json from 1.0.113 to 1.0.114. ([\#16961](https://github.com/element-hq/synapse/issues/16961))
+* Bump types-jsonschema from 4.21.0.20240118 to 4.21.0.20240311. ([\#17007](https://github.com/element-hq/synapse/issues/17007))
+* Bump types-psycopg2 from 2.9.21.16 to 2.9.21.20240311. ([\#16995](https://github.com/element-hq/synapse/issues/16995))
+* Bump types-pyopenssl from 23.3.0.0 to 24.0.0.20240311. ([\#17003](https://github.com/element-hq/synapse/issues/17003))
+
+# Synapse 1.103.0 (2024-03-19)
+
+No significant changes since 1.103.0rc1.
+
+
+
+
+# Synapse 1.103.0rc1 (2024-03-12)
+
+### Features
+
+- Add a new [List Accounts v3](https://element-hq.github.io/synapse/v1.103/admin_api/user_admin_api.html#list-accounts-v3) Admin API with improved deactivated user filtering capabilities. ([\#16874](https://github.com/element-hq/synapse/issues/16874))
+- Include `Retry-After` header by default per [MSC4041](https://github.com/matrix-org/matrix-spec-proposals/pull/4041). Contributed by @clokep. ([\#16947](https://github.com/element-hq/synapse/issues/16947))
+
+### Bugfixes
+
+- Fix joining remote rooms when a module uses the `on_new_event` callback. This callback may now pass partial state events instead of the full state for remote rooms. Introduced in v1.76.0. ([\#16973](https://github.com/element-hq/synapse/issues/16973))
+- Fix performance issue when joining very large rooms that can cause the server to lock up. Introduced in v1.100.0. Contributed by @ggogel. ([\#16968](https://github.com/element-hq/synapse/issues/16968))
+
+### Improved Documentation
+
+- Add HAProxy example for single port operation to reverse proxy documentation. Contributed by Georg Pfuetzenreuter (@tacerus). ([\#16768](https://github.com/element-hq/synapse/issues/16768))
+- Improve the documentation around running Complement tests with new configuration parameters. ([\#16946](https://github.com/element-hq/synapse/issues/16946))
+- Add docs on upgrading from a very old version. ([\#16951](https://github.com/element-hq/synapse/issues/16951))
+
+
+### Updates to locked dependencies
+
+* Bump JasonEtco/create-an-issue from 2.9.1 to 2.9.2. ([\#16934](https://github.com/element-hq/synapse/issues/16934))
+* Bump anyhow from 1.0.79 to 1.0.80. ([\#16935](https://github.com/element-hq/synapse/issues/16935))
+* Bump dawidd6/action-download-artifact from 3.0.0 to 3.1.1. ([\#16933](https://github.com/element-hq/synapse/issues/16933))
+* Bump furo from 2023.9.10 to 2024.1.29. ([\#16939](https://github.com/element-hq/synapse/issues/16939))
+* Bump pyopenssl from 23.3.0 to 24.0.0. ([\#16937](https://github.com/element-hq/synapse/issues/16937))
+* Bump types-netaddr from 0.10.0.20240106 to 1.2.0.20240219. ([\#16938](https://github.com/element-hq/synapse/issues/16938))
+
+
+# Synapse 1.102.0 (2024-03-05)
+
+### Bugfixes
+
+- Revert https://github.com/element-hq/synapse/pull/16756, which caused incorrect notification counts on mobile clients since v1.100.0. ([\#16979](https://github.com/element-hq/synapse/issues/16979))
+
+
+# Synapse 1.102.0rc1 (2024-02-20)
+
+### Features
+
+- A metric was added for emails sent by Synapse, broken down by type: `synapse_emails_sent_total`. Contributed by Remi Rampin. ([\#16881](https://github.com/element-hq/synapse/issues/16881))
+
+### Bugfixes
+
+- Do not send multiple concurrent requests for keys for the same server. ([\#16894](https://github.com/element-hq/synapse/issues/16894))
+- Fix performance issue when joining very large rooms that can cause the server to lock up. Introduced in v1.100.0. ([\#16903](https://github.com/element-hq/synapse/issues/16903))
+- Always prefer unthreaded receipt when >1 exist ([MSC4102](https://github.com/matrix-org/matrix-spec-proposals/pull/4102)). ([\#16927](https://github.com/element-hq/synapse/issues/16927))
+
+### Improved Documentation
+
+- Fix a small typo in the Rooms section of the Admin API documentation. Contributed by @RainerZufall187. ([\#16857](https://github.com/element-hq/synapse/issues/16857))
+
+### Internal Changes
+
+- Don't invalidate the entire event cache when we purge history. ([\#16905](https://github.com/element-hq/synapse/issues/16905))
+- Add experimental config option to not send device list updates for specific users. ([\#16909](https://github.com/element-hq/synapse/issues/16909))
+- Fix incorrect docker hub link in release script. ([\#16910](https://github.com/element-hq/synapse/issues/16910))
+
+
+
+### Updates to locked dependencies
+
+* Bump attrs from 23.1.0 to 23.2.0. ([\#16899](https://github.com/element-hq/synapse/issues/16899))
+* Bump bcrypt from 4.0.1 to 4.1.2. ([\#16900](https://github.com/element-hq/synapse/issues/16900))
+* Bump pygithub from 2.1.1 to 2.2.0. ([\#16902](https://github.com/element-hq/synapse/issues/16902))
+* Bump sentry-sdk from 1.40.0 to 1.40.3. ([\#16898](https://github.com/element-hq/synapse/issues/16898))
+
 # Synapse 1.101.0 (2024-02-13)
 
 ### Bugfixes

debian/changelog

@@ -1,3 +1,39 @@
+matrix-synapse-py3 (1.104.0) stable; urgency=medium
+
+  * New Synapse release 1.104.0.
+
+ -- Synapse Packaging team <[email protected]>  Tue, 02 Apr 2024 17:15:45 +0100
+
+matrix-synapse-py3 (1.104.0~rc1) stable; urgency=medium
+
+  * New Synapse release 1.104.0rc1.
+
+ -- Synapse Packaging team <[email protected]>  Tue, 26 Mar 2024 11:48:58 +0000
+
+matrix-synapse-py3 (1.103.0) stable; urgency=medium
+
+  * New Synapse release 1.103.0.
+
+ -- Synapse Packaging team <[email protected]>  Tue, 19 Mar 2024 12:24:36 +0000
+
+matrix-synapse-py3 (1.103.0~rc1) stable; urgency=medium
+
+  * New Synapse release 1.103.0rc1.
+
+ -- Synapse Packaging team <[email protected]>  Tue, 12 Mar 2024 15:02:56 +0000
+
+matrix-synapse-py3 (1.102.0) stable; urgency=medium
+
+  * New Synapse release 1.102.0.
+
+ -- Synapse Packaging team <[email protected]>  Tue, 05 Mar 2024 14:47:03 +0000
+
+matrix-synapse-py3 (1.102.0~rc1) stable; urgency=medium
+
+  * New Synapse release 1.102.0rc1.
+
+ -- Synapse Packaging team <[email protected]>  Tue, 20 Feb 2024 15:50:36 +0000
+
 matrix-synapse-py3 (1.101.0) stable; urgency=medium
 
   * New Synapse release 1.101.0.

docker/complement/README.md

@@ -30,3 +30,14 @@ Consult `scripts-dev/complement.sh` in the repository root for a real example.
 
 [complement]: https://github.com/matrix-org/complement
 [complementEnv]: https://github.com/matrix-org/complement/pull/382
+
+## How to modify homeserver.yaml for Complement tests
+
+It's common for MSCs to be gated behind a feature flag like this:
+```yaml
+experimental_features:
+  faster_joins: true
+```
+To modify this for the Complement image, modify `./conf/workers-shared-extra.yaml.j2`. Despite the name,
+this will affect non-worker mode as well. Remember to _rebuild_ the image (so don't use `-e` if using
+`complement.sh`).

docker/complement/conf/postgres.supervisord.conf

@@ -1,7 +1,7 @@
 [program:postgres]
 command=/usr/local/bin/prefix-log gosu postgres postgres
 
-# Only start if START_POSTGRES=1
+# Only start if START_POSTGRES=true
 autostart=%(ENV_START_POSTGRES)s
 
 # Lower priority number = starts first

docker/complement/conf/start_for_complement.sh

@@ -32,8 +32,9 @@ case "$SYNAPSE_COMPLEMENT_DATABASE" in
     ;;
 
   sqlite|"")
-    # Configure supervisord not to start Postgres, as we don't need it
-    export START_POSTGRES=false
+    # Set START_POSTGRES to false unless it has already been set
+    # (i.e. by another container image inheriting our own).
+    export START_POSTGRES=${START_POSTGRES:-false}
     ;;
 
   *)

docker/prefix-log

@@ -7,6 +7,9 @@
 #   prefix-log command [args...]
 #
 
-exec 1> >(awk '{print "'"${SUPERVISOR_PROCESS_NAME}"' | "$0}' >&1)
-exec 2> >(awk '{print "'"${SUPERVISOR_PROCESS_NAME}"' | "$0}' >&2)
+# '-W interactive' is a `mawk` extension which disables buffering on stdout and sets line-buffered reads on
+# stdin. The effect is that the output is flushed after each line, rather than being batched, which helps reduce
+# confusion due to to interleaving of the different processes.
+exec 1> >(awk -W interactive '{print "'"${SUPERVISOR_PROCESS_NAME}"' | "$0 }' >&1)
+exec 2> >(awk -W interactive '{print "'"${SUPERVISOR_PROCESS_NAME}"' | "$0 }' >&2)
 exec "$@"