Merge tag 'v1.53.0' into babolivier/dinsic_1.58

Synapse 1.53.0 (2022-02-22)
===========================

No significant changes.

Synapse 1.53.0rc1 (2022-02-15)
==============================

Features
--------

- Add experimental support for sending to-device messages to application services, as specified by [MSC2409](matrix-org/matrix-spec-proposals#2409). ([\#11215](matrix-org/synapse#11215), [\#11966](matrix-org/synapse#11966))
- Remove account data (including client config, push rules and ignored users) upon user deactivation. ([\#11655](matrix-org/synapse#11655))
- Experimental support for [MSC3666](matrix-org/matrix-spec-proposals#3666): including bundled aggregations in server side search results. ([\#11837](matrix-org/synapse#11837))
- Enable cache time-based expiry by default. The `expiry_time` config flag has been superseded by `expire_caches` and `cache_entry_ttl`. ([\#11849](matrix-org/synapse#11849))
- Add a callback to allow modules to allow or forbid a 3PID (email address, phone number) from being associated to a local account. ([\#11854](matrix-org/synapse#11854))
- Stabilize support and remove unstable endpoints for [MSC3231](matrix-org/matrix-spec-proposals#3231). Clients must switch to the stable identifier and endpoint. See the [upgrade notes](https://matrix-org.github.io/synapse/develop/upgrade#stablisation-of-msc3231) for more information. ([\#11867](matrix-org/synapse#11867))
- Allow modules to retrieve the current instance's server name and worker name. ([\#11868](matrix-org/synapse#11868))
- Use a dedicated configurable rate limiter for 3PID invites. ([\#11892](matrix-org/synapse#11892))
- Support the stable API endpoint for [MSC3283](matrix-org/matrix-spec-proposals#3283): new settings in `/capabilities` endpoint. ([\#11933](matrix-org/synapse#11933), [\#11989](matrix-org/synapse#11989))
- Support the `dir` parameter on the `/relations` endpoint, per [MSC3715](matrix-org/matrix-spec-proposals#3715). ([\#11941](matrix-org/synapse#11941))
- Experimental implementation of [MSC3706](matrix-org/matrix-spec-proposals#3706): extensions to `/send_join` to support reduced response size. ([\#11967](matrix-org/synapse#11967))

Bugfixes
--------

- Fix [MSC2716](matrix-org/matrix-spec-proposals#2716) historical messages backfilling in random order on remote homeservers. ([\#11114](matrix-org/synapse#11114))
- Fix a bug introduced in Synapse 1.51.0 where incoming federation transactions containing at least one EDU would be dropped if debug logging was enabled for `synapse.8631_debug`. ([\#11890](matrix-org/synapse#11890))
- Fix a long-standing bug where some unknown endpoints would return HTML error pages instead of JSON `M_UNRECOGNIZED` errors. ([\#11930](matrix-org/synapse#11930))
- Implement an allow list of content types for which we will attempt to preview a URL. This prevents Synapse from making useless longer-lived connections to streaming media servers. ([\#11936](matrix-org/synapse#11936))
- Fix a long-standing bug where pagination tokens from `/sync` and `/messages` could not be provided to the `/relations` API. ([\#11952](matrix-org/synapse#11952))
- Require that modules register their callbacks using keyword arguments. ([\#11975](matrix-org/synapse#11975))
- Fix a long-standing bug where `M_WRONG_ROOM_KEYS_VERSION` errors would not include the specced `current_version` field. ([\#11988](matrix-org/synapse#11988))

Improved Documentation
----------------------

- Fix typo in User Admin API: unpind -> unbind. ([\#11859](matrix-org/synapse#11859))
- Document images returned by the User List Media Admin API can include those generated by URL previews. ([\#11862](matrix-org/synapse#11862))
- Remove outdated MSC1711 FAQ document. ([\#11907](matrix-org/synapse#11907))
- Correct the structured logging configuration example. Contributed by Brad Jones. ([\#11946](matrix-org/synapse#11946))
- Add information on the Synapse release cycle. ([\#11954](matrix-org/synapse#11954))
- Fix broken link in the README to the admin API for password reset. ([\#11955](matrix-org/synapse#11955))

Deprecations and Removals
-------------------------

- Drop support for `webclient` listeners and configuring `web_client_location` to a non-HTTP(S) URL. Deprecated configurations are a configuration error. ([\#11895](matrix-org/synapse#11895))
- Remove deprecated `user_may_create_room_with_invites` spam checker callback. See the [upgrade notes](https://matrix-org.github.io/synapse/latest/upgrade.html#removal-of-user_may_create_room_with_invites) for more information. ([\#11950](matrix-org/synapse#11950))
- No longer build `.deb` packages for Ubuntu 21.04 Hirsute Hippo, which has now EOLed. ([\#11961](matrix-org/synapse#11961))

Internal Changes
----------------

- Enhance user registration test helpers to make them more useful for tests involving application services and devices. ([\#11615](matrix-org/synapse#11615), [\#11616](matrix-org/synapse#11616))
- Improve performance when fetching bundled aggregations for multiple events. ([\#11660](matrix-org/synapse#11660), [\#11752](matrix-org/synapse#11752))
- Fix type errors introduced by new annotations in the Prometheus Client library. ([\#11832](matrix-org/synapse#11832))
- Add missing type hints to replication code. ([\#11856](matrix-org/synapse#11856), [\#11938](matrix-org/synapse#11938))
- Ensure that `opentracing` scopes are activated and closed at the right time. ([\#11869](matrix-org/synapse#11869))
- Improve opentracing for incoming federation requests. ([\#11870](matrix-org/synapse#11870))
- Improve internal docstrings in `synapse.util.caches`. ([\#11876](matrix-org/synapse#11876))
- Do not needlessly clear the `get_users_in_room` and `get_users_in_room_with_profiles` caches when any room state changes. ([\#11878](matrix-org/synapse#11878))
- Convert `ApplicationServiceTestCase` to use `simple_async_mock`. ([\#11880](matrix-org/synapse#11880))
- Remove experimental changes to the default push rules which were introduced in Synapse 1.19.0 but never enabled. ([\#11884](matrix-org/synapse#11884))
- Disable coverage calculation for olddeps build. ([\#11888](matrix-org/synapse#11888))
- Preparation to support sending device list updates to application services. ([\#11905](matrix-org/synapse#11905))
- Add a test that checks users receive their own device list updates down `/sync`. ([\#11909](matrix-org/synapse#11909))
- Run Complement tests sequentially. ([\#11910](matrix-org/synapse#11910))
- Various refactors to the application service notifier code. ([\#11911](matrix-org/synapse#11911), [\#11912](matrix-org/synapse#11912))
- Tests: replace mocked `Authenticator` with the real thing. ([\#11913](matrix-org/synapse#11913))
- Various refactors to the typing notifications code. ([\#11914](matrix-org/synapse#11914))
- Use the proper type for the `Content-Length` header in the `UploadResource`. ([\#11927](matrix-org/synapse#11927))
- Remove an unnecessary ignoring of type hints due to fixes in upstream packages. ([\#11939](matrix-org/synapse#11939))
- Add missing type hints. ([\#11953](matrix-org/synapse#11953))
- Fix an import cycle in `synapse.event_auth`. ([\#11965](matrix-org/synapse#11965))
- Unpin `frozendict` but exclude the known bad version 2.1.2. ([\#11969](matrix-org/synapse#11969))
- Prepare for rename of default Complement branch. ([\#11971](matrix-org/synapse#11971))
- Fetch Synapse's version using a helper from `matrix-common`. ([\#11979](matrix-org/synapse#11979))

Author: babolivier

.ci/scripts/test_old_deps.sh

@@ -15,4 +15,4 @@ export LANG="C.UTF-8"
 # Prevent virtualenv from auto-updating pip to an incompatible version
 export VIRTUALENV_NO_DOWNLOAD=1
 
-exec tox -e py3-old,combine
+exec tox -e py3-old

.github/workflows/tests.yml

@@ -338,7 +338,7 @@ jobs:
           path: synapse
 
       # Attempt to check out the same branch of Complement as the PR. If it
-      # doesn't exist, fallback to master.
+      # doesn't exist, fallback to HEAD.
       - name: Checkout complement
         shell: bash
         run: |
@@ -351,8 +351,8 @@ jobs:
           #    for pull requests, otherwise GITHUB_REF).
           # 2. Attempt to use the base branch, e.g. when merging into release-vX.Y
           #    (GITHUB_BASE_REF for pull requests).
-          # 3. Use the default complement branch ("master").
-          for BRANCH_NAME in "$GITHUB_HEAD_REF" "$GITHUB_BASE_REF" "${GITHUB_REF#refs/heads/}" "master"; do
+          # 3. Use the default complement branch ("HEAD").
+          for BRANCH_NAME in "$GITHUB_HEAD_REF" "$GITHUB_BASE_REF" "${GITHUB_REF#refs/heads/}" "HEAD"; do
             # Skip empty branch names and merge commits.
             if [[ -z "$BRANCH_NAME" || $BRANCH_NAME =~ ^refs/pull/.* ]]; then
               continue
@@ -376,7 +376,7 @@ jobs:
       # Run Complement
       - run: |
           set -o pipefail
-          go test -v -json -tags synapse_blacklist,msc2403 ./tests/... 2>&1 | gotestfmt
+          go test -v -json -p 1 -tags synapse_blacklist,msc2403 ./tests/... 2>&1 | gotestfmt
         shell: bash
         name: Run Complement Tests
         env: