Merge tag 'v1.51.0'

Synapse 1.51.0 (2022-01-25)
===========================

No significant changes since 1.51.0rc2.

Synapse 1.51.0 deprecates `webclient` listeners and non-HTTP(S) `web_client_location`s. Support for these will be removed in Synapse 1.53.0, at which point Synapse will not be capable of directly serving a web client for Matrix.

Synapse 1.51.0rc2 (2022-01-24)
==============================

Bugfixes
--------

- Fix a bug introduced in Synapse 1.40.0 that caused Synapse to fail to process incoming federation traffic after handling a large amount of events in a v1 room. ([\#11806](#11806))

Synapse 1.51.0rc1 (2022-01-21)
==============================

Features
--------

- Add `track_puppeted_user_ips` config flag to record client IP addresses against puppeted users, and include the puppeted users in monthly active user counts. ([\#11561](#11561), [\#11749](#11749), [\#11757](#11757))
- Include whether the requesting user has participated in a thread when generating a summary for [MSC3440](matrix-org/matrix-spec-proposals#3440). ([\#11577](#11577))
- Return an `M_FORBIDDEN` error code instead of `M_UNKNOWN` when a spam checker module prevents a user from creating a room. ([\#11672](#11672))
- Add a flag to the `synapse_review_recent_signups` script to ignore and filter appservice users. ([\#11675](#11675), [\#11770](#11770))

Bugfixes
--------

- Fix a long-standing issue which could cause Synapse to incorrectly accept data in the unsigned field of events
  received over federation. ([\#11530](#11530))
- Fix a long-standing bug where Synapse wouldn't cache a response indicating that a remote user has no devices. ([\#11587](#11587))
- Fix an error that occurs whilst trying to get the federation status of a destination server that was working normally. This admin API was newly introduced in Synapse v1.49.0. ([\#11593](#11593))
- Fix bundled aggregations not being included in the `/sync` response, per [MSC2675](matrix-org/matrix-spec-proposals#2675). ([\#11612](#11612), [\#11659](#11659), [\#11791](#11791))
- Fix the `/_matrix/client/v1/room/{roomId}/hierarchy` endpoint returning incorrect fields which have been present since Synapse 1.49.0. ([\#11667](#11667))
- Fix preview of some GIF URLs (like tenor.com). Contributed by Philippe Daouadi. ([\#11669](#11669))
- Fix a bug where only the first 50 rooms from a space were returned from the `/hierarchy` API. This has existed since the introduction of the API in Synapse v1.41.0. ([\#11695](#11695))
- Fix a bug introduced in Synapse v1.18.0 where password reset and address validation emails would not be sent if their subject was configured to use the 'app' template variable. Contributed by @br4nnigan. ([\#11710](#11710), [\#11745](#11745))
- Make the 'List Rooms' Admin API sort stable. Contributed by Daniël Sonck. ([\#11737](#11737))
- Fix a long-standing bug where space hierarchy over federation would only work correctly some of the time. ([\#11775](#11775))
- Fix a bug introduced in Synapse v1.46.0 that prevented `on_logged_out` module callbacks from being correctly awaited by Synapse. ([\#11786](#11786))

Improved Documentation
----------------------

- Warn against using a Let's Encrypt certificate for TLS/DTLS TURN server client connections, and suggest using ZeroSSL certificate instead. This works around client-side connectivity errors caused by WebRTC libraries that reject Let's Encrypt certificates. Contibuted by @AndrewFerr. ([\#11686](#11686))
- Document the new `SYNAPSE_TEST_PERSIST_SQLITE_DB` environment variable in the contributing guide. ([\#11715](#11715))
- Document that the minimum supported PostgreSQL version is now 10. ([\#11725](#11725))
- Fix typo in demo docs: differnt. ([\#11735](#11735))
- Update room spec URL in config files. ([\#11739](#11739))
- Mention `python3-venv` and `libpq-dev` dependencies in the contribution guide. ([\#11740](#11740))
- Update documentation for configuring login with Facebook. ([\#11755](#11755))
- Update installation instructions to note that Python 3.6 is no longer supported. ([\#11781](#11781))

Deprecations and Removals
-------------------------

- Remove the unstable `/send_relation` endpoint. ([\#11682](#11682))
- Remove `python_twisted_reactor_pending_calls` Prometheus metric. ([\#11724](#11724))
- Remove the `password_hash` field from the response dictionaries of the [Users Admin API](https://matrix-org.github.io/synapse/latest/admin_api/user_admin_api.html). ([\#11576](#11576))
- **Deprecate support for `webclient` listeners and non-HTTP(S) `web_client_location` configuration. ([\#11774](#11774), [\#11783](#11783

Internal Changes
----------------

- Run `pyupgrade --py37-plus --keep-percent-format` on Synapse. ([\#11685](#11685))
- Use buildkit's cache feature to speed up docker builds. ([\#11691](#11691))
- Use `auto_attribs` and native type hints for attrs classes. ([\#11692](#11692), [\#11768](#11768))
- Remove debug logging for #4422, which has been closed since Synapse 0.99. ([\#11693](#11693))
- Remove fallback code for Python 2. ([\#11699](#11699))
- Add a test for [an edge case](#11532 (comment)) in the `/sync` logic. ([\#11701](#11701))
- Add the option to write SQLite test dbs to disk when running tests. ([\#11702](#11702))
- Improve Complement test output for Gitub Actions. ([\#11707](#11707))
- Fix docstring on `add_account_data_for_user`. ([\#11716](#11716))
- Complement environment variable name change and update `.gitignore`. ([\#11718](#11718))
- Simplify calculation of Prometheus metrics for garbage collection. ([\#11723](#11723))
- Improve accuracy of `python_twisted_reactor_tick_time` Prometheus metric. ([\#11724](#11724), [\#11771](#11771))
- Minor efficiency improvements when inserting many values into the database. ([\#11742](#11742))
- Invite PR authors to give themselves credit in the changelog. ([\#11744](#11744))
- Add optional debugging to investigate [issue 8631](#8631). ([\#11760](#11760))
- Remove `log_function` utility function and its uses. ([\#11761](#11761))
- Add a unit test that checks both `client` and `webclient` resources will function when simultaneously enabled. ([\#11765](#11765))
- Allow overriding complement commit using `COMPLEMENT_REF`. ([\#11766](#11766))
- Add some comments and type annotations for `_update_outliers_txn`. ([\#11776](#11776))

Author: David Robertson

.github/PULL_REQUEST_TEMPLATE.md

@@ -8,6 +8,7 @@
   - Use markdown where necessary, mostly for `code blocks`.
   - End with either a period (.) or an exclamation mark (!).
   - Start with a capital letter.
+  - Feel free to credit yourself, by adding a sentence "Contributed by @github_username." or "Contributed by [Your Name]." to the end of the entry.
 * [ ] Pull request includes a [sign off](https://matrix-org.github.io/synapse/latest/development/contributing_guide.html#sign-off)
 * [ ] [Code style](https://matrix-org.github.io/synapse/latest/code_style.html) is correct
   (run the [linters](https://matrix-org.github.io/synapse/latest/development/contributing_guide.html#run-the-linters))

.github/workflows/tests.yml

@@ -366,6 +366,8 @@ jobs:
       # Build initial Synapse image
       - run: docker build -t matrixdotorg/synapse:latest -f docker/Dockerfile .
         working-directory: synapse
+        env:
+          DOCKER_BUILDKIT: 1
 
       # Build a ready-to-run Synapse image based on the initial image above.
       # This new image includes a config file, keys for signing and TLS, and
@@ -374,7 +376,8 @@ jobs:
         working-directory: complement/dockerfiles
 
       # Run Complement
-      - run: go test -v -tags synapse_blacklist,msc2403 ./tests/...
+      - run: set -o pipefail && go test -v -json -tags synapse_blacklist,msc2403 ./tests/... 2>&1 | gotestfmt
+        shell: bash
         env:
           COMPLEMENT_BASE_IMAGE: complement-synapse:latest
         working-directory: complement