lint, but mypy sad

will sytest spot bugs?

.github/PULL_REQUEST_TEMPLATE.md

@@ -1,13 +1,12 @@
 ### Pull Request Checklist
 
-<!-- Please read https://matrix-org.github.io/synapse/latest/development/contributing_guide.html before submitting your pull request -->
+<!-- Please read CONTRIBUTING.md before submitting your pull request -->
 
 * [ ] Pull request is based on the develop branch
-* [ ] Pull request includes a [changelog file](https://matrix-org.github.io/synapse/latest/development/contributing_guide.html#changelog). The entry should:
+* [ ] Pull request includes a [changelog file](https://github.com/matrix-org/synapse/blob/master/CONTRIBUTING.md#changelog). The entry should:
   - Be a short description of your change which makes sense to users. "Fixed a bug that prevented receiving messages from other servers." instead of "Moved X method from `EventStore` to `EventWorkerStore`.".
   - Use markdown where necessary, mostly for `code blocks`.
   - End with either a period (.) or an exclamation mark (!).
   - Start with a capital letter.
-* [ ] Pull request includes a [sign off](https://matrix-org.github.io/synapse/latest/development/contributing_guide.html#sign-off)
-* [ ] [Code style](https://matrix-org.github.io/synapse/latest/code_style.html) is correct
-  (run the [linters](https://matrix-org.github.io/synapse/latest/development/contributing_guide.html#run-the-linters))
+* [ ] Pull request includes a [sign off](https://github.com/matrix-org/synapse/blob/master/CONTRIBUTING.md#sign-off)
+* [ ] Code style is correct (run the [linters](https://github.com/matrix-org/synapse/blob/master/CONTRIBUTING.md#code-style))

.github/workflows/docker.yml

@@ -5,7 +5,7 @@ name: Build docker images
 on:
   push:
     tags: ["v*"]
-    branches: [ master, main, develop ]
+    branches: [ master, main ]
   workflow_dispatch:
 
 permissions:
@@ -38,9 +38,6 @@ jobs:
         id: set-tag
         run: |
           case "${GITHUB_REF}" in
-              refs/heads/develop)
-                  tag=develop
-                  ;;
               refs/heads/master|refs/heads/main)
                   tag=latest
                   ;;

CHANGES.md

@@ -1,238 +1,8 @@
-Synapse 1.48.0 (2021-11-30)
-===========================
-
-This release removes support for the long-deprecated `trust_identity_server_for_password_resets` configuration flag.
-
-This release also fixes some performance issues with some background database updates introduced in Synapse 1.47.0.
-
-No significant changes since 1.48.0rc1.
-
-Synapse 1.48.0rc1 (2021-11-25)
-==============================
-
-Features
---------
-
-- Experimental support for the thread relation defined in [MSC3440](https://github.com/matrix-org/matrix-doc/pull/3440). ([\#11161](https://github.com/matrix-org/synapse/issues/11161))
-- Support filtering by relation senders & types per [MSC3440](https://github.com/matrix-org/matrix-doc/pull/3440). ([\#11236](https://github.com/matrix-org/synapse/issues/11236))
-- Add support for the `/_matrix/client/v3` and `/_matrix/media/v3` APIs from Matrix v1.1. ([\#11318](https://github.com/matrix-org/synapse/issues/11318), [\#11371](https://github.com/matrix-org/synapse/issues/11371))
-- Support the stable version of [MSC2778](https://github.com/matrix-org/matrix-doc/pull/2778): the `m.login.application_service` login type. Contributed by @tulir. ([\#11335](https://github.com/matrix-org/synapse/issues/11335))
-- Add a new version of delete room admin API `DELETE /_synapse/admin/v2/rooms/<room_id>` to run it in the background. Contributed by @dklimpel. ([\#11223](https://github.com/matrix-org/synapse/issues/11223))
-- Allow the admin [Delete Room API](https://matrix-org.github.io/synapse/latest/admin_api/rooms.html#delete-room-api) to block a room without the need to join it. ([\#11228](https://github.com/matrix-org/synapse/issues/11228))
-- Add an admin API to un-shadow-ban a user. ([\#11347](https://github.com/matrix-org/synapse/issues/11347))
-- Add an admin API to run background database schema updates. ([\#11352](https://github.com/matrix-org/synapse/issues/11352))
-- Add an admin API for blocking a room. ([\#11324](https://github.com/matrix-org/synapse/issues/11324))
-- Update the JWT login type to support custom a `sub` claim. ([\#11361](https://github.com/matrix-org/synapse/issues/11361))
-- Store and allow querying of arbitrary event relations. ([\#11391](https://github.com/matrix-org/synapse/issues/11391))
-
-
-Bugfixes
---------
-
-- Fix a long-standing bug wherein display names or avatar URLs containing null bytes cause an internal server error when stored in the DB. ([\#11230](https://github.com/matrix-org/synapse/issues/11230))
-- Prevent [MSC2716](https://github.com/matrix-org/matrix-doc/pull/2716) historical state events from being pushed to an application service via `/transactions`. ([\#11265](https://github.com/matrix-org/synapse/issues/11265))
-- Fix a long-standing bug where uploading extremely thin images (e.g. 1000x1) would fail. Contributed by @Neeeflix. ([\#11288](https://github.com/matrix-org/synapse/issues/11288))
-- Fix a bug, introduced in Synapse 1.46.0, which caused the `check_3pid_auth` and `on_logged_out` callbacks in legacy password authentication provider modules to not be registered. Modules using the generic module interface were not affected. ([\#11340](https://github.com/matrix-org/synapse/issues/11340))
-- Fix a bug introduced in 1.41.0 where space hierarchy responses would be incorrectly reused if multiple users were to make the same request at the same time. ([\#11355](https://github.com/matrix-org/synapse/issues/11355))
-- Fix a bug introduced in 1.45.0 where the `read_templates` method of the module API would error. ([\#11377](https://github.com/matrix-org/synapse/issues/11377))
-- Fix an issue introduced in 1.47.0 which prevented servers re-joining rooms they had previously left, if their signing keys were replaced. ([\#11379](https://github.com/matrix-org/synapse/issues/11379))
-- Fix a bug introduced in 1.13.0 where creating and publishing a room could cause errors if `room_list_publication_rules` is configured. ([\#11392](https://github.com/matrix-org/synapse/issues/11392))
-- Improve performance of various background database updates. ([\#11421](https://github.com/matrix-org/synapse/issues/11421), [\#11422](https://github.com/matrix-org/synapse/issues/11422))
-
-
-Improved Documentation
-----------------------
-
-- Suggest users of the Debian packages add configuration to `/etc/matrix-synapse/conf.d/` to prevent, upon upgrade, being asked to choose between their configuration and the maintainer's. ([\#11281](https://github.com/matrix-org/synapse/issues/11281))
-- Fix typos in the documentation for the `username_available` admin API. Contributed by Stanislav Motylkov. ([\#11286](https://github.com/matrix-org/synapse/issues/11286))
-- Add Single Sign-On, SAML and CAS pages to the documentation. ([\#11298](https://github.com/matrix-org/synapse/issues/11298))
-- Change the word 'Home server' as one word 'homeserver' in documentation. ([\#11320](https://github.com/matrix-org/synapse/issues/11320))
-- Fix missing quotes for wildcard domains in `federation_certificate_verification_whitelist`. ([\#11381](https://github.com/matrix-org/synapse/issues/11381))
-
-
-Deprecations and Removals
--------------------------
-
-- Remove deprecated `trust_identity_server_for_password_resets` configuration flag. ([\#11333](https://github.com/matrix-org/synapse/issues/11333), [\#11395](https://github.com/matrix-org/synapse/issues/11395))
-
-
-Internal Changes
-----------------
-
-- Add type annotations to `synapse.metrics`. ([\#10847](https://github.com/matrix-org/synapse/issues/10847))
-- Split out federated PDU retrieval function into a non-cached version. ([\#11242](https://github.com/matrix-org/synapse/issues/11242))
-- Clean up code relating to to-device messages and sending ephemeral events to application services. ([\#11247](https://github.com/matrix-org/synapse/issues/11247))
-- Fix a small typo in the error response when a relation type other than 'm.annotation' is passed to `GET /rooms/{room_id}/aggregations/{event_id}`. ([\#11278](https://github.com/matrix-org/synapse/issues/11278))
-- Drop unused database tables `room_stats_historical` and `user_stats_historical`. ([\#11280](https://github.com/matrix-org/synapse/issues/11280))
-- Require all files in synapse/ and tests/ to pass mypy unless specifically excluded. ([\#11282](https://github.com/matrix-org/synapse/issues/11282), [\#11285](https://github.com/matrix-org/synapse/issues/11285), [\#11359](https://github.com/matrix-org/synapse/issues/11359))
-- Add missing type hints to `synapse.app`. ([\#11287](https://github.com/matrix-org/synapse/issues/11287))
-- Remove unused parameters on `FederationEventHandler._check_event_auth`. ([\#11292](https://github.com/matrix-org/synapse/issues/11292))
-- Add type hints to `synapse._scripts`. ([\#11297](https://github.com/matrix-org/synapse/issues/11297))
-- Fix an issue which prevented the `remove_deleted_devices_from_device_inbox` background database schema update from running when updating from a recent Synapse version. ([\#11303](https://github.com/matrix-org/synapse/issues/11303))
-- Add type hints to storage classes. ([\#11307](https://github.com/matrix-org/synapse/issues/11307), [\#11310](https://github.com/matrix-org/synapse/issues/11310), [\#11311](https://github.com/matrix-org/synapse/issues/11311), [\#11312](https://github.com/matrix-org/synapse/issues/11312), [\#11313](https://github.com/matrix-org/synapse/issues/11313), [\#11314](https://github.com/matrix-org/synapse/issues/11314), [\#11316](https://github.com/matrix-org/synapse/issues/11316), [\#11322](https://github.com/matrix-org/synapse/issues/11322), [\#11332](https://github.com/matrix-org/synapse/issues/11332), [\#11339](https://github.com/matrix-org/synapse/issues/11339), [\#11342](https://github.com/matrix-org/synapse/issues/11342))
-- Add type hints to `synapse.util`. ([\#11321](https://github.com/matrix-org/synapse/issues/11321), [\#11328](https://github.com/matrix-org/synapse/issues/11328))
-- Improve type annotations in Synapse's test suite. ([\#11323](https://github.com/matrix-org/synapse/issues/11323), [\#11330](https://github.com/matrix-org/synapse/issues/11330))
-- Test that room alias deletion works as intended. ([\#11327](https://github.com/matrix-org/synapse/issues/11327))
-- Add type annotations for some methods and properties in the module API. ([\#11341](https://github.com/matrix-org/synapse/issues/11341))
-- Fix running `scripts-dev/complement.sh`, which was broken in v1.47.0rc1. ([\#11368](https://github.com/matrix-org/synapse/issues/11368))
-- Rename internal functions for token generation to better reflect what they do. ([\#11369](https://github.com/matrix-org/synapse/issues/11369), [\#11370](https://github.com/matrix-org/synapse/issues/11370))
-- Add type hints to configuration classes. ([\#11377](https://github.com/matrix-org/synapse/issues/11377))
-- Publish a `develop` image to Docker Hub. ([\#11380](https://github.com/matrix-org/synapse/issues/11380))
-- Keep fallback key marked as used if it's re-uploaded. ([\#11382](https://github.com/matrix-org/synapse/issues/11382))
-- Use `auto_attribs` on the `attrs` class `RefreshTokenLookupResult`. ([\#11386](https://github.com/matrix-org/synapse/issues/11386))
-- Rename unstable `access_token_lifetime` configuration option to `refreshable_access_token_lifetime` to make it clear it only concerns refreshable access tokens. ([\#11388](https://github.com/matrix-org/synapse/issues/11388))
-- Do not run the broken MSC2716 tests when running `scripts-dev/complement.sh`. ([\#11389](https://github.com/matrix-org/synapse/issues/11389))
-- Remove dead code from supporting ACME. ([\#11393](https://github.com/matrix-org/synapse/issues/11393))
-- Refactor including the bundled relations when serializing an event. ([\#11408](https://github.com/matrix-org/synapse/issues/11408))
-
-
-Synapse 1.47.1 (2021-11-23)
-===========================
-
-This release fixes a security issue in the media store, affecting all prior releases of Synapse. Server administrators are encouraged to update Synapse as soon as possible. We are not aware of these vulnerabilities being exploited in the wild.
-
-Server administrators who are unable to update Synapse may use the workarounds described in the linked GitHub Security Advisory below.
-
-Security advisory
------------------
-
-The following issue is fixed in 1.47.1.
-
-- **[GHSA-3hfw-x7gx-437c](https://github.com/matrix-org/synapse/security/advisories/GHSA-3hfw-x7gx-437c) / [CVE-2021-41281](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2021-41281): Path traversal when downloading remote media.**
-
-  Synapse instances with the media repository enabled can be tricked into downloading a file from a remote server into an arbitrary directory, potentially outside the media store directory.
-
-  The last two directories and file name of the path are chosen randomly by Synapse and cannot be controlled by an attacker, which limits the impact.
-
-  Homeservers with the media repository disabled are unaffected. Homeservers configured with a federation whitelist are also unaffected.
-
-  Fixed by [91f2bd090](https://github.com/matrix-org/synapse/commit/91f2bd090).
-
-
-Synapse 1.47.0 (2021-11-17)
-===========================
-
-No significant changes since 1.47.0rc3.
-
-
-Synapse 1.47.0rc3 (2021-11-16)
-==============================
-
-Bugfixes
---------
-
-- Fix a bug introduced in 1.47.0rc1 which caused worker processes to not halt startup in the presence of outstanding database migrations. ([\#11346](https://github.com/matrix-org/synapse/issues/11346))
-- Fix a bug introduced in 1.47.0rc1 which prevented the 'remove deleted devices from `device_inbox` column' background process from running when updating from a recent Synapse version. ([\#11303](https://github.com/matrix-org/synapse/issues/11303), [\#11353](https://github.com/matrix-org/synapse/issues/11353))
-
-
-Synapse 1.47.0rc2 (2021-11-10)
-==============================
-
-This fixes an issue with publishing the Debian packages for 1.47.0rc1.
-It is otherwise identical to 1.47.0rc1.
-
-
-Synapse 1.47.0rc1 (2021-11-09)
-==============================
-
-Deprecations and Removals
--------------------------
-
-- The `user_may_create_room_with_invites` module callback is now deprecated. Please refer to the [upgrade notes](https://matrix-org.github.io/synapse/develop/upgrade#upgrading-to-v1470) for more information. ([\#11206](https://github.com/matrix-org/synapse/issues/11206))
-- Remove deprecated admin API to delete rooms (`POST /_synapse/admin/v1/rooms/<room_id>/delete`). ([\#11213](https://github.com/matrix-org/synapse/issues/11213))
-
-
-Features
---------
-
-- Advertise support for Client-Server API r0.6.1. ([\#11097](https://github.com/matrix-org/synapse/issues/11097))
-- Add search by room ID and room alias to the List Room admin API. ([\#11099](https://github.com/matrix-org/synapse/issues/11099))
-- Add an `on_new_event` third-party rules callback to allow Synapse modules to act after an event has been sent into a room. ([\#11126](https://github.com/matrix-org/synapse/issues/11126))
-- Add a module API method to update a user's membership in a room. ([\#11147](https://github.com/matrix-org/synapse/issues/11147))
-- Add metrics for thread pool usage. ([\#11178](https://github.com/matrix-org/synapse/issues/11178))
-- Support the stable room type field for [MSC3288](https://github.com/matrix-org/matrix-doc/pull/3288). ([\#11187](https://github.com/matrix-org/synapse/issues/11187))
-- Add a module API method to retrieve the current state of a room. ([\#11204](https://github.com/matrix-org/synapse/issues/11204))
-- Calculate a default value for `public_baseurl` based on `server_name`. ([\#11210](https://github.com/matrix-org/synapse/issues/11210))
-- Add support for serving `/.well-known/matrix/server` files, to redirect federation traffic to port 443. ([\#11211](https://github.com/matrix-org/synapse/issues/11211))
-- Add admin APIs to pause, start and check the status of background updates. ([\#11263](https://github.com/matrix-org/synapse/issues/11263))
-
-
-Bugfixes
---------
-
-- Fix a long-standing bug which allowed hidden devices to receive to-device messages, resulting in unnecessary database bloat. ([\#10097](https://github.com/matrix-org/synapse/issues/10097))
-- Fix a long-standing bug where messages in the `device_inbox` table for deleted devices would persist indefinitely. Contributed by @dklimpel and @JohannesKleine. ([\#10969](https://github.com/matrix-org/synapse/issues/10969), [\#11212](https://github.com/matrix-org/synapse/issues/11212))
-- Do not accept events if a third-party rule `check_event_allowed` callback raises an exception. ([\#11033](https://github.com/matrix-org/synapse/issues/11033))
-- Fix long-standing bug where verification requests could fail in certain cases if a federation whitelist was in place but did not include your own homeserver. ([\#11129](https://github.com/matrix-org/synapse/issues/11129))
-- Allow an empty list of `state_events_at_start` to be sent when using the [MSC2716](https://github.com/matrix-org/matrix-doc/pull/2716) `/batch_send` endpoint and the author of the historical messages is already part of the current room state at the given `?prev_event_id`. ([\#11188](https://github.com/matrix-org/synapse/issues/11188))
-- Fix a bug introduced in Synapse 1.45.0 which prevented the `synapse_review_recent_signups` script from running. Contributed by @samuel-p. ([\#11191](https://github.com/matrix-org/synapse/issues/11191))
-- Delete `to_device` messages for hidden devices that will never be read, reducing database size. ([\#11199](https://github.com/matrix-org/synapse/issues/11199))
-- Fix a long-standing bug wherein a missing `Content-Type` header when downloading remote media would cause Synapse to throw an error. ([\#11200](https://github.com/matrix-org/synapse/issues/11200))
-- Fix a long-standing bug which could result in serialization errors and potentially duplicate transaction data when sending ephemeral events to application services. Contributed by @Fizzadar at Beeper. ([\#11207](https://github.com/matrix-org/synapse/issues/11207))
-- Fix a bug introduced in Synapse 1.35.0 which made it impossible to join rooms that return a `send_join` response containing floats. ([\#11217](https://github.com/matrix-org/synapse/issues/11217))
-- Fix long-standing bug where cross signing keys were not included in the response to `/r0/keys/query` the first time a remote user was queried. ([\#11234](https://github.com/matrix-org/synapse/issues/11234))
-- Fix a long-standing bug where all requests that read events from the database could get stuck as a result of losing the database connection. ([\#11240](https://github.com/matrix-org/synapse/issues/11240))
-- Fix a bug preventing Synapse from being rolled back to an earlier version when using workers. ([\#11255](https://github.com/matrix-org/synapse/issues/11255), [\#11276](https://github.com/matrix-org/synapse/issues/11276))
-- Fix a bug introduced in Synapse 1.37.1 which caused a remote event being processed by a worker to not get processed on restart if the worker was killed. ([\#11262](https://github.com/matrix-org/synapse/issues/11262))
-- Only allow old Element/Riot Android clients to send read receipts without a request body. All other clients must include a request body as required by the specification. Contributed by @rogersheu. ([\#11157](https://github.com/matrix-org/synapse/issues/11157))
-
-
-Updates to the Docker image
----------------------------
-
-- Avoid changing user ID when started as a non-root user, and no explicit `UID` is set. ([\#11209](https://github.com/matrix-org/synapse/issues/11209))
-
-
-Improved Documentation
-----------------------
-
-- Improve example HAProxy config in the docs to properly handle HTTP `Host` headers with port information. This is required for federation over port 443 to work correctly. ([\#11128](https://github.com/matrix-org/synapse/issues/11128))
-- Add documentation for using Authentik as an OpenID Connect Identity Provider. Contributed by @samip5. ([\#11151](https://github.com/matrix-org/synapse/issues/11151))
-- Clarify lack of support for Windows. ([\#11198](https://github.com/matrix-org/synapse/issues/11198))
-- Improve code formatting and fix a few typos in docs. Contributed by @sumnerevans at Beeper. ([\#11221](https://github.com/matrix-org/synapse/issues/11221))
-- Add documentation for using LemonLDAP as an OpenID Connect Identity Provider. Contributed by @l00ptr. ([\#11257](https://github.com/matrix-org/synapse/issues/11257))
-
-
-Internal Changes
-----------------
-
-- Add type annotations for the `log_function` decorator. ([\#10943](https://github.com/matrix-org/synapse/issues/10943))
-- Add type hints to `synapse.events`. ([\#11098](https://github.com/matrix-org/synapse/issues/11098))
-- Remove and document unnecessary `RoomStreamToken` checks in application service ephemeral event code. ([\#11137](https://github.com/matrix-org/synapse/issues/11137))
-- Add type hints so that `synapse.http` passes `mypy` checks. ([\#11164](https://github.com/matrix-org/synapse/issues/11164))
-- Update scripts to pass Shellcheck lints. ([\#11166](https://github.com/matrix-org/synapse/issues/11166))
-- Add knock information in admin export. Contributed by Rafael Gonçalves. ([\#11171](https://github.com/matrix-org/synapse/issues/11171))
-- Add tests to check that `ClientIpStore.get_last_client_ip_by_device` and `get_user_ip_and_agents` combine database and in-memory data correctly. ([\#11179](https://github.com/matrix-org/synapse/issues/11179))
-- Refactor `Filter` to check different fields depending on the data type. ([\#11194](https://github.com/matrix-org/synapse/issues/11194))
-- Improve type hints for the relations datastore. ([\#11205](https://github.com/matrix-org/synapse/issues/11205))
-- Replace outdated links in the pull request checklist with links to the rendered documentation. ([\#11225](https://github.com/matrix-org/synapse/issues/11225))
-- Fix a bug in unit test `test_block_room_and_not_purge`. ([\#11226](https://github.com/matrix-org/synapse/issues/11226))
-- In `ObservableDeferred`, run observers in the order they were registered. ([\#11229](https://github.com/matrix-org/synapse/issues/11229))
-- Minor speed up to start up times and getting updates for groups by adding missing index to `local_group_updates.stream_id`. ([\#11231](https://github.com/matrix-org/synapse/issues/11231))
-- Add `twine` and `towncrier` as dev dependencies, as they're used by the release script. ([\#11233](https://github.com/matrix-org/synapse/issues/11233))
-- Allow `stream_writers.typing` config to be a list of one worker. ([\#11237](https://github.com/matrix-org/synapse/issues/11237))
-- Remove debugging statement in tests. ([\#11239](https://github.com/matrix-org/synapse/issues/11239))
-- Fix [MSC2716](https://github.com/matrix-org/matrix-doc/pull/2716) historical messages backfilling in random order on remote homeservers. ([\#11244](https://github.com/matrix-org/synapse/issues/11244))
-- Add an additional test for the `cachedList` method decorator. ([\#11246](https://github.com/matrix-org/synapse/issues/11246))
-- Make minor correction to the type of `auth_checkers` callbacks. ([\#11253](https://github.com/matrix-org/synapse/issues/11253))
-- Clean up trivial aspects of the Debian package build tooling. ([\#11269](https://github.com/matrix-org/synapse/issues/11269), [\#11273](https://github.com/matrix-org/synapse/issues/11273))
-- Blacklist new SyTest that checks that key uploads are valid pending the validation being implemented in Synapse. ([\#11270](https://github.com/matrix-org/synapse/issues/11270))
-
-
-Synapse 1.46.0 (2021-11-02)
-===========================
-
-The cause of the [performance regression affecting Synapse 1.44](https://github.com/matrix-org/synapse/issues/11049) has been identified and fixed. ([\#11177](https://github.com/matrix-org/synapse/issues/11177))
-
-Bugfixes
---------
-
-- Fix a bug introduced in v1.46.0rc1 where URL previews of some XML documents would fail. ([\#11196](https://github.com/matrix-org/synapse/issues/11196))
-
-
 Synapse 1.46.0rc1 (2021-10-27)
 ==============================
 
+The cause of the [performance regression affecting Synapse 1.44](https://github.com/matrix-org/synapse/issues/11049) has been identified and fixed. ([\#11177](https://github.com/matrix-org/synapse/issues/11177))
+
 Features
 --------
 
@@ -8805,14 +8575,14 @@ General:
 
 Federation:
 
--   Add key distribution mechanisms for fetching public keys of unavailable remote homeservers. See [Retrieving Server Keys](https://github.com/matrix-org/matrix-doc/blob/6f2698/specification/30_server_server_api.rst#retrieving-server-keys) in the spec.
+-   Add key distribution mechanisms for fetching public keys of unavailable remote home servers. See [Retrieving Server Keys](https://github.com/matrix-org/matrix-doc/blob/6f2698/specification/30_server_server_api.rst#retrieving-server-keys) in the spec.
 
 Configuration:
 
 -   Add support for multiple config files.
 -   Add support for dictionaries in config files.
 -   Remove support for specifying config options on the command line, except for:
-    -   `--daemonize` - Daemonize the homeserver.
+    -   `--daemonize` - Daemonize the home server.
     -   `--manhole` - Turn on the twisted telnet manhole service on the given port.
     -   `--database-path` - The path to a sqlite database to use.
     -   `--verbose` - The verbosity level.
@@ -9017,7 +8787,7 @@ This version adds support for using a TURN server. See docs/turn-howto.rst on ho
 Homeserver:
 
 -   Add support for redaction of messages.
--   Fix bug where inviting a user on a remote homeserver could take up to 20-30s.
+-   Fix bug where inviting a user on a remote home server could take up to 20-30s.
 -   Implement a get current room state API.
 -   Add support specifying and retrieving turn server configuration.
 
@@ -9107,7 +8877,7 @@ Changes in synapse 0.2.3 (2014-09-12)
 
 Homeserver:
 
--   Fix bug where we stopped sending events to remote homeservers if a user from that homeserver left, even if there were some still in the room.
+-   Fix bug where we stopped sending events to remote home servers if a user from that home server left, even if there were some still in the room.
 -   Fix bugs in the state conflict resolution where it was incorrectly rejecting events.
 
 Webclient:

changelog.d/10097.bugfix

@@ -0,0 +1 @@
+Fix a long-standing bug which allowed hidden devices to receive to-device messages, resulting in unnecessary database bloat.

changelog.d/10943.misc

@@ -0,0 +1 @@
+Add type annotations for the `log_function` decorator.

changelog.d/10969.bugfix

@@ -0,0 +1 @@
+Fix a long-standing bug where messages in the `device_inbox` table for deleted devices would persist indefinitely. Contributed by @dklimpel and @JohannesKleine.

changelog.d/11033.bugfix

@@ -0,0 +1 @@
+Do not accept events if a third-party rule module API callback raises an exception.

changelog.d/11097.feature

@@ -0,0 +1 @@
+Advertise support for Client-Server API r0.6.1.

changelog.d/11126.feature

@@ -0,0 +1 @@
+Add an `on_new_event` third-party rules callback to allow Synapse modules to act after an event has been sent into a room.

changelog.d/11128.doc

@@ -0,0 +1 @@
+Improve example HAProxy config in the docs to properly handle host headers with port information. This is required for federation over port 443 to work correctly.

changelog.d/11129.bugfix

@@ -0,0 +1 @@
+Fix long-standing bug where verification requests could fail in certain cases if whitelist was in place but did not include your own homeserver.

changelog.d/11147.feature

@@ -0,0 +1 @@
+Add a module API method to update a user's membership in a room.

changelog.d/11151.doc

@@ -0,0 +1 @@
+Add documentation for using Authentik as an OpenID Connect Identity Provider. Contributed by @samip5.

changelog.d/11164.misc

@@ -0,0 +1 @@
+Add type hints so that `synapse.http` passes `mypy` checks.

changelog.d/11166.misc

@@ -0,0 +1 @@
+Update scripts to pass Shellcheck lints.

changelog.d/11171.misc

@@ -0,0 +1 @@
+Add knock information in admin export. Contributed by Rafael Gonçalves.

changelog.d/11178.feature

@@ -0,0 +1 @@
+Add metrics for thread pool usage.

changelog.d/11179.misc

@@ -0,0 +1 @@
+Add tests to check that `ClientIpStore.get_last_client_ip_by_device` and `get_user_ip_and_agents` combine database and in-memory data correctly.

changelog.d/11187.feature

@@ -0,0 +1 @@
+Support the stable room type field for [MSC3288](https://github.com/matrix-org/matrix-doc/pull/3288).

changelog.d/11191.bugfix

@@ -0,0 +1 @@
+Fix a bug introduced in Synapse 1.45.0 which prevented the `synapse_review_recent_signups` script from running. Contributed by @samuel-p.

changelog.d/11194.misc

@@ -0,0 +1 @@
+Refactor `Filter` to check different fields depending on the data type.

changelog.d/11198.doc

@@ -0,0 +1 @@
+Clarify lack of support for Windows.

changelog.d/11204.feature

@@ -0,0 +1 @@
+Add a module API method to retrieve the current state of a room.

changelog.d/11205.misc

@@ -0,0 +1 @@
+Improve type hints for the relations datastore.

changelog.d/11206.removal

@@ -0,0 +1 @@
+The `user_may_create_room_with_invites` module callback is now deprecated. Please refer to the [upgrade notes](https://matrix-org.github.io/synapse/develop/upgrade#upgrading-to-v1470) for more information.

changelog.d/11209.docker

@@ -0,0 +1 @@
+Avoid changing userid when started as a non-root user, and no explicit `UID` is set.

changelog.d/11211.feature

@@ -0,0 +1 @@
+Add support for serving `/.well-known/matrix/server` files, to redirect federation traffic to port 443.

changelog.d/11212.bugfix

@@ -0,0 +1 @@
+Fix a long-standing bug where messages in the `device_inbox` table for deleted devices would persist indefinitely. Contributed by @dklimpel and @JohannesKleine.

changelog.d/11213.removal

@@ -0,0 +1 @@
+Remove deprecated admin API to delete rooms (`POST /_synapse/admin/v1/rooms/<room_id>/delete`).

changelog.d/11221.doc

@@ -0,0 +1 @@
+Improve code formatting and fix a few typos in docs. Contributed by @sumnerevans at Beeper.

changelog.d/11226.misc

@@ -0,0 +1 @@
+Fix a bug in unit test `test_block_room_and_not_purge`.

changelog.d/11232.misc

@@ -0,0 +1 @@
+TODO: write a proper changelog.

debian/build_virtualenv

@@ -40,7 +40,6 @@ dh_virtualenv \
     --upgrade-pip \
     --preinstall="lxml" \
     --preinstall="mock" \
-    --preinstall="wheel" \
     --extra-pip-arg="--no-cache-dir" \
     --extra-pip-arg="--compile" \
     --extras="all,systemd,test"

debian/changelog

@@ -1,56 +1,8 @@
-matrix-synapse-py3 (1.48.0) stable; urgency=medium
+matrix-synapse-py3 (1.47.0+nmu1) UNRELEASED; urgency=medium
 
-  * New synapse release 1.48.0.
-
- -- Synapse Packaging team <packages@matrix.org>  Tue, 30 Nov 2021 11:24:15 +0000
-
-matrix-synapse-py3 (1.48.0~rc1) stable; urgency=medium
-
-  * New synapse release 1.48.0~rc1.
-
- -- Synapse Packaging team <packages@matrix.org>  Thu, 25 Nov 2021 15:56:03 +0000
-
-matrix-synapse-py3 (1.47.1) stable; urgency=medium
-
-  * New synapse release 1.47.1.
-
- -- Synapse Packaging team <packages@matrix.org>  Fri, 19 Nov 2021 13:44:32 +0000
-
-matrix-synapse-py3 (1.47.0) stable; urgency=medium
-
-  * New synapse release 1.47.0.
-
- -- Synapse Packaging team <packages@matrix.org>  Wed, 17 Nov 2021 13:09:43 +0000
-
-matrix-synapse-py3 (1.47.0~rc3) stable; urgency=medium
-
-  * New synapse release 1.47.0~rc3.
-
- -- Synapse Packaging team <packages@matrix.org>  Tue, 16 Nov 2021 14:32:47 +0000
-
-matrix-synapse-py3 (1.47.0~rc2) stable; urgency=medium
-
-  [ Dan Callahan ]
   * Update scripts to pass Shellcheck lints.
-  * Remove unused Vagrant scripts from debian/ directory.
-  * Allow building Debian packages for any architecture, not just amd64.
-  * Preinstall the "wheel" package when building virtualenvs.
-  * Do not error if /etc/default/matrix-synapse is missing.
 
-  [ Synapse Packaging team ]
-  * New synapse release 1.47.0~rc2.
-
- -- Synapse Packaging team <packages@matrix.org>  Wed, 10 Nov 2021 09:41:01 +0000
-
-matrix-synapse-py3 (1.46.0) stable; urgency=medium
-
-  [ Richard van der Hoff ]
-  * Compress debs with xz, to fix incompatibility of impish debs with reprepro.
-
-  [ Synapse Packaging team ]
-  * New synapse release 1.46.0.
-
- -- Synapse Packaging team <packages@matrix.org>  Tue, 02 Nov 2021 13:22:53 +0000
+ -- root <root@cae79a6e79d7>  Fri, 22 Oct 2021 22:20:31 +0000
 
 matrix-synapse-py3 (1.46.0~rc1) stable; urgency=medium
 

debian/control

@@ -19,7 +19,7 @@ Standards-Version: 3.9.8
 Homepage: https://github.com/matrix-org/synapse
 
 Package: matrix-synapse-py3
-Architecture: any
+Architecture: amd64
 Provides: matrix-synapse
 Conflicts:
  matrix-synapse (<< 0.34.0.1-0matrix2),

debian/matrix-synapse.service

@@ -5,7 +5,7 @@ Description=Synapse Matrix homeserver
 Type=notify
 User=matrix-synapse
 WorkingDirectory=/var/lib/matrix-synapse
-EnvironmentFile=-/etc/default/matrix-synapse
+EnvironmentFile=/etc/default/matrix-synapse
 ExecStartPre=/opt/venvs/matrix-synapse/bin/python -m synapse.app.homeserver --config-path=/etc/matrix-synapse/homeserver.yaml --config-path=/etc/matrix-synapse/conf.d/ --generate-keys
 ExecStart=/opt/venvs/matrix-synapse/bin/python -m synapse.app.homeserver --config-path=/etc/matrix-synapse/homeserver.yaml --config-path=/etc/matrix-synapse/conf.d/
 ExecReload=/bin/kill -HUP $MAINPID

debian/rules

@@ -51,11 +51,5 @@ override_dh_shlibdeps:
 override_dh_virtualenv:
 	./debian/build_virtualenv
 
-override_dh_builddeb:
-        # force the compression to xzip, to stop dpkg-deb on impish defaulting to zstd
-        # (which requires reprepro 5.3.0-1.3, which is currently only in 'experimental' in Debian:
-        # https://metadata.ftp-master.debian.org/changelogs/main/r/reprepro/reprepro_5.3.0-1.3_changelog)
-	dh_builddeb -- -Zxz
-
 %:
 	dh $@ --with python-virtualenv

debian/test/.gitignore

@@ -0,0 +1,2 @@
+.vagrant
+*.log

debian/test/provision.sh

@@ -0,0 +1,24 @@
+#!/bin/bash
+#
+# provisioning script for vagrant boxes for testing the matrix-synapse debs.
+#
+# Will install the most recent matrix-synapse-py3 deb for this platform from
+# the /debs directory.
+
+set -e
+
+apt-get update
+apt-get install -y lsb-release
+
+deb=$(find /debs -name "matrix-synapse-py3_*+$(lsb_release -cs)*.deb" | sort | tail -n1)
+
+debconf-set-selections <<EOF
+matrix-synapse matrix-synapse/report-stats boolean false
+matrix-synapse matrix-synapse/server-name string localhost:18448
+EOF
+
+dpkg -i "$deb"
+
+sed -i -e 's/port: 8448$/port: 18448/; s/port: 8008$/port: 18008' /etc/matrix-synapse/homeserver.yaml
+echo 'registration_shared_secret: secret' >> /etc/matrix-synapse/homeserver.yaml
+systemctl restart matrix-synapse

debian/test/stretch/Vagrantfile

@@ -0,0 +1,13 @@
+# -*- mode: ruby -*-
+# vi: set ft=ruby :
+
+ver = `cd ../../..; dpkg-parsechangelog -S Version`.strip()
+
+Vagrant.configure("2") do |config|
+  config.vm.box = "debian/stretch64"
+
+  config.vm.synced_folder ".", "/vagrant", disabled: true
+  config.vm.synced_folder "../../../../debs", "/debs", type: "nfs"
+
+  config.vm.provision "shell", path: "../provision.sh"
+end

debian/test/xenial/Vagrantfile

@@ -0,0 +1,10 @@
+# -*- mode: ruby -*-
+# vi: set ft=ruby :
+
+Vagrant.configure("2") do |config|
+  config.vm.box = "ubuntu/xenial64"
+
+  config.vm.synced_folder ".", "/vagrant", disabled: true
+  config.vm.synced_folder "../../../../debs", "/debs"
+  config.vm.provision "shell", path: "../provision.sh"
+end

docker/conf/homeserver.yaml

@@ -148,6 +148,14 @@ bcrypt_rounds: 12
 allow_guest_access: {{ "True" if SYNAPSE_ALLOW_GUEST else "False" }}
 enable_group_creation: true
 
+# The list of identity servers trusted to verify third party
+# identifiers by this server.
+#
+# Also defines the ID server which will be called when an account is
+# deactivated (one will be picked arbitrarily).
+trusted_third_party_id_servers:
+    - matrix.org
+    - vector.im
 
 ## Metrics ###
 

docker/configure_workers_and_start.py

@@ -48,7 +48,7 @@ WORKERS_CONFIG = {
         "app": "synapse.app.user_dir",
         "listener_resources": ["client"],
         "endpoint_patterns": [
-            "^/_matrix/client/(api/v1|r0|v3|unstable)/user_directory/search$"
+            "^/_matrix/client/(api/v1|r0|unstable)/user_directory/search$"
         ],
         "shared_extra_conf": {"update_user_directory": False},
         "worker_extra_conf": "",
@@ -85,10 +85,10 @@ WORKERS_CONFIG = {
         "app": "synapse.app.generic_worker",
         "listener_resources": ["client"],
         "endpoint_patterns": [
-            "^/_matrix/client/(v2_alpha|r0|v3)/sync$",
-            "^/_matrix/client/(api/v1|v2_alpha|r0|v3)/events$",
-            "^/_matrix/client/(api/v1|r0|v3)/initialSync$",
-            "^/_matrix/client/(api/v1|r0|v3)/rooms/[^/]+/initialSync$",
+            "^/_matrix/client/(v2_alpha|r0)/sync$",
+            "^/_matrix/client/(api/v1|v2_alpha|r0)/events$",
+            "^/_matrix/client/(api/v1|r0)/initialSync$",
+            "^/_matrix/client/(api/v1|r0)/rooms/[^/]+/initialSync$",
         ],
         "shared_extra_conf": {},
         "worker_extra_conf": "",
@@ -146,11 +146,11 @@ WORKERS_CONFIG = {
         "app": "synapse.app.generic_worker",
         "listener_resources": ["client"],
         "endpoint_patterns": [
-            "^/_matrix/client/(api/v1|r0|v3|unstable)/rooms/.*/redact",
-            "^/_matrix/client/(api/v1|r0|v3|unstable)/rooms/.*/send",
-            "^/_matrix/client/(api/v1|r0|v3|unstable)/rooms/.*/(join|invite|leave|ban|unban|kick)$",
-            "^/_matrix/client/(api/v1|r0|v3|unstable)/join/",
-            "^/_matrix/client/(api/v1|r0|v3|unstable)/profile/",
+            "^/_matrix/client/(api/v1|r0|unstable)/rooms/.*/redact",
+            "^/_matrix/client/(api/v1|r0|unstable)/rooms/.*/send",
+            "^/_matrix/client/(api/v1|r0|unstable)/rooms/.*/(join|invite|leave|ban|unban|kick)$",
+            "^/_matrix/client/(api/v1|r0|unstable)/join/",
+            "^/_matrix/client/(api/v1|r0|unstable)/profile/",
         ],
         "shared_extra_conf": {},
         "worker_extra_conf": "",
@@ -158,7 +158,7 @@ WORKERS_CONFIG = {
     "frontend_proxy": {
         "app": "synapse.app.frontend_proxy",
         "listener_resources": ["client", "replication"],
-        "endpoint_patterns": ["^/_matrix/client/(api/v1|r0|v3|unstable)/keys/upload"],
+        "endpoint_patterns": ["^/_matrix/client/(api/v1|r0|unstable)/keys/upload"],
         "shared_extra_conf": {},
         "worker_extra_conf": (
             "worker_main_http_uri: http://127.0.0.1:%d"

docs/README.md

@@ -50,10 +50,8 @@ build the documentation with:
 mdbook build
 ```
 
-The rendered contents will be outputted to a new `book/` directory at the root of the repository. Please note that 
-index.html is not built by default, it is created by copying over the file `welcome_and_overview.html` to `index.html`
-during deployment. Thus, when running `mdbook serve` locally the book will initially show a 404 in place of the index
-due to the above. Do not be alarmed!
+The rendered contents will be outputted to a new `book/` directory at the root of the repository. You can
+browse the book by opening `book/index.html` in a web browser.
 
 You can also have mdbook host the docs on a local webserver with hot-reload functionality via:
 

docs/SUMMARY.md

@@ -23,10 +23,10 @@
     - [Structured Logging](structured_logging.md)
     - [Templates](templates.md)
     - [User Authentication](usage/configuration/user_authentication/README.md)
-      - [Single-Sign On](usage/configuration/user_authentication/single_sign_on/README.md)
+      - [Single-Sign On]()
         - [OpenID Connect](openid.md)
-        - [SAML](usage/configuration/user_authentication/single_sign_on/saml.md)
-        - [CAS](usage/configuration/user_authentication/single_sign_on/cas.md)
+        - [SAML]()
+        - [CAS]()
         - [SSO Mapping Providers](sso_mapping_providers.md)
       - [Password Auth Providers](password_auth_providers.md)
       - [JSON Web Tokens](jwt.md)
@@ -51,7 +51,6 @@
   - [Administration](usage/administration/README.md)
     - [Admin API](usage/administration/admin_api/README.md)
       - [Account Validity](admin_api/account_validity.md)
-      - [Background Updates](usage/administration/admin_api/background_updates.md)
       - [Delete Group](admin_api/delete_group.md)
       - [Event Reports](admin_api/event_reports.md)
       - [Media](admin_api/media_admin_api.md)

docs/admin_api/purge_history_api.md

@@ -70,8 +70,6 @@ This API returns a JSON body like the following:
 
 The status will be one of `active`, `complete`, or `failed`.
 
-If `status` is `failed` there will be a string `error` with the error message.
-
 ## Reclaim disk space (Postgres)
 
 To reclaim the disk space and return it to the operating system, you need to run

docs/admin_api/rooms.md

@@ -3,11 +3,7 @@
 - [Room Details API](#room-details-api)
 - [Room Members API](#room-members-api)
 - [Room State API](#room-state-api)
-- [Block Room API](#block-room-api)
 - [Delete Room API](#delete-room-api)
-  * [Version 1 (old version)](#version-1-old-version)
-  * [Version 2 (new version)](#version-2-new-version)
-  * [Status of deleting rooms](#status-of-deleting-rooms)
   * [Undoing room shutdowns](#undoing-room-shutdowns)
 - [Make Room Admin API](#make-room-admin-api)
 - [Forward Extremities Admin API](#forward-extremities-admin-api)
@@ -42,14 +38,9 @@ The following query parameters are available:
   - `history_visibility` - Rooms are ordered alphabetically by visibility of history of the room.
   - `state_events` - Rooms are ordered by number of state events. Largest to smallest.
 * `dir` - Direction of room order. Either `f` for forwards or `b` for backwards. Setting
-  this value to `b` will reverse the above sort order. Defaults to `f`.
-* `search_term` - Filter rooms by their room name, canonical alias and room id.
-  Specifically, rooms are selected if the search term is contained in
-  - the room's name,
-  - the local part of the room's canonical alias, or
-  - the complete (local and server part) room's id (case sensitive).
-
-  Defaults to no filtering.
+          this value to `b` will reverse the above sort order. Defaults to `f`.
+* `search_term` - Filter rooms by their room name. Search term can be contained in any
+                  part of the room name. Defaults to no filtering.
 
 **Response**
 
@@ -387,86 +378,9 @@ A response body like the following is returned:
 }
 ```
 
-# Block Room API
-The Block Room admin API allows server admins to block and unblock rooms,
-and query to see if a given room is blocked.
-This API can be used to pre-emptively block a room, even if it's unknown to this
-homeserver. Users will be prevented from joining a blocked room.
-
-## Block or unblock a room
-
-The API is:
-
-```
-PUT /_synapse/admin/v1/rooms/<room_id>/block
-```
-
-with a body of:
-
-```json
-{
-    "block": true
-}
-```
-
-A response body like the following is returned:
-
-```json
-{
-    "block": true
-}
-```
-
-**Parameters**
-
-The following parameters should be set in the URL:
-
-- `room_id` - The ID of the room.
-
-The following JSON body parameters are available:
-
-- `block` - If `true` the room will be blocked and if `false` the room will be unblocked.
-
-**Response**
-
-The following fields are possible in the JSON response body:
-
-- `block` - A boolean. `true` if the room is blocked, otherwise `false`
-
-## Get block status
-
-The API is:
-
-```
-GET /_synapse/admin/v1/rooms/<room_id>/block
-```
-
-A response body like the following is returned:
-
-```json
-{
-    "block": true,
-    "user_id": "<user_id>"
-}
-```
-
-**Parameters**
-
-The following parameters should be set in the URL:
-
-- `room_id` - The ID of the room.
-
-**Response**
-
-The following fields are possible in the JSON response body:
-
-- `block` - A boolean. `true` if the room is blocked, otherwise `false`
-- `user_id` - An optional string. If the room is blocked (`block` is `true`) shows
-  the user who has add the room to blocking list. Otherwise it is not displayed.
-
 # Delete Room API
 
-The Delete Room admin API allows server admins to remove rooms from the server
+The Delete Room admin API allows server admins to remove rooms from server
 and block these rooms.
 
 Shuts down a room. Moves all local users and room aliases automatically to a
@@ -477,33 +391,18 @@ The new room will be created with the user specified by the `new_room_user_id` p
 as room administrator and will contain a message explaining what happened. Users invited
 to the new room will have power level `-10` by default, and thus be unable to speak.
 
-If `block` is `true`, users will be prevented from joining the old room.
-This option can in [Version 1](#version-1-old-version) also be used to pre-emptively
-block a room, even if it's unknown to this homeserver. In this case, the room will be
-blocked, and no further action will be taken. If `block` is `false`, attempting to
-delete an unknown room is invalid and will be rejected as a bad request.
+If `block` is `True` it prevents new joins to the old room.
 
 This API will remove all trace of the old room from your database after removing
 all local users. If `purge` is `true` (the default), all traces of the old room will
 be removed from your database after removing all local users. If you do not want
 this to happen, set `purge` to `false`.
-Depending on the amount of history being purged, a call to the API may take
+Depending on the amount of history being purged a call to the API may take
 several minutes or longer.
 
 The local server will only have the power to move local user and room aliases to
 the new room. Users on other servers will be unaffected.
 
-To use it, you will need to authenticate by providing an ``access_token`` for a
-server admin: see [Admin API](../usage/administration/admin_api).
-
-## Version 1 (old version)
-
-This version works synchronously. That means you only get the response once the server has
-finished the action, which may take a long time. If you request the same action
-a second time, and the server has not finished the first one, the second request will block.
-This is fixed in version 2 of this API. The parameters are the same in both APIs.
-This API will become deprecated in the future.
-
 The API is:
 
 ```
@@ -522,6 +421,9 @@ with a body of:
 }
 ```
 
+To use it, you will need to authenticate by providing an ``access_token`` for a
+server admin: see [Admin API](../usage/administration/admin_api).
+
 A response body like the following is returned:
 
 ```json
@@ -538,44 +440,6 @@ A response body like the following is returned:
 }
 ```
 
-The parameters and response values have the same format as
-[version 2](#version-2-new-version) of the API.
-
-## Version 2 (new version)
-
-**Note**: This API is new, experimental and "subject to change".
-
-This version works asynchronously, meaning you get the response from server immediately
-while the server works on that task in background. You can then request the status of the action
-to check if it has completed.
-
-The API is:
-
-```
-DELETE /_synapse/admin/v2/rooms/<room_id>
-```
-
-with a body of:
-
-```json
-{
-    "new_room_user_id": "@someuser:example.com",
-    "room_name": "Content Violation Notification",
-    "message": "Bad Room has been shutdown due to content violations on this server. Please review our Terms of Service.",
-    "block": true,
-    "purge": true
-}
-```
-
-The API starts the shut down and purge running, and returns immediately with a JSON body with
-a purge id:
-
-```json
-{
-    "delete_id": "<opaque id>"
-}
-```
-
 **Parameters**
 
 The following parameters should be set in the URL:
@@ -595,10 +459,8 @@ The following JSON body parameters are available:
               `new_room_user_id` in the new room. Ideally this will clearly convey why the
                original room was shut down. Defaults to `Sharing illegal content on this server
                is not permitted and rooms in violation will be blocked.`
-* `block` - Optional. If set to `true`, this room will be added to a blocking list,
-            preventing future attempts to join the room. Rooms can be blocked
-            even if they're not yet known to the homeserver (only with
-            [Version 1](#version-1-old-version) of the API). Defaults to `false`.
+* `block` - Optional. If set to `true`, this room will be added to a blocking list, preventing
+            future attempts to join the room. Defaults to `false`.
 * `purge` - Optional. If set to `true`, it will remove all traces of the room from your database.
             Defaults to `true`.
 * `force_purge` - Optional, and ignored unless `purge` is `true`. If set to `true`, it
@@ -608,124 +470,16 @@ The following JSON body parameters are available:
 
 The JSON body must not be empty. The body must be at least `{}`.
 
-## Status of deleting rooms
-
-**Note**: This API is new, experimental and "subject to change".
-
-It is possible to query the status of the background task for deleting rooms.
-The status can be queried up to 24 hours after completion of the task,
-or until Synapse is restarted (whichever happens first).
-
-### Query by `room_id`
-
-With this API you can get the status of all active deletion tasks, and all those completed in the last 24h,
-for the given `room_id`.
-
-The API is:
-
-```
-GET /_synapse/admin/v2/rooms/<room_id>/delete_status
-```
-
-A response body like the following is returned:
-
-```json
-{
-    "results": [
-        {
-            "delete_id": "delete_id1",
-            "status": "failed",
-            "error": "error message",
-            "shutdown_room": {
-                "kicked_users": [],
-                "failed_to_kick_users": [],
-                "local_aliases": [],
-                "new_room_id": null
-            }
-        }, {
-            "delete_id": "delete_id2",
-            "status": "purging",
-            "shutdown_room": {
-                "kicked_users": [
-                    "@foobar:example.com"
-                ],
-                "failed_to_kick_users": [],
-                "local_aliases": [
-                    "#badroom:example.com",
-                    "#evilsaloon:example.com"
-                ],
-                "new_room_id": "!newroomid:example.com"
-            }
-        }
-    ]
-}
-```
-
-**Parameters**
-
-The following parameters should be set in the URL:
-
-* `room_id` - The ID of the room.
-
-### Query by `delete_id`
-
-With this API you can get the status of one specific task by `delete_id`.
-
-The API is:
-
-```
-GET /_synapse/admin/v2/rooms/delete_status/<delete_id>
-```
-
-A response body like the following is returned:
-
-```json
-{
-    "status": "purging",
-    "shutdown_room": {
-        "kicked_users": [
-            "@foobar:example.com"
-        ],
-        "failed_to_kick_users": [],
-        "local_aliases": [
-            "#badroom:example.com",
-            "#evilsaloon:example.com"
-        ],
-        "new_room_id": "!newroomid:example.com"
-    }
-}
-```
-
-**Parameters**
-
-The following parameters should be set in the URL:
-
-* `delete_id` - The ID for this delete.
-
-### Response
+**Response**
 
 The following fields are returned in the JSON response body:
 
-- `results` - An array of objects, each containing information about one task.
-  This field is omitted from the result when you query by `delete_id`.
-  Task objects contain the following fields:
-  - `delete_id` - The ID for this purge if you query by `room_id`.
-  - `status` - The status will be one of:
-    - `shutting_down` - The process is removing users from the room.
-    - `purging` - The process is purging the room and event data from database.
-    - `complete` - The process has completed successfully.
-    - `failed` - The process is aborted, an error has occurred.
-  - `error` - A string that shows an error message if `status` is `failed`.
-    Otherwise this field is hidden.
-  - `shutdown_room` - An object containing information about the result of shutting down the room.
-    *Note:* The result is shown after removing the room members.
-    The delete process can still be running. Please pay attention to the `status`.
-    - `kicked_users` - An array of users (`user_id`) that were kicked.
-    - `failed_to_kick_users` - An array of users (`user_id`) that that were not kicked.
-    - `local_aliases` - An array of strings representing the local aliases that were
-      migrated from the old room to the new.
-    - `new_room_id` - A string representing the room ID of the new room, or `null` if
-      no such room was created.
+* `kicked_users` - An array of users (`user_id`) that were kicked.
+* `failed_to_kick_users` - An array of users (`user_id`) that that were not kicked.
+* `local_aliases` - An array of strings representing the local aliases that were migrated from
+                    the old room to the new.
+* `new_room_id` - A string representing the room ID of the new room.
+
 
 ## Undoing room deletions
 

docs/admin_api/user_admin_api.md

@@ -948,7 +948,7 @@ The following fields are returned in the JSON response body:
 See also the
 [Client-Server API Spec on pushers](https://matrix.org/docs/spec/client_server/latest#get-matrix-client-r0-pushers).
 
-## Controlling whether a user is shadow-banned
+## Shadow-banning users
 
 Shadow-banning is a useful tool for moderating malicious or egregiously abusive users.
 A shadow-banned users receives successful responses to their client-server API requests,
@@ -961,22 +961,16 @@ or broken behaviour for the client. A shadow-banned user will not receive any
 notification and it is generally more appropriate to ban or kick abusive users.
 A shadow-banned user will be unable to contact anyone on the server.
 
-To shadow-ban a user the API is:
+The API is:
 
 ```
 POST /_synapse/admin/v1/users/<user_id>/shadow_ban
 ```
 
-To un-shadow-ban a user the API is:
-
-```
-DELETE /_synapse/admin/v1/users/<user_id>/shadow_ban
-```
-
 To use it, you will need to authenticate by providing an `access_token` for a
 server admin: [Admin API](../usage/administration/admin_api)
 
-An empty JSON dict is returned in both cases.
+An empty JSON dict is returned.
 
 **Parameters**
 
@@ -1113,7 +1107,7 @@ This endpoint will work even if registration is disabled on the server, unlike
 The API is:
 
 ```
-GET /_synapse/admin/v1/username_available?username=$localpart
+POST /_synapse/admin/v1/username_availabile?username=$localpart
 ```
 
 The request and response format is the same as the [/_matrix/client/r0/register/available](https://matrix.org/docs/spec/client_server/r0.6.0#get-matrix-client-r0-register-available) API.

docs/ancient_architecture_notes.md

@@ -7,7 +7,7 @@
 
 ## Server to Server Stack
 
-To use the server to server stack, homeservers should only need to
+To use the server to server stack, home servers should only need to
 interact with the Messaging layer.
 
 The server to server side of things is designed into 4 distinct layers:
@@ -23,7 +23,7 @@ Server with a domain specific API.
 
 1. **Messaging Layer**
 
-    This is what the rest of the homeserver hits to send messages, join rooms,
+    This is what the rest of the Home Server hits to send messages, join rooms,
     etc. It also allows you to register callbacks for when it get's notified by
     lower levels that e.g. a new message has been received.
 
@@ -45,7 +45,7 @@ Server with a domain specific API.
 
     For incoming PDUs, it has to check the PDUs it references to see
     if we have missed any. If we have go and ask someone (another
-    homeserver) for it.
+    home server) for it.
 
 3. **Transaction Layer**
 

docs/jwt.md

@@ -22,9 +22,8 @@ will be removed in a future version of Synapse.
 
 The `token` field should include the JSON web token with the following claims:
 
-* A claim that encodes the local part of the user ID is required. By default,
-  the `sub` (subject) claim is used, or a custom claim can be set in the
-  configuration file.
+* The `sub` (subject) claim is required and should encode the local part of the
+  user ID.
 * The expiration time (`exp`), not before time (`nbf`), and issued at (`iat`)
   claims are optional, but validated if present.
 * The issuer (`iss`) claim is optional, but required and validated if configured.

docs/modules/password_auth_provider_callbacks.md

@@ -11,7 +11,7 @@ registered by using the Module API's `register_password_auth_provider_callbacks`
 _First introduced in Synapse v1.46.0_
 
 ```python
-auth_checkers: Dict[Tuple[str, Tuple[str, ...]], Callable]
+ auth_checkers: Dict[Tuple[str,Tuple], Callable]
 ```
 
 A dict mapping from tuples of a login type identifier (such as `m.login.password`) and a

docs/openid.md

@@ -22,7 +22,6 @@ such as [Github][github-idp].
 [google-idp]: https://developers.google.com/identity/protocols/oauth2/openid-connect
 [auth0]: https://auth0.com/
 [authentik]: https://goauthentik.io/
-[lemonldap]: https://lemonldap-ng.org/
 [okta]: https://www.okta.com/
 [dex-idp]: https://github.com/dexidp/dex
 [keycloak-idp]: https://www.keycloak.org/docs/latest/server_admin/#sso-protocols
@@ -244,43 +243,6 @@ oidc_providers:
         display_name_template: "{{ user.preferred_username|capitalize }}" # TO BE FILLED: If your users have names in Authentik and you want those in Synapse, this should be replaced with user.name|capitalize.
 ```
 
-### LemonLDAP
-
-[LemonLDAP::NG][lemonldap] is an open-source IdP solution.
-
-1. Create an OpenID Connect Relying Parties in LemonLDAP::NG
-2. The parameters are:
-- Client ID under the basic menu of the new Relying Parties (`Options > Basic >
-  Client ID`)
-- Client secret (`Options > Basic > Client secret`)
-- JWT Algorithm: RS256 within the security menu of the new Relying Parties
-  (`Options > Security > ID Token signature algorithm` and `Options > Security >
-  Access Token signature algorithm`)
-- Scopes: OpenID, Email and Profile
-- Allowed redirection addresses for login (`Options > Basic > Allowed
-  redirection addresses for login` ) :
-  `[synapse public baseurl]/_synapse/client/oidc/callback`
-
-Synapse config:
-```yaml
-oidc_providers:
-  - idp_id: lemonldap
-    idp_name: lemonldap
-    discover: true
-    issuer: "https://auth.example.org/" # TO BE FILLED: replace with your domain
-    client_id: "your client id" # TO BE FILLED
-    client_secret: "your client secret" # TO BE FILLED
-    scopes:
-      - "openid"
-      - "profile"
-      - "email"
-    user_mapping_provider:
-      config:
-        localpart_template: "{{ user.preferred_username }}}"
-        # TO BE FILLED: If your users have names in LemonLDAP::NG and you want those in Synapse, this should be replaced with user.name|capitalize or any valid filter.
-        display_name_template: "{{ user.preferred_username|capitalize }}"
-```
-
 ### GitHub
 
 [GitHub][github-idp] is a bit special as it is not an OpenID Connect compliant provider, but

docs/password_auth_providers.md

@@ -1,7 +1,7 @@
 <h2 style="color:red">
 This page of the Synapse documentation is now deprecated. For up to date
 documentation on setting up or writing a password auth provider module, please see
-<a href="modules/index.md">this page</a>.
+<a href="modules.md">this page</a>.
 </h2>
 
 # Password auth provider modules

docs/sample_config.yaml

@@ -91,8 +91,6 @@ pid_file: DATADIR/homeserver.pid
 # Otherwise, it should be the URL to reach Synapse's client HTTP listener (see
 # 'listeners' below).
 #
-# Defaults to 'https://<server_name>/'.
-#
 #public_baseurl: https://example.com/
 
 # Uncomment the following to tell other servers to send federation traffic on
@@ -647,8 +645,8 @@ retention:
 #
 #federation_certificate_verification_whitelist:
 #  - lon.example.com
-#  - "*.domain.com"
-#  - "*.onion"
+#  - *.domain.com
+#  - *.onion
 
 # List of custom certificate authorities for federation traffic.
 #
@@ -1267,7 +1265,7 @@ oembed:
 # in on this server.
 #
 # (By default, no suggestion is made, so it is left up to the client.
-# This setting is ignored unless public_baseurl is also explicitly set.)
+# This setting is ignored unless public_baseurl is also set.)
 #
 #default_identity_server: https://matrix.org
 
@@ -1292,6 +1290,8 @@ oembed:
 # by the Matrix Identity Service API specification:
 # https://matrix.org/docs/spec/identity_service/latest
 #
+# If a delegate is specified, the config option public_baseurl must also be filled out.
+#
 account_threepid_delegates:
     #email: https://example.com     # Delegate email sending to example.com
     #msisdn: http://localhost:8090  # Delegate SMS sending to this local process
@@ -1981,10 +1981,11 @@ sso:
     # phishing attacks from evil.site. To avoid this, include a slash after the
     # hostname: "https://my.client/".
     #
-    # The login fallback page (used by clients that don't natively support the
-    # required login flows) is whitelisted in addition to any URLs in this list.
+    # If public_baseurl is set, then the login fallback page (used by clients
+    # that don't natively support the required login flows) is whitelisted in
+    # addition to any URLs in this list.
     #
-    # By default, this list contains only the login fallback page.
+    # By default, this list is empty.
     #
     #client_whitelist:
     #  - https://riot.im/develop
@@ -2039,12 +2040,6 @@ sso:
     #
     #algorithm: "provided-by-your-issuer"
 
-    # Name of the claim containing a unique identifier for the user.
-    #
-    # Optional, defaults to `sub`.
-    #
-    #subject_claim: "sub"
-
     # The issuer to validate the "iss" claim against.
     #
     # Optional, if provided the "iss" claim will be required and
@@ -2366,8 +2361,8 @@ user_directory:
     # indexes were (re)built was before Synapse 1.44, you'll have to
     # rebuild the indexes in order to search through all known users.
     # These indexes are built the first time Synapse starts; admins can
-    # manually trigger a rebuild via API following the instructions at
-    #     https://matrix-org.github.io/synapse/latest/usage/administration/admin_api/background_updates.html#run
+    # manually trigger a rebuild following the instructions at
+    #     https://matrix-org.github.io/synapse/latest/user_directory.html
     #
     # Uncomment to return search results containing all known users, even if that
     # user does not share a room with the requester.

docs/setup/installation.md

@@ -76,12 +76,6 @@ The fingerprint of the repository signing key (as shown by `gpg
 /usr/share/keyrings/matrix-org-archive-keyring.gpg`) is
 `AAF9AE843A7584B5A3E4CD2BCF45A512DE2DA058`.
 
-When installing with Debian packages, you might prefer to place files in
-`/etc/matrix-synapse/conf.d/` to override your configuration without editing
-the main configuration file at `/etc/matrix-synapse/homeserver.yaml`.
-By doing that, you won't be asked if you want to replace your configuration
-file when you upgrade the Debian package to a later version.
-
 ##### Downstream Debian packages
 
 We do not recommend using the packages from the default Debian `buster`

docs/systemd-with-workers/system/matrix-synapse-worker@.service

@@ -15,7 +15,7 @@ Type=notify
 NotifyAccess=main
 User=matrix-synapse
 WorkingDirectory=/var/lib/matrix-synapse
-EnvironmentFile=-/etc/default/matrix-synapse
+EnvironmentFile=/etc/default/matrix-synapse
 ExecStart=/opt/venvs/matrix-synapse/bin/python -m synapse.app.generic_worker --config-path=/etc/matrix-synapse/homeserver.yaml --config-path=/etc/matrix-synapse/conf.d/ --config-path=/etc/matrix-synapse/workers/%i.yaml
 ExecReload=/bin/kill -HUP $MAINPID
 Restart=always

docs/systemd-with-workers/system/matrix-synapse.service

@@ -10,7 +10,7 @@ Type=notify
 NotifyAccess=main
 User=matrix-synapse
 WorkingDirectory=/var/lib/matrix-synapse
-EnvironmentFile=-/etc/default/matrix-synapse
+EnvironmentFile=/etc/default/matrix-synapse
 ExecStartPre=/opt/venvs/matrix-synapse/bin/python -m synapse.app.homeserver --config-path=/etc/matrix-synapse/homeserver.yaml --config-path=/etc/matrix-synapse/conf.d/ --generate-keys
 ExecStart=/opt/venvs/matrix-synapse/bin/python -m synapse.app.homeserver --config-path=/etc/matrix-synapse/homeserver.yaml --config-path=/etc/matrix-synapse/conf.d/
 ExecReload=/bin/kill -HUP $MAINPID

docs/turn-howto.md

@@ -1,12 +1,12 @@
 # Overview
 
-This document explains how to enable VoIP relaying on your homeserver with
+This document explains how to enable VoIP relaying on your Home Server with
 TURN.
 
-The synapse Matrix homeserver supports integration with TURN server via the
+The synapse Matrix Home Server supports integration with TURN server via the
 [TURN server REST API](<https://tools.ietf.org/html/draft-uberti-behave-turn-rest-00>). This
-allows the homeserver to generate credentials that are valid for use on the
-TURN server through the use of a secret shared between the homeserver and the
+allows the Home Server to generate credentials that are valid for use on the
+TURN server through the use of a secret shared between the Home Server and the
 TURN server.
 
 The following sections describe how to install [coturn](<https://github.com/coturn/coturn>) (which implements the TURN REST API) and integrate it with synapse.
@@ -165,18 +165,18 @@ This will install and start a systemd service called `coturn`.
 
 ## Synapse setup
 
-Your homeserver configuration file needs the following extra keys:
+Your home server configuration file needs the following extra keys:
 
 1.  "`turn_uris`": This needs to be a yaml list of public-facing URIs
     for your TURN server to be given out to your clients. Add separate
     entries for each transport your TURN server supports.
 2.  "`turn_shared_secret`": This is the secret shared between your
-    homeserver and your TURN server, so you should set it to the same
+    Home server and your TURN server, so you should set it to the same
     string you used in turnserver.conf.
 3.  "`turn_user_lifetime`": This is the amount of time credentials
-    generated by your homeserver are valid for (in milliseconds).
+    generated by your Home Server are valid for (in milliseconds).
     Shorter times offer less potential for abuse at the expense of
-    increased traffic between web clients and your homeserver to
+    increased traffic between web clients and your home server to
     refresh credentials. The TURN REST API specification recommends
     one day (86400000).
 4.  "`turn_allow_guests`": Whether to allow guest users to use the
@@ -220,7 +220,7 @@ Here are a few things to try:
    anyone who has successfully set this up.
 
  * Check that you have opened your firewall to allow TCP and UDP traffic to the
-   TURN ports (normally 3478 and 5349).
+   TURN ports (normally 3478 and 5479).
 
  * Check that you have opened your firewall to allow UDP traffic to the UDP
    relay ports (49152-65535 by default).

docs/usage/administration/admin_api/background_updates.md

@@ -1,109 +0,0 @@
-# Background Updates API
-
-This API allows a server administrator to manage the background updates being
-run against the database.
-
-## Status
-
-This API gets the current status of the background updates.
-
-
-The API is:
-
-```
-GET /_synapse/admin/v1/background_updates/status
-```
-
-Returning:
-
-```json
-{
-    "enabled": true,
-    "current_updates": {
-        "<db_name>": {
-            "name": "<background_update_name>",
-            "total_item_count": 50,
-            "total_duration_ms": 10000.0,
-            "average_items_per_ms": 2.2,
-        },
-    }
-}
-```
-
-`enabled` whether the background updates are enabled or disabled.
-
-`db_name` the database name (usually Synapse is configured with a single database named 'master').
-
-For each update:
-
-`name` the name of the update.
-`total_item_count` total number of "items" processed (the meaning of 'items' depends on the update in question).
-`total_duration_ms` how long the background process has been running, not including time spent sleeping.
-`average_items_per_ms` how many items are processed per millisecond based on an exponential average.
-
-
-## Enabled
-
-This API allow pausing background updates.
-
-Background updates should *not* be paused for significant periods of time, as
-this can affect the performance of Synapse.
-
-*Note*: This won't persist over restarts.
-
-*Note*: This won't cancel any update query that is currently running. This is
-usually fine since most queries are short lived, except for `CREATE INDEX`
-background updates which won't be cancelled once started.
-
-
-The API is:
-
-```
-POST /_synapse/admin/v1/background_updates/enabled
-```
-
-with the following body:
-
-```json
-{
-    "enabled": false
-}
-```
-
-`enabled` sets whether the background updates are enabled or disabled.
-
-The API returns the `enabled` param.
-
-```json
-{
-    "enabled": false
-}
-```
-
-There is also a `GET` version which returns the `enabled` state.
-
-
-## Run
-
-This API schedules a specific background update to run. The job starts immediately after calling the API.
-
-
-The API is:
-
-```
-POST /_synapse/admin/v1/background_updates/start_job
-```
-
-with the following body:
-
-```json
-{
-    "job_name": "populate_stats_process_rooms"
-}
-```
-
-The following JSON body parameters are available:
-
-- `job_name` - A string which job to run. Valid values are:
-  - `populate_stats_process_rooms` - Recalculate the stats for all rooms.
-  - `regenerate_directory` - Recalculate the [user directory](../../../user_directory.md) if it is stale or out of sync.

docs/usage/configuration/user_authentication/single_sign_on/README.md

@@ -1,5 +0,0 @@
-# Single Sign-On
-
-Synapse supports single sign-on through the SAML, Open ID Connect or CAS protocols. 
-LDAP and other login methods are supported through first and third-party password
-auth provider modules.

docs/usage/configuration/user_authentication/single_sign_on/cas.md

@@ -1,8 +0,0 @@
-# CAS
-
-Synapse supports authenticating users via the [Central Authentication
-Service protocol](https://en.wikipedia.org/wiki/Central_Authentication_Service)
-(CAS) natively.
-
-Please see the `cas_config` and `sso` sections of the [Synapse configuration
-file](../../../configuration/homeserver_sample_config.md) for more details.

docs/usage/configuration/user_authentication/single_sign_on/saml.md

@@ -1,8 +0,0 @@
-# SAML
-
-Synapse supports authenticating users via the [Security Assertion
-Markup Language](https://en.wikipedia.org/wiki/Security_Assertion_Markup_Language)
-(SAML) protocol natively.
-
-Please see the `saml2_config` and `sso` sections of the [Synapse configuration
-file](../../../configuration/homeserver_sample_config.md) for more details.

docs/user_directory.md

@@ -6,9 +6,9 @@ on this particular server - i.e. ones which your account shares a room with, or
 who are present in a publicly viewable room present on the server.
 
 The directory info is stored in various tables, which can (typically after
-DB corruption) get stale or out of sync. If this happens, for now the
-solution to fix it is to use the [admin API](usage/administration/admin_api/background_updates.md#run)
-and execute the job `regenerate_directory`. This should then start a background task to
+DB corruption) get stale or out of sync.  If this happens, for now the
+solution to fix it is to execute the SQL [here](https://github.com/matrix-org/synapse/blob/master/synapse/storage/schema/main/delta/53/user_dir_populate.sql)
+and then restart synapse. This should then start a background task to
 flush the current tables and regenerate the directory.
 
 Data model

docs/workers.md

@@ -182,10 +182,10 @@ This worker can handle API requests matching the following regular
 expressions:
 
     # Sync requests
-    ^/_matrix/client/(v2_alpha|r0|v3)/sync$
-    ^/_matrix/client/(api/v1|v2_alpha|r0|v3)/events$
-    ^/_matrix/client/(api/v1|r0|v3)/initialSync$
-    ^/_matrix/client/(api/v1|r0|v3)/rooms/[^/]+/initialSync$
+    ^/_matrix/client/(v2_alpha|r0)/sync$
+    ^/_matrix/client/(api/v1|v2_alpha|r0)/events$
+    ^/_matrix/client/(api/v1|r0)/initialSync$
+    ^/_matrix/client/(api/v1|r0)/rooms/[^/]+/initialSync$
 
     # Federation requests
     ^/_matrix/federation/v1/event/
@@ -216,40 +216,40 @@ expressions:
     ^/_matrix/federation/v1/send/
 
     # Client API requests
-    ^/_matrix/client/(api/v1|r0|v3|unstable)/createRoom$
-    ^/_matrix/client/(api/v1|r0|v3|unstable)/publicRooms$
-    ^/_matrix/client/(api/v1|r0|v3|unstable)/rooms/.*/joined_members$
-    ^/_matrix/client/(api/v1|r0|v3|unstable)/rooms/.*/context/.*$
-    ^/_matrix/client/(api/v1|r0|v3|unstable)/rooms/.*/members$
-    ^/_matrix/client/(api/v1|r0|v3|unstable)/rooms/.*/state$
+    ^/_matrix/client/(api/v1|r0|unstable)/createRoom$
+    ^/_matrix/client/(api/v1|r0|unstable)/publicRooms$
+    ^/_matrix/client/(api/v1|r0|unstable)/rooms/.*/joined_members$
+    ^/_matrix/client/(api/v1|r0|unstable)/rooms/.*/context/.*$
+    ^/_matrix/client/(api/v1|r0|unstable)/rooms/.*/members$
+    ^/_matrix/client/(api/v1|r0|unstable)/rooms/.*/state$
     ^/_matrix/client/unstable/org.matrix.msc2946/rooms/.*/spaces$
     ^/_matrix/client/unstable/org.matrix.msc2946/rooms/.*/hierarchy$
     ^/_matrix/client/unstable/im.nheko.summary/rooms/.*/summary$
-    ^/_matrix/client/(api/v1|r0|v3|unstable)/account/3pid$
-    ^/_matrix/client/(api/v1|r0|v3|unstable)/devices$
-    ^/_matrix/client/(api/v1|r0|v3|unstable)/keys/query$
-    ^/_matrix/client/(api/v1|r0|v3|unstable)/keys/changes$
+    ^/_matrix/client/(api/v1|r0|unstable)/account/3pid$
+    ^/_matrix/client/(api/v1|r0|unstable)/devices$
+    ^/_matrix/client/(api/v1|r0|unstable)/keys/query$
+    ^/_matrix/client/(api/v1|r0|unstable)/keys/changes$
     ^/_matrix/client/versions$
-    ^/_matrix/client/(api/v1|r0|v3|unstable)/voip/turnServer$
-    ^/_matrix/client/(api/v1|r0|v3|unstable)/joined_groups$
-    ^/_matrix/client/(api/v1|r0|v3|unstable)/publicised_groups$
-    ^/_matrix/client/(api/v1|r0|v3|unstable)/publicised_groups/
-    ^/_matrix/client/(api/v1|r0|v3|unstable)/rooms/.*/event/
-    ^/_matrix/client/(api/v1|r0|v3|unstable)/joined_rooms$
-    ^/_matrix/client/(api/v1|r0|v3|unstable)/search$
+    ^/_matrix/client/(api/v1|r0|unstable)/voip/turnServer$
+    ^/_matrix/client/(api/v1|r0|unstable)/joined_groups$
+    ^/_matrix/client/(api/v1|r0|unstable)/publicised_groups$
+    ^/_matrix/client/(api/v1|r0|unstable)/publicised_groups/
+    ^/_matrix/client/(api/v1|r0|unstable)/rooms/.*/event/
+    ^/_matrix/client/(api/v1|r0|unstable)/joined_rooms$
+    ^/_matrix/client/(api/v1|r0|unstable)/search$
 
     # Registration/login requests
-    ^/_matrix/client/(api/v1|r0|v3|unstable)/login$
-    ^/_matrix/client/(r0|v3|unstable)/register$
+    ^/_matrix/client/(api/v1|r0|unstable)/login$
+    ^/_matrix/client/(r0|unstable)/register$
     ^/_matrix/client/unstable/org.matrix.msc3231/register/org.matrix.msc3231.login.registration_token/validity$
 
     # Event sending requests
-    ^/_matrix/client/(api/v1|r0|v3|unstable)/rooms/.*/redact
-    ^/_matrix/client/(api/v1|r0|v3|unstable)/rooms/.*/send
-    ^/_matrix/client/(api/v1|r0|v3|unstable)/rooms/.*/state/
-    ^/_matrix/client/(api/v1|r0|v3|unstable)/rooms/.*/(join|invite|leave|ban|unban|kick)$
-    ^/_matrix/client/(api/v1|r0|v3|unstable)/join/
-    ^/_matrix/client/(api/v1|r0|v3|unstable)/profile/
+    ^/_matrix/client/(api/v1|r0|unstable)/rooms/.*/redact
+    ^/_matrix/client/(api/v1|r0|unstable)/rooms/.*/send
+    ^/_matrix/client/(api/v1|r0|unstable)/rooms/.*/state/
+    ^/_matrix/client/(api/v1|r0|unstable)/rooms/.*/(join|invite|leave|ban|unban|kick)$
+    ^/_matrix/client/(api/v1|r0|unstable)/join/
+    ^/_matrix/client/(api/v1|r0|unstable)/profile/
 
 
 Additionally, the following REST endpoints can be handled for GET requests:
@@ -261,14 +261,14 @@ room must be routed to the same instance. Additionally, care must be taken to
 ensure that the purge history admin API is not used while pagination requests
 for the room are in flight:
 
-    ^/_matrix/client/(api/v1|r0|v3|unstable)/rooms/.*/messages$
+    ^/_matrix/client/(api/v1|r0|unstable)/rooms/.*/messages$
 
 Additionally, the following endpoints should be included if Synapse is configured
 to use SSO (you only need to include the ones for whichever SSO provider you're
 using):
 
     # for all SSO providers
-    ^/_matrix/client/(api/v1|r0|v3|unstable)/login/sso/redirect
+    ^/_matrix/client/(api/v1|r0|unstable)/login/sso/redirect
     ^/_synapse/client/pick_idp$
     ^/_synapse/client/pick_username
     ^/_synapse/client/new_user_consent$
@@ -281,7 +281,7 @@ using):
     ^/_synapse/client/saml2/authn_response$
 
     # CAS requests.
-    ^/_matrix/client/(api/v1|r0|v3|unstable)/login/cas/ticket$
+    ^/_matrix/client/(api/v1|r0|unstable)/login/cas/ticket$
 
 Ensure that all SSO logins go to a single process.
 For multiple workers not handling the SSO endpoints properly, see
@@ -465,7 +465,7 @@ Note that if a reverse proxy is used , then `/_matrix/media/` must be routed for
 Handles searches in the user directory. It can handle REST endpoints matching
 the following regular expressions:
 
-    ^/_matrix/client/(api/v1|r0|v3|unstable)/user_directory/search$
+    ^/_matrix/client/(api/v1|r0|unstable)/user_directory/search$
 
 When using this worker you must also set `update_user_directory: False` in the
 shared configuration file to stop the main synapse running background
@@ -477,12 +477,12 @@ Proxies some frequently-requested client endpoints to add caching and remove
 load from the main synapse. It can handle REST endpoints matching the following
 regular expressions:
 
-    ^/_matrix/client/(api/v1|r0|v3|unstable)/keys/upload
+    ^/_matrix/client/(api/v1|r0|unstable)/keys/upload
 
 If `use_presence` is False in the homeserver config, it can also handle REST
 endpoints matching the following regular expressions:
 
-    ^/_matrix/client/(api/v1|r0|v3|unstable)/presence/[^/]+/status
+    ^/_matrix/client/(api/v1|r0|unstable)/presence/[^/]+/status
 
 This "stub" presence handler will pass through `GET` request but make the
 `PUT` effectively a no-op.

mypy.ini

@@ -10,150 +10,92 @@ warn_unreachable = True
 local_partial_types = True
 no_implicit_optional = True
 
+# To find all folders that pass mypy you run:
+#
+#   find synapse/* -type d -not -name __pycache__ -exec bash -c "mypy '{}' > /dev/null"  \; -print
+
 files =
   scripts-dev/sign_json,
-  setup.py,
-  synapse/,
-  tests/
-
-# Note: Better exclusion syntax coming in mypy > 0.910
-# https://github.com/python/mypy/pull/11329
-#
-# For now, set the (?x) flag enable "verbose" regexes
-# https://docs.python.org/3/library/re.html#re.X
-exclude = (?x)
-  ^(
-   |synapse/storage/databases/__init__.py
-   |synapse/storage/databases/main/__init__.py
-   |synapse/storage/databases/main/account_data.py
-   |synapse/storage/databases/main/cache.py
-   |synapse/storage/databases/main/devices.py
-   |synapse/storage/databases/main/e2e_room_keys.py
-   |synapse/storage/databases/main/end_to_end_keys.py
-   |synapse/storage/databases/main/event_federation.py
-   |synapse/storage/databases/main/event_push_actions.py
-   |synapse/storage/databases/main/events_bg_updates.py
-   |synapse/storage/databases/main/events_worker.py
-   |synapse/storage/databases/main/group_server.py
-   |synapse/storage/databases/main/metrics.py
-   |synapse/storage/databases/main/monthly_active_users.py
-   |synapse/storage/databases/main/presence.py
-   |synapse/storage/databases/main/purge_events.py
-   |synapse/storage/databases/main/push_rule.py
-   |synapse/storage/databases/main/receipts.py
-   |synapse/storage/databases/main/room.py
-   |synapse/storage/databases/main/roommember.py
-   |synapse/storage/databases/main/search.py
-   |synapse/storage/databases/main/state.py
-   |synapse/storage/databases/main/stats.py
-   |synapse/storage/databases/main/transactions.py
-   |synapse/storage/databases/main/user_directory.py
-   |synapse/storage/schema/
-
-   |tests/api/test_auth.py
-   |tests/api/test_ratelimiting.py
-   |tests/app/test_openid_listener.py
-   |tests/appservice/test_scheduler.py
-   |tests/config/test_cache.py
-   |tests/config/test_tls.py
-   |tests/crypto/test_keyring.py
-   |tests/events/test_presence_router.py
-   |tests/events/test_utils.py
-   |tests/federation/test_federation_catch_up.py
-   |tests/federation/test_federation_sender.py
-   |tests/federation/test_federation_server.py
-   |tests/federation/transport/test_knocking.py
-   |tests/federation/transport/test_server.py
-   |tests/handlers/test_cas.py
-   |tests/handlers/test_directory.py
-   |tests/handlers/test_e2e_keys.py
-   |tests/handlers/test_federation.py
-   |tests/handlers/test_oidc.py
-   |tests/handlers/test_presence.py
-   |tests/handlers/test_profile.py
-   |tests/handlers/test_saml.py
-   |tests/handlers/test_typing.py
-   |tests/http/federation/test_matrix_federation_agent.py
-   |tests/http/federation/test_srv_resolver.py
-   |tests/http/test_fedclient.py
-   |tests/http/test_proxyagent.py
-   |tests/http/test_servlet.py
-   |tests/http/test_site.py
-   |tests/logging/__init__.py
-   |tests/logging/test_terse_json.py
-   |tests/module_api/test_api.py
-   |tests/push/test_email.py
-   |tests/push/test_http.py
-   |tests/push/test_presentable_names.py
-   |tests/push/test_push_rule_evaluator.py
-   |tests/rest/admin/test_admin.py
-   |tests/rest/admin/test_device.py
-   |tests/rest/admin/test_media.py
-   |tests/rest/admin/test_server_notice.py
-   |tests/rest/admin/test_user.py
-   |tests/rest/admin/test_username_available.py
-   |tests/rest/client/test_account.py
-   |tests/rest/client/test_events.py
-   |tests/rest/client/test_filter.py
-   |tests/rest/client/test_groups.py
-   |tests/rest/client/test_register.py
-   |tests/rest/client/test_report_event.py
-   |tests/rest/client/test_rooms.py
-   |tests/rest/client/test_third_party_rules.py
-   |tests/rest/client/test_transactions.py
-   |tests/rest/client/test_typing.py
-   |tests/rest/client/utils.py
-   |tests/rest/key/v2/test_remote_key_resource.py
-   |tests/rest/media/v1/test_base.py
-   |tests/rest/media/v1/test_media_storage.py
-   |tests/rest/media/v1/test_url_preview.py
-   |tests/scripts/test_new_matrix_user.py
-   |tests/server.py
-   |tests/server_notices/test_resource_limits_server_notices.py
-   |tests/state/test_v2.py
-   |tests/storage/test_account_data.py
-   |tests/storage/test_appservice.py
-   |tests/storage/test_background_update.py
-   |tests/storage/test_base.py
-   |tests/storage/test_client_ips.py
-   |tests/storage/test_database.py
-   |tests/storage/test_event_federation.py
-   |tests/storage/test_id_generators.py
-   |tests/storage/test_roommember.py
-   |tests/test_metrics.py
-   |tests/test_phone_home.py
-   |tests/test_server.py
-   |tests/test_state.py
-   |tests/test_terms_auth.py
-   |tests/test_visibility.py
-   |tests/unittest.py
-   |tests/util/caches/test_cached_call.py
-   |tests/util/caches/test_deferred_cache.py
-   |tests/util/caches/test_descriptors.py
-   |tests/util/caches/test_response_cache.py
-   |tests/util/caches/test_ttlcache.py
-   |tests/util/test_async_helpers.py
-   |tests/util/test_batching_queue.py
-   |tests/util/test_dict_cache.py
-   |tests/util/test_expiring_cache.py
-   |tests/util/test_file_consumer.py
-   |tests/util/test_linearizer.py
-   |tests/util/test_logcontext.py
-   |tests/util/test_lrucache.py
-   |tests/util/test_rwlock.py
-   |tests/util/test_wheel_timer.py
-   |tests/utils.py
-   )$
+  synapse/__init__.py,
+  synapse/api,
+  synapse/appservice,
+  synapse/config,
+  synapse/crypto,
+  synapse/event_auth.py,
+  synapse/events/builder.py,
+  synapse/events/presence_router.py,
+  synapse/events/snapshot.py,
+  synapse/events/spamcheck.py,
+  synapse/events/third_party_rules.py,
+  synapse/events/utils.py,
+  synapse/events/validator.py,
+  synapse/federation,
+  synapse/groups,
+  synapse/handlers,
+  synapse/http,
+  synapse/logging,
+  synapse/metrics,
+  synapse/module_api,
+  synapse/notifier.py,
+  synapse/push,
+  synapse/replication,
+  synapse/rest,
+  synapse/server.py,
+  synapse/server_notices,
+  synapse/spam_checker_api,
+  synapse/state,
+  synapse/storage/__init__.py,
+  synapse/storage/_base.py,
+  synapse/storage/background_updates.py,
+  synapse/storage/databases/main/appservice.py,
+  synapse/storage/databases/main/client_ips.py,
+  synapse/storage/databases/main/events.py,
+  synapse/storage/databases/main/keys.py,
+  synapse/storage/databases/main/pusher.py,
+  synapse/storage/databases/main/registration.py,
+  synapse/storage/databases/main/relations.py,
+  synapse/storage/databases/main/session.py,
+  synapse/storage/databases/main/stream.py,
+  synapse/storage/databases/main/ui_auth.py,
+  synapse/storage/databases/state,
+  synapse/storage/database.py,
+  synapse/storage/engines,
+  synapse/storage/keys.py,
+  synapse/storage/persist_events.py,
+  synapse/storage/prepare_database.py,
+  synapse/storage/purge_events.py,
+  synapse/storage/push_rule.py,
+  synapse/storage/relations.py,
+  synapse/storage/roommember.py,
+  synapse/storage/state.py,
+  synapse/storage/types.py,
+  synapse/storage/util,
+  synapse/streams,
+  synapse/types.py,
+  synapse/util,
+  synapse/visibility.py,
+  tests/replication,
+  tests/test_event_auth.py,
+  tests/test_utils,
+  tests/handlers/test_password_providers.py,
+  tests/handlers/test_room.py,
+  tests/handlers/test_room_summary.py,
+  tests/handlers/test_send_email.py,
+  tests/handlers/test_sync.py,
+  tests/handlers/test_user_directory.py,
+  tests/rest/client/test_login.py,
+  tests/rest/client/test_auth.py,
+  tests/rest/client/test_relations.py,
+  tests/rest/media/v1/test_filepath.py,
+  tests/rest/media/v1/test_oembed.py,
+  tests/storage/test_state.py,
+  tests/storage/test_user_directory.py,
+  tests/util/test_itertools.py,
+  tests/util/test_stream_change_cache.py
 
 [mypy-synapse.api.*]
 disallow_untyped_defs = True
 
-[mypy-synapse.app.*]
-disallow_untyped_defs = True
-
-[mypy-synapse.config._base]
-disallow_untyped_defs = True
-
 [mypy-synapse.crypto.*]
 disallow_untyped_defs = True
 
@@ -163,9 +105,6 @@ disallow_untyped_defs = True
 [mypy-synapse.handlers.*]
 disallow_untyped_defs = True
 
-[mypy-synapse.metrics.*]
-disallow_untyped_defs = True
-
 [mypy-synapse.push.*]
 disallow_untyped_defs = True
 
@@ -181,45 +120,105 @@ disallow_untyped_defs = True
 [mypy-synapse.storage.databases.main.client_ips]
 disallow_untyped_defs = True
 
-[mypy-synapse.storage.databases.main.directory]
-disallow_untyped_defs = True
-
-[mypy-synapse.storage.databases.main.room_batch]
-disallow_untyped_defs = True
-
-[mypy-synapse.storage.databases.main.profile]
-disallow_untyped_defs = True
-
-[mypy-synapse.storage.databases.main.state_deltas]
-disallow_untyped_defs = True
-
-[mypy-synapse.storage.databases.main.user_erasure_store]
-disallow_untyped_defs = True
-
 [mypy-synapse.storage.util.*]
 disallow_untyped_defs = True
 
 [mypy-synapse.streams.*]
 disallow_untyped_defs = True
 
-[mypy-synapse.util.*]
+[mypy-synapse.util.batching_queue]
 disallow_untyped_defs = True
 
-[mypy-synapse.util.caches.treecache]
-disallow_untyped_defs = False
+[mypy-synapse.util.caches.cached_call]
+disallow_untyped_defs = True
+
+[mypy-synapse.util.caches.dictionary_cache]
+disallow_untyped_defs = True
+
+[mypy-synapse.util.caches.lrucache]
+disallow_untyped_defs = True
+
+[mypy-synapse.util.caches.response_cache]
+disallow_untyped_defs = True
+
+[mypy-synapse.util.caches.stream_change_cache]
+disallow_untyped_defs = True
+
+[mypy-synapse.util.caches.ttl_cache]
+disallow_untyped_defs = True
+
+[mypy-synapse.util.daemonize]
+disallow_untyped_defs = True
+
+[mypy-synapse.util.file_consumer]
+disallow_untyped_defs = True
+
+[mypy-synapse.util.frozenutils]
+disallow_untyped_defs = True
+
+[mypy-synapse.util.hash]
+disallow_untyped_defs = True
+
+[mypy-synapse.util.httpresourcetree]
+disallow_untyped_defs = True
+
+[mypy-synapse.util.iterutils]
+disallow_untyped_defs = True
+
+[mypy-synapse.util.linked_list]
+disallow_untyped_defs = True
+
+[mypy-synapse.util.logcontext]
+disallow_untyped_defs = True
+
+[mypy-synapse.util.logformatter]
+disallow_untyped_defs = True
+
+[mypy-synapse.util.macaroons]
+disallow_untyped_defs = True
+
+[mypy-synapse.util.manhole]
+disallow_untyped_defs = True
+
+[mypy-synapse.util.module_loader]
+disallow_untyped_defs = True
+
+[mypy-synapse.util.msisdn]
+disallow_untyped_defs = True
+
+[mypy-synapse.util.patch_inline_callbacks]
+disallow_untyped_defs = True
+
+[mypy-synapse.util.ratelimitutils]
+disallow_untyped_defs = True
+
+[mypy-synapse.util.retryutils]
+disallow_untyped_defs = True
+
+[mypy-synapse.util.rlimit]
+disallow_untyped_defs = True
+
+[mypy-synapse.util.stringutils]
+disallow_untyped_defs = True
+
+[mypy-synapse.util.templates]
+disallow_untyped_defs = True
+
+[mypy-synapse.util.threepids]
+disallow_untyped_defs = True
+
+[mypy-synapse.util.wheel_timer]
+disallow_untyped_defs = True
+
+[mypy-synapse.util.versionstring]
+disallow_untyped_defs = True
 
 [mypy-tests.handlers.test_user_directory]
 disallow_untyped_defs = True
 
-[mypy-tests.storage.test_profile]
-disallow_untyped_defs = True
-
 [mypy-tests.storage.test_user_directory]
 disallow_untyped_defs = True
 
-[mypy-tests.rest.client.test_directory]
-disallow_untyped_defs = True
-
 ;; Dependencies without annotations
 ;; Before ignoring a module, check to see if type stubs are available.
 ;; The `typeshed` project maintains stubs here:
@@ -279,9 +278,6 @@ ignore_missing_imports = True
 [mypy-opentracing]
 ignore_missing_imports = True
 
-[mypy-parameterized.*]
-ignore_missing_imports = True
-
 [mypy-phonenumbers.*]
 ignore_missing_imports = True
 

scripts-dev/complement.sh

@@ -24,7 +24,7 @@
 set -e
 
 # Change to the repository root
-cd "$(dirname $0)/.."
+cd "$(dirname "$0")/.."
 
 # Check for a user-specified Complement checkout
 if [[ -z "$COMPLEMENT_DIR" ]]; then
@@ -61,8 +61,8 @@ cd "$COMPLEMENT_DIR"
 EXTRA_COMPLEMENT_ARGS=""
 if [[ -n "$1" ]]; then
   # A test name regex has been set, supply it to Complement
-  EXTRA_COMPLEMENT_ARGS+="-run $1 "
+  EXTRA_COMPLEMENT_ARGS=(-run "$1")
 fi
 
 # Run the tests!
-go test -v -tags synapse_blacklist,msc2946,msc3083,msc2403 -count=1 $EXTRA_COMPLEMENT_ARGS ./tests/...
+go test -v -tags synapse_blacklist,msc2946,msc3083,msc2403,msc2716 -count=1 "${EXTRA_COMPLEMENT_ARGS[@]}" ./tests/...

scripts/synapse_port_db

@@ -43,7 +43,6 @@ from synapse.storage.databases.main.end_to_end_keys import EndToEndKeyBackground
 from synapse.storage.databases.main.events_bg_updates import (
     EventsBackgroundUpdatesStore,
 )
-from synapse.storage.databases.main.group_server import GroupServerWorkerStore
 from synapse.storage.databases.main.media_repository import (
     MediaRepositoryBackgroundUpdateStore,
 )
@@ -182,7 +181,6 @@ class Store(
     StatsStore,
     PusherWorkerStore,
     PresenceBackgroundUpdateStore,
-    GroupServerWorkerStore,
 ):
     def execute(self, f, *args, **kwargs):
         return self.db_pool.runInteraction(f.__name__, f, *args, **kwargs)

setup.py

@@ -17,7 +17,6 @@
 # limitations under the License.
 import glob
 import os
-from typing import Any, Dict
 
 from setuptools import Command, find_packages, setup
 
@@ -50,6 +49,8 @@ here = os.path.abspath(os.path.dirname(__file__))
 # [1]: http://tox.readthedocs.io/en/2.5.0/example/basic.html#integration-with-setup-py-test-command
 # [2]: https://pypi.python.org/pypi/setuptools_trial
 class TestCommand(Command):
+    user_options = []
+
     def initialize_options(self):
         pass
 
@@ -74,7 +75,7 @@ def read_file(path_segments):
 
 def exec_file(path_segments):
     """Execute a single python file to get the variables defined in it"""
-    result: Dict[str, Any] = {}
+    result = {}
     code = read_file(path_segments)
     exec(code, result)
     return result
@@ -110,7 +111,6 @@ CONDITIONAL_REQUIREMENTS["mypy"] = [
     "types-Pillow>=8.3.4",
     "types-pyOpenSSL>=20.0.7",
     "types-PyYAML>=5.4.10",
-    "types-requests>=2.26.0",
     "types-setuptools>=57.4.0",
 ]
 
@@ -132,9 +132,6 @@ CONDITIONAL_REQUIREMENTS["dev"] = (
         "GitPython==3.1.14",
         "commonmark==0.9.1",
         "pygithub==1.55",
-        # The following are executed as commands by the release script.
-        "twine",
-        "towncrier",
     ]
 )
 

synapse/__init__.py

@@ -47,7 +47,7 @@ try:
 except ImportError:
     pass
 
-__version__ = "1.48.0"
+__version__ = "1.46.0rc1"
 
 if bool(os.environ.get("SYNAPSE_TEST_PATCH_LOG_CONTEXTS", False)):
     # We import here so that we don't have to install a bunch of deps when

synapse/_scripts/register_new_matrix_user.py

@@ -1,6 +1,5 @@
 # Copyright 2015, 2016 OpenMarket Ltd
 # Copyright 2018 New Vector
-# Copyright 2021 The Matrix.org Foundation C.I.C.
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -20,23 +19,22 @@ import hashlib
 import hmac
 import logging
 import sys
-from typing import Callable, Optional
 
 import requests as _requests
 import yaml
 
 
 def request_registration(
-    user: str,
-    password: str,
-    server_location: str,
-    shared_secret: str,
-    admin: bool = False,
-    user_type: Optional[str] = None,
+    user,
+    password,
+    server_location,
+    shared_secret,
+    admin=False,
+    user_type=None,
     requests=_requests,
-    _print: Callable[[str], None] = print,
-    exit: Callable[[int], None] = sys.exit,
-) -> None:
+    _print=print,
+    exit=sys.exit,
+):
 
     url = "%s/_synapse/admin/v1/register" % (server_location.rstrip("/"),)
 
@@ -67,13 +65,13 @@ def request_registration(
         mac.update(b"\x00")
         mac.update(user_type.encode("utf8"))
 
-    hex_mac = mac.hexdigest()
+    mac = mac.hexdigest()
 
     data = {
         "nonce": nonce,
         "username": user,
         "password": password,
-        "mac": hex_mac,
+        "mac": mac,
         "admin": admin,
         "user_type": user_type,
     }
@@ -93,17 +91,10 @@ def request_registration(
     _print("Success!")
 
 
-def register_new_user(
-    user: str,
-    password: str,
-    server_location: str,
-    shared_secret: str,
-    admin: Optional[bool],
-    user_type: Optional[str],
-) -> None:
+def register_new_user(user, password, server_location, shared_secret, admin, user_type):
     if not user:
         try:
-            default_user: Optional[str] = getpass.getuser()
+            default_user = getpass.getuser()
         except Exception:
             default_user = None
 
@@ -132,8 +123,8 @@ def register_new_user(
             sys.exit(1)
 
     if admin is None:
-        admin_inp = input("Make admin [no]: ")
-        if admin_inp in ("y", "yes", "true"):
+        admin = input("Make admin [no]: ")
+        if admin in ("y", "yes", "true"):
             admin = True
         else:
             admin = False
@@ -143,7 +134,7 @@ def register_new_user(
     )
 
 
-def main() -> None:
+def main():
 
     logging.captureWarnings(True)
 

synapse/_scripts/review_recent_signups.py

@@ -92,7 +92,7 @@ def get_recent_users(txn: LoggingTransaction, since_ms: int) -> List[UserInfo]:
     return user_infos
 
 
-def main() -> None:
+def main():
     parser = argparse.ArgumentParser()
     parser.add_argument(
         "-c",
@@ -142,8 +142,7 @@ def main() -> None:
     engine = create_engine(database_config.config)
 
     with make_conn(database_config, engine, "review_recent_signups") as db_conn:
-        # This generates a type of Cursor, not LoggingTransaction.
-        user_infos = get_recent_users(db_conn.cursor(), since_ms)  # type: ignore[arg-type]
+        user_infos = get_recent_users(db_conn.cursor(), since_ms)
 
     for user_info in user_infos:
         if exclude_users_with_email and user_info.emails:

synapse/api/filtering.py

@@ -1,7 +1,7 @@
 # Copyright 2015, 2016 OpenMarket Ltd
 # Copyright 2017 Vector Creations Ltd
 # Copyright 2018-2019 New Vector Ltd
-# Copyright 2019-2021 The Matrix.org Foundation C.I.C.
+# Copyright 2019 The Matrix.org Foundation C.I.C.
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -86,9 +86,6 @@ ROOM_EVENT_FILTER_SCHEMA = {
         # cf https://github.com/matrix-org/matrix-doc/pull/2326
         "org.matrix.labels": {"type": "array", "items": {"type": "string"}},
         "org.matrix.not_labels": {"type": "array", "items": {"type": "string"}},
-        # MSC3440, filtering by event relations.
-        "io.element.relation_senders": {"type": "array", "items": {"type": "string"}},
-        "io.element.relation_types": {"type": "array", "items": {"type": "string"}},
     },
 }
 
@@ -149,16 +146,14 @@ def matrix_user_id_validator(user_id_str: str) -> UserID:
 
 class Filtering:
     def __init__(self, hs: "HomeServer"):
-        self._hs = hs
+        super().__init__()
         self.store = hs.get_datastore()
 
-        self.DEFAULT_FILTER_COLLECTION = FilterCollection(hs, {})
-
     async def get_user_filter(
         self, user_localpart: str, filter_id: Union[int, str]
     ) -> "FilterCollection":
         result = await self.store.get_user_filter(user_localpart, filter_id)
-        return FilterCollection(self._hs, result)
+        return FilterCollection(result)
 
     def add_user_filter(
         self, user_localpart: str, user_filter: JsonDict
@@ -196,22 +191,21 @@ FilterEvent = TypeVar("FilterEvent", EventBase, UserPresenceState, JsonDict)
 
 
 class FilterCollection:
-    def __init__(self, hs: "HomeServer", filter_json: JsonDict):
+    def __init__(self, filter_json: JsonDict):
         self._filter_json = filter_json
 
         room_filter_json = self._filter_json.get("room", {})
 
         self._room_filter = Filter(
-            hs,
-            {k: v for k, v in room_filter_json.items() if k in ("rooms", "not_rooms")},
+            {k: v for k, v in room_filter_json.items() if k in ("rooms", "not_rooms")}
         )
 
-        self._room_timeline_filter = Filter(hs, room_filter_json.get("timeline", {}))
-        self._room_state_filter = Filter(hs, room_filter_json.get("state", {}))
-        self._room_ephemeral_filter = Filter(hs, room_filter_json.get("ephemeral", {}))
-        self._room_account_data = Filter(hs, room_filter_json.get("account_data", {}))
-        self._presence_filter = Filter(hs, filter_json.get("presence", {}))
-        self._account_data = Filter(hs, filter_json.get("account_data", {}))
+        self._room_timeline_filter = Filter(room_filter_json.get("timeline", {}))
+        self._room_state_filter = Filter(room_filter_json.get("state", {}))
+        self._room_ephemeral_filter = Filter(room_filter_json.get("ephemeral", {}))
+        self._room_account_data = Filter(room_filter_json.get("account_data", {}))
+        self._presence_filter = Filter(filter_json.get("presence", {}))
+        self._account_data = Filter(filter_json.get("account_data", {}))
 
         self.include_leave = filter_json.get("room", {}).get("include_leave", False)
         self.event_fields = filter_json.get("event_fields", [])
@@ -238,37 +232,25 @@ class FilterCollection:
     def include_redundant_members(self) -> bool:
         return self._room_state_filter.include_redundant_members
 
-    async def filter_presence(
+    def filter_presence(
         self, events: Iterable[UserPresenceState]
     ) -> List[UserPresenceState]:
-        return await self._presence_filter.filter(events)
+        return self._presence_filter.filter(events)
 
-    async def filter_account_data(self, events: Iterable[JsonDict]) -> List[JsonDict]:
-        return await self._account_data.filter(events)
+    def filter_account_data(self, events: Iterable[JsonDict]) -> List[JsonDict]:
+        return self._account_data.filter(events)
 
-    async def filter_room_state(self, events: Iterable[EventBase]) -> List[EventBase]:
-        return await self._room_state_filter.filter(
-            await self._room_filter.filter(events)
-        )
+    def filter_room_state(self, events: Iterable[EventBase]) -> List[EventBase]:
+        return self._room_state_filter.filter(self._room_filter.filter(events))
 
-    async def filter_room_timeline(
-        self, events: Iterable[EventBase]
-    ) -> List[EventBase]:
-        return await self._room_timeline_filter.filter(
-            await self._room_filter.filter(events)
-        )
+    def filter_room_timeline(self, events: Iterable[EventBase]) -> List[EventBase]:
+        return self._room_timeline_filter.filter(self._room_filter.filter(events))
 
-    async def filter_room_ephemeral(self, events: Iterable[JsonDict]) -> List[JsonDict]:
-        return await self._room_ephemeral_filter.filter(
-            await self._room_filter.filter(events)
-        )
+    def filter_room_ephemeral(self, events: Iterable[JsonDict]) -> List[JsonDict]:
+        return self._room_ephemeral_filter.filter(self._room_filter.filter(events))
 
-    async def filter_room_account_data(
-        self, events: Iterable[JsonDict]
-    ) -> List[JsonDict]:
-        return await self._room_account_data.filter(
-            await self._room_filter.filter(events)
-        )
+    def filter_room_account_data(self, events: Iterable[JsonDict]) -> List[JsonDict]:
+        return self._room_account_data.filter(self._room_filter.filter(events))
 
     def blocks_all_presence(self) -> bool:
         return (
@@ -292,9 +274,7 @@ class FilterCollection:
 
 
 class Filter:
-    def __init__(self, hs: "HomeServer", filter_json: JsonDict):
-        self._hs = hs
-        self._store = hs.get_datastore()
+    def __init__(self, filter_json: JsonDict):
         self.filter_json = filter_json
 
         self.limit = filter_json.get("limit", 10)
@@ -317,20 +297,6 @@ class Filter:
         self.labels = filter_json.get("org.matrix.labels", None)
         self.not_labels = filter_json.get("org.matrix.not_labels", [])
 
-        # Ideally these would be rejected at the endpoint if they were provided
-        # and not supported, but that would involve modifying the JSON schema
-        # based on the homeserver configuration.
-        if hs.config.experimental.msc3440_enabled:
-            self.relation_senders = self.filter_json.get(
-                "io.element.relation_senders", None
-            )
-            self.relation_types = self.filter_json.get(
-                "io.element.relation_types", None
-            )
-        else:
-            self.relation_senders = None
-            self.relation_types = None
-
     def filters_all_types(self) -> bool:
         return "*" in self.not_types
 
@@ -340,7 +306,7 @@ class Filter:
     def filters_all_rooms(self) -> bool:
         return "*" in self.not_rooms
 
-    def _check(self, event: FilterEvent) -> bool:
+    def check(self, event: FilterEvent) -> bool:
         """Checks whether the filter matches the given event.
 
         Args:
@@ -454,30 +420,8 @@ class Filter:
 
         return room_ids
 
-    async def _check_event_relations(
-        self, events: Iterable[FilterEvent]
-    ) -> List[FilterEvent]:
-        # The event IDs to check, mypy doesn't understand the ifinstance check.
-        event_ids = [event.event_id for event in events if isinstance(event, EventBase)]  # type: ignore[attr-defined]
-        event_ids_to_keep = set(
-            await self._store.events_have_relations(
-                event_ids, self.relation_senders, self.relation_types
-            )
-        )
-
-        return [
-            event
-            for event in events
-            if not isinstance(event, EventBase) or event.event_id in event_ids_to_keep
-        ]
-
-    async def filter(self, events: Iterable[FilterEvent]) -> List[FilterEvent]:
-        result = [event for event in events if self._check(event)]
-
-        if self.relation_senders or self.relation_types:
-            return await self._check_event_relations(result)
-
-        return result
+    def filter(self, events: Iterable[FilterEvent]) -> List[FilterEvent]:
+        return list(filter(self.check, events))
 
     def with_room_ids(self, room_ids: Iterable[str]) -> "Filter":
         """Returns a new filter with the given room IDs appended.
@@ -489,7 +433,7 @@ class Filter:
             filter: A new filter including the given rooms and the old
                     filter's rooms.
         """
-        newFilter = Filter(self._hs, self.filter_json)
+        newFilter = Filter(self.filter_json)
         newFilter.rooms += room_ids
         return newFilter
 
@@ -500,3 +444,6 @@ def _matches_wildcard(actual_value: Optional[str], filter_value: str) -> bool:
         return actual_value.startswith(type_prefix)
     else:
         return actual_value == filter_value
+
+
+DEFAULT_FILTER_COLLECTION = FilterCollection({})

synapse/api/urls.py

@@ -30,8 +30,7 @@ FEDERATION_UNSTABLE_PREFIX = FEDERATION_PREFIX + "/unstable"
 STATIC_PREFIX = "/_matrix/static"
 WEB_CLIENT_PREFIX = "/_matrix/client"
 SERVER_KEY_V2_PREFIX = "/_matrix/key/v2"
-MEDIA_R0_PREFIX = "/_matrix/media/r0"
-MEDIA_V3_PREFIX = "/_matrix/media/v3"
+MEDIA_PREFIX = "/_matrix/media/r0"
 LEGACY_MEDIA_PREFIX = "/_matrix/media/v1"
 
 
@@ -39,6 +38,9 @@ class ConsentURIBuilder:
     def __init__(self, hs_config: HomeServerConfig):
         if hs_config.key.form_secret is None:
             raise ConfigError("form_secret not set in config")
+        if hs_config.server.public_baseurl is None:
+            raise ConfigError("public_baseurl not set in config")
+
         self._hmac_secret = hs_config.key.form_secret.encode("utf-8")
         self._public_baseurl = hs_config.server.public_baseurl
 

synapse/app/__init__.py

@@ -13,7 +13,6 @@
 # limitations under the License.
 import logging
 import sys
-from typing import Container
 
 from synapse import python_dependencies  # noqa: E402
 
@@ -28,9 +27,7 @@ except python_dependencies.DependencyException as e:
     sys.exit(1)
 
 
-def check_bind_error(
-    e: Exception, address: str, bind_addresses: Container[str]
-) -> None:
+def check_bind_error(e, address, bind_addresses):
     """
     This method checks an exception occurred while binding on 0.0.0.0.
     If :: is specified in the bind addresses a warning is shown.
@@ -41,9 +38,9 @@ def check_bind_error(
     When binding on 0.0.0.0 after :: this can safely be ignored.
 
     Args:
-        e: Exception that was caught.
-        address: Address on which binding was attempted.
-        bind_addresses: Addresses on which the service listens.
+        e (Exception): Exception that was caught.
+        address (str): Address on which binding was attempted.
+        bind_addresses (list): Addresses on which the service listens.
     """
     if address == "0.0.0.0" and "::" in bind_addresses:
         logger.warning(

synapse/app/_base.py

@@ -22,27 +22,13 @@ import socket
 import sys
 import traceback
 import warnings
-from typing import (
-    TYPE_CHECKING,
-    Any,
-    Awaitable,
-    Callable,
-    Collection,
-    Dict,
-    Iterable,
-    List,
-    NoReturn,
-    Tuple,
-    cast,
-)
+from typing import TYPE_CHECKING, Awaitable, Callable, Iterable
 
 from cryptography.utils import CryptographyDeprecationWarning
+from typing_extensions import NoReturn
 
 import twisted
-from twisted.internet import defer, error, reactor as _reactor
-from twisted.internet.interfaces import IOpenSSLContextFactory, IReactorSSL, IReactorTCP
-from twisted.internet.protocol import ServerFactory
-from twisted.internet.tcp import Port
+from twisted.internet import defer, error, reactor
 from twisted.logger import LoggingFile, LogLevel
 from twisted.protocols.tls import TLSMemoryBIOFactory
 from twisted.python.threadpool import ThreadPool
@@ -62,7 +48,6 @@ from synapse.logging.context import PreserveLoggingContext
 from synapse.metrics import register_threadpool
 from synapse.metrics.background_process_metrics import wrap_as_background_process
 from synapse.metrics.jemalloc import setup_jemalloc_stats
-from synapse.types import ISynapseReactor
 from synapse.util.caches.lrucache import setup_expire_lru_cache_entries
 from synapse.util.daemonize import daemonize_process
 from synapse.util.gai_resolver import GAIResolver
@@ -72,44 +57,33 @@ from synapse.util.versionstring import get_version_string
 if TYPE_CHECKING:
     from synapse.server import HomeServer
 
-# Twisted injects the global reactor to make it easier to import, this confuses
-# mypy which thinks it is a module. Tell it that it a more proper type.
-reactor = cast(ISynapseReactor, _reactor)
-
-
 logger = logging.getLogger(__name__)
 
 # list of tuples of function, args list, kwargs dict
-_sighup_callbacks: List[
-    Tuple[Callable[..., None], Tuple[Any, ...], Dict[str, Any]]
-] = []
+_sighup_callbacks = []
 
 
-def register_sighup(func: Callable[..., None], *args: Any, **kwargs: Any) -> None:
+def register_sighup(func, *args, **kwargs):
     """
     Register a function to be called when a SIGHUP occurs.
 
     Args:
-        func: Function to be called when sent a SIGHUP signal.
+        func (function): Function to be called when sent a SIGHUP signal.
         *args, **kwargs: args and kwargs to be passed to the target function.
     """
     _sighup_callbacks.append((func, args, kwargs))
 
 
-def start_worker_reactor(
-    appname: str,
-    config: HomeServerConfig,
-    run_command: Callable[[], None] = reactor.run,
-) -> None:
+def start_worker_reactor(appname, config, run_command=reactor.run):
     """Run the reactor in the main process
 
     Daemonizes if necessary, and then configures some resources, before starting
     the reactor. Pulls configuration from the 'worker' settings in 'config'.
 
     Args:
-        appname: application name which will be sent to syslog
-        config: config object
-        run_command: callable that actually runs the reactor
+        appname (str): application name which will be sent to syslog
+        config (synapse.config.Config): config object
+        run_command (Callable[]): callable that actually runs the reactor
     """
 
     logger = logging.getLogger(config.worker.worker_app)
@@ -127,32 +101,32 @@ def start_worker_reactor(
 
 
 def start_reactor(
-    appname: str,
-    soft_file_limit: int,
-    gc_thresholds: Tuple[int, int, int],
-    pid_file: str,
-    daemonize: bool,
-    print_pidfile: bool,
-    logger: logging.Logger,
-    run_command: Callable[[], None] = reactor.run,
-) -> None:
+    appname,
+    soft_file_limit,
+    gc_thresholds,
+    pid_file,
+    daemonize,
+    print_pidfile,
+    logger,
+    run_command=reactor.run,
+):
     """Run the reactor in the main process
 
     Daemonizes if necessary, and then configures some resources, before starting
     the reactor
 
     Args:
-        appname: application name which will be sent to syslog
-        soft_file_limit:
+        appname (str): application name which will be sent to syslog
+        soft_file_limit (int):
         gc_thresholds:
-        pid_file: name of pid file to write to if daemonize is True
-        daemonize: true to run the reactor in a background process
-        print_pidfile: whether to print the pid file, if daemonize is True
-        logger: logger instance to pass to Daemonize
-        run_command: callable that actually runs the reactor
+        pid_file (str): name of pid file to write to if daemonize is True
+        daemonize (bool): true to run the reactor in a background process
+        print_pidfile (bool): whether to print the pid file, if daemonize is True
+        logger (logging.Logger): logger instance to pass to Daemonize
+        run_command (Callable[]): callable that actually runs the reactor
     """
 
-    def run() -> None:
+    def run():
         logger.info("Running")
         setup_jemalloc_stats()
         change_resource_limit(soft_file_limit)
@@ -211,7 +185,7 @@ def redirect_stdio_to_logs() -> None:
     print("Redirected stdout/stderr to logs")
 
 
-def register_start(cb: Callable[..., Awaitable], *args: Any, **kwargs: Any) -> None:
+def register_start(cb: Callable[..., Awaitable], *args, **kwargs) -> None:
     """Register a callback with the reactor, to be called once it is running
 
     This can be used to initialise parts of the system which require an asynchronous
@@ -221,7 +195,7 @@ def register_start(cb: Callable[..., Awaitable], *args: Any, **kwargs: Any) -> N
     will exit.
     """
 
-    async def wrapper() -> None:
+    async def wrapper():
         try:
             await cb(*args, **kwargs)
         except Exception:
@@ -250,7 +224,7 @@ def register_start(cb: Callable[..., Awaitable], *args: Any, **kwargs: Any) -> N
     reactor.callWhenRunning(lambda: defer.ensureDeferred(wrapper()))
 
 
-def listen_metrics(bind_addresses: Iterable[str], port: int) -> None:
+def listen_metrics(bind_addresses, port):
     """
     Start Prometheus metrics server.
     """
@@ -262,11 +236,11 @@ def listen_metrics(bind_addresses: Iterable[str], port: int) -> None:
 
 
 def listen_manhole(
-    bind_addresses: Collection[str],
+    bind_addresses: Iterable[str],
     port: int,
     manhole_settings: ManholeConfig,
     manhole_globals: dict,
-) -> None:
+):
     # twisted.conch.manhole 21.1.0 uses "int_from_bytes", which produces a confusing
     # warning. It's fixed by https://github.com/twisted/twisted/pull/1522), so
     # suppress the warning for now.
@@ -285,18 +259,12 @@ def listen_manhole(
     )
 
 
-def listen_tcp(
-    bind_addresses: Collection[str],
-    port: int,
-    factory: ServerFactory,
-    reactor: IReactorTCP = reactor,
-    backlog: int = 50,
-) -> List[Port]:
+def listen_tcp(bind_addresses, port, factory, reactor=reactor, backlog=50):
     """
     Create a TCP socket for a port and several addresses
 
     Returns:
-        list of twisted.internet.tcp.Port listening for TCP connections
+        list[twisted.internet.tcp.Port]: listening for TCP connections
     """
     r = []
     for address in bind_addresses:
@@ -305,19 +273,12 @@ def listen_tcp(
         except error.CannotListenError as e:
             check_bind_error(e, address, bind_addresses)
 
-    # IReactorTCP returns an object implementing IListeningPort from listenTCP,
-    # but we know it will be a Port instance.
-    return r  # type: ignore[return-value]
+    return r
 
 
 def listen_ssl(
-    bind_addresses: Collection[str],
-    port: int,
-    factory: ServerFactory,
-    context_factory: IOpenSSLContextFactory,
-    reactor: IReactorSSL = reactor,
-    backlog: int = 50,
-) -> List[Port]:
+    bind_addresses, port, factory, context_factory, reactor=reactor, backlog=50
+):
     """
     Create an TLS-over-TCP socket for a port and several addresses
 
@@ -333,13 +294,10 @@ def listen_ssl(
         except error.CannotListenError as e:
             check_bind_error(e, address, bind_addresses)
 
-    # IReactorSSL incorrectly declares that an int is returned from listenSSL,
-    # it actually returns an object implementing IListeningPort, but we know it
-    # will be a Port instance.
-    return r  # type: ignore[return-value]
+    return r
 
 
-def refresh_certificate(hs: "HomeServer") -> None:
+def refresh_certificate(hs: "HomeServer"):
     """
     Refresh the TLS certificates that Synapse is using by re-reading them from
     disk and updating the TLS context factories to use them.
@@ -371,7 +329,7 @@ def refresh_certificate(hs: "HomeServer") -> None:
         logger.info("Context factories updated.")
 
 
-async def start(hs: "HomeServer") -> None:
+async def start(hs: "HomeServer"):
     """
     Start a Synapse server or worker.
 
@@ -402,7 +360,7 @@ async def start(hs: "HomeServer") -> None:
     if hasattr(signal, "SIGHUP"):
 
         @wrap_as_background_process("sighup")
-        async def handle_sighup(*args: Any, **kwargs: Any) -> None:
+        def handle_sighup(*args, **kwargs):
             # Tell systemd our state, if we're using it. This will silently fail if
             # we're not using systemd.
             sdnotify(b"RELOADING=1")
@@ -415,7 +373,7 @@ async def start(hs: "HomeServer") -> None:
         # We defer running the sighup handlers until next reactor tick. This
         # is so that we're in a sane state, e.g. flushing the logs may fail
         # if the sighup happens in the middle of writing a log entry.
-        def run_sighup(*args: Any, **kwargs: Any) -> None:
+        def run_sighup(*args, **kwargs):
             # `callFromThread` should be "signal safe" as well as thread
             # safe.
             reactor.callFromThread(handle_sighup, *args, **kwargs)
@@ -478,8 +436,12 @@ async def start(hs: "HomeServer") -> None:
         atexit.register(gc.freeze)
 
 
-def setup_sentry(hs: "HomeServer") -> None:
-    """Enable sentry integration, if enabled in configuration"""
+def setup_sentry(hs: "HomeServer"):
+    """Enable sentry integration, if enabled in configuration
+
+    Args:
+        hs
+    """
 
     if not hs.config.metrics.sentry_enabled:
         return
@@ -504,7 +466,7 @@ def setup_sentry(hs: "HomeServer") -> None:
         scope.set_tag("worker_name", name)
 
 
-def setup_sdnotify(hs: "HomeServer") -> None:
+def setup_sdnotify(hs: "HomeServer"):
     """Adds process state hooks to tell systemd what we are up to."""
 
     # Tell systemd our state, if we're using it. This will silently fail if
@@ -519,7 +481,7 @@ def setup_sdnotify(hs: "HomeServer") -> None:
 sdnotify_sockaddr = os.getenv("NOTIFY_SOCKET")
 
 
-def sdnotify(state: bytes) -> None:
+def sdnotify(state):
     """
     Send a notification to systemd, if the NOTIFY_SOCKET env var is set.
 
@@ -528,7 +490,7 @@ def sdnotify(state: bytes) -> None:
     package which many OSes don't include as a matter of principle.
 
     Args:
-        state: notification to send
+        state (bytes): notification to send
     """
     if not isinstance(state, bytes):
         raise TypeError("sdnotify should be called with a bytes")

synapse/app/admin_cmd.py

@@ -17,7 +17,6 @@ import logging
 import os
 import sys
 import tempfile
-from typing import List, Optional
 
 from twisted.internet import defer, task
 
@@ -26,7 +25,6 @@ from synapse.app import _base
 from synapse.config._base import ConfigError
 from synapse.config.homeserver import HomeServerConfig
 from synapse.config.logger import setup_logging
-from synapse.events import EventBase
 from synapse.handlers.admin import ExfiltrationWriter
 from synapse.replication.slave.storage._base import BaseSlavedStore
 from synapse.replication.slave.storage.account_data import SlavedAccountDataStore
@@ -42,7 +40,6 @@ from synapse.replication.slave.storage.receipts import SlavedReceiptsStore
 from synapse.replication.slave.storage.registration import SlavedRegistrationStore
 from synapse.server import HomeServer
 from synapse.storage.databases.main.room import RoomWorkerStore
-from synapse.types import StateMap
 from synapse.util.logcontext import LoggingContext
 from synapse.util.versionstring import get_version_string
 
@@ -68,11 +65,16 @@ class AdminCmdSlavedStore(
 
 
 class AdminCmdServer(HomeServer):
-    DATASTORE_CLASS = AdminCmdSlavedStore  # type: ignore
+    DATASTORE_CLASS = AdminCmdSlavedStore
 
 
-async def export_data_command(hs: HomeServer, args: argparse.Namespace) -> None:
-    """Export data for a user."""
+async def export_data_command(hs: HomeServer, args):
+    """Export data for a user.
+
+    Args:
+        hs
+        args (argparse.Namespace)
+    """
 
     user_id = args.user_id
     directory = args.output_directory
@@ -90,12 +92,12 @@ class FileExfiltrationWriter(ExfiltrationWriter):
     Note: This writes to disk on the main reactor thread.
 
     Args:
-        user_id: The user whose data is being exfiltrated.
-        directory: The directory to write the data to, if None then will write
-            to a temporary directory.
+        user_id (str): The user whose data is being exfiltrated.
+        directory (str|None): The directory to write the data to, if None then
+            will write to a temporary directory.
     """
 
-    def __init__(self, user_id: str, directory: Optional[str] = None):
+    def __init__(self, user_id, directory=None):
         self.user_id = user_id
 
         if directory:
@@ -109,7 +111,7 @@ class FileExfiltrationWriter(ExfiltrationWriter):
         if list(os.listdir(self.base_directory)):
             raise Exception("Directory must be empty")
 
-    def write_events(self, room_id: str, events: List[EventBase]) -> None:
+    def write_events(self, room_id, events):
         room_directory = os.path.join(self.base_directory, "rooms", room_id)
         os.makedirs(room_directory, exist_ok=True)
         events_file = os.path.join(room_directory, "events")
@@ -118,9 +120,7 @@ class FileExfiltrationWriter(ExfiltrationWriter):
             for event in events:
                 print(json.dumps(event.get_pdu_json()), file=f)
 
-    def write_state(
-        self, room_id: str, event_id: str, state: StateMap[EventBase]
-    ) -> None:
+    def write_state(self, room_id, event_id, state):
         room_directory = os.path.join(self.base_directory, "rooms", room_id)
         state_directory = os.path.join(room_directory, "state")
         os.makedirs(state_directory, exist_ok=True)
@@ -131,9 +131,7 @@ class FileExfiltrationWriter(ExfiltrationWriter):
             for event in state.values():
                 print(json.dumps(event.get_pdu_json()), file=f)
 
-    def write_invite(
-        self, room_id: str, event: EventBase, state: StateMap[EventBase]
-    ) -> None:
+    def write_invite(self, room_id, event, state):
         self.write_events(room_id, [event])
 
         # We write the invite state somewhere else as they aren't full events
@@ -147,9 +145,7 @@ class FileExfiltrationWriter(ExfiltrationWriter):
             for event in state.values():
                 print(json.dumps(event), file=f)
 
-    def write_knock(
-        self, room_id: str, event: EventBase, state: StateMap[EventBase]
-    ) -> None:
+    def write_knock(self, room_id, event, state):
         self.write_events(room_id, [event])
 
         # We write the knock state somewhere else as they aren't full events
@@ -163,11 +159,11 @@ class FileExfiltrationWriter(ExfiltrationWriter):
             for event in state.values():
                 print(json.dumps(event), file=f)
 
-    def finished(self) -> str:
+    def finished(self):
         return self.base_directory
 
 
-def start(config_options: List[str]) -> None:
+def start(config_options):
     parser = argparse.ArgumentParser(description="Synapse Admin Command")
     HomeServerConfig.add_arguments_to_parser(parser)
 
@@ -235,7 +231,7 @@ def start(config_options: List[str]) -> None:
     # We also make sure that `_base.start` gets run before we actually run the
     # command.
 
-    async def run() -> None:
+    async def run():
         with LoggingContext("command"):
             await _base.start(ss)
             await args.func(ss, args)

synapse/app/generic_worker.py

@@ -14,10 +14,11 @@
 # limitations under the License.
 import logging
 import sys
-from typing import Dict, List, Optional, Tuple
+from typing import Dict, Optional
 
 from twisted.internet import address
-from twisted.web.resource import Resource
+from twisted.web.resource import IResource
+from twisted.web.server import Request
 
 import synapse
 import synapse.events
@@ -26,8 +27,7 @@ from synapse.api.urls import (
     CLIENT_API_PREFIX,
     FEDERATION_PREFIX,
     LEGACY_MEDIA_PREFIX,
-    MEDIA_R0_PREFIX,
-    MEDIA_V3_PREFIX,
+    MEDIA_PREFIX,
     SERVER_KEY_V2_PREFIX,
 )
 from synapse.app import _base
@@ -44,7 +44,7 @@ from synapse.config.server import ListenerConfig
 from synapse.federation.transport.server import TransportLayerServer
 from synapse.http.server import JsonResource, OptionsResource
 from synapse.http.servlet import RestServlet, parse_json_object_from_request
-from synapse.http.site import SynapseRequest, SynapseSite
+from synapse.http.site import SynapseSite
 from synapse.logging.context import LoggingContext
 from synapse.metrics import METRICS_PREFIX, MetricsResource, RegistryProxy
 from synapse.replication.http import REPLICATION_PREFIX, ReplicationRestResource
@@ -119,7 +119,6 @@ from synapse.storage.databases.main.stats import StatsStore
 from synapse.storage.databases.main.transactions import TransactionWorkerStore
 from synapse.storage.databases.main.ui_auth import UIAuthWorkerStore
 from synapse.storage.databases.main.user_directory import UserDirectoryStore
-from synapse.types import JsonDict
 from synapse.util.httpresourcetree import create_resource_tree
 from synapse.util.versionstring import get_version_string
 
@@ -144,9 +143,7 @@ class KeyUploadServlet(RestServlet):
         self.http_client = hs.get_simple_http_client()
         self.main_uri = hs.config.worker.worker_main_http_uri
 
-    async def on_POST(
-        self, request: SynapseRequest, device_id: Optional[str]
-    ) -> Tuple[int, JsonDict]:
+    async def on_POST(self, request: Request, device_id: Optional[str]):
         requester = await self.auth.get_user_by_req(request, allow_guest=True)
         user_id = requester.user.to_string()
         body = parse_json_object_from_request(request)
@@ -190,8 +187,9 @@ class KeyUploadServlet(RestServlet):
                 # If the header exists, add to the comma-separated list of the first
                 # instance of the header. Otherwise, generate a new header.
                 if x_forwarded_for:
-                    x_forwarded_for = [x_forwarded_for[0] + b", " + previous_host]
-                    x_forwarded_for.extend(x_forwarded_for[1:])
+                    x_forwarded_for = [
+                        x_forwarded_for[0] + b", " + previous_host
+                    ] + x_forwarded_for[1:]
                 else:
                     x_forwarded_for = [previous_host]
             headers[b"X-Forwarded-For"] = x_forwarded_for
@@ -255,16 +253,13 @@ class GenericWorkerSlavedStore(
     SessionStore,
     BaseSlavedStore,
 ):
-    # Properties that multiple storage classes define. Tell mypy what the
-    # expected type is.
-    server_name: str
-    config: HomeServerConfig
+    pass
 
 
 class GenericWorkerServer(HomeServer):
-    DATASTORE_CLASS = GenericWorkerSlavedStore  # type: ignore
+    DATASTORE_CLASS = GenericWorkerSlavedStore
 
-    def _listen_http(self, listener_config: ListenerConfig) -> None:
+    def _listen_http(self, listener_config: ListenerConfig):
         port = listener_config.port
         bind_addresses = listener_config.bind_addresses
 
@@ -272,10 +267,10 @@ class GenericWorkerServer(HomeServer):
 
         site_tag = listener_config.http_options.tag
         if site_tag is None:
-            site_tag = str(port)
+            site_tag = port
 
         # We always include a health resource.
-        resources: Dict[str, Resource] = {"/health": HealthResource()}
+        resources: Dict[str, IResource] = {"/health": HealthResource()}
 
         for res in listener_config.http_options.resources:
             for name in res.names:
@@ -339,8 +334,7 @@ class GenericWorkerServer(HomeServer):
 
                         resources.update(
                             {
-                                MEDIA_R0_PREFIX: media_repo,
-                                MEDIA_V3_PREFIX: media_repo,
+                                MEDIA_PREFIX: media_repo,
                                 LEGACY_MEDIA_PREFIX: media_repo,
                                 "/_synapse/admin": admin_resource,
                             }
@@ -392,7 +386,7 @@ class GenericWorkerServer(HomeServer):
 
         logger.info("Synapse worker now listening on port %d", port)
 
-    def start_listening(self) -> None:
+    def start_listening(self):
         for listener in self.config.worker.worker_listeners:
             if listener.type == "http":
                 self._listen_http(listener)
@@ -417,7 +411,7 @@ class GenericWorkerServer(HomeServer):
         self.get_tcp_replication().start_replication(self)
 
 
-def start(config_options: List[str]) -> None:
+def start(config_options):
     try:
         config = HomeServerConfig.load_config("Synapse worker", config_options)
     except ConfigError as e:

synapse/app/homeserver.py

@@ -16,10 +16,10 @@
 import logging
 import os
 import sys
-from typing import Dict, Iterable, Iterator, List
+from typing import Iterator
 
-from twisted.internet.tcp import Port
-from twisted.web.resource import EncodingResourceWrapper, Resource
+from twisted.internet import reactor
+from twisted.web.resource import EncodingResourceWrapper, IResource
 from twisted.web.server import GzipEncoderFactory
 from twisted.web.static import File
 
@@ -29,8 +29,7 @@ from synapse import events
 from synapse.api.urls import (
     FEDERATION_PREFIX,
     LEGACY_MEDIA_PREFIX,
-    MEDIA_R0_PREFIX,
-    MEDIA_V3_PREFIX,
+    MEDIA_PREFIX,
     SERVER_KEY_V2_PREFIX,
     STATIC_PREFIX,
     WEB_CLIENT_PREFIX,
@@ -77,27 +76,23 @@ from synapse.util.versionstring import get_version_string
 logger = logging.getLogger("synapse.app.homeserver")
 
 
-def gz_wrap(r: Resource) -> Resource:
+def gz_wrap(r):
     return EncodingResourceWrapper(r, [GzipEncoderFactory()])
 
 
 class SynapseHomeServer(HomeServer):
-    DATASTORE_CLASS = DataStore  # type: ignore
+    DATASTORE_CLASS = DataStore
 
-    def _listener_http(
-        self, config: HomeServerConfig, listener_config: ListenerConfig
-    ) -> Iterable[Port]:
+    def _listener_http(self, config: HomeServerConfig, listener_config: ListenerConfig):
         port = listener_config.port
         bind_addresses = listener_config.bind_addresses
         tls = listener_config.tls
-        # Must exist since this is an HTTP listener.
-        assert listener_config.http_options is not None
         site_tag = listener_config.http_options.tag
         if site_tag is None:
             site_tag = str(port)
 
         # We always include a health resource.
-        resources: Dict[str, Resource] = {"/health": HealthResource()}
+        resources = {"/health": HealthResource()}
 
         for res in listener_config.http_options.resources:
             for name in res.names:
@@ -116,7 +111,7 @@ class SynapseHomeServer(HomeServer):
                 ("listeners", site_tag, "additional_resources", "<%s>" % (path,)),
             )
             handler = handler_cls(config, module_api)
-            if isinstance(handler, Resource):
+            if IResource.providedBy(handler):
                 resource = handler
             elif hasattr(handler, "handle_request"):
                 resource = AdditionalResource(self, handler.handle_request)
@@ -133,7 +128,7 @@ class SynapseHomeServer(HomeServer):
 
         # try to find something useful to redirect '/' to
         if WEB_CLIENT_PREFIX in resources:
-            root_resource: Resource = RootOptionsRedirectResource(WEB_CLIENT_PREFIX)
+            root_resource = RootOptionsRedirectResource(WEB_CLIENT_PREFIX)
         elif STATIC_PREFIX in resources:
             root_resource = RootOptionsRedirectResource(STATIC_PREFIX)
         else:
@@ -150,8 +145,6 @@ class SynapseHomeServer(HomeServer):
         )
 
         if tls:
-            # refresh_certificate should have been called before this.
-            assert self.tls_server_context_factory is not None
             ports = listen_ssl(
                 bind_addresses,
                 port,
@@ -172,21 +165,20 @@ class SynapseHomeServer(HomeServer):
 
         return ports
 
-    def _configure_named_resource(
-        self, name: str, compress: bool = False
-    ) -> Dict[str, Resource]:
+    def _configure_named_resource(self, name, compress=False):
         """Build a resource map for a named resource
 
         Args:
-            name: named resource: one of "client", "federation", etc
-            compress: whether to enable gzip compression for this resource
+            name (str): named resource: one of "client", "federation", etc
+            compress (bool): whether to enable gzip compression for this
+                resource
 
         Returns:
-            map from path to HTTP resource
+            dict[str, Resource]: map from path to HTTP resource
         """
-        resources: Dict[str, Resource] = {}
+        resources = {}
         if name == "client":
-            client_resource: Resource = ClientRestResource(self)
+            client_resource = ClientRestResource(self)
             if compress:
                 client_resource = gz_wrap(client_resource)
 
@@ -194,7 +186,6 @@ class SynapseHomeServer(HomeServer):
                 {
                     "/_matrix/client/api/v1": client_resource,
                     "/_matrix/client/r0": client_resource,
-                    "/_matrix/client/v3": client_resource,
                     "/_matrix/client/unstable": client_resource,
                     "/_matrix/client/v2_alpha": client_resource,
                     "/_matrix/client/versions": client_resource,
@@ -216,7 +207,7 @@ class SynapseHomeServer(HomeServer):
         if name == "consent":
             from synapse.rest.consent.consent_resource import ConsentResource
 
-            consent_resource: Resource = ConsentResource(self)
+            consent_resource = ConsentResource(self)
             if compress:
                 consent_resource = gz_wrap(consent_resource)
             resources.update({"/_matrix/consent": consent_resource})
@@ -246,11 +237,7 @@ class SynapseHomeServer(HomeServer):
             if self.config.server.enable_media_repo:
                 media_repo = self.get_media_repository_resource()
                 resources.update(
-                    {
-                        MEDIA_R0_PREFIX: media_repo,
-                        MEDIA_V3_PREFIX: media_repo,
-                        LEGACY_MEDIA_PREFIX: media_repo,
-                    }
+                    {MEDIA_PREFIX: media_repo, LEGACY_MEDIA_PREFIX: media_repo}
                 )
             elif name == "media":
                 raise ConfigError(
@@ -290,7 +277,7 @@ class SynapseHomeServer(HomeServer):
 
         return resources
 
-    def start_listening(self) -> None:
+    def start_listening(self):
         if self.config.redis.redis_enabled:
             # If redis is enabled we connect via the replication command handler
             # in the same way as the workers (since we're effectively a client
@@ -316,9 +303,7 @@ class SynapseHomeServer(HomeServer):
                     ReplicationStreamProtocolFactory(self),
                 )
                 for s in services:
-                    self.get_reactor().addSystemEventTrigger(
-                        "before", "shutdown", s.stopListening
-                    )
+                    reactor.addSystemEventTrigger("before", "shutdown", s.stopListening)
             elif listener.type == "metrics":
                 if not self.config.metrics.enable_metrics:
                     logger.warning(
@@ -333,13 +318,14 @@ class SynapseHomeServer(HomeServer):
                 logger.warning("Unrecognized listener type: %s", listener.type)
 
 
-def setup(config_options: List[str]) -> SynapseHomeServer:
+def setup(config_options):
     """
     Args:
-        config_options_options: The options passed to Synapse. Usually `sys.argv[1:]`.
+        config_options_options: The options passed to Synapse. Usually
+            `sys.argv[1:]`.
 
     Returns:
-        A homeserver instance.
+        HomeServer
     """
     try:
         config = HomeServerConfig.load_or_generate_config(
@@ -378,7 +364,7 @@ def setup(config_options: List[str]) -> SynapseHomeServer:
     except Exception as e:
         handle_startup_exception(e)
 
-    async def start() -> None:
+    async def start():
         # Load the OIDC provider metadatas, if OIDC is enabled.
         if hs.config.oidc.oidc_enabled:
             oidc = hs.get_oidc_handler()
@@ -418,15 +404,39 @@ def format_config_error(e: ConfigError) -> Iterator[str]:
 
     yield ":\n  %s" % (e.msg,)
 
-    parent_e = e.__cause__
+    e = e.__cause__
     indent = 1
-    while parent_e:
+    while e:
         indent += 1
-        yield ":\n%s%s" % ("  " * indent, str(parent_e))
-        parent_e = parent_e.__cause__
+        yield ":\n%s%s" % ("  " * indent, str(e))
+        e = e.__cause__
 
 
-def run(hs: HomeServer) -> None:
+def run(hs: HomeServer):
+    PROFILE_SYNAPSE = False
+    if PROFILE_SYNAPSE:
+
+        def profile(func):
+            from cProfile import Profile
+            from threading import current_thread
+
+            def profiled(*args, **kargs):
+                profile = Profile()
+                profile.enable()
+                func(*args, **kargs)
+                profile.disable()
+                ident = current_thread().ident
+                profile.dump_stats(
+                    "/tmp/%s.%s.%i.pstat" % (hs.hostname, func.__name__, ident)
+                )
+
+            return profiled
+
+        from twisted.python.threadpool import ThreadPool
+
+        ThreadPool._worker = profile(ThreadPool._worker)
+        reactor.run = profile(reactor.run)
+
     _base.start_reactor(
         "synapse-homeserver",
         soft_file_limit=hs.config.server.soft_file_limit,
@@ -438,7 +448,7 @@ def run(hs: HomeServer) -> None:
     )
 
 
-def main() -> None:
+def main():
     with LoggingContext("main"):
         # check base requirements
         check_requirements()

synapse/app/phone_stats_home.py

@@ -15,12 +15,11 @@ import logging
 import math
 import resource
 import sys
-from typing import TYPE_CHECKING, List, Sized, Tuple
+from typing import TYPE_CHECKING
 
 from prometheus_client import Gauge
 
 from synapse.metrics.background_process_metrics import wrap_as_background_process
-from synapse.types import JsonDict
 
 if TYPE_CHECKING:
     from synapse.server import HomeServer
@@ -29,7 +28,7 @@ logger = logging.getLogger("synapse.app.homeserver")
 
 # Contains the list of processes we will be monitoring
 # currently either 0 or 1
-_stats_process: List[Tuple[int, "resource.struct_rusage"]] = []
+_stats_process = []
 
 # Gauges to expose monthly active user control metrics
 current_mau_gauge = Gauge("synapse_admin_mau:current", "Current MAU")
@@ -46,15 +45,9 @@ registered_reserved_users_mau_gauge = Gauge(
 
 
 @wrap_as_background_process("phone_stats_home")
-async def phone_stats_home(
-    hs: "HomeServer",
-    stats: JsonDict,
-    stats_process: List[Tuple[int, "resource.struct_rusage"]] = _stats_process,
-) -> None:
+async def phone_stats_home(hs: "HomeServer", stats, stats_process=_stats_process):
     logger.info("Gathering stats for reporting")
     now = int(hs.get_clock().time())
-    # Ensure the homeserver has started.
-    assert hs.start_time is not None
     uptime = int(now - hs.start_time)
     if uptime < 0:
         uptime = 0
@@ -153,15 +146,15 @@ async def phone_stats_home(
         logger.warning("Error reporting stats: %s", e)
 
 
-def start_phone_stats_home(hs: "HomeServer") -> None:
+def start_phone_stats_home(hs: "HomeServer"):
     """
     Start the background tasks which report phone home stats.
     """
     clock = hs.get_clock()
 
-    stats: JsonDict = {}
+    stats = {}
 
-    def performance_stats_init() -> None:
+    def performance_stats_init():
         _stats_process.clear()
         _stats_process.append(
             (int(hs.get_clock().time()), resource.getrusage(resource.RUSAGE_SELF))
@@ -177,10 +170,10 @@ def start_phone_stats_home(hs: "HomeServer") -> None:
     hs.get_datastore().reap_monthly_active_users()
 
     @wrap_as_background_process("generate_monthly_active_users")
-    async def generate_monthly_active_users() -> None:
+    async def generate_monthly_active_users():
         current_mau_count = 0
         current_mau_count_by_service = {}
-        reserved_users: Sized = ()
+        reserved_users = ()
         store = hs.get_datastore()
         if hs.config.server.limit_usage_by_mau or hs.config.server.mau_stats_only:
             current_mau_count = await store.get_monthly_active_count()

synapse/appservice/api.py

@@ -231,32 +231,13 @@ class ApplicationServiceApi(SimpleHttpClient):
                 json_body=body,
                 args={"access_token": service.hs_token},
             )
-            if logger.isEnabledFor(logging.DEBUG):
-                logger.debug(
-                    "push_bulk to %s succeeded! events=%s",
-                    uri,
-                    [event.get("event_id") for event in events],
-                )
             sent_transactions_counter.labels(service.id).inc()
             sent_events_counter.labels(service.id).inc(len(events))
             return True
         except CodeMessageException as e:
-            logger.warning(
-                "push_bulk to %s received code=%s msg=%s",
-                uri,
-                e.code,
-                e.msg,
-                exc_info=logger.isEnabledFor(logging.DEBUG),
-            )
+            logger.warning("push_bulk to %s received %s", uri, e.code)
         except Exception as ex:
-            logger.warning(
-                "push_bulk to %s threw exception(%s) %s args=%s",
-                uri,
-                type(ex).__name__,
-                ex,
-                ex.args,
-                exc_info=logger.isEnabledFor(logging.DEBUG),
-            )
+            logger.warning("push_bulk to %s threw exception %s", uri, ex)
         failed_transactions_counter.labels(service.id).inc()
         return False
 

synapse/config/_base.py

@@ -20,18 +20,7 @@ import os
 from collections import OrderedDict
 from hashlib import sha256
 from textwrap import dedent
-from typing import (
-    Any,
-    Dict,
-    Iterable,
-    List,
-    MutableMapping,
-    Optional,
-    Tuple,
-    Type,
-    TypeVar,
-    Union,
-)
+from typing import Any, Iterable, List, MutableMapping, Optional, Union
 
 import attr
 import jinja2
@@ -89,7 +78,7 @@ CONFIG_FILE_HEADER = """\
 """
 
 
-def path_exists(file_path: str) -> bool:
+def path_exists(file_path):
     """Check if a file exists
 
     Unlike os.path.exists, this throws an exception if there is an error
@@ -97,7 +86,7 @@ def path_exists(file_path: str) -> bool:
     the parent dir).
 
     Returns:
-        True if the file exists; False if not.
+        bool: True if the file exists; False if not.
     """
     try:
         os.stat(file_path)
@@ -113,15 +102,15 @@ class Config:
     A configuration section, containing configuration keys and values.
 
     Attributes:
-        section: The section title of this config object, such as
+        section (str): The section title of this config object, such as
             "tls" or "logger". This is used to refer to it on the root
             logger (for example, `config.tls.some_option`). Must be
             defined in subclasses.
     """
 
-    section: str
+    section = None
 
-    def __init__(self, root_config: "RootConfig" = None):
+    def __init__(self, root_config=None):
         self.root = root_config
 
         # Get the path to the default Synapse template directory
@@ -130,7 +119,7 @@ class Config:
         )
 
     @staticmethod
-    def parse_size(value: Union[str, int]) -> int:
+    def parse_size(value):
         if isinstance(value, int):
             return value
         sizes = {"K": 1024, "M": 1024 * 1024}
@@ -173,15 +162,15 @@ class Config:
         return int(value) * size
 
     @staticmethod
-    def abspath(file_path: str) -> str:
+    def abspath(file_path):
         return os.path.abspath(file_path) if file_path else file_path
 
     @classmethod
-    def path_exists(cls, file_path: str) -> bool:
+    def path_exists(cls, file_path):
         return path_exists(file_path)
 
     @classmethod
-    def check_file(cls, file_path: Optional[str], config_name: str) -> str:
+    def check_file(cls, file_path, config_name):
         if file_path is None:
             raise ConfigError("Missing config for %s." % (config_name,))
         try:
@@ -194,7 +183,7 @@ class Config:
         return cls.abspath(file_path)
 
     @classmethod
-    def ensure_directory(cls, dir_path: str) -> str:
+    def ensure_directory(cls, dir_path):
         dir_path = cls.abspath(dir_path)
         os.makedirs(dir_path, exist_ok=True)
         if not os.path.isdir(dir_path):
@@ -202,7 +191,7 @@ class Config:
         return dir_path
 
     @classmethod
-    def read_file(cls, file_path: Any, config_name: str) -> str:
+    def read_file(cls, file_path, config_name):
         """Deprecated: call read_file directly"""
         return read_file(file_path, (config_name,))
 
@@ -295,9 +284,6 @@ class Config:
         return [env.get_template(filename) for filename in filenames]
 
 
-TRootConfig = TypeVar("TRootConfig", bound="RootConfig")
-
-
 class RootConfig:
     """
     Holder of an application's configuration.
@@ -322,9 +308,7 @@ class RootConfig:
                 raise Exception("Failed making %s: %r" % (config_class.section, e))
             setattr(self, config_class.section, conf)
 
-    def invoke_all(
-        self, func_name: str, *args: Any, **kwargs: Any
-    ) -> MutableMapping[str, Any]:
+    def invoke_all(self, func_name: str, *args, **kwargs) -> MutableMapping[str, Any]:
         """
         Invoke a function on all instantiated config objects this RootConfig is
         configured to use.
@@ -333,7 +317,6 @@ class RootConfig:
             func_name: Name of function to invoke
             *args
             **kwargs
-
         Returns:
             ordered dictionary of config section name and the result of the
             function from it.
@@ -349,7 +332,7 @@ class RootConfig:
         return res
 
     @classmethod
-    def invoke_all_static(cls, func_name: str, *args: Any, **kwargs: any) -> None:
+    def invoke_all_static(cls, func_name: str, *args, **kwargs):
         """
         Invoke a static function on config objects this RootConfig is
         configured to use.
@@ -358,7 +341,6 @@ class RootConfig:
             func_name: Name of function to invoke
             *args
             **kwargs
-
         Returns:
             ordered dictionary of config section name and the result of the
             function from it.
@@ -369,16 +351,16 @@ class RootConfig:
 
     def generate_config(
         self,
-        config_dir_path: str,
-        data_dir_path: str,
-        server_name: str,
-        generate_secrets: bool = False,
-        report_stats: Optional[bool] = None,
-        open_private_ports: bool = False,
-        listeners: Optional[List[dict]] = None,
-        tls_certificate_path: Optional[str] = None,
-        tls_private_key_path: Optional[str] = None,
-    ) -> str:
+        config_dir_path,
+        data_dir_path,
+        server_name,
+        generate_secrets=False,
+        report_stats=None,
+        open_private_ports=False,
+        listeners=None,
+        tls_certificate_path=None,
+        tls_private_key_path=None,
+    ):
         """
         Build a default configuration file
 
@@ -386,27 +368,27 @@ class RootConfig:
         (eg with --generate_config).
 
         Args:
-            config_dir_path: The path where the config files are kept. Used to
+            config_dir_path (str): The path where the config files are kept. Used to
                 create filenames for things like the log config and the signing key.
 
-            data_dir_path: The path where the data files are kept. Used to create
+            data_dir_path (str): The path where the data files are kept. Used to create
                 filenames for things like the database and media store.
 
-            server_name: The server name. Used to initialise the server_name
+            server_name (str): The server name. Used to initialise the server_name
                 config param, but also used in the names of some of the config files.
 
-            generate_secrets: True if we should generate new secrets for things
+            generate_secrets (bool): True if we should generate new secrets for things
                 like the macaroon_secret_key. If False, these parameters will be left
                 unset.
 
-            report_stats: Initial setting for the report_stats setting.
+            report_stats (bool|None): Initial setting for the report_stats setting.
                 If None, report_stats will be left unset.
 
-            open_private_ports: True to leave private ports (such as the non-TLS
+            open_private_ports (bool): True to leave private ports (such as the non-TLS
                 HTTP listener) open to the internet.
 
-            listeners: A list of descriptions of the listeners synapse should
-                start with each of which specifies a port (int), a list of
+            listeners (list(dict)|None): A list of descriptions of the listeners
+                synapse should start with each of which specifies a port (str), a list of
                 resources (list(str)), tls (bool) and type (str). For example:
                 [{
                     "port": 8448,
@@ -421,12 +403,16 @@ class RootConfig:
                     "type": "http",
                 }],
 
-            tls_certificate_path: The path to the tls certificate.
 
-            tls_private_key_path: The path to the tls private key.
+            database (str|None): The database type to configure, either `psycog2`
+                or `sqlite3`.
+
+            tls_certificate_path (str|None): The path to the tls certificate.
+
+            tls_private_key_path (str|None): The path to the tls private key.
 
         Returns:
-            The yaml config file
+            str: the yaml config file
         """
 
         return CONFIG_FILE_HEADER + "\n\n".join(
@@ -446,15 +432,12 @@ class RootConfig:
         )
 
     @classmethod
-    def load_config(
-        cls: Type[TRootConfig], description: str, argv: List[str]
-    ) -> TRootConfig:
+    def load_config(cls, description, argv):
         """Parse the commandline and config files
 
         Doesn't support config-file-generation: used by the worker apps.
 
-        Returns:
-            Config object.
+        Returns: Config object.
         """
         config_parser = argparse.ArgumentParser(description=description)
         cls.add_arguments_to_parser(config_parser)
@@ -463,7 +446,7 @@ class RootConfig:
         return obj
 
     @classmethod
-    def add_arguments_to_parser(cls, config_parser: argparse.ArgumentParser) -> None:
+    def add_arguments_to_parser(cls, config_parser):
         """Adds all the config flags to an ArgumentParser.
 
         Doesn't support config-file-generation: used by the worker apps.
@@ -471,7 +454,7 @@ class RootConfig:
         Used for workers where we want to add extra flags/subcommands.
 
         Args:
-            config_parser: App description
+            config_parser (ArgumentParser): App description
         """
 
         config_parser.add_argument(
@@ -494,9 +477,7 @@ class RootConfig:
         cls.invoke_all_static("add_arguments", config_parser)
 
     @classmethod
-    def load_config_with_parser(
-        cls: Type[TRootConfig], parser: argparse.ArgumentParser, argv: List[str]
-    ) -> Tuple[TRootConfig, argparse.Namespace]:
+    def load_config_with_parser(cls, parser, argv):
         """Parse the commandline and config files with the given parser
 
         Doesn't support config-file-generation: used by the worker apps.
@@ -504,12 +485,13 @@ class RootConfig:
         Used for workers where we want to add extra flags/subcommands.
 
         Args:
-            parser
-            argv
+            parser (ArgumentParser)
+            argv (list[str])
 
         Returns:
-            Returns the parsed config object and the parsed argparse.Namespace
-            object from parser.parse_args(..)`
+            tuple[HomeServerConfig, argparse.Namespace]: Returns the parsed
+            config object and the parsed argparse.Namespace object from
+            `parser.parse_args(..)`
         """
 
         obj = cls()
@@ -538,15 +520,12 @@ class RootConfig:
         return obj, config_args
 
     @classmethod
-    def load_or_generate_config(
-        cls: Type[TRootConfig], description: str, argv: List[str]
-    ) -> Optional[TRootConfig]:
+    def load_or_generate_config(cls, description, argv):
         """Parse the commandline and config files
 
         Supports generation of config files, so is used for the main homeserver app.
 
-        Returns:
-            Config object, or None if --generate-config or --generate-keys was set
+        Returns: Config object, or None if --generate-config or --generate-keys was set
         """
         parser = argparse.ArgumentParser(description=description)
         parser.add_argument(
@@ -701,21 +680,16 @@ class RootConfig:
 
         return obj
 
-    def parse_config_dict(
-        self,
-        config_dict: Dict[str, Any],
-        config_dir_path: Optional[str] = None,
-        data_dir_path: Optional[str] = None,
-    ) -> None:
+    def parse_config_dict(self, config_dict, config_dir_path=None, data_dir_path=None):
         """Read the information from the config dict into this Config object.
 
         Args:
-            config_dict: Configuration data, as read from the yaml
+            config_dict (dict): Configuration data, as read from the yaml
 
-            config_dir_path: The path where the config files are kept. Used to
+            config_dir_path (str): The path where the config files are kept. Used to
                 create filenames for things like the log config and the signing key.
 
-            data_dir_path: The path where the data files are kept. Used to create
+            data_dir_path (str): The path where the data files are kept. Used to create
                 filenames for things like the database and media store.
         """
         self.invoke_all(
@@ -725,20 +699,17 @@ class RootConfig:
             data_dir_path=data_dir_path,
         )
 
-    def generate_missing_files(
-        self, config_dict: Dict[str, Any], config_dir_path: str
-    ) -> None:
+    def generate_missing_files(self, config_dict, config_dir_path):
         self.invoke_all("generate_files", config_dict, config_dir_path)
 
 
-def read_config_files(config_files: Iterable[str]) -> Dict[str, Any]:
+def read_config_files(config_files):
     """Read the config files into a dict
 
     Args:
-        config_files: A list of the config files to read
+        config_files (iterable[str]): A list of the config files to read
 
-    Returns:
-        The configuration dictionary.
+    Returns: dict
     """
     specified_config = {}
     for config_file in config_files:
@@ -762,17 +733,17 @@ def read_config_files(config_files: Iterable[str]) -> Dict[str, Any]:
     return specified_config
 
 
-def find_config_files(search_paths: List[str]) -> List[str]:
+def find_config_files(search_paths):
     """Finds config files using a list of search paths. If a path is a file
     then that file path is added to the list. If a search path is a directory
     then all the "*.yaml" files in that directory are added to the list in
     sorted order.
 
     Args:
-        search_paths: A list of paths to search.
+        search_paths(list(str)): A list of paths to search.
 
     Returns:
-        A list of file paths.
+        list(str): A list of file paths.
     """
 
     config_files = []
@@ -806,7 +777,7 @@ def find_config_files(search_paths: List[str]) -> List[str]:
     return config_files
 
 
-@attr.s(auto_attribs=True)
+@attr.s
 class ShardedWorkerHandlingConfig:
     """Algorithm for choosing which instance is responsible for handling some
     sharded work.
@@ -816,7 +787,7 @@ class ShardedWorkerHandlingConfig:
     below).
     """
 
-    instances: List[str]
+    instances = attr.ib(type=List[str])
 
     def should_handle(self, instance_name: str, key: str) -> bool:
         """Whether this instance is responsible for handling the given key."""

synapse/config/_base.pyi

@@ -1,18 +1,4 @@
-import argparse
-from typing import (
-    Any,
-    Dict,
-    Iterable,
-    List,
-    MutableMapping,
-    Optional,
-    Tuple,
-    Type,
-    TypeVar,
-    Union,
-)
-
-import jinja2
+from typing import Any, Iterable, List, Optional
 
 from synapse.config import (
     account_validity,
@@ -33,7 +19,6 @@ from synapse.config import (
     logger,
     metrics,
     modules,
-    oembed,
     oidc,
     password_auth_providers,
     push,
@@ -42,7 +27,6 @@ from synapse.config import (
     registration,
     repository,
     retention,
-    room,
     room_directory,
     saml2,
     server,
@@ -67,9 +51,7 @@ MISSING_REPORT_STATS_CONFIG_INSTRUCTIONS: str
 MISSING_REPORT_STATS_SPIEL: str
 MISSING_SERVER_NAME: str
 
-def path_exists(file_path: str) -> bool: ...
-
-TRootConfig = TypeVar("TRootConfig", bound="RootConfig")
+def path_exists(file_path: str): ...
 
 class RootConfig:
     server: server.ServerConfig
@@ -79,7 +61,6 @@ class RootConfig:
     logging: logger.LoggingConfig
     ratelimiting: ratelimiting.RatelimitConfig
     media: repository.ContentRepositoryConfig
-    oembed: oembed.OembedConfig
     captcha: captcha.CaptchaConfig
     voip: voip.VoipConfig
     registration: registration.RegistrationConfig
@@ -99,7 +80,6 @@ class RootConfig:
     authproviders: password_auth_providers.PasswordAuthProviderConfig
     push: push.PushConfig
     spamchecker: spam_checker.SpamCheckerConfig
-    room: room.RoomConfig
     groups: groups.GroupsConfig
     userdirectory: user_directory.UserDirectoryConfig
     consent: consent.ConsentConfig
@@ -107,85 +87,72 @@ class RootConfig:
     servernotices: server_notices.ServerNoticesConfig
     roomdirectory: room_directory.RoomDirectoryConfig
     thirdpartyrules: third_party_event_rules.ThirdPartyRulesConfig
-    tracing: tracer.TracerConfig
+    tracer: tracer.TracerConfig
     redis: redis.RedisConfig
     modules: modules.ModulesConfig
     caches: cache.CacheConfig
     federation: federation.FederationConfig
     retention: retention.RetentionConfig
 
-    config_classes: List[Type["Config"]] = ...
+    config_classes: List = ...
     def __init__(self) -> None: ...
-    def invoke_all(
-        self, func_name: str, *args: Any, **kwargs: Any
-    ) -> MutableMapping[str, Any]: ...
+    def invoke_all(self, func_name: str, *args: Any, **kwargs: Any): ...
     @classmethod
     def invoke_all_static(cls, func_name: str, *args: Any, **kwargs: Any) -> None: ...
+    def __getattr__(self, item: str): ...
     def parse_config_dict(
         self,
-        config_dict: Dict[str, Any],
-        config_dir_path: Optional[str] = ...,
-        data_dir_path: Optional[str] = ...,
+        config_dict: Any,
+        config_dir_path: Optional[Any] = ...,
+        data_dir_path: Optional[Any] = ...,
     ) -> None: ...
+    read_config: Any = ...
     def generate_config(
         self,
         config_dir_path: str,
         data_dir_path: str,
         server_name: str,
         generate_secrets: bool = ...,
-        report_stats: Optional[bool] = ...,
+        report_stats: Optional[str] = ...,
         open_private_ports: bool = ...,
         listeners: Optional[Any] = ...,
+        database_conf: Optional[Any] = ...,
         tls_certificate_path: Optional[str] = ...,
         tls_private_key_path: Optional[str] = ...,
-    ) -> str: ...
+    ): ...
     @classmethod
-    def load_or_generate_config(
-        cls: Type[TRootConfig], description: str, argv: List[str]
-    ) -> Optional[TRootConfig]: ...
+    def load_or_generate_config(cls, description: Any, argv: Any): ...
     @classmethod
-    def load_config(
-        cls: Type[TRootConfig], description: str, argv: List[str]
-    ) -> TRootConfig: ...
+    def load_config(cls, description: Any, argv: Any): ...
     @classmethod
-    def add_arguments_to_parser(
-        cls, config_parser: argparse.ArgumentParser
-    ) -> None: ...
+    def add_arguments_to_parser(cls, config_parser: Any) -> None: ...
     @classmethod
-    def load_config_with_parser(
-        cls: Type[TRootConfig], parser: argparse.ArgumentParser, argv: List[str]
-    ) -> Tuple[TRootConfig, argparse.Namespace]: ...
+    def load_config_with_parser(cls, parser: Any, argv: Any): ...
     def generate_missing_files(
         self, config_dict: dict, config_dir_path: str
     ) -> None: ...
 
 class Config:
     root: RootConfig
-    default_template_dir: str
     def __init__(self, root_config: Optional[RootConfig] = ...) -> None: ...
+    def __getattr__(self, item: str, from_root: bool = ...): ...
     @staticmethod
-    def parse_size(value: Union[str, int]) -> int: ...
+    def parse_size(value: Any): ...
     @staticmethod
-    def parse_duration(value: Union[str, int]) -> int: ...
+    def parse_duration(value: Any): ...
     @staticmethod
-    def abspath(file_path: Optional[str]) -> str: ...
+    def abspath(file_path: Optional[str]): ...
     @classmethod
-    def path_exists(cls, file_path: str) -> bool: ...
+    def path_exists(cls, file_path: str): ...
     @classmethod
-    def check_file(cls, file_path: str, config_name: str) -> str: ...
+    def check_file(cls, file_path: str, config_name: str): ...
     @classmethod
-    def ensure_directory(cls, dir_path: str) -> str: ...
+    def ensure_directory(cls, dir_path: str): ...
     @classmethod
-    def read_file(cls, file_path: str, config_name: str) -> str: ...
-    def read_template(self, filenames: str) -> jinja2.Template: ...
-    def read_templates(
-        self,
-        filenames: List[str],
-        custom_template_directories: Optional[Iterable[str]] = None,
-    ) -> List[jinja2.Template]: ...
+    def read_file(cls, file_path: str, config_name: str): ...
 
-def read_config_files(config_files: Iterable[str]) -> Dict[str, Any]: ...
-def find_config_files(search_paths: List[str]) -> List[str]: ...
+def read_config_files(config_files: List[str]): ...
+def find_config_files(search_paths: List[str]): ...
 
 class ShardedWorkerHandlingConfig:
     instances: List[str]

synapse/config/account_validity.py

@@ -75,6 +75,10 @@ class AccountValidityConfig(Config):
                 self.account_validity_period * 10.0 / 100.0
             )
 
+        if self.account_validity_renew_by_email_enabled:
+            if not self.root.server.public_baseurl:
+                raise ConfigError("Can't send renewal emails without 'public_baseurl'")
+
         # Load account validity templates.
         account_validity_template_dir = account_validity_config.get("template_dir")
         if account_validity_template_dir is not None:

synapse/config/cache.py

@@ -15,7 +15,7 @@
 import os
 import re
 import threading
-from typing import Callable, Dict, Optional
+from typing import Callable, Dict
 
 from synapse.python_dependencies import DependencyException, check_requirements
 
@@ -217,7 +217,7 @@ class CacheConfig(Config):
 
         expiry_time = cache_config.get("expiry_time")
         if expiry_time:
-            self.expiry_time_msec: Optional[int] = self.parse_duration(expiry_time)
+            self.expiry_time_msec = self.parse_duration(expiry_time)
         else:
             self.expiry_time_msec = None
 

synapse/config/cas.py

@@ -16,7 +16,7 @@ from typing import Any, List
 
 from synapse.config.sso import SsoAttributeRequirement
 
-from ._base import Config
+from ._base import Config, ConfigError
 from ._util import validate_config
 
 
@@ -35,10 +35,14 @@ class CasConfig(Config):
         if self.cas_enabled:
             self.cas_server_url = cas_config["server_url"]
 
-            # TODO Update this to a _synapse URL.
+            # The public baseurl is required because it is used by the redirect
+            # template.
             public_baseurl = self.root.server.public_baseurl
-            self.cas_service_url = public_baseurl + "_matrix/client/r0/login/cas/ticket"
+            if not public_baseurl:
+                raise ConfigError("cas_config requires a public_baseurl to be set")
 
+            # TODO Update this to a _synapse URL.
+            self.cas_service_url = public_baseurl + "_matrix/client/r0/login/cas/ticket"
             self.cas_displayname_attribute = cas_config.get("displayname_attribute")
             required_attributes = cas_config.get("required_attributes") or {}
             self.cas_required_attributes = _parsed_required_attributes_def(

synapse/config/emailconfig.py

@@ -137,14 +137,33 @@ class EmailConfig(Config):
             if self.root.registration.account_threepid_delegate_email
             else ThreepidBehaviour.LOCAL
         )
+        # Prior to Synapse v1.4.0, there was another option that defined whether Synapse would
+        # use an identity server to password reset tokens on its behalf. We now warn the user
+        # if they have this set and tell them to use the updated option, while using a default
+        # identity server in the process.
+        self.using_identity_server_from_trusted_list = False
+        if (
+            not self.root.registration.account_threepid_delegate_email
+            and config.get("trust_identity_server_for_password_resets", False) is True
+        ):
+            # Use the first entry in self.trusted_third_party_id_servers instead
+            if self.trusted_third_party_id_servers:
+                # XXX: It's a little confusing that account_threepid_delegate_email is modified
+                # both in RegistrationConfig and here. We should factor this bit out
 
-        if config.get("trust_identity_server_for_password_resets"):
-            raise ConfigError(
-                'The config option "trust_identity_server_for_password_resets" '
-                'has been replaced by "account_threepid_delegate". '
-                "Please consult the sample config at docs/sample_config.yaml for "
-                "details and update your config file."
-            )
+                first_trusted_identity_server = self.trusted_third_party_id_servers[0]
+
+                # trusted_third_party_id_servers does not contain a scheme whereas
+                # account_threepid_delegate_email is expected to. Presume https
+                self.root.registration.account_threepid_delegate_email = (
+                    "https://" + first_trusted_identity_server
+                )
+                self.using_identity_server_from_trusted_list = True
+            else:
+                raise ConfigError(
+                    "Attempted to use an identity server from"
+                    '"trusted_third_party_id_servers" but it is empty.'
+                )
 
         self.local_threepid_handling_disabled_due_to_email_config = False
         if (
@@ -167,6 +186,11 @@ class EmailConfig(Config):
             if not self.email_notif_from:
                 missing.append("email.notif_from")
 
+            # public_baseurl is required to build password reset and validation links that
+            # will be emailed to users
+            if config.get("public_baseurl") is None:
+                missing.append("public_baseurl")
+
             if missing:
                 raise ConfigError(
                     MISSING_PASSWORD_RESET_CONFIG_ERROR % (", ".join(missing),)
@@ -272,6 +296,9 @@ class EmailConfig(Config):
             if not self.email_notif_from:
                 missing.append("email.notif_from")
 
+            if config.get("public_baseurl") is None:
+                missing.append("public_baseurl")
+
             if missing:
                 raise ConfigError(
                     "email.enable_notifs is True but required keys are missing: %s"

synapse/config/jwt.py

@@ -31,8 +31,6 @@ class JWTConfig(Config):
             self.jwt_secret = jwt_config["secret"]
             self.jwt_algorithm = jwt_config["algorithm"]
 
-            self.jwt_subject_claim = jwt_config.get("subject_claim", "sub")
-
             # The issuer and audiences are optional, if provided, it is asserted
             # that the claims exist on the JWT.
             self.jwt_issuer = jwt_config.get("issuer")
@@ -48,7 +46,6 @@ class JWTConfig(Config):
             self.jwt_enabled = False
             self.jwt_secret = None
             self.jwt_algorithm = None
-            self.jwt_subject_claim = None
             self.jwt_issuer = None
             self.jwt_audiences = None
 
@@ -91,12 +88,6 @@ class JWTConfig(Config):
             #
             #algorithm: "provided-by-your-issuer"
 
-            # Name of the claim containing a unique identifier for the user.
-            #
-            # Optional, defaults to `sub`.
-            #
-            #subject_claim: "sub"
-
             # The issuer to validate the "iss" claim against.
             #
             # Optional, if provided the "iss" claim will be required and

synapse/config/key.py

@@ -16,7 +16,6 @@
 import hashlib
 import logging
 import os
-from typing import Any, Dict
 
 import attr
 import jsonschema
@@ -313,7 +312,7 @@ class KeyConfig(Config):
                 )
         return keys
 
-    def generate_files(self, config: Dict[str, Any], config_dir_path: str) -> None:
+    def generate_files(self, config, config_dir_path):
         if "signing_key" in config:
             return
 

synapse/config/logger.py

@@ -18,7 +18,7 @@ import os
 import sys
 import threading
 from string import Template
-from typing import TYPE_CHECKING, Any, Dict
+from typing import TYPE_CHECKING
 
 import yaml
 from zope.interface import implementer
@@ -185,7 +185,7 @@ class LoggingConfig(Config):
             help=argparse.SUPPRESS,
         )
 
-    def generate_files(self, config: Dict[str, Any], config_dir_path: str) -> None:
+    def generate_files(self, config, config_dir_path):
         log_config = config.get("log_config")
         if log_config and not os.path.exists(log_config):
             log_file = self.abspath("homeserver.log")

synapse/config/oidc.py

@@ -59,6 +59,8 @@ class OIDCConfig(Config):
                 )
 
         public_baseurl = self.root.server.public_baseurl
+        if public_baseurl is None:
+            raise ConfigError("oidc_config requires a public_baseurl to be set")
         self.oidc_callback_url = public_baseurl + "_synapse/client/oidc/callback"
 
     @property

synapse/config/registration.py

@@ -39,10 +39,23 @@ class RegistrationConfig(Config):
         self.registration_shared_secret = config.get("registration_shared_secret")
 
         self.bcrypt_rounds = config.get("bcrypt_rounds", 12)
-
+        self.trusted_third_party_id_servers = config.get(
+            "trusted_third_party_id_servers", ["matrix.org", "vector.im"]
+        )
         account_threepid_delegates = config.get("account_threepid_delegates") or {}
         self.account_threepid_delegate_email = account_threepid_delegates.get("email")
         self.account_threepid_delegate_msisdn = account_threepid_delegates.get("msisdn")
+        if (
+            self.account_threepid_delegate_msisdn
+            and not self.root.server.public_baseurl
+        ):
+            raise ConfigError(
+                "The configuration option `public_baseurl` is required if "
+                "`account_threepid_delegate.msisdn` is set, such that "
+                "clients know where to submit validation tokens to. Please "
+                "configure `public_baseurl`."
+            )
+
         self.default_identity_server = config.get("default_identity_server")
         self.allow_guest_access = config.get("allow_guest_access", False)
 
@@ -112,32 +125,25 @@ class RegistrationConfig(Config):
             session_lifetime = self.parse_duration(session_lifetime)
         self.session_lifetime = session_lifetime
 
-        # The `refreshable_access_token_lifetime` applies for tokens that can be renewed
+        # The `access_token_lifetime` applies for tokens that can be renewed
         # using a refresh token, as per MSC2918. If it is `None`, the refresh
         # token mechanism is disabled.
         #
         # Since it is incompatible with the `session_lifetime` mechanism, it is set to
         # `None` by default if a `session_lifetime` is set.
-        refreshable_access_token_lifetime = config.get(
-            "refreshable_access_token_lifetime",
-            "5m" if session_lifetime is None else None,
+        access_token_lifetime = config.get(
+            "access_token_lifetime", "5m" if session_lifetime is None else None
         )
-        if refreshable_access_token_lifetime is not None:
-            refreshable_access_token_lifetime = self.parse_duration(
-                refreshable_access_token_lifetime
-            )
-        self.refreshable_access_token_lifetime = refreshable_access_token_lifetime
+        if access_token_lifetime is not None:
+            access_token_lifetime = self.parse_duration(access_token_lifetime)
+        self.access_token_lifetime = access_token_lifetime
 
-        if (
-            session_lifetime is not None
-            and refreshable_access_token_lifetime is not None
-        ):
+        if session_lifetime is not None and access_token_lifetime is not None:
             raise ConfigError(
                 "The refresh token mechanism is incompatible with the "
                 "`session_lifetime` option. Consider disabling the "
                 "`session_lifetime` option or disabling the refresh token "
-                "mechanism by removing the `refreshable_access_token_lifetime` "
-                "option."
+                "mechanism by removing the `access_token_lifetime` option."
             )
 
         # The fallback template used for authenticating using a registration token
@@ -234,7 +240,7 @@ class RegistrationConfig(Config):
         # in on this server.
         #
         # (By default, no suggestion is made, so it is left up to the client.
-        # This setting is ignored unless public_baseurl is also explicitly set.)
+        # This setting is ignored unless public_baseurl is also set.)
         #
         #default_identity_server: https://matrix.org
 
@@ -259,6 +265,8 @@ class RegistrationConfig(Config):
         # by the Matrix Identity Service API specification:
         # https://matrix.org/docs/spec/identity_service/latest
         #
+        # If a delegate is specified, the config option public_baseurl must also be filled out.
+        #
         account_threepid_delegates:
             #email: https://example.com     # Delegate email sending to example.com
             #msisdn: http://localhost:8090  # Delegate SMS sending to this local process

synapse/config/room_directory.py

@@ -1,5 +1,4 @@
 # Copyright 2018 New Vector Ltd
-# Copyright 2021 Matrix.org Foundation C.I.C.
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -13,9 +12,6 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-from typing import List
-
-from synapse.types import JsonDict
 from synapse.util import glob_to_regex
 
 from ._base import Config, ConfigError
@@ -24,7 +20,7 @@ from ._base import Config, ConfigError
 class RoomDirectoryConfig(Config):
     section = "roomdirectory"
 
-    def read_config(self, config, **kwargs) -> None:
+    def read_config(self, config, **kwargs):
         self.enable_room_list_search = config.get("enable_room_list_search", True)
 
         alias_creation_rules = config.get("alias_creation_rules")
@@ -51,7 +47,7 @@ class RoomDirectoryConfig(Config):
                 _RoomDirectoryRule("room_list_publication_rules", {"action": "allow"})
             ]
 
-    def generate_config_section(self, config_dir_path, server_name, **kwargs) -> str:
+    def generate_config_section(self, config_dir_path, server_name, **kwargs):
         return """
         # Uncomment to disable searching the public room list. When disabled
         # blocks searching local and remote room lists for local and remote
@@ -117,16 +113,16 @@ class RoomDirectoryConfig(Config):
         #    action: allow
         """
 
-    def is_alias_creation_allowed(self, user_id: str, room_id: str, alias: str) -> bool:
+    def is_alias_creation_allowed(self, user_id, room_id, alias):
         """Checks if the given user is allowed to create the given alias
 
         Args:
-            user_id: The user to check.
-            room_id: The room ID for the alias.
-            alias: The alias being created.
+            user_id (str)
+            room_id (str)
+            alias (str)
 
         Returns:
-            True if user is allowed to create the alias
+            boolean: True if user is allowed to create the alias
         """
         for rule in self._alias_creation_rules:
             if rule.matches(user_id, room_id, [alias]):
@@ -134,18 +130,16 @@ class RoomDirectoryConfig(Config):
 
         return False
 
-    def is_publishing_room_allowed(
-        self, user_id: str, room_id: str, aliases: List[str]
-    ) -> bool:
+    def is_publishing_room_allowed(self, user_id, room_id, aliases):
         """Checks if the given user is allowed to publish the room
 
         Args:
-            user_id: The user ID publishing the room.
-            room_id: The room being published.
-            aliases: any local aliases associated with the room
+            user_id (str)
+            room_id (str)
+            aliases (list[str]): any local aliases associated with the room
 
         Returns:
-            True if user can publish room
+            boolean: True if user can publish room
         """
         for rule in self._room_list_publication_rules:
             if rule.matches(user_id, room_id, aliases):
@@ -159,11 +153,11 @@ class _RoomDirectoryRule:
     creating an alias or publishing a room.
     """
 
-    def __init__(self, option_name: str, rule: JsonDict):
+    def __init__(self, option_name, rule):
         """
         Args:
-            option_name: Name of the config option this rule belongs to
-            rule: The rule as specified in the config
+            option_name (str): Name of the config option this rule belongs to
+            rule (dict): The rule as specified in the config
         """
 
         action = rule["action"]
@@ -187,18 +181,18 @@ class _RoomDirectoryRule:
         except Exception as e:
             raise ConfigError("Failed to parse glob into regex") from e
 
-    def matches(self, user_id: str, room_id: str, aliases: List[str]) -> bool:
+    def matches(self, user_id, room_id, aliases):
         """Tests if this rule matches the given user_id, room_id and aliases.
 
         Args:
-            user_id: The user ID to check.
-            room_id: The room ID to check.
-            aliases: The associated aliases to the room. Will be a single element
-                for testing alias creation, and can be empty for testing room
-                publishing.
+            user_id (str)
+            room_id (str)
+            aliases (list[str]): The associated aliases to the room. Will be a
+                single element for testing alias creation, and can be empty for
+                testing room publishing.
 
         Returns:
-            True if the rule matches.
+            boolean
         """
 
         # Note: The regexes are anchored at both ends

synapse/config/saml2.py

@@ -199,11 +199,14 @@ class SAML2Config(Config):
         """
         import saml2
 
+        public_baseurl = self.root.server.public_baseurl
+        if public_baseurl is None:
+            raise ConfigError("saml2_config requires a public_baseurl to be set")
+
         if self.saml2_grandfathered_mxid_source_attribute:
             optional_attributes.add(self.saml2_grandfathered_mxid_source_attribute)
         optional_attributes -= required_attributes
 
-        public_baseurl = self.root.server.public_baseurl
         metadata_url = public_baseurl + "_synapse/client/saml2/metadata.xml"
         response_url = public_baseurl + "_synapse/client/saml2/authn_response"
         return {

synapse/config/server.py

@@ -16,7 +16,6 @@ import itertools
 import logging
 import os.path
 import re
-import urllib.parse
 from textwrap import indent
 from typing import Any, Dict, Iterable, List, Optional, Set, Tuple, Union
 
@@ -265,44 +264,10 @@ class ServerConfig(Config):
         self.use_frozen_dicts = config.get("use_frozen_dicts", False)
         self.serve_server_wellknown = config.get("serve_server_wellknown", False)
 
-        # Whether we should serve a "client well-known":
-        #  (a) at .well-known/matrix/client on our client HTTP listener
-        #  (b) in the response to /login
-        #
-        # ... which together help ensure that clients use our public_baseurl instead of
-        # whatever they were told by the user.
-        #
-        # For the sake of backwards compatibility with existing installations, this is
-        # True if public_baseurl is specified explicitly, and otherwise False. (The
-        # reasoning here is that we have no way of knowing that the default
-        # public_baseurl is actually correct for existing installations - many things
-        # will not work correctly, but that's (probably?) better than sending clients
-        # to a completely broken URL.
-        self.serve_client_wellknown = False
-
-        public_baseurl = config.get("public_baseurl")
-        if public_baseurl is None:
-            public_baseurl = f"https://{self.server_name}/"
-            logger.info("Using default public_baseurl %s", public_baseurl)
-        else:
-            self.serve_client_wellknown = True
-            if public_baseurl[-1] != "/":
-                public_baseurl += "/"
-        self.public_baseurl = public_baseurl
-
-        # check that public_baseurl is valid
-        try:
-            splits = urllib.parse.urlsplit(self.public_baseurl)
-        except Exception as e:
-            raise ConfigError(f"Unable to parse URL: {e}", ("public_baseurl",))
-        if splits.scheme not in ("https", "http"):
-            raise ConfigError(
-                f"Invalid scheme '{splits.scheme}': only https and http are supported"
-            )
-        if splits.query or splits.fragment:
-            raise ConfigError(
-                "public_baseurl cannot contain query parameters or a #-fragment"
-            )
+        self.public_baseurl = config.get("public_baseurl")
+        if self.public_baseurl is not None:
+            if self.public_baseurl[-1] != "/":
+                self.public_baseurl += "/"
 
         # Whether to enable user presence.
         presence_config = config.get("presence") or {}
@@ -421,7 +386,7 @@ class ServerConfig(Config):
         # before redacting them.
         redaction_retention_period = config.get("redaction_retention_period", "7d")
         if redaction_retention_period is not None:
-            self.redaction_retention_period: Optional[int] = self.parse_duration(
+            self.redaction_retention_period = self.parse_duration(
                 redaction_retention_period
             )
         else:
@@ -430,7 +395,7 @@ class ServerConfig(Config):
         # How long to keep entries in the `users_ips` table.
         user_ips_max_age = config.get("user_ips_max_age", "28d")
         if user_ips_max_age is not None:
-            self.user_ips_max_age: Optional[int] = self.parse_duration(user_ips_max_age)
+            self.user_ips_max_age = self.parse_duration(user_ips_max_age)
         else:
             self.user_ips_max_age = None
 
@@ -808,8 +773,6 @@ class ServerConfig(Config):
         # Otherwise, it should be the URL to reach Synapse's client HTTP listener (see
         # 'listeners' below).
         #
-        # Defaults to 'https://<server_name>/'.
-        #
         #public_baseurl: https://example.com/
 
         # Uncomment the following to tell other servers to send federation traffic on

synapse/config/sso.py

@@ -101,10 +101,13 @@ class SSOConfig(Config):
         # gracefully to the client). This would make it pointless to ask the user for
         # confirmation, since the URL the confirmation page would be showing wouldn't be
         # the client's.
-        login_fallback_url = (
-            self.root.server.public_baseurl + "_matrix/static/client/login"
-        )
-        self.sso_client_whitelist.append(login_fallback_url)
+        # public_baseurl is an optional setting, so we only add the fallback's URL to the
+        # list if it's provided (because we can't figure out what that URL is otherwise).
+        if self.root.server.public_baseurl:
+            login_fallback_url = (
+                self.root.server.public_baseurl + "_matrix/static/client/login"
+            )
+            self.sso_client_whitelist.append(login_fallback_url)
 
     def generate_config_section(self, **kwargs):
         return """\
@@ -125,10 +128,11 @@ class SSOConfig(Config):
             # phishing attacks from evil.site. To avoid this, include a slash after the
             # hostname: "https://my.client/".
             #
-            # The login fallback page (used by clients that don't natively support the
-            # required login flows) is whitelisted in addition to any URLs in this list.
+            # If public_baseurl is set, then the login fallback page (used by clients
+            # that don't natively support the required login flows) is whitelisted in
+            # addition to any URLs in this list.
             #
-            # By default, this list contains only the login fallback page.
+            # By default, this list is empty.
             #
             #client_whitelist:
             #  - https://riot.im/develop

synapse/config/tls.py

@@ -14,6 +14,7 @@
 
 import logging
 import os
+from datetime import datetime
 from typing import List, Optional, Pattern
 
 from OpenSSL import SSL, crypto
@@ -132,6 +133,55 @@ class TlsConfig(Config):
         self.tls_certificate: Optional[crypto.X509] = None
         self.tls_private_key: Optional[crypto.PKey] = None
 
+    def is_disk_cert_valid(self, allow_self_signed=True):
+        """
+        Is the certificate we have on disk valid, and if so, for how long?
+
+        Args:
+            allow_self_signed (bool): Should we allow the certificate we
+                read to be self signed?
+
+        Returns:
+            int: Days remaining of certificate validity.
+            None: No certificate exists.
+        """
+        if not os.path.exists(self.tls_certificate_file):
+            return None
+
+        try:
+            with open(self.tls_certificate_file, "rb") as f:
+                cert_pem = f.read()
+        except Exception as e:
+            raise ConfigError(
+                "Failed to read existing certificate file %s: %s"
+                % (self.tls_certificate_file, e)
+            )
+
+        try:
+            tls_certificate = crypto.load_certificate(crypto.FILETYPE_PEM, cert_pem)
+        except Exception as e:
+            raise ConfigError(
+                "Failed to parse existing certificate file %s: %s"
+                % (self.tls_certificate_file, e)
+            )
+
+        if not allow_self_signed:
+            if tls_certificate.get_subject() == tls_certificate.get_issuer():
+                raise ValueError(
+                    "TLS Certificate is self signed, and this is not permitted"
+                )
+
+        # YYYYMMDDhhmmssZ -- in UTC
+        expiry_data = tls_certificate.get_notAfter()
+        if expiry_data is None:
+            raise ValueError(
+                "TLS Certificate has no expiry date, and this is not permitted"
+            )
+        expires_on = datetime.strptime(expiry_data.decode("ascii"), "%Y%m%d%H%M%SZ")
+        now = datetime.utcnow()
+        days_remaining = (expires_on - now).days
+        return days_remaining
+
     def read_certificate_from_disk(self):
         """
         Read the certificates and private key from disk.
@@ -213,8 +263,8 @@ class TlsConfig(Config):
         #
         #federation_certificate_verification_whitelist:
         #  - lon.example.com
-        #  - "*.domain.com"
-        #  - "*.onion"
+        #  - *.domain.com
+        #  - *.onion
 
         # List of custom certificate authorities for federation traffic.
         #
@@ -245,7 +295,7 @@ class TlsConfig(Config):
         cert_path = self.tls_certificate_file
         logger.info("Loading TLS certificate from %s", cert_path)
         cert_pem = self.read_file(cert_path, "tls_certificate_path")
-        cert = crypto.load_certificate(crypto.FILETYPE_PEM, cert_pem.encode())
+        cert = crypto.load_certificate(crypto.FILETYPE_PEM, cert_pem)
 
         return cert
 

synapse/config/user_directory.py

@@ -53,8 +53,8 @@ class UserDirectoryConfig(Config):
             # indexes were (re)built was before Synapse 1.44, you'll have to
             # rebuild the indexes in order to search through all known users.
             # These indexes are built the first time Synapse starts; admins can
-            # manually trigger a rebuild via API following the instructions at
-            #     https://matrix-org.github.io/synapse/latest/usage/administration/admin_api/background_updates.html#run
+            # manually trigger a rebuild following the instructions at
+            #     https://matrix-org.github.io/synapse/latest/user_directory.html
             #
             # Uncomment to return search results containing all known users, even if that
             # user does not share a room with the requester.

synapse/config/workers.py

@@ -63,8 +63,7 @@ class WriterLocations:
 
     Attributes:
         events: The instances that write to the event and backfill streams.
-        typing: The instances that write to the typing stream. Currently
-            can only be a single instance.
+        typing: The instance that writes to the typing stream.
         to_device: The instances that write to the to_device stream. Currently
             can only be a single instance.
         account_data: The instances that write to the account data streams. Currently
@@ -76,15 +75,9 @@ class WriterLocations:
     """
 
     events = attr.ib(
-        default=["master"],
-        type=List[str],
-        converter=_instance_to_list_converter,
-    )
-    typing = attr.ib(
-        default=["master"],
-        type=List[str],
-        converter=_instance_to_list_converter,
+        default=["master"], type=List[str], converter=_instance_to_list_converter
     )
+    typing = attr.ib(default="master", type=str)
     to_device = attr.ib(
         default=["master"],
         type=List[str],
@@ -224,11 +217,6 @@ class WorkerConfig(Config):
                         % (instance, stream)
                     )
 
-        if len(self.writers.typing) != 1:
-            raise ConfigError(
-                "Must only specify one instance to handle `typing` messages."
-            )
-
         if len(self.writers.to_device) != 1:
             raise ConfigError(
                 "Must only specify one instance to handle `to_device` messages."

synapse/crypto/keyring.py

@@ -1,4 +1,5 @@
-# Copyright 2014-2021 The Matrix.org Foundation C.I.C.
+# Copyright 2014-2016 OpenMarket Ltd
+# Copyright 2017, 2018 New Vector Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -119,6 +120,16 @@ class VerifyJsonRequest:
             key_ids=key_ids,
         )
 
+    def to_fetch_key_request(self) -> "_FetchKeyRequest":
+        """Create a key fetch request for all keys needed to satisfy the
+        verification request.
+        """
+        return _FetchKeyRequest(
+            server_name=self.server_name,
+            minimum_valid_until_ts=self.minimum_valid_until_ts,
+            key_ids=self.key_ids,
+        )
+
 
 class KeyLookupError(ValueError):
     pass
@@ -168,22 +179,8 @@ class Keyring:
             clock=hs.get_clock(),
             process_batch_callback=self._inner_fetch_key_requests,
         )
-
-        self._hostname = hs.hostname
-
-        # build a FetchKeyResult for each of our own keys, to shortcircuit the
-        # fetcher.
-        self._local_verify_keys: Dict[str, FetchKeyResult] = {}
-        for key_id, key in hs.config.key.old_signing_keys.items():
-            self._local_verify_keys[key_id] = FetchKeyResult(
-                verify_key=key, valid_until_ts=key.expired_ts
-            )
-
-        vk = get_verify_key(hs.signing_key)
-        self._local_verify_keys[f"{vk.alg}:{vk.version}"] = FetchKeyResult(
-            verify_key=vk,
-            valid_until_ts=2 ** 63,  # fake future timestamp
-        )
+        self.verify_key = get_verify_key(hs.signing_key)
+        self.hostname = hs.hostname
 
     async def verify_json_for_server(
         self,
@@ -270,32 +267,22 @@ class Keyring:
                 Codes.UNAUTHORIZED,
             )
 
-        found_keys: Dict[str, FetchKeyResult] = {}
+        # If we are the originating server don't fetch verify key for self over federation
+        if verify_request.server_name == self.hostname:
+            await self._process_json(self.verify_key, verify_request)
+            return
 
-        # If we are the originating server, short-circuit the key-fetch for any keys
-        # we already have
-        if verify_request.server_name == self._hostname:
-            for key_id in verify_request.key_ids:
-                if key_id in self._local_verify_keys:
-                    found_keys[key_id] = self._local_verify_keys[key_id]
+        # Add the keys we need to verify to the queue for retrieval. We queue
+        # up requests for the same server so we don't end up with many in flight
+        # requests for the same keys.
+        key_request = verify_request.to_fetch_key_request()
+        found_keys_by_server = await self._server_queue.add_to_queue(
+            key_request, key=verify_request.server_name
+        )
 
-        key_ids_to_find = set(verify_request.key_ids) - found_keys.keys()
-        if key_ids_to_find:
-            # Add the keys we need to verify to the queue for retrieval. We queue
-            # up requests for the same server so we don't end up with many in flight
-            # requests for the same keys.
-            key_request = _FetchKeyRequest(
-                server_name=verify_request.server_name,
-                minimum_valid_until_ts=verify_request.minimum_valid_until_ts,
-                key_ids=list(key_ids_to_find),
-            )
-            found_keys_by_server = await self._server_queue.add_to_queue(
-                key_request, key=verify_request.server_name
-            )
-
-            # Since we batch up requests the returned set of keys may contain keys
-            # from other servers, so we pull out only the ones we care about.
-            found_keys.update(found_keys_by_server.get(verify_request.server_name, {}))
+        # Since we batch up requests the returned set of keys may contain keys
+        # from other servers, so we pull out only the ones we care about.s
+        found_keys = found_keys_by_server.get(verify_request.server_name, {})
 
         # Verify each signature we got valid keys for, raising if we can't
         # verify any of them.

synapse/events/__init__.py

@@ -16,23 +16,8 @@
 
 import abc
 import os
-from typing import (
-    TYPE_CHECKING,
-    Any,
-    Dict,
-    Generic,
-    Iterable,
-    List,
-    Optional,
-    Sequence,
-    Tuple,
-    Type,
-    TypeVar,
-    Union,
-    overload,
-)
+from typing import Dict, Optional, Tuple, Type
 
-from typing_extensions import Literal
 from unpaddedbase64 import encode_base64
 
 from synapse.api.room_versions import EventFormatVersions, RoomVersion, RoomVersions
@@ -41,9 +26,6 @@ from synapse.util.caches import intern_dict
 from synapse.util.frozenutils import freeze
 from synapse.util.stringutils import strtobool
 
-if TYPE_CHECKING:
-    from synapse.events.builder import EventBuilder
-
 # Whether we should use frozen_dict in FrozenEvent. Using frozen_dicts prevents
 # bugs where we accidentally share e.g. signature dicts. However, converting a
 # dict to frozen_dicts is expensive.
@@ -55,23 +37,7 @@ if TYPE_CHECKING:
 USE_FROZEN_DICTS = strtobool(os.environ.get("SYNAPSE_USE_FROZEN_DICTS", "0"))
 
 
-T = TypeVar("T")
-
-
-# DictProperty (and DefaultDictProperty) require the classes they're used with to
-# have a _dict property to pull properties from.
-#
-# TODO _DictPropertyInstance should not include EventBuilder but due to
-# https://github.com/python/mypy/issues/5570 it thinks the DictProperty and
-# DefaultDictProperty get applied to EventBuilder when it is in a Union with
-# EventBase. This is the least invasive hack to get mypy to comply.
-#
-# Note that DictProperty/DefaultDictProperty cannot actually be used with
-# EventBuilder as it lacks a _dict property.
-_DictPropertyInstance = Union["_EventInternalMetadata", "EventBase", "EventBuilder"]
-
-
-class DictProperty(Generic[T]):
+class DictProperty:
     """An object property which delegates to the `_dict` within its parent object."""
 
     __slots__ = ["key"]
@@ -79,33 +45,12 @@ class DictProperty(Generic[T]):
     def __init__(self, key: str):
         self.key = key
 
-    @overload
-    def __get__(
-        self,
-        instance: Literal[None],
-        owner: Optional[Type[_DictPropertyInstance]] = None,
-    ) -> "DictProperty":
-        ...
-
-    @overload
-    def __get__(
-        self,
-        instance: _DictPropertyInstance,
-        owner: Optional[Type[_DictPropertyInstance]] = None,
-    ) -> T:
-        ...
-
-    def __get__(
-        self,
-        instance: Optional[_DictPropertyInstance],
-        owner: Optional[Type[_DictPropertyInstance]] = None,
-    ) -> Union[T, "DictProperty"]:
+    def __get__(self, instance, owner=None):
         # if the property is accessed as a class property rather than an instance
         # property, return the property itself rather than the value
         if instance is None:
             return self
         try:
-            assert isinstance(instance, (EventBase, _EventInternalMetadata))
             return instance._dict[self.key]
         except KeyError as e1:
             # We want this to look like a regular attribute error (mostly so that
@@ -120,12 +65,10 @@ class DictProperty(Generic[T]):
                 "'%s' has no '%s' property" % (type(instance), self.key)
             ) from e1.__context__
 
-    def __set__(self, instance: _DictPropertyInstance, v: T) -> None:
-        assert isinstance(instance, (EventBase, _EventInternalMetadata))
+    def __set__(self, instance, v):
         instance._dict[self.key] = v
 
-    def __delete__(self, instance: _DictPropertyInstance) -> None:
-        assert isinstance(instance, (EventBase, _EventInternalMetadata))
+    def __delete__(self, instance):
         try:
             del instance._dict[self.key]
         except KeyError as e1:
@@ -134,7 +77,7 @@ class DictProperty(Generic[T]):
             ) from e1.__context__
 
 
-class DefaultDictProperty(DictProperty, Generic[T]):
+class DefaultDictProperty(DictProperty):
     """An extension of DictProperty which provides a default if the property is
     not present in the parent's _dict.
 
@@ -143,34 +86,13 @@ class DefaultDictProperty(DictProperty, Generic[T]):
 
     __slots__ = ["default"]
 
-    def __init__(self, key: str, default: T):
+    def __init__(self, key, default):
         super().__init__(key)
         self.default = default
 
-    @overload
-    def __get__(
-        self,
-        instance: Literal[None],
-        owner: Optional[Type[_DictPropertyInstance]] = None,
-    ) -> "DefaultDictProperty":
-        ...
-
-    @overload
-    def __get__(
-        self,
-        instance: _DictPropertyInstance,
-        owner: Optional[Type[_DictPropertyInstance]] = None,
-    ) -> T:
-        ...
-
-    def __get__(
-        self,
-        instance: Optional[_DictPropertyInstance],
-        owner: Optional[Type[_DictPropertyInstance]] = None,
-    ) -> Union[T, "DefaultDictProperty"]:
+    def __get__(self, instance, owner=None):
         if instance is None:
             return self
-        assert isinstance(instance, (EventBase, _EventInternalMetadata))
         return instance._dict.get(self.key, self.default)
 
 
@@ -189,22 +111,22 @@ class _EventInternalMetadata:
         # in the DAG)
         self.outlier = False
 
-    out_of_band_membership: DictProperty[bool] = DictProperty("out_of_band_membership")
-    send_on_behalf_of: DictProperty[str] = DictProperty("send_on_behalf_of")
-    recheck_redaction: DictProperty[bool] = DictProperty("recheck_redaction")
-    soft_failed: DictProperty[bool] = DictProperty("soft_failed")
-    proactively_send: DictProperty[bool] = DictProperty("proactively_send")
-    redacted: DictProperty[bool] = DictProperty("redacted")
-    txn_id: DictProperty[str] = DictProperty("txn_id")
-    token_id: DictProperty[int] = DictProperty("token_id")
-    historical: DictProperty[bool] = DictProperty("historical")
+    out_of_band_membership: bool = DictProperty("out_of_band_membership")
+    send_on_behalf_of: str = DictProperty("send_on_behalf_of")
+    recheck_redaction: bool = DictProperty("recheck_redaction")
+    soft_failed: bool = DictProperty("soft_failed")
+    proactively_send: bool = DictProperty("proactively_send")
+    redacted: bool = DictProperty("redacted")
+    txn_id: str = DictProperty("txn_id")
+    token_id: int = DictProperty("token_id")
+    historical: bool = DictProperty("historical")
 
     # XXX: These are set by StreamWorkerStore._set_before_and_after.
     # I'm pretty sure that these are never persisted to the database, so shouldn't
     # be here
-    before: DictProperty[RoomStreamToken] = DictProperty("before")
-    after: DictProperty[RoomStreamToken] = DictProperty("after")
-    order: DictProperty[Tuple[int, int]] = DictProperty("order")
+    before: RoomStreamToken = DictProperty("before")
+    after: RoomStreamToken = DictProperty("after")
+    order: Tuple[int, int] = DictProperty("order")
 
     def get_dict(self) -> JsonDict:
         return dict(self._dict)
@@ -240,6 +162,9 @@ class _EventInternalMetadata:
 
         If the sender of the redaction event is allowed to redact any event
         due to auth rules, then this will always return false.
+
+        Returns:
+            bool
         """
         return self._dict.get("recheck_redaction", False)
 
@@ -251,23 +176,32 @@ class _EventInternalMetadata:
                sent to clients.
             2. They should not be added to the forward extremities (and
                therefore not to current state).
+
+        Returns:
+            bool
         """
         return self._dict.get("soft_failed", False)
 
-    def should_proactively_send(self) -> bool:
+    def should_proactively_send(self):
         """Whether the event, if ours, should be sent to other clients and
         servers.
 
         This is used for sending dummy events internally. Servers and clients
         can still explicitly fetch the event.
+
+        Returns:
+            bool
         """
         return self._dict.get("proactively_send", True)
 
-    def is_redacted(self) -> bool:
+    def is_redacted(self):
         """Whether the event has been redacted.
 
         This is used for efficiently checking whether an event has been
         marked as redacted without needing to make another database call.
+
+        Returns:
+            bool
         """
         return self._dict.get("redacted", False)
 
@@ -307,31 +241,29 @@ class EventBase(metaclass=abc.ABCMeta):
 
         self.internal_metadata = _EventInternalMetadata(internal_metadata_dict)
 
-    depth: DictProperty[int] = DictProperty("depth")
-    content: DictProperty[JsonDict] = DictProperty("content")
-    hashes: DictProperty[Dict[str, str]] = DictProperty("hashes")
-    origin: DictProperty[str] = DictProperty("origin")
-    origin_server_ts: DictProperty[int] = DictProperty("origin_server_ts")
-    redacts: DefaultDictProperty[Optional[str]] = DefaultDictProperty("redacts", None)
-    room_id: DictProperty[str] = DictProperty("room_id")
-    sender: DictProperty[str] = DictProperty("sender")
-    # TODO state_key should be Optional[str], this is generally asserted in Synapse
-    # by calling is_state() first (which ensures this), but it is hard (not possible?)
-    # to properly annotate that calling is_state() asserts that state_key exists
-    # and is non-None.
-    state_key: DictProperty[str] = DictProperty("state_key")
-    type: DictProperty[str] = DictProperty("type")
-    user_id: DictProperty[str] = DictProperty("sender")
+    auth_events = DictProperty("auth_events")
+    depth = DictProperty("depth")
+    content = DictProperty("content")
+    hashes = DictProperty("hashes")
+    origin = DictProperty("origin")
+    origin_server_ts = DictProperty("origin_server_ts")
+    prev_events = DictProperty("prev_events")
+    redacts = DefaultDictProperty("redacts", None)
+    room_id = DictProperty("room_id")
+    sender = DictProperty("sender")
+    state_key = DictProperty("state_key")
+    type = DictProperty("type")
+    user_id = DictProperty("sender")
 
     @property
     def event_id(self) -> str:
         raise NotImplementedError()
 
     @property
-    def membership(self) -> str:
+    def membership(self):
         return self.content["membership"]
 
-    def is_state(self) -> bool:
+    def is_state(self):
         return hasattr(self, "state_key") and self.state_key is not None
 
     def get_dict(self) -> JsonDict:
@@ -340,13 +272,13 @@ class EventBase(metaclass=abc.ABCMeta):
 
         return d
 
-    def get(self, key: str, default: Optional[Any] = None) -> Any:
+    def get(self, key, default=None):
         return self._dict.get(key, default)
 
-    def get_internal_metadata_dict(self) -> JsonDict:
+    def get_internal_metadata_dict(self):
         return self.internal_metadata.get_dict()
 
-    def get_pdu_json(self, time_now: Optional[int] = None) -> JsonDict:
+    def get_pdu_json(self, time_now=None) -> JsonDict:
         pdu_json = self.get_dict()
 
         if time_now is not None and "age_ts" in pdu_json["unsigned"]:
@@ -373,46 +305,49 @@ class EventBase(metaclass=abc.ABCMeta):
 
         return template_json
 
-    def __getitem__(self, field: str) -> Optional[Any]:
+    def __set__(self, instance, value):
+        raise AttributeError("Unrecognized attribute %s" % (instance,))
+
+    def __getitem__(self, field):
         return self._dict[field]
 
-    def __contains__(self, field: str) -> bool:
+    def __contains__(self, field):
         return field in self._dict
 
-    def items(self) -> List[Tuple[str, Optional[Any]]]:
+    def items(self):
         return list(self._dict.items())
 
-    def keys(self) -> Iterable[str]:
+    def keys(self):
         return self._dict.keys()
 
-    def prev_event_ids(self) -> Sequence[str]:
+    def prev_event_ids(self):
         """Returns the list of prev event IDs. The order matches the order
         specified in the event, though there is no meaning to it.
 
         Returns:
-            The list of event IDs of this event's prev_events
+            list[str]: The list of event IDs of this event's prev_events
         """
-        return [e for e, _ in self._dict["prev_events"]]
+        return [e for e, _ in self.prev_events]
 
-    def auth_event_ids(self) -> Sequence[str]:
+    def auth_event_ids(self):
         """Returns the list of auth event IDs. The order matches the order
         specified in the event, though there is no meaning to it.
 
         Returns:
-            The list of event IDs of this event's auth_events
+            list[str]: The list of event IDs of this event's auth_events
         """
-        return [e for e, _ in self._dict["auth_events"]]
+        return [e for e, _ in self.auth_events]
 
-    def freeze(self) -> None:
+    def freeze(self):
         """'Freeze' the event dict, so it cannot be modified by accident"""
 
         # this will be a no-op if the event dict is already frozen.
         self._dict = freeze(self._dict)
 
-    def __str__(self) -> str:
+    def __str__(self):
         return self.__repr__()
 
-    def __repr__(self) -> str:
+    def __repr__(self):
         rejection = f"REJECTED={self.rejected_reason}, " if self.rejected_reason else ""
 
         return (
@@ -508,7 +443,7 @@ class FrozenEventV2(EventBase):
         else:
             frozen_dict = event_dict
 
-        self._event_id: Optional[str] = None
+        self._event_id = None
 
         super().__init__(
             frozen_dict,
@@ -520,7 +455,7 @@ class FrozenEventV2(EventBase):
         )
 
     @property
-    def event_id(self) -> str:
+    def event_id(self):
         # We have to import this here as otherwise we get an import loop which
         # is hard to break.
         from synapse.crypto.event_signing import compute_event_reference_hash
@@ -530,23 +465,23 @@ class FrozenEventV2(EventBase):
         self._event_id = "$" + encode_base64(compute_event_reference_hash(self)[1])
         return self._event_id
 
-    def prev_event_ids(self) -> Sequence[str]:
+    def prev_event_ids(self):
         """Returns the list of prev event IDs. The order matches the order
         specified in the event, though there is no meaning to it.
 
         Returns:
-            The list of event IDs of this event's prev_events
+            list[str]: The list of event IDs of this event's prev_events
         """
-        return self._dict["prev_events"]
+        return self.prev_events
 
-    def auth_event_ids(self) -> Sequence[str]:
+    def auth_event_ids(self):
         """Returns the list of auth event IDs. The order matches the order
         specified in the event, though there is no meaning to it.
 
         Returns:
-            The list of event IDs of this event's auth_events
+            list[str]: The list of event IDs of this event's auth_events
         """
-        return self._dict["auth_events"]
+        return self.auth_events
 
 
 class FrozenEventV3(FrozenEventV2):
@@ -555,7 +490,7 @@ class FrozenEventV3(FrozenEventV2):
     format_version = EventFormatVersions.V3  # All events of this type are V3
 
     @property
-    def event_id(self) -> str:
+    def event_id(self):
         # We have to import this here as otherwise we get an import loop which
         # is hard to break.
         from synapse.crypto.event_signing import compute_event_reference_hash
@@ -568,14 +503,12 @@ class FrozenEventV3(FrozenEventV2):
         return self._event_id
 
 
-def _event_type_from_format_version(
-    format_version: int,
-) -> Type[Union[FrozenEvent, FrozenEventV2, FrozenEventV3]]:
+def _event_type_from_format_version(format_version: int) -> Type[EventBase]:
     """Returns the python type to use to construct an Event object for the
     given event format version.
 
     Args:
-        format_version: The event format version
+        format_version (int): The event format version
 
     Returns:
         type: A type that can be initialized as per the initializer of