Changelog

This looks like a copy/paste error: the function doesn't reject anything, but instead allows the action count to go through regardless. The remainder of the function's documentation appears correct.

.github/workflows/docker.yml

@@ -30,7 +30,7 @@ jobs:
         run: docker buildx inspect
 
       - name: Install Cosign
-        uses: sigstore/cosign-installer@v3.6.0
+        uses: sigstore/cosign-installer@v3.5.0
 
       - name: Checkout repository
         uses: actions/checkout@v4

.github/workflows/tests.yml

@@ -305,7 +305,7 @@ jobs:
       - lint-readme
     runs-on: ubuntu-latest
     steps:
-      - uses: matrix-org/done-action@v3
+      - uses: matrix-org/done-action@v2
         with:
           needs: ${{ toJSON(needs) }}
 
@@ -737,7 +737,7 @@ jobs:
       - linting-done
     runs-on: ubuntu-latest
     steps:
-      - uses: matrix-org/done-action@v3
+      - uses: matrix-org/done-action@v2
         with:
           needs: ${{ toJSON(needs) }}
 

CHANGES.md

@@ -1,175 +1,3 @@
-# Synapse 1.113.0rc1 (2024-08-06)
-
-### Features
-
-- Track which rooms have been sent to clients in the experimental [MSC3575](https://github.com/matrix-org/matrix-spec-proposals/pull/3575) Sliding Sync `/sync` endpoint. ([\#17447](https://github.com/element-hq/synapse/issues/17447))
-- Add Account Data extension support to experimental [MSC3575](https://github.com/matrix-org/matrix-spec-proposals/pull/3575) Sliding Sync `/sync` endpoint. ([\#17477](https://github.com/element-hq/synapse/issues/17477))
-- Add receipts extension support to experimental [MSC3575](https://github.com/matrix-org/matrix-spec-proposals/pull/3575) Sliding Sync `/sync` endpoint. ([\#17489](https://github.com/element-hq/synapse/issues/17489))
-- Add typing notification extension support to experimental [MSC3575](https://github.com/matrix-org/matrix-spec-proposals/pull/3575) Sliding Sync `/sync` endpoint. ([\#17505](https://github.com/element-hq/synapse/issues/17505))
-
-### Bugfixes
-
-- Update experimental [MSC3575](https://github.com/matrix-org/matrix-spec-proposals/pull/3575) Sliding Sync `/sync` endpoint to handle invite/knock rooms when filtering. ([\#17450](https://github.com/element-hq/synapse/issues/17450))
-- Fix a bug introduced in v1.110.0 which caused `/keys/query` to return incomplete results, leading to high network activity and CPU usage on Matrix clients. ([\#17499](https://github.com/element-hq/synapse/issues/17499))
-
-### Improved Documentation
-
-- Update the [`allowed_local_3pids`](https://element-hq.github.io/synapse/v1.112/usage/configuration/config_documentation.html#allowed_local_3pids) config option's msisdn address to a working example. ([\#17476](https://github.com/element-hq/synapse/issues/17476))
-
-### Internal Changes
-
-- Change sliding sync to use their own token format in preparation for storing per-connection state. ([\#17452](https://github.com/element-hq/synapse/issues/17452))
-- Ensure we don't send down negative `bump_stamp` in experimental sliding sync endpoint. ([\#17478](https://github.com/element-hq/synapse/issues/17478))
-- Do not send down empty room entries down experimental sliding sync endpoint. ([\#17479](https://github.com/element-hq/synapse/issues/17479))
-- Refactor Sliding Sync tests to better utilize the `SlidingSyncBase`. ([\#17481](https://github.com/element-hq/synapse/issues/17481), [\#17482](https://github.com/element-hq/synapse/issues/17482))
-- Add some opentracing tags and logging to the experimental sliding sync implementation. ([\#17501](https://github.com/element-hq/synapse/issues/17501))
-- Split and move Sliding Sync tests so we have some more sane test file sizes. ([\#17504](https://github.com/element-hq/synapse/issues/17504))
-- Update the `limited` field description in the Sliding Sync response to accurately describe what it actually represents. ([\#17507](https://github.com/element-hq/synapse/issues/17507))
-- Easier to understand `timeline` assertions in Sliding Sync tests. ([\#17511](https://github.com/element-hq/synapse/issues/17511))
-- Reset the sliding sync connection if we don't recognize the per-connection state position. ([\#17529](https://github.com/element-hq/synapse/issues/17529))
-
-
-
-### Updates to locked dependencies
-
-* Bump bcrypt from 4.1.3 to 4.2.0. ([\#17495](https://github.com/element-hq/synapse/issues/17495))
-* Bump black from 24.4.2 to 24.8.0. ([\#17522](https://github.com/element-hq/synapse/issues/17522))
-* Bump phonenumbers from 8.13.39 to 8.13.42. ([\#17521](https://github.com/element-hq/synapse/issues/17521))
-* Bump ruff from 0.5.4 to 0.5.5. ([\#17494](https://github.com/element-hq/synapse/issues/17494))
-* Bump serde_json from 1.0.120 to 1.0.121. ([\#17493](https://github.com/element-hq/synapse/issues/17493))
-* Bump serde_json from 1.0.121 to 1.0.122. ([\#17525](https://github.com/element-hq/synapse/issues/17525))
-* Bump towncrier from 23.11.0 to 24.7.1. ([\#17523](https://github.com/element-hq/synapse/issues/17523))
-* Bump types-pyopenssl from 24.1.0.20240425 to 24.1.0.20240722. ([\#17496](https://github.com/element-hq/synapse/issues/17496))
-* Bump types-setuptools from 70.1.0.20240627 to 71.1.0.20240726. ([\#17497](https://github.com/element-hq/synapse/issues/17497))
-
-# Synapse 1.112.0 (2024-07-30)
-
-This security release is to update our locked dependency on Twisted to 24.7.0rc1, which includes a security fix for [CVE-2024-41671 / GHSA-c8m8-j448-xjx7: Disordered HTTP pipeline response in twisted.web, again](https://github.com/twisted/twisted/security/advisories/GHSA-c8m8-j448-xjx7).
-
-Note that this security fix is also available as **Synapse 1.111.1**, which does not include the rest of the changes in Synapse 1.112.0.
-
-This issue means that, if multiple HTTP requests are pipelined in the same TCP connection, Synapse can send responses to the wrong HTTP request.
-If a reverse proxy was configured to use HTTP pipelining, this could result in responses being sent to the wrong user, severely harming confidentiality.
-
-With that said, despite being a high severity issue, **we consider it unlikely that Synapse installations will be affected**.
-The use of HTTP pipelining in this fashion would cause worse performance for clients (request-response latencies would be increased as users' responses would be artificially blocked behind other users' slow requests). Further, Nginx and Haproxy, two common reverse proxies, do not appear to support configuring their upstreams to use HTTP pipelining and thus would not be affected. For both of these reasons, we consider it unlikely that a Synapse deployment would be set up in such a configuration.
-
-Despite that, we cannot rule out that some installations may exist with this unusual setup and so we are releasing this security update today.
-
-**pip users:** Note that by default, upgrading Synapse using pip will not automatically upgrade Twisted. **Please manually install the new version of Twisted** using `pip install Twisted==24.7.0rc1`. Note also that even the `--upgrade-strategy=eager` flag to `pip install -U matrix-synapse` will not upgrade Twisted to a patched version because it is only a release candidate at this time.
-
-### Internal Changes
-
-- Upgrade locked dependency on Twisted to 24.7.0rc1. ([\#17502](https://github.com/element-hq/synapse/issues/17502))
-
-
-# Synapse 1.112.0rc1 (2024-07-23)
-
-Please note that this release candidate does not include the security dependency update
-included in version 1.111.1 as this version was released before 1.111.1.
-The same security fix can be found in the full release of 1.112.0.
-
-### Features
-
-- Add to-device extension support to experimental [MSC3575](https://github.com/matrix-org/matrix-spec-proposals/pull/3575) Sliding Sync `/sync` endpoint. ([\#17416](https://github.com/element-hq/synapse/issues/17416))
-- Populate `name`/`avatar` fields in experimental [MSC3575](https://github.com/matrix-org/matrix-spec-proposals/pull/3575) Sliding Sync `/sync` endpoint. ([\#17418](https://github.com/element-hq/synapse/issues/17418))
-- Populate `heroes` and room summary fields (`joined_count`, `invited_count`) in experimental [MSC3575](https://github.com/matrix-org/matrix-spec-proposals/pull/3575) Sliding Sync `/sync` endpoint. ([\#17419](https://github.com/element-hq/synapse/issues/17419))
-- Populate `is_dm` room field in experimental [MSC3575](https://github.com/matrix-org/matrix-spec-proposals/pull/3575) Sliding Sync `/sync` endpoint. ([\#17429](https://github.com/element-hq/synapse/issues/17429))
-- Add room subscriptions to experimental [MSC3575](https://github.com/matrix-org/matrix-spec-proposals/pull/3575) Sliding Sync `/sync` endpoint. ([\#17432](https://github.com/element-hq/synapse/issues/17432))
-- Prepare for authenticated media freeze. ([\#17433](https://github.com/element-hq/synapse/issues/17433))
-- Add E2EE extension support to experimental [MSC3575](https://github.com/matrix-org/matrix-spec-proposals/pull/3575) Sliding Sync `/sync` endpoint. ([\#17454](https://github.com/element-hq/synapse/issues/17454))
-
-### Bugfixes
-
-- Add configurable option to always include offline users in presence sync results. Contributed by @Michael-Hollister. ([\#17231](https://github.com/element-hq/synapse/issues/17231))
-- Fix bug in experimental [MSC3575](https://github.com/matrix-org/matrix-spec-proposals/pull/3575) Sliding Sync `/sync` endpoint when using room type filters and the user has one or more remote invites. ([\#17434](https://github.com/element-hq/synapse/issues/17434))
-- Order `heroes` by `stream_ordering` as the Matrix specification states (applies to `/sync`). ([\#17435](https://github.com/element-hq/synapse/issues/17435))
-- Fix rare bug where `/sync` would break for a user when using workers with multiple stream writers. ([\#17438](https://github.com/element-hq/synapse/issues/17438))
-
-### Improved Documentation
-
-- Update the readme image to have a white background, so that it is readable in dark mode. ([\#17387](https://github.com/element-hq/synapse/issues/17387))
-- Add Red Hat Enterprise Linux and Rocky Linux 8 and 9 installation instructions. ([\#17423](https://github.com/element-hq/synapse/issues/17423))
-- Improve documentation for the [`default_power_level_content_override`](https://element-hq.github.io/synapse/latest/usage/configuration/config_documentation.html#default_power_level_content_override) config option. ([\#17451](https://github.com/element-hq/synapse/issues/17451))
-
-### Internal Changes
-
-- Make sure we always use the right logic for enabling the media repo. ([\#17424](https://github.com/element-hq/synapse/issues/17424))
-- Fix argument documentation for method `RateLimiter.record_action`. ([\#17426](https://github.com/element-hq/synapse/issues/17426))
-- Reduce volume of 'Waiting for current token' logs, which were introduced in v1.109.0. ([\#17428](https://github.com/element-hq/synapse/issues/17428))
-- Limit concurrent remote downloads to 6 per IP address, and decrement remote downloads without a content-length from the ratelimiter after the download is complete. ([\#17439](https://github.com/element-hq/synapse/issues/17439))
-- Remove unnecessary call to resume producing in fake channel. ([\#17449](https://github.com/element-hq/synapse/issues/17449))
-- Update experimental [MSC3575](https://github.com/matrix-org/matrix-spec-proposals/pull/3575) Sliding Sync `/sync` endpoint to bump room when it is created. ([\#17453](https://github.com/element-hq/synapse/issues/17453))
-- Speed up generating sliding sync responses. ([\#17458](https://github.com/element-hq/synapse/issues/17458))
-- Add cache to `get_rooms_for_local_user_where_membership_is` to speed up sliding sync. ([\#17460](https://github.com/element-hq/synapse/issues/17460))
-- Speed up fetching room keys from backup. ([\#17461](https://github.com/element-hq/synapse/issues/17461))
-- Speed up sorting of the room list in sliding sync. ([\#17468](https://github.com/element-hq/synapse/issues/17468))
-- Implement handling of `$ME` as a state key in sliding sync. ([\#17469](https://github.com/element-hq/synapse/issues/17469))
-
-
-
-### Updates to locked dependencies
-
-* Bump bytes from 1.6.0 to 1.6.1. ([\#17441](https://github.com/element-hq/synapse/issues/17441))
-* Bump hiredis from 2.3.2 to 3.0.0. ([\#17464](https://github.com/element-hq/synapse/issues/17464))
-* Bump jsonschema from 4.22.0 to 4.23.0. ([\#17444](https://github.com/element-hq/synapse/issues/17444))
-* Bump matrix-org/done-action from 2 to 3. ([\#17440](https://github.com/element-hq/synapse/issues/17440))
-* Bump mypy from 1.9.0 to 1.10.1. ([\#17445](https://github.com/element-hq/synapse/issues/17445))
-* Bump pyopenssl from 24.1.0 to 24.2.1. ([\#17465](https://github.com/element-hq/synapse/issues/17465))
-* Bump ruff from 0.5.0 to 0.5.4. ([\#17466](https://github.com/element-hq/synapse/issues/17466))
-* Bump sentry-sdk from 2.6.0 to 2.8.0. ([\#17456](https://github.com/element-hq/synapse/issues/17456))
-* Bump sentry-sdk from 2.8.0 to 2.10.0. ([\#17467](https://github.com/element-hq/synapse/issues/17467))
-* Bump setuptools from 67.6.0 to 70.0.0. ([\#17448](https://github.com/element-hq/synapse/issues/17448))
-* Bump twine from 5.1.0 to 5.1.1. ([\#17443](https://github.com/element-hq/synapse/issues/17443))
-* Bump types-jsonschema from 4.22.0.20240610 to 4.23.0.20240712. ([\#17446](https://github.com/element-hq/synapse/issues/17446))
-* Bump ulid from 1.1.2 to 1.1.3. ([\#17442](https://github.com/element-hq/synapse/issues/17442))
-* Bump zipp from 3.15.0 to 3.19.1. ([\#17427](https://github.com/element-hq/synapse/issues/17427))
-
-
-# Synapse 1.111.1 (2024-07-30)
-
-This security release is to update our locked dependency on Twisted to 24.7.0rc1, which includes a security fix for [CVE-2024-41671 / GHSA-c8m8-j448-xjx7: Disordered HTTP pipeline response in twisted.web, again](https://github.com/twisted/twisted/security/advisories/GHSA-c8m8-j448-xjx7).
-
-This issue means that, if multiple HTTP requests are pipelined in the same TCP connection, Synapse can send responses to the wrong HTTP request.
-If a reverse proxy was configured to use HTTP pipelining, this could result in responses being sent to the wrong user, severely harming confidentiality.
-
-With that said, despite being a high severity issue, **we consider it unlikely that Synapse installations will be affected**.
-The use of HTTP pipelining in this fashion would cause worse performance for clients (request-response latencies would be increased as users' responses would be artificially blocked behind other users' slow requests). Further, Nginx and Haproxy, two common reverse proxies, do not appear to support configuring their upstreams to use HTTP pipelining and thus would not be affected. For both of these reasons, we consider it unlikely that a Synapse deployment would be set up in such a configuration.
-
-Despite that, we cannot rule out that some installations may exist with this unusual setup and so we are releasing this security update today.
-
-**pip users:** Note that by default, upgrading Synapse using pip will not automatically upgrade Twisted. **Please manually install the new version of Twisted** using `pip install Twisted==24.7.0rc1`. Note also that even the `--upgrade-strategy=eager` flag to `pip install -U matrix-synapse` will not upgrade Twisted to a patched version because it is only a release candidate at this time.
-
-
-### Internal Changes
-
-- Upgrade locked dependency on Twisted to 24.7.0rc1. ([\#17502](https://github.com/element-hq/synapse/issues/17502))
-
-
-# Synapse 1.111.0 (2024-07-16)
-
-No significant changes since 1.111.0rc2.
-
-
-
-
-# Synapse 1.111.0rc2 (2024-07-10)
-
-### Bugfixes
-
-- Fix bug where using `synapse.app.media_repository` worker configuration would break the new media endpoints. ([\#17420](https://github.com/element-hq/synapse/issues/17420))
-
-### Improved Documentation
-
-- Document the new federation media worker endpoints in the [upgrade notes](https://element-hq.github.io/synapse/v1.111/upgrade.html) and [worker docs](https://element-hq.github.io/synapse/v1.111/workers.html). ([\#17421](https://github.com/element-hq/synapse/issues/17421))
-
-### Internal Changes
-
-- Route authenticated federation media requests to media repository workers in Complement tests. ([\#17422](https://github.com/element-hq/synapse/issues/17422))
-
-
-
-
 # Synapse 1.111.0rc1 (2024-07-09)
 
 ### Features

Cargo.lock

@@ -67,9 +67,9 @@ checksum = "79296716171880943b8470b5f8d03aa55eb2e645a4874bdbb28adb49162e012c"
 
 [[package]]
 name = "bytes"
-version = "1.7.1"
+version = "1.6.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "8318a53db07bb3f8dca91a600466bdb3f2eaadeedfdbcf02e1accbad9271ba50"
+checksum = "514de17de45fdb8dc022b1a7975556c53c86f9f0aa5f534b98977b171857c2c9"
 
 [[package]]
 name = "cfg-if"
@@ -444,9 +444,9 @@ dependencies = [
 
 [[package]]
 name = "regex"
-version = "1.10.6"
+version = "1.10.5"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "4219d74c6b67a3654a9fbebc4b419e22126d13d2f3c4a07ee0cb61ff79a79619"
+checksum = "b91213439dad192326a0d7c6ee3955910425f441d7038e0d6933b0aec5c4517f"
 dependencies = [
  "aho-corasick",
  "memchr",
@@ -485,18 +485,18 @@ checksum = "94143f37725109f92c262ed2cf5e59bce7498c01bcc1502d7b9afe439a4e9f49"
 
 [[package]]
 name = "serde"
-version = "1.0.206"
+version = "1.0.204"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "5b3e4cd94123dd520a128bcd11e34d9e9e423e7e3e50425cb1b4b1e3549d0284"
+checksum = "bc76f558e0cbb2a839d37354c575f1dc3fdc6546b5be373ba43d95f231bf7c12"
 dependencies = [
  "serde_derive",
 ]
 
 [[package]]
 name = "serde_derive"
-version = "1.0.206"
+version = "1.0.204"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "fabfb6138d2383ea8208cf98ccf69cdfb1aff4088460681d84189aa259762f97"
+checksum = "e0cd7e117be63d3c3678776753929474f3b04a43a080c744d6b0ae2a8c28e222"
 dependencies = [
  "proc-macro2",
  "quote",
@@ -505,12 +505,11 @@ dependencies = [
 
 [[package]]
 name = "serde_json"
-version = "1.0.124"
+version = "1.0.120"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "66ad62847a56b3dba58cc891acd13884b9c61138d330c0d7b6181713d4fce38d"
+checksum = "4e0d21c9a8cae1235ad58a00c11cb40d4b1e5c784f1ef2c537876ed6ffd8b7c5"
 dependencies = [
  "itoa",
- "memchr",
  "ryu",
  "serde",
 ]
@@ -598,9 +597,9 @@ checksum = "42ff0bf0c66b8238c6f3b578df37d0b7848e55df8577b3f74f92a69acceeb825"
 
 [[package]]
 name = "ulid"
-version = "1.1.3"
+version = "1.1.2"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "04f903f293d11f31c0c29e4148f6dc0d033a7f80cebc0282bea147611667d289"
+checksum = "34778c17965aa2a08913b57e1f34db9b4a63f5de31768b55bf20d2795f921259"
 dependencies = [
  "getrandom",
  "rand",

README.rst

@@ -1,4 +1,4 @@
-.. image:: ./docs/element_logo_white_bg.svg
+.. image:: https://github.com/element-hq/product/assets/87339233/7abf477a-5277-47f3-be44-ea44917d8ed7
    :height: 60px
 
 **Element Synapse - Matrix homeserver implementation**

changelog.d/17418.feature

@@ -0,0 +1 @@
+Populate `name`/`avatar` fields in experimental [MSC3575](https://github.com/matrix-org/matrix-spec-proposals/pull/3575) Sliding Sync `/sync` endpoint.

changelog.d/17426.misc

@@ -0,0 +1 @@
+Fix documentation on `RateLimiter#record_action`.

changelog.d/17483.bugfix

@@ -1 +0,0 @@
-Start handlers for new media endpoints when media resource configured.

changelog.d/17510.bugfix

@@ -1 +0,0 @@
-Fix timeline ordering (using `stream_ordering` instead of topological ordering) in experimental [MSC3575](https://github.com/matrix-org/matrix-spec-proposals/pull/3575) Sliding Sync `/sync` endpoint.

changelog.d/17514.misc

@@ -1 +0,0 @@
-Add more tracing to experimental [MSC3575](https://github.com/matrix-org/matrix-spec-proposals/pull/3575) Sliding Sync `/sync` endpoint.

changelog.d/17515.doc

@@ -1,3 +0,0 @@
-Clarify default behaviour of the
-[`auto_accept_invites.worker_to_run_on`](https://element-hq.github.io/synapse/develop/usage/configuration/config_documentation.html#auto-accept-invites)
-option.

changelog.d/17531.misc

@@ -1 +0,0 @@
-Fixup comment in sliding sync implementation.

changelog.d/17535.bugfix

@@ -1 +0,0 @@
-Fix experimental sliding sync implementation to remember any updates in rooms that were not sent down immediately.

changelog.d/17536.misc

@@ -1 +0,0 @@
-Replace override of deprecated method `HTTPAdapter.get_connection` with `get_connection_with_tls_context`.

changelog.d/17537.misc

@@ -1 +0,0 @@
-Fix performance of device lists in `/key/changes` and sliding sync.

changelog.d/17542.misc

@@ -1 +0,0 @@
-Bump setuptools from 67.6.0 to 72.1.0.

changelog.d/17558.misc

@@ -1 +0,0 @@
-Speed up responding to media requests.

changelog.d/17562.misc

@@ -1 +0,0 @@
-Test github token before running release script steps.

debian/changelog

@@ -1,39 +1,3 @@
-matrix-synapse-py3 (1.113.0~rc1) stable; urgency=medium
-
-  * New Synapse release 1.113.0rc1.
-
- -- Synapse Packaging team <packages@matrix.org>  Tue, 06 Aug 2024 12:23:23 +0100
-
-matrix-synapse-py3 (1.112.0) stable; urgency=medium
-
-  * New Synapse release 1.112.0.
-
- -- Synapse Packaging team <packages@matrix.org>  Tue, 30 Jul 2024 17:15:48 +0100
-
-matrix-synapse-py3 (1.112.0~rc1) stable; urgency=medium
-
-  * New Synapse release 1.112.0rc1.
-
- -- Synapse Packaging team <packages@matrix.org>  Tue, 23 Jul 2024 08:58:55 -0600
-
-matrix-synapse-py3 (1.111.1) stable; urgency=medium
-
-  * New Synapse release 1.111.1.
-
- -- Synapse Packaging team <packages@matrix.org>  Tue, 30 Jul 2024 16:13:52 +0100
-
-matrix-synapse-py3 (1.111.0) stable; urgency=medium
-
-  * New Synapse release 1.111.0.
-
- -- Synapse Packaging team <packages@matrix.org>  Tue, 16 Jul 2024 12:42:46 +0200
-
-matrix-synapse-py3 (1.111.0~rc2) stable; urgency=medium
-
-  * New synapse release 1.111.0rc2.
-
- -- Synapse Packaging team <packages@matrix.org>  Wed, 10 Jul 2024 08:46:54 +0000
-
 matrix-synapse-py3 (1.111.0~rc1) stable; urgency=medium
 
   * New synapse release 1.111.0rc1.

debian/templates

@@ -5,7 +5,7 @@ _Description: Name of the server:
  servers via federation. This is normally the public hostname of the
  server running synapse, but can be different if you set up delegation.
  Please refer to the delegation documentation in this case:
- https://element-hq.github.io/synapse/latest/delegate.html.
+ https://github.com/element-hq/synapse/blob/master/docs/delegate.md.
 
 Template: matrix-synapse/report-stats
 Type: boolean

docker/Dockerfile

@@ -27,7 +27,7 @@ ARG PYTHON_VERSION=3.11
 ###
 # We hardcode the use of Debian bookworm here because this could change upstream
 # and other Dockerfiles used for testing are expecting bookworm.
-FROM docker.io/library/python:${PYTHON_VERSION}-slim-bookworm AS requirements
+FROM docker.io/library/python:${PYTHON_VERSION}-slim-bookworm as requirements
 
 # RUN --mount is specific to buildkit and is documented at
 # https://github.com/moby/buildkit/blob/master/frontend/dockerfile/docs/syntax.md#build-mounts-run---mount.
@@ -87,7 +87,7 @@ RUN if [ -z "$TEST_ONLY_IGNORE_POETRY_LOCKFILE" ]; then \
 ###
 ### Stage 1: builder
 ###
-FROM docker.io/library/python:${PYTHON_VERSION}-slim-bookworm AS builder
+FROM docker.io/library/python:${PYTHON_VERSION}-slim-bookworm as builder
 
 # install the OS build deps
 RUN \

docker/Dockerfile-dhvirtualenv

@@ -24,7 +24,7 @@ ARG distro=""
 # https://launchpad.net/~jyrki-pulliainen/+archive/ubuntu/dh-virtualenv, but
 # it's not obviously easier to use that than to build our own.)
 
-FROM docker.io/library/${distro} AS builder
+FROM docker.io/library/${distro} as builder
 
 RUN apt-get update -qq -o Acquire::Languages=none
 RUN env DEBIAN_FRONTEND=noninteractive apt-get install \

docker/configure_workers_and_start.py

@@ -126,7 +126,6 @@ WORKERS_CONFIG: Dict[str, Dict[str, Any]] = {
             "^/_synapse/admin/v1/media/.*$",
             "^/_synapse/admin/v1/quarantine_media/.*$",
             "^/_matrix/client/v1/media/.*$",
-            "^/_matrix/federation/v1/media/.*$",
         ],
         # The first configured media worker will run the media background jobs
         "shared_extra_conf": {

docs/development/room-dag-concepts.md

@@ -21,10 +21,8 @@ incrementing integer, but backfilled events start with `stream_ordering=-1` and
 
 ---
 
- - Incremental `/sync?since=xxx` returns things in the order they arrive at the server
-   (`stream_ordering`).
- - Initial `/sync`, `/messages` (and `/backfill` in the federation API) return them in
-   the order determined by the event graph `(topological_ordering, stream_ordering)`.
+ - `/sync` returns things in the order they arrive at the server (`stream_ordering`).
+ - `/messages` (and `/backfill` in the federation API) return them in the order determined by the event graph `(topological_ordering, stream_ordering)`.
 
 The general idea is that, if you're following a room in real-time (i.e.
 `/sync`), you probably want to see the messages as they arrive at your server,

docs/element_logo_white_bg.svg

@@ -1,94 +0,0 @@
-<?xml version="1.0" encoding="UTF-8" standalone="no"?>
-<!-- Created with Inkscape (http://www.inkscape.org/) -->
-
-<svg
-   width="41.440346mm"
-   height="10.383124mm"
-   viewBox="0 0 41.440346 10.383125"
-   version="1.1"
-   id="svg1"
-   xml:space="preserve"
-   sodipodi:docname="element_logo_white_bg.svg"
-   xmlns:inkscape="http://www.inkscape.org/namespaces/inkscape"
-   xmlns:sodipodi="http://sodipodi.sourceforge.net/DTD/sodipodi-0.dtd"
-   xmlns="http://www.w3.org/2000/svg"
-   xmlns:svg="http://www.w3.org/2000/svg"><sodipodi:namedview
-     id="namedview1"
-     pagecolor="#ffffff"
-     bordercolor="#000000"
-     borderopacity="0.25"
-     inkscape:showpageshadow="2"
-     inkscape:pageopacity="0.0"
-     inkscape:pagecheckerboard="0"
-     inkscape:deskcolor="#d1d1d1"
-     inkscape:document-units="mm"
-     showgrid="false"
-     inkscape:export-bgcolor="#ffffffff" /><defs
-     id="defs1" /><g
-     id="layer1"
-     transform="translate(-84.803844,-143.2075)"
-     inkscape:export-filename="element_logo_white_bg.svg"
-     inkscape:export-xdpi="96"
-     inkscape:export-ydpi="96"><g
-       style="fill:none"
-       id="g1"
-       transform="matrix(0.26458333,0,0,0.26458333,85.841658,144.26667)"><rect
-         style="display:inline;fill:#ffffff;fill-opacity:1;stroke:#ffffff;stroke-width:1.31041;stroke-dasharray:none;stroke-opacity:1"
-         id="rect20"
-         width="155.31451"
-         height="37.932892"
-         x="-3.2672384"
-         y="-3.3479743"
-         rx="3.3718522"
-         ry="3.7915266"
-         transform="translate(-2.1259843e-6)"
-         inkscape:label="rect20"
-         inkscape:export-filename="rect20.svg"
-         inkscape:export-xdpi="96"
-         inkscape:export-ydpi="96" /><path
-         fill-rule="evenodd"
-         clip-rule="evenodd"
-         d="M 16,32 C 24.8366,32 32,24.8366 32,16 32,7.16344 24.8366,0 16,0 7.16344,0 0,7.16344 0,16 0,24.8366 7.16344,32 16,32 Z"
-         fill="#0dbd8b"
-         id="path1" /><path
-         fill-rule="evenodd"
-         clip-rule="evenodd"
-         d="m 13.0756,7.455 c 0,-0.64584 0.5247,-1.1694 1.1719,-1.1694 4.3864,0 7.9423,3.54853 7.9423,7.9259 0,0.6458 -0.5246,1.1694 -1.1718,1.1694 -0.6472,0 -1.1719,-0.5236 -1.1719,-1.1694 0,-3.0857 -2.5066,-5.58711 -5.5986,-5.58711 -0.6472,0 -1.1719,-0.52355 -1.1719,-1.16939 z"
-         fill="#ffffff"
-         id="path2" /><path
-         fill-rule="evenodd"
-         clip-rule="evenodd"
-         d="m 24.5424,13.042 c 0.6472,0 1.1719,0.5235 1.1719,1.1694 0,4.3773 -3.5559,7.9258 -7.9424,7.9258 -0.6472,0 -1.1718,-0.5235 -1.1718,-1.1693 0,-0.6459 0.5246,-1.1694 1.1718,-1.1694 3.0921,0 5.5987,-2.5015 5.5987,-5.5871 0,-0.6459 0.5247,-1.1694 1.1718,-1.1694 z"
-         fill="#ffffff"
-         id="path3" /><path
-         fill-rule="evenodd"
-         clip-rule="evenodd"
-         d="m 18.9446,24.5446 c 0,0.6459 -0.5247,1.1694 -1.1718,1.1694 -4.3865,0 -7.94239,-3.5485 -7.94239,-7.9258 0,-0.6459 0.52469,-1.1694 1.17179,-1.1694 0.6472,0 1.1719,0.5235 1.1719,1.1694 0,3.0856 2.5066,5.587 5.5987,5.587 0.6471,0 1.1718,0.5236 1.1718,1.1694 z"
-         fill="#ffffff"
-         id="path4" /><path
-         fill-rule="evenodd"
-         clip-rule="evenodd"
-         d="m 7.45823,18.9576 c -0.64718,0 -1.17183,-0.5235 -1.17183,-1.1694 0,-4.3773 3.55591,-7.92581 7.9423,-7.92581 0.6472,0 1.1719,0.52351 1.1719,1.16941 0,0.6458 -0.5247,1.1694 -1.1719,1.1694 -3.092,0 -5.59864,2.5014 -5.59864,5.587 0,0.6459 -0.52465,1.1694 -1.17183,1.1694 z"
-         fill="#ffffff"
-         id="path5" /><path
-         d="M 56.2856,18.1428 H 44.9998 c 0.1334,1.181 0.5619,2.1238 1.2858,2.8286 0.7238,0.6857 1.6761,1.0286 2.8571,1.0286 0.7809,0 1.4857,-0.1905 2.1143,-0.5715 0.6286,-0.3809 1.0762,-0.8952 1.3428,-1.5428 h 3.4286 c -0.4571,1.5047 -1.3143,2.7238 -2.5714,3.6571 -1.2381,0.9143 -2.7048,1.3715 -4.4,1.3715 -2.2095,0 -4,-0.7334 -5.3714,-2.2 -1.3524,-1.4667 -2.0286,-3.3239 -2.0286,-5.5715 0,-2.1905 0.6857,-4.0285 2.0571,-5.5143 1.3715,-1.4857 3.1429,-2.22853 5.3143,-2.22853 2.1714,0 3.9238,0.73333 5.2572,2.20003 1.3523,1.4476 2.0285,3.2762 2.0285,5.4857 z m -7.2572,-5.9714 c -1.0667,0 -1.9524,0.3143 -2.6571,0.9429 -0.7048,0.6285 -1.1429,1.4666 -1.3143,2.5142 h 7.8857 c -0.1524,-1.0476 -0.5714,-1.8857 -1.2571,-2.5142 -0.6858,-0.6286 -1.5715,-0.9429 -2.6572,-0.9429 z"
-         fill="#000000"
-         id="path6" /><path
-         d="M 58.6539,20.1428 V 3.14282 h 3.4 V 20.2 c 0,0.7619 0.419,1.1428 1.2571,1.1428 l 0.6,-0.0285 v 3.2285 c -0.3238,0.0572 -0.6667,0.0857 -1.0286,0.0857 -1.4666,0 -2.5428,-0.3714 -3.2285,-1.1142 -0.6667,-0.7429 -1,-1.8667 -1,-3.3715 z"
-         fill="#000000"
-         id="path7" /><path
-         d="M 79.7454,18.1428 H 68.4597 c 0.1333,1.181 0.5619,2.1238 1.2857,2.8286 0.7238,0.6857 1.6762,1.0286 2.8571,1.0286 0.781,0 1.4857,-0.1905 2.1143,-0.5715 0.6286,-0.3809 1.0762,-0.8952 1.3429,-1.5428 h 3.4285 c -0.4571,1.5047 -1.3143,2.7238 -2.5714,3.6571 -1.2381,0.9143 -2.7048,1.3715 -4.4,1.3715 -2.2095,0 -4,-0.7334 -5.3714,-2.2 -1.3524,-1.4667 -2.0286,-3.3239 -2.0286,-5.5715 0,-2.1905 0.6857,-4.0285 2.0571,-5.5143 1.3715,-1.4857 3.1429,-2.22853 5.3143,-2.22853 2.1715,0 3.9238,0.73333 5.2572,2.20003 1.3524,1.4476 2.0285,3.2762 2.0285,5.4857 z m -7.2572,-5.9714 c -1.0666,0 -1.9524,0.3143 -2.6571,0.9429 -0.7048,0.6285 -1.1429,1.4666 -1.3143,2.5142 h 7.8857 c -0.1524,-1.0476 -0.5714,-1.8857 -1.2571,-2.5142 -0.6857,-0.6286 -1.5715,-0.9429 -2.6572,-0.9429 z"
-         fill="#000000"
-         id="path8" /><path
-         d="m 95.0851,16.0571 v 8.5143 h -3.4 v -8.8857 c 0,-2.2476 -0.9333,-3.3714 -2.8,-3.3714 -1.0095,0 -1.819,0.3238 -2.4286,0.9714 -0.5904,0.6476 -0.8857,1.5333 -0.8857,2.6571 v 8.6286 h -3.4 V 9.74282 h 3.1429 v 1.97148 c 0.3619,-0.6667 0.9143,-1.2191 1.6571,-1.6572 0.7429,-0.43809 1.6667,-0.65713 2.7714,-0.65713 2.0572,0 3.5429,0.78093 4.4572,2.34283 1.2571,-1.5619 2.9333,-2.34283 5.0286,-2.34283 1.733,0 3.067,0.54285 4,1.62853 0.933,1.0667 1.4,2.4762 1.4,4.2286 v 9.3143 h -3.4 v -8.8857 c 0,-2.2476 -0.933,-3.3714 -2.8,-3.3714 -1.0286,0 -1.8477,0.3333 -2.4572,1 -0.5905,0.6476 -0.8857,1.5619 -0.8857,2.7428 z"
-         fill="#000000"
-         id="path9" /><path
-         d="m 121.537,18.1428 h -11.286 c 0.133,1.181 0.562,2.1238 1.286,2.8286 0.723,0.6857 1.676,1.0286 2.857,1.0286 0.781,0 1.486,-0.1905 2.114,-0.5715 0.629,-0.3809 1.076,-0.8952 1.343,-1.5428 h 3.429 c -0.458,1.5047 -1.315,2.7238 -2.572,3.6571 -1.238,0.9143 -2.705,1.3715 -4.4,1.3715 -2.209,0 -4,-0.7334 -5.371,-2.2 -1.353,-1.4667 -2.029,-3.3239 -2.029,-5.5715 0,-2.1905 0.686,-4.0285 2.057,-5.5143 1.372,-1.4857 3.143,-2.22853 5.315,-2.22853 2.171,0 3.923,0.73333 5.257,2.20003 1.352,1.4476 2.028,3.2762 2.028,5.4857 z m -7.257,-5.9714 c -1.067,0 -1.953,0.3143 -2.658,0.9429 -0.704,0.6285 -1.142,1.4666 -1.314,2.5142 h 7.886 c -0.153,-1.0476 -0.572,-1.8857 -1.257,-2.5142 -0.686,-0.6286 -1.572,-0.9429 -2.657,-0.9429 z"
-         fill="#000000"
-         id="path10" /><path
-         d="m 127.105,9.74282 v 1.97148 c 0.343,-0.6477 0.905,-1.1905 1.686,-1.6286 0.8,-0.45716 1.762,-0.68573 2.885,-0.68573 1.753,0 3.105,0.53333 4.058,1.60003 0.971,1.0666 1.457,2.4857 1.457,4.2571 v 9.3143 h -3.4 v -8.8857 c 0,-1.0476 -0.248,-1.8667 -0.743,-2.4572 -0.476,-0.6095 -1.21,-0.9142 -2.2,-0.9142 -1.086,0 -1.943,0.3238 -2.572,0.9714 -0.609,0.6476 -0.914,1.5428 -0.914,2.6857 v 8.6 h -3.4 V 9.74282 Z"
-         fill="#000000"
-         id="path11" /><path
-         d="m 147.12,21.5428 v 2.9429 c -0.419,0.1143 -1.009,0.1714 -1.771,0.1714 -2.895,0 -4.343,-1.4571 -4.343,-4.3714 v -7.8286 h -2.257 V 9.74282 h 2.257 V 5.88568 h 3.4 v 3.85714 h 2.772 v 2.71428 h -2.772 v 7.4857 c 0,1.1619 0.552,1.7429 1.657,1.7429 z"
-         fill="#000000"
-         id="path12" /></g></g></svg>

docs/setup/forward_proxy.md

@@ -67,7 +67,7 @@ in Synapse can be deactivated.
 **NOTE**: This has an impact on security and is for testing purposes only!
 
 To deactivate the certificate validation, the following setting must be added to
-your [homeserver.yaml](../usage/configuration/homeserver_sample_config.md).
+your [homserver.yaml](../usage/configuration/homeserver_sample_config.md).
 
 ```yaml
 use_insecure_ssl_client_just_for_testing_do_not_use: true

docs/setup/installation.md

@@ -309,62 +309,7 @@ sudo dnf install libtiff-devel libjpeg-devel libzip-devel freetype-devel \
                  libwebp-devel libxml2-devel libxslt-devel libpq-devel \
                  python3-virtualenv libffi-devel openssl-devel python3-devel \
                  libicu-devel
-sudo dnf group install "Development Tools"
-```
-
-##### Red Hat Enterprise Linux / Rocky Linux
-
-*Note: The term "RHEL" below refers to both Red Hat Enterprise Linux and Rocky Linux. The distributions are 1:1 binary compatible.*
-
-It's recommended to use the latest Python versions. 
-
-RHEL 8 in particular ships with Python 3.6 by default which is EOL and therefore no longer supported by Synapse. RHEL 9 ship with Python 3.9 which is still supported by the Python core team as of this writing. However, newer Python versions provide significant performance improvements and they're available in official distributions' repositories. Therefore it's recommended to use them.
-
-Python 3.11 and 3.12 are available for both RHEL 8 and 9.
-
-These commands should be run as root user.
-
-RHEL 8
-```bash
-# Enable PowerTools repository
-dnf config-manager --set-enabled powertools
-```
-RHEL 9
-```bash
-# Enable CodeReady Linux Builder repository
-crb enable
-```
-
-Install new version of Python. You only need one of these:
-```bash
-# Python 3.11
-dnf install python3.11 python3.11-devel
-```
-```bash
-# Python 3.12
-dnf install python3.12 python3.12-devel
-```
-Finally, install common prerequisites
-```bash
-dnf install libicu libicu-devel libpq5 libpq5-devel lz4 pkgconf 
-dnf group install "Development Tools"
-```
-###### Using venv module instead of virtualenv command
-
-It's recommended to use Python venv module directly rather than the virtualenv command.
-* On RHEL 9, virtualenv is only available on [EPEL](https://docs.fedoraproject.org/en-US/epel/).
-* On RHEL 8, virtualenv is based on Python 3.6. It does not support creating 3.11/3.12 virtual environments.
-
-Here's an example of creating Python 3.12 virtual environment and installing Synapse from PyPI. 
-
-```bash
-mkdir -p ~/synapse
-# To use Python 3.11, simply use the command "python3.11" instead.
-python3.12 -m venv ~/synapse/env
-source ~/synapse/env/bin/activate
-pip install --upgrade pip
-pip install --upgrade setuptools
-pip install matrix-synapse
+sudo dnf groupinstall "Development Tools"
 ```
 
 ##### macOS

docs/upgrade.md

@@ -119,14 +119,13 @@ stacking them up. You can monitor the currently running background updates with
 
 # Upgrading to v1.111.0
 
-## New worker endpoints for authenticated client and federation media
+## New worker endpoints for authenticated client media
 
 [Media repository workers](./workers.md#synapseappmedia_repository) handling
-Media APIs can now handle the following endpoint patterns:
+Media APIs can now handle the following endpoint pattern:
 
 ```
 ^/_matrix/client/v1/media/.*$
-^/_matrix/federation/v1/media/.*$
 ```
 
 Please update your reverse proxy configuration.

docs/usage/configuration/config_documentation.md

@@ -246,7 +246,6 @@ Example configuration:
 ```yaml
 presence:
   enabled: false
-  include_offline_users_on_sync: false
 ```
 
 `enabled` can also be set to a special value of "untracked" which ignores updates
@@ -255,10 +254,6 @@ received via clients and federation, while still accepting updates from the
 
 *The "untracked" option was added in Synapse 1.96.0.*
 
-When clients perform an initial or `full_state` sync, presence results for offline users are
-not included by default. Setting `include_offline_users_on_sync` to `true` will always include
-offline users in the results. Defaults to false.
-
 ---
 ### `require_auth_for_profile_requests`
 
@@ -1868,18 +1863,6 @@ federation_rr_transactions_per_room_per_second: 40
 ## Media Store
 Config options related to Synapse's media store.
 
----
-### `enable_authenticated_media`
-
-When set to true, all subsequent media uploads will be marked as authenticated, and will not be available over legacy
-unauthenticated media endpoints (`/_matrix/media/(r0|v3|v1)/download` and `/_matrix/media/(r0|v3|v1)/thumbnail`) - requests for authenticated media over these endpoints will result in a 404. All media, including authenticated media, will be available over the authenticated media endpoints `_matrix/client/v1/media/download` and `_matrix/client/v1/media/thumbnail`. Media uploaded prior to setting this option to true will still be available over the legacy endpoints. Note if the setting is switched to false
-after enabling, media marked as authenticated will be available over legacy endpoints. Defaults to false, but
-this will change to true in a future Synapse release.
-
-Example configuration:
-```yaml
-enable_authenticated_media: true
-```
 ---
 ### `enable_media_repo`
 
@@ -2386,7 +2369,7 @@ enable_registration_without_verification: true
 ---
 ### `registrations_require_3pid`
 
-If this is set, users must provide all of the specified types of [3PID](https://spec.matrix.org/latest/appendices/#3pid-types) when registering an account.
+If this is set, users must provide all of the specified types of 3PID when registering an account.
 
 Note that [`enable_registration`](#enable_registration) must also be set to allow account registration.
 
@@ -2411,9 +2394,6 @@ disable_msisdn_registration: true
 
 Mandate that users are only allowed to associate certain formats of
 3PIDs with accounts on this server, as specified by the `medium` and `pattern` sub-options.
-`pattern` is a [Perl-like regular expression](https://docs.python.org/3/library/re.html#module-re).
-
-More information about 3PIDs, allowed `medium` types and their `address` syntax can be found [in the Matrix spec](https://spec.matrix.org/latest/appendices/#3pid-types).
 
 Example configuration:
 ```yaml
@@ -2423,7 +2403,7 @@ allowed_local_3pids:
   - medium: email
     pattern: '^[^@]+@vector\.im$'
   - medium: msisdn
-    pattern: '^44\d{10}$'
+    pattern: '\+44'
 ```
 ---
 ### `enable_3pid_lookup`
@@ -4154,38 +4134,6 @@ default_power_level_content_override:
    trusted_private_chat: null
    public_chat: null
 ```
-
-The default power levels for each preset are:
-```yaml
-"m.room.name": 50
-"m.room.power_levels": 100
-"m.room.history_visibility": 100
-"m.room.canonical_alias": 50
-"m.room.avatar": 50
-"m.room.tombstone": 100
-"m.room.server_acl": 100
-"m.room.encryption": 100
-```
-
-So a complete example where the default power-levels for a preset are maintained
-but the power level for a new key is set is:
-```yaml
-default_power_level_content_override:
-   private_chat:
-    events:
-      "com.example.foo": 0
-      "m.room.name": 50
-      "m.room.power_levels": 100
-      "m.room.history_visibility": 100
-      "m.room.canonical_alias": 50
-      "m.room.avatar": 50
-      "m.room.tombstone": 100
-      "m.room.server_acl": 100
-      "m.room.encryption": 100
-   trusted_private_chat: null
-   public_chat: null
-```
-
 ---
 ### `forget_rooms_on_leave`
 
@@ -4685,9 +4633,7 @@ This setting has the following sub-options:
 * `only_for_direct_messages`: Whether invites should be automatically accepted for all room types, or only
    for direct messages. Defaults to false.
 * `only_from_local_users`: Whether to only automatically accept invites from users on this homeserver. Defaults to false.
-* `worker_to_run_on`: Which worker to run this module on. This must match 
-  the "worker_name". If not set or `null`, invites will be accepted on the
-  main process.
+* `worker_to_run_on`: Which worker to run this module on. This must match the "worker_name".
 
 NOTE: Care should be taken not to enable this setting if the `synapse_auto_accept_invite` module is enabled and installed.
 The two modules will compete to perform the same task and may result in undesired behaviour. For example, multiple join

docs/workers.md

@@ -740,7 +740,6 @@ Handles the media repository. It can handle all endpoints starting with:
 
     /_matrix/media/
     /_matrix/client/v1/media/
-    /_matrix/federation/v1/media/
 
 ... and the following regular expressions matching media-specific administration APIs:
 

pyproject.toml

@@ -97,7 +97,7 @@ module-name = "synapse.synapse_rust"
 
 [tool.poetry]
 name = "matrix-synapse"
-version = "1.113.0rc1"
+version = "1.111.0rc1"
 description = "Homeserver for the Matrix decentralised comms protocol"
 authors = ["Matrix.org Team and Contributors <packages@matrix.org>"]
 license = "AGPL-3.0-or-later"
@@ -201,8 +201,8 @@ netaddr = ">=0.7.18"
 # add a lower bound to the Jinja2 dependency.
 Jinja2 = ">=3.0"
 bleach = ">=1.4.3"
-# We use `assert_never`, which were added in `typing-extensions` 4.1.
-typing-extensions = ">=4.1"
+# We use `Self`, which were added in `typing-extensions` 4.0.
+typing-extensions = ">=4.0"
 # We enforce that we have a `cryptography` version that bundles an `openssl`
 # with the latest security patches.
 cryptography = ">=3.4.7"
@@ -322,7 +322,7 @@ all = [
 # This helps prevents merge conflicts when running a batch of dependabot updates.
 isort = ">=5.10.1"
 black = ">=22.7.0"
-ruff = "0.5.5"
+ruff = "0.5.0"
 # Type checking only works with the pydantic.v1 compat module from pydantic v2
 pydantic = "^2"
 

scripts-dev/federation_client.py

@@ -43,7 +43,7 @@ import argparse
 import base64
 import json
 import sys
-from typing import Any, Dict, Mapping, Optional, Tuple, Union
+from typing import Any, Dict, Optional, Tuple
 from urllib import parse as urlparse
 
 import requests
@@ -75,7 +75,7 @@ def encode_canonical_json(value: object) -> bytes:
         value,
         # Encode code-points outside of ASCII as UTF-8 rather than \u escapes
         ensure_ascii=False,
-        # Remove unnecessary white space.
+        # Remove unecessary white space.
         separators=(",", ":"),
         # Sort the keys of dictionaries.
         sort_keys=True,
@@ -298,23 +298,12 @@ class MatrixConnectionAdapter(HTTPAdapter):
 
         return super().send(request, *args, **kwargs)
 
-    def get_connection_with_tls_context(
-        self,
-        request: PreparedRequest,
-        verify: Optional[Union[bool, str]],
-        proxies: Optional[Mapping[str, str]] = None,
-        cert: Optional[Union[Tuple[str, str], str]] = None,
+    def get_connection(
+        self, url: str, proxies: Optional[Dict[str, str]] = None
     ) -> HTTPConnectionPool:
-        # overrides the get_connection_with_tls_context() method in the base class
-        parsed = urlparse.urlsplit(request.url)
-
-        # Extract the server name from the request URL, and ensure it's a str.
-        hostname = parsed.netloc
-        if isinstance(hostname, bytes):
-            hostname = hostname.decode("utf-8")
-        assert isinstance(hostname, str)
-
-        (host, port, ssl_server_name) = self._lookup(hostname)
+        # overrides the get_connection() method in the base class
+        parsed = urlparse.urlsplit(url)
+        (host, port, ssl_server_name) = self._lookup(parsed.netloc)
         print(
             f"Connecting to {host}:{port} with SNI {ssl_server_name}", file=sys.stderr
         )

scripts-dev/release.py

@@ -324,11 +324,6 @@ def tag(gh_token: Optional[str]) -> None:
 def _tag(gh_token: Optional[str]) -> None:
     """Tags the release and generates a draft GitHub release"""
 
-    if gh_token:
-        # Test that the GH Token is valid before continuing.
-        gh = Github(gh_token)
-        gh.get_user()
-
     # Make sure we're in a git repo.
     repo = get_repo_and_check_clean_checkout()
 
@@ -423,11 +418,6 @@ def publish(gh_token: str) -> None:
 def _publish(gh_token: str) -> None:
     """Publish release on GitHub."""
 
-    if gh_token:
-        # Test that the GH Token is valid before continuing.
-        gh = Github(gh_token)
-        gh.get_user()
-
     # Make sure we're in a git repo.
     get_repo_and_check_clean_checkout()
 
@@ -470,11 +460,6 @@ def upload(gh_token: Optional[str]) -> None:
 def _upload(gh_token: Optional[str]) -> None:
     """Upload release to pypi."""
 
-    if gh_token:
-        # Test that the GH Token is valid before continuing.
-        gh = Github(gh_token)
-        gh.get_user()
-
     current_version = get_package_version()
     tag_name = f"v{current_version}"
 
@@ -570,11 +555,6 @@ def wait_for_actions(gh_token: Optional[str]) -> None:
 
 
 def _wait_for_actions(gh_token: Optional[str]) -> None:
-    if gh_token:
-        # Test that the GH Token is valid before continuing.
-        gh = Github(gh_token)
-        gh.get_user()
-
     # Find out the version and tag name.
     current_version = get_package_version()
     tag_name = f"v{current_version}"
@@ -731,11 +711,6 @@ Ask the designated people to do the blog and tweets."""
 @cli.command()
 @click.option("--gh-token", envvar=["GH_TOKEN", "GITHUB_TOKEN"], required=True)
 def full(gh_token: str) -> None:
-    if gh_token:
-        # Test that the GH Token is valid before continuing.
-        gh = Github(gh_token)
-        gh.get_user()
-
     click.echo("1. If this is a security release, read the security wiki page.")
     click.echo("2. Check for any release blockers before proceeding.")
     click.echo("    https://github.com/element-hq/synapse/labels/X-Release-Blocker")

synapse/_scripts/synapse_port_db.py

@@ -119,19 +119,18 @@ BOOLEAN_COLUMNS = {
     "e2e_room_keys": ["is_verified"],
     "event_edges": ["is_state"],
     "events": ["processed", "outlier", "contains_url"],
-    "local_media_repository": ["safe_from_quarantine", "authenticated"],
-    "per_user_experimental_features": ["enabled"],
+    "local_media_repository": ["safe_from_quarantine"],
     "presence_list": ["accepted"],
     "presence_stream": ["currently_active"],
     "public_room_list_stream": ["visibility"],
     "pushers": ["enabled"],
     "redactions": ["have_censored"],
-    "remote_media_cache": ["authenticated"],
     "room_stats_state": ["is_federatable"],
     "rooms": ["is_public", "has_auth_chain_index"],
     "users": ["shadow_banned", "approved", "locked", "suspended"],
     "un_partial_stated_event_stream": ["rejection_status_changed"],
     "users_who_share_rooms": ["share_private"],
+    "per_user_experimental_features": ["enabled"],
 }
 
 

synapse/api/constants.py

@@ -50,7 +50,7 @@ class Membership:
     KNOCK: Final = "knock"
     LEAVE: Final = "leave"
     BAN: Final = "ban"
-    LIST: Final = frozenset((INVITE, JOIN, KNOCK, LEAVE, BAN))
+    LIST: Final = {INVITE, JOIN, KNOCK, LEAVE, BAN}
 
 
 class PresenceState:
@@ -225,11 +225,6 @@ class EventContentFields:
     # This is deprecated in MSC2175.
     ROOM_CREATOR: Final = "creator"
 
-    # The version of the room for `m.room.create` events.
-    ROOM_VERSION: Final = "room_version"
-
-    ROOM_NAME: Final = "name"
-
     # Used in m.room.guest_access events.
     GUEST_ACCESS: Final = "guest_access"
 
@@ -242,9 +237,6 @@ class EventContentFields:
     # an unspecced field added to to-device messages to identify them uniquely-ish
     TO_DEVICE_MSGID: Final = "org.matrix.msgid"
 
-    # `m.room.encryption`` algorithm field
-    ENCRYPTION_ALGORITHM: Final = "algorithm"
-
 
 class EventUnsignedContentFields:
     """Fields found inside the 'unsigned' data on events"""

synapse/api/errors.py

@@ -128,10 +128,6 @@ class Codes(str, Enum):
     # MSC2677
     DUPLICATE_ANNOTATION = "M_DUPLICATE_ANNOTATION"
 
-    # MSC3575 we are telling the client they need to expire their sliding sync
-    # connection.
-    UNKNOWN_POS = "M_UNKNOWN_POS"
-
 
 class CodeMessageException(RuntimeError):
     """An exception with integer code, a message string attributes and optional headers.
@@ -851,17 +847,3 @@ class PartialStateConflictError(SynapseError):
             msg=PartialStateConflictError.message(),
             errcode=Codes.UNKNOWN,
         )
-
-
-class SlidingSyncUnknownPosition(SynapseError):
-    """An error that Synapse can return to signal to the client to expire their
-    sliding sync connection (i.e. send a new request without a `?since=`
-    param).
-    """
-
-    def __init__(self) -> None:
-        super().__init__(
-            HTTPStatus.BAD_REQUEST,
-            msg="Unknown position",
-            errcode=Codes.UNKNOWN_POS,
-        )

synapse/app/generic_worker.py

@@ -206,21 +206,6 @@ class GenericWorkerServer(HomeServer):
                                 "/_synapse/admin": admin_resource,
                             }
                         )
-
-                        if "federation" not in res.names:
-                            # Only load the federation media resource separately if federation
-                            # resource is not specified since federation resource includes media
-                            # resource.
-                            resources[FEDERATION_PREFIX] = TransportLayerServer(
-                                self, servlet_groups=["media"]
-                            )
-                        if "client" not in res.names:
-                            # Only load the client media resource separately if client
-                            # resource is not specified since client resource includes media
-                            # resource.
-                            resources[CLIENT_API_PREFIX] = ClientRestResource(
-                                self, servlet_groups=["media"]
-                            )
                     else:
                         logger.warning(
                             "A 'media' listener is configured but the media"

synapse/app/homeserver.py

@@ -101,12 +101,6 @@ class SynapseHomeServer(HomeServer):
                     # Skip loading openid resource if federation is defined
                     # since federation resource will include openid
                     continue
-                if name == "media" and (
-                    "federation" in res.names or "client" in res.names
-                ):
-                    # Skip loading media resource if federation or client are defined
-                    # since federation & client resources will include media
-                    continue
                 if name == "health":
                     # Skip loading, health resource is always included
                     continue
@@ -223,7 +217,7 @@ class SynapseHomeServer(HomeServer):
             )
 
         if name in ["media", "federation", "client"]:
-            if self.config.media.can_load_media_repo:
+            if self.config.server.enable_media_repo:
                 media_repo = self.get_media_repository_resource()
                 resources.update(
                     {
@@ -237,14 +231,6 @@ class SynapseHomeServer(HomeServer):
                     "'media' resource conflicts with enable_media_repo=False"
                 )
 
-        if name == "media":
-            resources[FEDERATION_PREFIX] = TransportLayerServer(
-                self, servlet_groups=["media"]
-            )
-            resources[CLIENT_API_PREFIX] = ClientRestResource(
-                self, servlet_groups=["media"]
-            )
-
         if name in ["keys", "federation"]:
             resources[SERVER_KEY_PREFIX] = KeyResource(self)
 

synapse/config/repository.py

@@ -126,7 +126,7 @@ class ContentRepositoryConfig(Config):
         # Only enable the media repo if either the media repo is enabled or the
         # current worker app is the media repo.
         if (
-            config.get("enable_media_repo", True) is False
+            self.root.server.enable_media_repo is False
             and config.get("worker_app") != "synapse.app.media_repository"
         ):
             self.can_load_media_repo = False
@@ -272,10 +272,6 @@ class ContentRepositoryConfig(Config):
                 remote_media_lifetime
             )
 
-        self.enable_authenticated_media = config.get(
-            "enable_authenticated_media", False
-        )
-
     def generate_config_section(self, data_dir_path: str, **kwargs: Any) -> str:
         assert data_dir_path is not None
         media_store = os.path.join(data_dir_path, "media_store")

synapse/config/server.py

@@ -384,11 +384,6 @@ class ServerConfig(Config):
         # Whether to internally track presence, requires that presence is enabled,
         self.track_presence = self.presence_enabled and presence_enabled != "untracked"
 
-        # Determines if presence results for offline users are included on initial/full sync
-        self.presence_include_offline_users_on_sync = presence_config.get(
-            "include_offline_users_on_sync", False
-        )
-
         # Custom presence router module
         # This is the legacy way of configuring it (the config should now be put in the modules section)
         self.presence_router_module_class = None
@@ -400,6 +395,12 @@ class ServerConfig(Config):
                 self.presence_router_config,
             ) = load_module(presence_router_config, ("presence", "presence_router"))
 
+        # whether to enable the media repository endpoints. This should be set
+        # to false if the media repository is running as a separate endpoint;
+        # doing so ensures that we will not run cache cleanup jobs on the
+        # master, potentially causing inconsistency.
+        self.enable_media_repo = config.get("enable_media_repo", True)
+
         # Whether to require authentication to retrieve profile data (avatars,
         # display names) of other users through the client API.
         self.require_auth_for_profile_requests = config.get(

synapse/events/__init__.py

@@ -554,22 +554,3 @@ def relation_from_event(event: EventBase) -> Optional[_EventRelation]:
             aggregation_key = None
 
     return _EventRelation(parent_id, rel_type, aggregation_key)
-
-
-@attr.s(slots=True, frozen=True, auto_attribs=True)
-class StrippedStateEvent:
-    """
-    A stripped down state event. Usually used for remote invite/knocks so the user can
-    make an informed decision on whether they want to join.
-
-    Attributes:
-        type: Event `type`
-        state_key: Event `state_key`
-        sender: Event `sender`
-        content: Event `content`
-    """
-
-    type: str
-    state_key: str
-    sender: str
-    content: Dict[str, Any]

synapse/events/utils.py

@@ -49,7 +49,7 @@ from synapse.api.errors import Codes, SynapseError
 from synapse.api.room_versions import RoomVersion
 from synapse.types import JsonDict, Requester
 
-from . import EventBase, StrippedStateEvent, make_event_from_dict
+from . import EventBase, make_event_from_dict
 
 if TYPE_CHECKING:
     from synapse.handlers.relations import BundledAggregations
@@ -854,30 +854,3 @@ def strip_event(event: EventBase) -> JsonDict:
         "content": event.content,
         "sender": event.sender,
     }
-
-
-def parse_stripped_state_event(raw_stripped_event: Any) -> Optional[StrippedStateEvent]:
-    """
-    Given a raw value from an event's `unsigned` field, attempt to parse it into a
-    `StrippedStateEvent`.
-    """
-    if isinstance(raw_stripped_event, dict):
-        # All of these fields are required
-        type = raw_stripped_event.get("type")
-        state_key = raw_stripped_event.get("state_key")
-        sender = raw_stripped_event.get("sender")
-        content = raw_stripped_event.get("content")
-        if (
-            isinstance(type, str)
-            and isinstance(state_key, str)
-            and isinstance(sender, str)
-            and isinstance(content, dict)
-        ):
-            return StrippedStateEvent(
-                type=type,
-                state_key=state_key,
-                sender=sender,
-                content=content,
-            )
-
-    return None

synapse/federation/transport/server/__init__.py

@@ -271,10 +271,6 @@ SERVLET_GROUPS: Dict[str, Iterable[Type[BaseFederationServlet]]] = {
     "federation": FEDERATION_SERVLET_CLASSES,
     "room_list": (PublicRoomList,),
     "openid": (OpenIdUserInfo,),
-    "media": (
-        FederationMediaDownloadServlet,
-        FederationMediaThumbnailServlet,
-    ),
 }
 
 
@@ -325,7 +321,7 @@ def register_servlets(
                 servletclass == FederationMediaDownloadServlet
                 or servletclass == FederationMediaThumbnailServlet
             ):
-                if not hs.config.media.can_load_media_repo:
+                if not hs.config.server.enable_media_repo:
                     continue
 
             servletclass(

synapse/federation/transport/server/federation.py

@@ -912,4 +912,6 @@ FEDERATION_SERVLET_CLASSES: Tuple[Type[BaseFederationServlet], ...] = (
     FederationV1SendKnockServlet,
     FederationMakeKnockServlet,
     FederationAccountStatusServlet,
+    FederationMediaDownloadServlet,
+    FederationMediaThumbnailServlet,
 )

synapse/handlers/admin.py

@@ -197,14 +197,8 @@ class AdminHandler:
             # events that we have and then filtering, this isn't the most
             # efficient method perhaps but it does guarantee we get everything.
             while True:
-                events, _ = (
-                    await self._store.paginate_room_events_by_topological_ordering(
-                        room_id=room_id,
-                        from_key=from_key,
-                        to_key=to_key,
-                        limit=100,
-                        direction=Direction.FORWARDS,
-                    )
+                events, _ = await self._store.paginate_room_events(
+                    room_id, from_key, to_key, limit=100, direction=Direction.FORWARDS
                 )
                 if not events:
                     break

synapse/handlers/device.py

@@ -20,20 +20,10 @@
 #
 #
 import logging
-from typing import (
-    TYPE_CHECKING,
-    AbstractSet,
-    Dict,
-    Iterable,
-    List,
-    Mapping,
-    Optional,
-    Set,
-    Tuple,
-)
+from typing import TYPE_CHECKING, Dict, Iterable, List, Mapping, Optional, Set, Tuple
 
 from synapse.api import errors
-from synapse.api.constants import EduTypes, EventTypes, Membership
+from synapse.api.constants import EduTypes, EventTypes
 from synapse.api.errors import (
     Codes,
     FederationDeniedError,
@@ -48,9 +38,7 @@ from synapse.metrics.background_process_metrics import (
     wrap_as_background_process,
 )
 from synapse.storage.databases.main.client_ips import DeviceLastConnectionInfo
-from synapse.storage.databases.main.state_deltas import StateDelta
 from synapse.types import (
-    DeviceListUpdates,
     JsonDict,
     JsonMapping,
     ScheduledTask,
@@ -226,214 +214,138 @@ class DeviceWorkerHandler:
     @cancellable
     async def get_user_ids_changed(
         self, user_id: str, from_token: StreamToken
-    ) -> DeviceListUpdates:
+    ) -> JsonDict:
         """Get list of users that have had the devices updated, or have newly
         joined a room, that `user_id` may be interested in.
         """
 
         set_tag("user_id", user_id)
         set_tag("from_token", str(from_token))
+        now_room_key = self.store.get_room_max_token()
 
-        now_token = self._event_sources.get_current_token()
+        room_ids = await self.store.get_rooms_for_user(user_id)
 
-        # We need to work out all the different membership changes for the user
-        # and user they share a room with, to pass to
-        # `generate_sync_entry_for_device_list`. See its docstring for details
-        # on the data required.
-
-        joined_room_ids = await self.store.get_rooms_for_user(user_id)
-
-        # Get the set of rooms that the user has joined/left
-        membership_changes = (
-            await self.store.get_current_state_delta_membership_changes_for_user(
-                user_id, from_key=from_token.room_key, to_key=now_token.room_key
-            )
+        changed = await self.get_device_changes_in_shared_rooms(
+            user_id, room_ids, from_token
         )
 
-        # Check for newly joined or left rooms. We need to make sure that we add
-        # to newly joined in the case membership goes from join -> leave -> join
-        # again.
-        newly_joined_rooms: Set[str] = set()
-        newly_left_rooms: Set[str] = set()
-        for change in membership_changes:
-            # We check for changes in "joinedness", i.e. if the membership has
-            # changed to or from JOIN.
-            if change.membership == Membership.JOIN:
-                if change.prev_membership != Membership.JOIN:
-                    newly_joined_rooms.add(change.room_id)
-                    newly_left_rooms.discard(change.room_id)
-            elif change.prev_membership == Membership.JOIN:
-                newly_joined_rooms.discard(change.room_id)
-                newly_left_rooms.add(change.room_id)
+        # Then work out if any users have since joined
+        rooms_changed = self.store.get_rooms_that_changed(room_ids, from_token.room_key)
 
-        # We now work out if any other users have since joined or left the rooms
-        # the user is currently in. First we filter out rooms that we know
-        # haven't changed recently.
-        rooms_changed = self.store.get_rooms_that_changed(
-            joined_room_ids, from_token.room_key
+        member_events = await self.store.get_membership_changes_for_user(
+            user_id, from_token.room_key, now_room_key
         )
+        rooms_changed.update(event.room_id for event in member_events)
 
-        # List of membership changes per room
-        room_to_deltas: Dict[str, List[StateDelta]] = {}
-        # The set of event IDs of membership events (so we can fetch their
-        # associated membership).
-        memberships_to_fetch: Set[str] = set()
+        stream_ordering = from_token.room_key.stream
+
+        possibly_changed = set(changed)
+        possibly_left = set()
         for room_id in rooms_changed:
-            # TODO: Only pull out membership events?
-            state_changes = await self.store.get_current_state_deltas_for_room(
-                room_id, from_token=from_token.room_key, to_token=now_token.room_key
+            # Check if the forward extremities have changed. If not then we know
+            # the current state won't have changed, and so we can skip this room.
+            try:
+                if not await self.store.have_room_forward_extremities_changed_since(
+                    room_id, stream_ordering
+                ):
+                    continue
+            except errors.StoreError:
+                pass
+
+            current_state_ids = await self._state_storage.get_current_state_ids(
+                room_id, await_full_state=False
             )
-            for delta in state_changes:
-                if delta.event_type != EventTypes.Member:
+
+            # The user may have left the room
+            # TODO: Check if they actually did or if we were just invited.
+            if room_id not in room_ids:
+                for etype, state_key in current_state_ids.keys():
+                    if etype != EventTypes.Member:
+                        continue
+                    possibly_left.add(state_key)
+                continue
+
+            # Fetch the current state at the time.
+            try:
+                event_ids = await self.store.get_forward_extremities_for_room_at_stream_ordering(
+                    room_id, stream_ordering=stream_ordering
+                )
+            except errors.StoreError:
+                # we have purged the stream_ordering index since the stream
+                # ordering: treat it the same as a new room
+                event_ids = []
+
+            # special-case for an empty prev state: include all members
+            # in the changed list
+            if not event_ids:
+                log_kv(
+                    {"event": "encountered empty previous state", "room_id": room_id}
+                )
+                for etype, state_key in current_state_ids.keys():
+                    if etype != EventTypes.Member:
+                        continue
+                    possibly_changed.add(state_key)
+                continue
+
+            current_member_id = current_state_ids.get((EventTypes.Member, user_id))
+            if not current_member_id:
+                continue
+
+            # mapping from event_id -> state_dict
+            prev_state_ids = await self._state_storage.get_state_ids_for_events(
+                event_ids,
+                await_full_state=False,
+            )
+
+            # Check if we've joined the room? If so we just blindly add all the users to
+            # the "possibly changed" users.
+            for state_dict in prev_state_ids.values():
+                member_event = state_dict.get((EventTypes.Member, user_id), None)
+                if not member_event or member_event != current_member_id:
+                    for etype, state_key in current_state_ids.keys():
+                        if etype != EventTypes.Member:
+                            continue
+                        possibly_changed.add(state_key)
+                    break
+
+            # If there has been any change in membership, include them in the
+            # possibly changed list. We'll check if they are joined below,
+            # and we're not toooo worried about spuriously adding users.
+            for key, event_id in current_state_ids.items():
+                etype, state_key = key
+                if etype != EventTypes.Member:
                     continue
 
-                room_to_deltas.setdefault(room_id, []).append(delta)
-                if delta.event_id:
-                    memberships_to_fetch.add(delta.event_id)
-                if delta.prev_event_id:
-                    memberships_to_fetch.add(delta.prev_event_id)
+                # check if this member has changed since any of the extremities
+                # at the stream_ordering, and add them to the list if so.
+                for state_dict in prev_state_ids.values():
+                    prev_event_id = state_dict.get(key, None)
+                    if not prev_event_id or prev_event_id != event_id:
+                        if state_key != user_id:
+                            possibly_changed.add(state_key)
+                        break
 
-        # Fetch all the memberships for the membership events
-        event_id_to_memberships = await self.store.get_membership_from_event_ids(
-            memberships_to_fetch
-        )
+        if possibly_changed or possibly_left:
+            possibly_joined = possibly_changed
+            possibly_left = possibly_changed | possibly_left
 
-        joined_invited_knocked = (
-            Membership.JOIN,
-            Membership.INVITE,
-            Membership.KNOCK,
-        )
+            # Double check if we still share rooms with the given user.
+            users_rooms = await self.store.get_rooms_for_users(possibly_left)
+            for changed_user_id, entries in users_rooms.items():
+                if any(rid in room_ids for rid in entries):
+                    possibly_left.discard(changed_user_id)
+                else:
+                    possibly_joined.discard(changed_user_id)
 
-        # We now want to find any user that have newly joined/invited/knocked,
-        # or newly left, similarly to above.
-        newly_joined_or_invited_or_knocked_users: Set[str] = set()
-        newly_left_users: Set[str] = set()
-        for _, deltas in room_to_deltas.items():
-            for delta in deltas:
-                # Get the prev/new memberships for the delta
-                new_membership = None
-                prev_membership = None
-                if delta.event_id:
-                    m = event_id_to_memberships.get(delta.event_id)
-                    if m is not None:
-                        new_membership = m.membership
-                if delta.prev_event_id:
-                    m = event_id_to_memberships.get(delta.prev_event_id)
-                    if m is not None:
-                        prev_membership = m.membership
+        else:
+            possibly_joined = set()
+            possibly_left = set()
 
-                # Check if a user has newly joined/invited/knocked, or left.
-                if new_membership in joined_invited_knocked:
-                    if prev_membership not in joined_invited_knocked:
-                        newly_joined_or_invited_or_knocked_users.add(delta.state_key)
-                        newly_left_users.discard(delta.state_key)
-                elif prev_membership in joined_invited_knocked:
-                    newly_joined_or_invited_or_knocked_users.discard(delta.state_key)
-                    newly_left_users.add(delta.state_key)
+        result = {"changed": list(possibly_joined), "left": list(possibly_left)}
 
-        # Now we actually calculate the device list entry with the information
-        # calculated above.
-        device_list_updates = await self.generate_sync_entry_for_device_list(
-            user_id=user_id,
-            since_token=from_token,
-            now_token=now_token,
-            joined_room_ids=joined_room_ids,
-            newly_joined_rooms=newly_joined_rooms,
-            newly_joined_or_invited_or_knocked_users=newly_joined_or_invited_or_knocked_users,
-            newly_left_rooms=newly_left_rooms,
-            newly_left_users=newly_left_users,
-        )
+        log_kv(result)
 
-        log_kv(
-            {
-                "changed": device_list_updates.changed,
-                "left": device_list_updates.left,
-            }
-        )
-
-        return device_list_updates
-
-    @measure_func("_generate_sync_entry_for_device_list")
-    async def generate_sync_entry_for_device_list(
-        self,
-        user_id: str,
-        since_token: StreamToken,
-        now_token: StreamToken,
-        joined_room_ids: AbstractSet[str],
-        newly_joined_rooms: AbstractSet[str],
-        newly_joined_or_invited_or_knocked_users: AbstractSet[str],
-        newly_left_rooms: AbstractSet[str],
-        newly_left_users: AbstractSet[str],
-    ) -> DeviceListUpdates:
-        """Generate the DeviceListUpdates section of sync
-
-        Args:
-            sync_result_builder
-            newly_joined_rooms: Set of rooms user has joined since previous sync
-            newly_joined_or_invited_or_knocked_users: Set of users that have joined,
-                been invited to a room or are knocking on a room since
-                previous sync.
-            newly_left_rooms: Set of rooms user has left since previous sync
-            newly_left_users: Set of users that have left a room we're in since
-                previous sync
-        """
-        # Take a copy since these fields will be mutated later.
-        newly_joined_or_invited_or_knocked_users = set(
-            newly_joined_or_invited_or_knocked_users
-        )
-        newly_left_users = set(newly_left_users)
-
-        # We want to figure out what user IDs the client should refetch
-        # device keys for, and which users we aren't going to track changes
-        # for anymore.
-        #
-        # For the first step we check:
-        #   a. if any users we share a room with have updated their devices,
-        #      and
-        #   b. we also check if we've joined any new rooms, or if a user has
-        #      joined a room we're in.
-        #
-        # For the second step we just find any users we no longer share a
-        # room with by looking at all users that have left a room plus users
-        # that were in a room we've left.
-
-        users_that_have_changed = set()
-
-        # Step 1a, check for changes in devices of users we share a room
-        # with
-        users_that_have_changed = await self.get_device_changes_in_shared_rooms(
-            user_id,
-            joined_room_ids,
-            from_token=since_token,
-            now_token=now_token,
-        )
-
-        # Step 1b, check for newly joined rooms
-        for room_id in newly_joined_rooms:
-            joined_users = await self.store.get_users_in_room(room_id)
-            newly_joined_or_invited_or_knocked_users.update(joined_users)
-
-        # TODO: Check that these users are actually new, i.e. either they
-        # weren't in the previous sync *or* they left and rejoined.
-        users_that_have_changed.update(newly_joined_or_invited_or_knocked_users)
-
-        user_signatures_changed = await self.store.get_users_whose_signatures_changed(
-            user_id, since_token.device_list_key
-        )
-        users_that_have_changed.update(user_signatures_changed)
-
-        # Now find users that we no longer track
-        for room_id in newly_left_rooms:
-            left_users = await self.store.get_users_in_room(room_id)
-            newly_left_users.update(left_users)
-
-        # Remove any users that we still share a room with.
-        left_users_rooms = await self.store.get_rooms_for_users(newly_left_users)
-        for user_id, entries in left_users_rooms.items():
-            if any(rid in joined_room_ids for rid in entries):
-                newly_left_users.discard(user_id)
-
-        return DeviceListUpdates(changed=users_that_have_changed, left=newly_left_users)
+        return result
 
     async def on_federation_query_user_devices(self, user_id: str) -> JsonDict:
         if not self.hs.is_mine(UserID.from_string(user_id)):

synapse/handlers/e2e_keys.py

@@ -291,20 +291,13 @@ class E2eKeysHandler:
 
             # Only try and fetch keys for destinations that are not marked as
             # down.
-            unfiltered_destinations = remote_queries_not_in_cache.keys()
-            filtered_destinations = set(
-                await filter_destinations_by_retry_limiter(
-                    unfiltered_destinations,
-                    self.clock,
-                    self.store,
-                    # Let's give an arbitrary grace period for those hosts that are
-                    # only recently down
-                    retry_due_within_ms=60 * 1000,
-                )
-            )
-            failures.update(
-                (dest, _NOT_READY_FOR_RETRY_FAILURE)
-                for dest in (unfiltered_destinations - filtered_destinations)
+            filtered_destinations = await filter_destinations_by_retry_limiter(
+                remote_queries_not_in_cache.keys(),
+                self.clock,
+                self.store,
+                # Let's give an arbitrary grace period for those hosts that are
+                # only recently down
+                retry_due_within_ms=60 * 1000,
             )
 
             await concurrently_execute(
@@ -1648,9 +1641,6 @@ def _check_device_signature(
         raise SynapseError(400, "Invalid signature", Codes.INVALID_SIGNATURE)
 
 
-_NOT_READY_FOR_RETRY_FAILURE = {"status": 503, "message": "Not ready for retry"}
-
-
 def _exception_to_failure(e: Exception) -> JsonDict:
     if isinstance(e, SynapseError):
         return {"status": e.code, "errcode": e.errcode, "message": str(e)}
@@ -1659,7 +1649,7 @@ def _exception_to_failure(e: Exception) -> JsonDict:
         return {"status": e.code, "message": str(e)}
 
     if isinstance(e, NotRetryingDestination):
-        return _NOT_READY_FOR_RETRY_FAILURE
+        return {"status": 503, "message": "Not ready for retry"}
 
     # include ConnectionRefused and other errors
     #

synapse/handlers/e2e_room_keys.py

@@ -34,7 +34,7 @@ from synapse.api.errors import (
 from synapse.logging.opentracing import log_kv, trace
 from synapse.storage.databases.main.e2e_room_keys import RoomKey
 from synapse.types import JsonDict
-from synapse.util.async_helpers import ReadWriteLock
+from synapse.util.async_helpers import Linearizer
 
 if TYPE_CHECKING:
     from synapse.server import HomeServer
@@ -58,7 +58,7 @@ class E2eRoomKeysHandler:
         # clients belonging to a user will receive and try to upload a new session at
         # roughly the same time.  Also used to lock out uploads when the key is being
         # changed.
-        self._upload_lock = ReadWriteLock()
+        self._upload_linearizer = Linearizer("upload_room_keys_lock")
 
     @trace
     async def get_room_keys(
@@ -89,7 +89,7 @@ class E2eRoomKeysHandler:
 
         # we deliberately take the lock to get keys so that changing the version
         # works atomically
-        async with self._upload_lock.read(user_id):
+        async with self._upload_linearizer.queue(user_id):
             # make sure the backup version exists
             try:
                 await self.store.get_e2e_room_keys_version_info(user_id, version)
@@ -132,7 +132,7 @@ class E2eRoomKeysHandler:
         """
 
         # lock for consistency with uploading
-        async with self._upload_lock.write(user_id):
+        async with self._upload_linearizer.queue(user_id):
             # make sure the backup version exists
             try:
                 version_info = await self.store.get_e2e_room_keys_version_info(
@@ -193,7 +193,7 @@ class E2eRoomKeysHandler:
         # TODO: Validate the JSON to make sure it has the right keys.
 
         # XXX: perhaps we should use a finer grained lock here?
-        async with self._upload_lock.write(user_id):
+        async with self._upload_linearizer.queue(user_id):
             # Check that the version we're trying to upload is the current version
             try:
                 version_info = await self.store.get_e2e_room_keys_version_info(user_id)
@@ -355,7 +355,7 @@ class E2eRoomKeysHandler:
         # TODO: Validate the JSON to make sure it has the right keys.
 
         # lock everyone out until we've switched version
-        async with self._upload_lock.write(user_id):
+        async with self._upload_linearizer.queue(user_id):
             new_version = await self.store.create_e2e_room_keys_version(
                 user_id, version_info
             )
@@ -382,7 +382,7 @@ class E2eRoomKeysHandler:
         }
         """
 
-        async with self._upload_lock.read(user_id):
+        async with self._upload_linearizer.queue(user_id):
             try:
                 res = await self.store.get_e2e_room_keys_version_info(user_id, version)
             except StoreError as e:
@@ -407,7 +407,7 @@ class E2eRoomKeysHandler:
             NotFoundError: if this backup version doesn't exist
         """
 
-        async with self._upload_lock.write(user_id):
+        async with self._upload_linearizer.queue(user_id):
             try:
                 await self.store.delete_e2e_room_keys_version(user_id, version)
             except StoreError as e:
@@ -437,7 +437,7 @@ class E2eRoomKeysHandler:
             raise SynapseError(
                 400, "Version in body does not match", Codes.INVALID_PARAM
             )
-        async with self._upload_lock.write(user_id):
+        async with self._upload_linearizer.queue(user_id):
             try:
                 old_info = await self.store.get_e2e_room_keys_version_info(
                     user_id, version

synapse/handlers/pagination.py

@@ -507,15 +507,13 @@ class PaginationHandler:
 
         # Initially fetch the events from the database. With any luck, we can return
         # these without blocking on backfill (handled below).
-        events, next_key = (
-            await self.store.paginate_room_events_by_topological_ordering(
-                room_id=room_id,
-                from_key=from_token.room_key,
-                to_key=to_room_key,
-                direction=pagin_config.direction,
-                limit=pagin_config.limit,
-                event_filter=event_filter,
-            )
+        events, next_key = await self.store.paginate_room_events(
+            room_id=room_id,
+            from_key=from_token.room_key,
+            to_key=to_room_key,
+            direction=pagin_config.direction,
+            limit=pagin_config.limit,
+            event_filter=event_filter,
         )
 
         if pagin_config.direction == Direction.BACKWARDS:
@@ -584,15 +582,13 @@ class PaginationHandler:
                 # If we did backfill something, refetch the events from the database to
                 # catch anything new that might have been added since we last fetched.
                 if did_backfill:
-                    events, next_key = (
-                        await self.store.paginate_room_events_by_topological_ordering(
-                            room_id=room_id,
-                            from_key=from_token.room_key,
-                            to_key=to_room_key,
-                            direction=pagin_config.direction,
-                            limit=pagin_config.limit,
-                            event_filter=event_filter,
-                        )
+                    events, next_key = await self.store.paginate_room_events(
+                        room_id=room_id,
+                        from_key=from_token.room_key,
+                        to_key=to_room_key,
+                        direction=pagin_config.direction,
+                        limit=pagin_config.limit,
+                        event_filter=event_filter,
                     )
             else:
                 # Otherwise, we can backfill in the background for eventual

synapse/handlers/receipts.py

@@ -286,14 +286,8 @@ class ReceiptEventSource(EventSource[MultiWriterStreamToken, JsonMapping]):
         room_ids: Iterable[str],
         is_guest: bool,
         explicit_room_id: Optional[str] = None,
-        to_key: Optional[MultiWriterStreamToken] = None,
     ) -> Tuple[List[JsonMapping], MultiWriterStreamToken]:
-        """
-        Find read receipts for given rooms (> `from_token` and <= `to_token`)
-        """
-
-        if to_key is None:
-            to_key = self.get_current_key()
+        to_key = self.get_current_key()
 
         if from_key == to_key:
             return [], to_key

synapse/handlers/room.py

@@ -1188,8 +1188,6 @@ class RoomCreationHandler:
             )
             events_to_send.append((power_event, power_context))
         else:
-            # Please update the docs for `default_power_level_content_override` when
-            # updating the `events` dict below
             power_level_content: JsonDict = {
                 "users": {creator_id: 100},
                 "users_default": 0,
@@ -1750,7 +1748,7 @@ class RoomEventSource(EventSource[RoomStreamToken, EventBase]):
                 from_key=from_key,
                 to_key=to_key,
                 limit=limit or 10,
-                direction=Direction.FORWARDS,
+                order="ASC",
             )
 
             events = list(room_events)

synapse/handlers/stats.py

@@ -293,9 +293,7 @@ class StatsHandler:
                     "history_visibility"
                 )
             elif delta.event_type == EventTypes.RoomEncryption:
-                room_state["encryption"] = event_content.get(
-                    EventContentFields.ENCRYPTION_ALGORITHM
-                )
+                room_state["encryption"] = event_content.get("algorithm")
             elif delta.event_type == EventTypes.Name:
                 room_state["name"] = event_content.get("name")
             elif delta.event_type == EventTypes.Topic:

synapse/handlers/sync.py

@@ -43,7 +43,6 @@ from prometheus_client import Counter
 
 from synapse.api.constants import (
     AccountDataTypes,
-    Direction,
     EventContentFields,
     EventTypes,
     JoinRules,
@@ -65,7 +64,6 @@ from synapse.logging.opentracing import (
 )
 from synapse.storage.databases.main.event_push_actions import RoomNotifCounts
 from synapse.storage.databases.main.roommember import extract_heroes_from_room_summary
-from synapse.storage.databases.main.stream import PaginateFunction
 from synapse.storage.roommember import MemberSummary
 from synapse.types import (
     DeviceListUpdates,
@@ -86,7 +84,7 @@ from synapse.util.async_helpers import concurrently_execute
 from synapse.util.caches.expiringcache import ExpiringCache
 from synapse.util.caches.lrucache import LruCache
 from synapse.util.caches.response_cache import ResponseCache, ResponseCacheContext
-from synapse.util.metrics import Measure
+from synapse.util.metrics import Measure, measure_func
 from synapse.visibility import filter_events_for_client
 
 if TYPE_CHECKING:
@@ -881,49 +879,22 @@ class SyncHandler:
                 since_key = since_token.room_key
 
             while limited and len(recents) < timeline_limit and max_repeat:
-                # For initial `/sync`, we want to view a historical section of the
-                # timeline; to fetch events by `topological_ordering` (best
-                # representation of the room DAG as others were seeing it at the time).
-                # This also aligns with the order that `/messages` returns events in.
-                #
-                # For incremental `/sync`, we want to get all updates for rooms since
-                # the last `/sync` (regardless if those updates arrived late or happened
-                # a while ago in the past); to fetch events by `stream_ordering` (in the
-                # order they were received by the server).
-                #
-                # Relevant spec issue: https://github.com/matrix-org/matrix-spec/issues/1917
-                #
-                # FIXME: Using workaround for mypy,
-                # https://github.com/python/mypy/issues/10740#issuecomment-1997047277 and
-                # https://github.com/python/mypy/issues/17479
-                paginate_room_events_by_topological_ordering: PaginateFunction = (
-                    self.store.paginate_room_events_by_topological_ordering
-                )
-                paginate_room_events_by_stream_ordering: PaginateFunction = (
-                    self.store.paginate_room_events_by_stream_ordering
-                )
-                pagination_method: PaginateFunction = (
-                    # Use `topographical_ordering` for historical events
-                    paginate_room_events_by_topological_ordering
-                    if since_key is None
-                    # Use `stream_ordering` for updates
-                    else paginate_room_events_by_stream_ordering
-                )
-                events, end_key = await pagination_method(
-                    room_id=room_id,
-                    # The bounds are reversed so we can paginate backwards
-                    # (from newer to older events) starting at to_bound.
-                    # This ensures we fill the `limit` with the newest events first,
-                    from_key=end_key,
-                    to_key=since_key,
-                    direction=Direction.BACKWARDS,
-                    # We add one so we can determine if there are enough events to saturate
-                    # the limit or not (see `limited`)
-                    limit=load_limit + 1,
-                )
-                # We want to return the events in ascending order (the last event is the
-                # most recent).
-                events.reverse()
+                # If we have a since_key then we are trying to get any events
+                # that have happened since `since_key` up to `end_key`, so we
+                # can just use `get_room_events_stream_for_room`.
+                # Otherwise, we want to return the last N events in the room
+                # in topological ordering.
+                if since_key:
+                    events, end_key = await self.store.get_room_events_stream_for_room(
+                        room_id,
+                        limit=load_limit + 1,
+                        from_key=since_key,
+                        to_key=end_key,
+                    )
+                else:
+                    events, end_key = await self.store.get_recent_events_for_room(
+                        room_id, limit=load_limit + 1, end_token=end_key
+                    )
 
                 log_kv({"loaded_recents": len(events)})
 
@@ -1779,15 +1750,8 @@ class SyncHandler:
                     )
 
                 if include_device_list_updates:
-                    # include_device_list_updates can only be True if we have a
-                    # since token.
-                    assert since_token is not None
-
-                    device_lists = await self._device_handler.generate_sync_entry_for_device_list(
-                        user_id=user_id,
-                        since_token=since_token,
-                        now_token=sync_result_builder.now_token,
-                        joined_room_ids=sync_result_builder.joined_room_ids,
+                    device_lists = await self._generate_sync_entry_for_device_list(
+                        sync_result_builder,
                         newly_joined_rooms=newly_joined_rooms,
                         newly_joined_or_invited_or_knocked_users=newly_joined_or_invited_or_knocked_users,
                         newly_left_rooms=newly_left_rooms,
@@ -1899,14 +1863,8 @@ class SyncHandler:
                 newly_left_users,
             ) = sync_result_builder.calculate_user_changes()
 
-            # include_device_list_updates can only be True if we have a
-            # since token.
-            assert since_token is not None
-            device_lists = await self._device_handler.generate_sync_entry_for_device_list(
-                user_id=user_id,
-                since_token=since_token,
-                now_token=sync_result_builder.now_token,
-                joined_room_ids=sync_result_builder.joined_room_ids,
+            device_lists = await self._generate_sync_entry_for_device_list(
+                sync_result_builder,
                 newly_joined_rooms=newly_joined_rooms,
                 newly_joined_or_invited_or_knocked_users=newly_joined_or_invited_or_knocked_users,
                 newly_left_rooms=newly_left_rooms,
@@ -2083,6 +2041,94 @@ class SyncHandler:
 
         return sync_result_builder
 
+    @measure_func("_generate_sync_entry_for_device_list")
+    async def _generate_sync_entry_for_device_list(
+        self,
+        sync_result_builder: "SyncResultBuilder",
+        newly_joined_rooms: AbstractSet[str],
+        newly_joined_or_invited_or_knocked_users: AbstractSet[str],
+        newly_left_rooms: AbstractSet[str],
+        newly_left_users: AbstractSet[str],
+    ) -> DeviceListUpdates:
+        """Generate the DeviceListUpdates section of sync
+
+        Args:
+            sync_result_builder
+            newly_joined_rooms: Set of rooms user has joined since previous sync
+            newly_joined_or_invited_or_knocked_users: Set of users that have joined,
+                been invited to a room or are knocking on a room since
+                previous sync.
+            newly_left_rooms: Set of rooms user has left since previous sync
+            newly_left_users: Set of users that have left a room we're in since
+                previous sync
+        """
+
+        user_id = sync_result_builder.sync_config.user.to_string()
+        since_token = sync_result_builder.since_token
+        assert since_token is not None
+
+        # Take a copy since these fields will be mutated later.
+        newly_joined_or_invited_or_knocked_users = set(
+            newly_joined_or_invited_or_knocked_users
+        )
+        newly_left_users = set(newly_left_users)
+
+        # We want to figure out what user IDs the client should refetch
+        # device keys for, and which users we aren't going to track changes
+        # for anymore.
+        #
+        # For the first step we check:
+        #   a. if any users we share a room with have updated their devices,
+        #      and
+        #   b. we also check if we've joined any new rooms, or if a user has
+        #      joined a room we're in.
+        #
+        # For the second step we just find any users we no longer share a
+        # room with by looking at all users that have left a room plus users
+        # that were in a room we've left.
+
+        users_that_have_changed = set()
+
+        joined_room_ids = sync_result_builder.joined_room_ids
+
+        # Step 1a, check for changes in devices of users we share a room
+        # with
+        users_that_have_changed = (
+            await self._device_handler.get_device_changes_in_shared_rooms(
+                user_id,
+                joined_room_ids,
+                from_token=since_token,
+                now_token=sync_result_builder.now_token,
+            )
+        )
+
+        # Step 1b, check for newly joined rooms
+        for room_id in newly_joined_rooms:
+            joined_users = await self.store.get_users_in_room(room_id)
+            newly_joined_or_invited_or_knocked_users.update(joined_users)
+
+        # TODO: Check that these users are actually new, i.e. either they
+        # weren't in the previous sync *or* they left and rejoined.
+        users_that_have_changed.update(newly_joined_or_invited_or_knocked_users)
+
+        user_signatures_changed = await self.store.get_users_whose_signatures_changed(
+            user_id, since_token.device_list_key
+        )
+        users_that_have_changed.update(user_signatures_changed)
+
+        # Now find users that we no longer track
+        for room_id in newly_left_rooms:
+            left_users = await self.store.get_users_in_room(room_id)
+            newly_left_users.update(left_users)
+
+        # Remove any users that we still share a room with.
+        left_users_rooms = await self.store.get_rooms_for_users(newly_left_users)
+        for user_id, entries in left_users_rooms.items():
+            if any(rid in joined_room_ids for rid in entries):
+                newly_left_users.discard(user_id)
+
+        return DeviceListUpdates(changed=users_that_have_changed, left=newly_left_users)
+
     @trace
     async def _generate_sync_entry_for_to_device(
         self, sync_result_builder: "SyncResultBuilder"
@@ -2224,11 +2270,7 @@ class SyncHandler:
             user=user,
             from_key=presence_key,
             is_guest=sync_config.is_guest,
-            include_offline=(
-                True
-                if self.hs_config.server.presence_include_offline_users_on_sync
-                else include_offline
-            ),
+            include_offline=include_offline,
         )
         assert presence_key
         sync_result_builder.now_token = now_token.copy_and_replace(
@@ -2595,10 +2637,9 @@ class SyncHandler:
         # a "gap" in the timeline, as described by the spec for /sync.
         room_to_events = await self.store.get_room_events_stream_for_rooms(
             room_ids=sync_result_builder.joined_room_ids,
-            from_key=now_token.room_key,
-            to_key=since_token.room_key,
+            from_key=since_token.room_key,
+            to_key=now_token.room_key,
             limit=timeline_limit + 1,
-            direction=Direction.BACKWARDS,
         )
 
         # We loop through all room ids, even if there are no new events, in case
@@ -2609,9 +2650,6 @@ class SyncHandler:
             newly_joined = room_id in newly_joined_rooms
             if room_entry:
                 events, start_key = room_entry
-                # We want to return the events in ascending order (the last event is the
-                # most recent).
-                events.reverse()
 
                 prev_batch_token = now_token.copy_and_replace(
                     StreamKeyType.ROOM, start_key

synapse/handlers/typing.py

@@ -565,12 +565,7 @@ class TypingNotificationEventSource(EventSource[int, JsonMapping]):
         room_ids: Iterable[str],
         is_guest: bool,
         explicit_room_id: Optional[str] = None,
-        to_key: Optional[int] = None,
     ) -> Tuple[List[JsonMapping], int]:
-        """
-        Find typing notifications for given rooms (> `from_token` and <= `to_token`)
-        """
-
         with Measure(self.clock, "typing.get_new_events"):
             from_key = int(from_key)
             handler = self.get_typing_handler()
@@ -579,9 +574,7 @@ class TypingNotificationEventSource(EventSource[int, JsonMapping]):
             for room_id in room_ids:
                 if room_id not in handler._room_serials:
                     continue
-                if handler._room_serials[room_id] <= from_key or (
-                    to_key is not None and handler._room_serials[room_id] > to_key
-                ):
+                if handler._room_serials[room_id] <= from_key:
                     continue
 
                 events.append(self._make_event_for(room_id))

synapse/http/matrixfederationclient.py

@@ -90,7 +90,7 @@ from synapse.logging.context import make_deferred_yieldable, run_in_background
 from synapse.logging.opentracing import set_tag, start_active_span, tags
 from synapse.types import JsonDict
 from synapse.util import json_decoder
-from synapse.util.async_helpers import AwakenableSleeper, Linearizer, timeout_deferred
+from synapse.util.async_helpers import AwakenableSleeper, timeout_deferred
 from synapse.util.metrics import Measure
 from synapse.util.stringutils import parse_and_validate_server_name
 
@@ -475,8 +475,6 @@ class MatrixFederationHttpClient:
             use_proxy=True,
         )
 
-        self.remote_download_linearizer = Linearizer("remote_download_linearizer", 6)
-
     def wake_destination(self, destination: str) -> None:
         """Called when the remote server may have come back online."""
 
@@ -1488,44 +1486,35 @@ class MatrixFederationHttpClient:
         )
 
         headers = dict(response.headers.getAllRawHeaders())
-        expected_size = response.length
 
+        expected_size = response.length
+        # if we don't get an expected length then use the max length
         if expected_size == UNKNOWN_LENGTH:
             expected_size = max_size
-        else:
-            if int(expected_size) > max_size:
-                msg = "Requested file is too large > %r bytes" % (max_size,)
-                logger.warning(
-                    "{%s} [%s] %s",
-                    request.txn_id,
-                    request.destination,
-                    msg,
-                )
-                raise SynapseError(HTTPStatus.BAD_GATEWAY, msg, Codes.TOO_LARGE)
-
-            read_body, _ = await download_ratelimiter.can_do_action(
-                requester=None,
-                key=ip_address,
-                n_actions=expected_size,
+            logger.debug(
+                f"File size unknown, assuming file is max allowable size: {max_size}"
             )
-            if not read_body:
-                msg = "Requested file size exceeds ratelimits"
-                logger.warning(
-                    "{%s} [%s] %s",
-                    request.txn_id,
-                    request.destination,
-                    msg,
-                )
-                raise SynapseError(
-                    HTTPStatus.TOO_MANY_REQUESTS, msg, Codes.LIMIT_EXCEEDED
-                )
+
+        read_body, _ = await download_ratelimiter.can_do_action(
+            requester=None,
+            key=ip_address,
+            n_actions=expected_size,
+        )
+        if not read_body:
+            msg = "Requested file size exceeds ratelimits"
+            logger.warning(
+                "{%s} [%s] %s",
+                request.txn_id,
+                request.destination,
+                msg,
+            )
+            raise SynapseError(HTTPStatus.TOO_MANY_REQUESTS, msg, Codes.LIMIT_EXCEEDED)
 
         try:
-            async with self.remote_download_linearizer.queue(ip_address):
-                # add a byte of headroom to max size as function errs at >=
-                d = read_body_with_max_size(response, output_stream, expected_size + 1)
-                d.addTimeout(self.default_timeout_seconds, self.reactor)
-                length = await make_deferred_yieldable(d)
+            # add a byte of headroom to max size as function errs at >=
+            d = read_body_with_max_size(response, output_stream, expected_size + 1)
+            d.addTimeout(self.default_timeout_seconds, self.reactor)
+            length = await make_deferred_yieldable(d)
         except BodyExceededMaxSize:
             msg = "Requested file is too large > %r bytes" % (expected_size,)
             logger.warning(
@@ -1571,13 +1560,6 @@ class MatrixFederationHttpClient:
             request.method,
             request.uri.decode("ascii"),
         )
-
-        # if we didn't know the length upfront, decrement the actual size from ratelimiter
-        if response.length == UNKNOWN_LENGTH:
-            download_ratelimiter.record_action(
-                requester=None, key=ip_address, n_actions=length
-            )
-
         return length, headers
 
     async def federation_get_file(
@@ -1648,37 +1630,29 @@ class MatrixFederationHttpClient:
         )
 
         headers = dict(response.headers.getAllRawHeaders())
-        expected_size = response.length
 
+        expected_size = response.length
+        # if we don't get an expected length then use the max length
         if expected_size == UNKNOWN_LENGTH:
             expected_size = max_size
-        else:
-            if int(expected_size) > max_size:
-                msg = "Requested file is too large > %r bytes" % (max_size,)
-                logger.warning(
-                    "{%s} [%s] %s",
-                    request.txn_id,
-                    request.destination,
-                    msg,
-                )
-                raise SynapseError(HTTPStatus.BAD_GATEWAY, msg, Codes.TOO_LARGE)
-
-            read_body, _ = await download_ratelimiter.can_do_action(
-                requester=None,
-                key=ip_address,
-                n_actions=expected_size,
+            logger.debug(
+                f"File size unknown, assuming file is max allowable size: {max_size}"
             )
-            if not read_body:
-                msg = "Requested file size exceeds ratelimits"
-                logger.warning(
-                    "{%s} [%s] %s",
-                    request.txn_id,
-                    request.destination,
-                    msg,
-                )
-                raise SynapseError(
-                    HTTPStatus.TOO_MANY_REQUESTS, msg, Codes.LIMIT_EXCEEDED
-                )
+
+        read_body, _ = await download_ratelimiter.can_do_action(
+            requester=None,
+            key=ip_address,
+            n_actions=expected_size,
+        )
+        if not read_body:
+            msg = "Requested file size exceeds ratelimits"
+            logger.warning(
+                "{%s} [%s] %s",
+                request.txn_id,
+                request.destination,
+                msg,
+            )
+            raise SynapseError(HTTPStatus.TOO_MANY_REQUESTS, msg, Codes.LIMIT_EXCEEDED)
 
         # this should be a multipart/mixed response with the boundary string in the header
         try:
@@ -1698,12 +1672,11 @@ class MatrixFederationHttpClient:
             raise SynapseError(HTTPStatus.BAD_GATEWAY, msg)
 
         try:
-            async with self.remote_download_linearizer.queue(ip_address):
-                # add a byte of headroom to max size as `_MultipartParserProtocol.dataReceived` errs at >=
-                deferred = read_multipart_response(
-                    response, output_stream, boundary, expected_size + 1
-                )
-                deferred.addTimeout(self.default_timeout_seconds, self.reactor)
+            # add a byte of headroom to max size as `_MultipartParserProtocol.dataReceived` errs at >=
+            deferred = read_multipart_response(
+                response, output_stream, boundary, expected_size + 1
+            )
+            deferred.addTimeout(self.default_timeout_seconds, self.reactor)
         except BodyExceededMaxSize:
             msg = "Requested file is too large > %r bytes" % (expected_size,)
             logger.warning(
@@ -1770,13 +1743,6 @@ class MatrixFederationHttpClient:
             request.method,
             request.uri.decode("ascii"),
         )
-
-        # if we didn't know the length upfront, decrement the actual size from ratelimiter
-        if response.length == UNKNOWN_LENGTH:
-            download_ratelimiter.record_action(
-                requester=None, key=ip_address, n_actions=length
-            )
-
         return length, headers, multipart_response.json
 
 

synapse/http/proxy.py

@@ -62,15 +62,6 @@ HOP_BY_HOP_HEADERS = {
     "Upgrade",
 }
 
-if hasattr(Headers, "_canonicalNameCaps"):
-    # Twisted < 24.7.0rc1
-    _canonicalHeaderName = Headers()._canonicalNameCaps  # type: ignore[attr-defined]
-else:
-    # Twisted >= 24.7.0rc1
-    # But note that `_encodeName` still exists on prior versions,
-    # it just encodes differently
-    _canonicalHeaderName = Headers()._encodeName
-
 
 def parse_connection_header_value(
     connection_header_value: Optional[bytes],
@@ -94,10 +85,11 @@ def parse_connection_header_value(
         The set of header names that should not be copied over from the remote response.
         The keys are capitalized in canonical capitalization.
     """
+    headers = Headers()
     extra_headers_to_remove: Set[str] = set()
     if connection_header_value:
         extra_headers_to_remove = {
-            _canonicalHeaderName(connection_option.strip()).decode("ascii")
+            headers._canonicalNameCaps(connection_option.strip()).decode("ascii")
             for connection_option in connection_header_value.split(b",")
         }
 

synapse/http/site.py

@@ -658,7 +658,7 @@ class SynapseSite(ProxySite):
         )
 
         self.site_tag = site_tag
-        self.reactor: ISynapseReactor = reactor
+        self.reactor = reactor
 
         assert config.http_options is not None
         proxied = config.http_options.x_forwarded
@@ -683,7 +683,7 @@ class SynapseSite(ProxySite):
         self.access_logger = logging.getLogger(logger_name)
         self.server_version_string = server_version_string.encode("ascii")
 
-    def log(self, request: SynapseRequest) -> None:  # type: ignore[override]
+    def log(self, request: SynapseRequest) -> None:
         pass
 
 

synapse/media/_base.py

@@ -22,14 +22,12 @@
 
 import logging
 import os
-import threading
 import urllib
 from abc import ABC, abstractmethod
 from types import TracebackType
 from typing import (
     TYPE_CHECKING,
     Awaitable,
-    BinaryIO,
     Dict,
     Generator,
     List,
@@ -39,19 +37,15 @@ from typing import (
 )
 
 import attr
-from zope.interface import implementer
 
-from twisted.internet import interfaces
-from twisted.internet.defer import Deferred
 from twisted.internet.interfaces import IConsumer
-from twisted.python.failure import Failure
+from twisted.protocols.basic import FileSender
 from twisted.web.server import Request
 
 from synapse.api.errors import Codes, cs_error
 from synapse.http.server import finish_request, respond_with_json
 from synapse.http.site import SynapseRequest
-from synapse.logging.context import defer_to_thread, make_deferred_yieldable
-from synapse.types import ISynapseReactor
+from synapse.logging.context import make_deferred_yieldable
 from synapse.util import Clock
 from synapse.util.stringutils import is_ascii
 
@@ -144,7 +138,7 @@ async def respond_with_file(
         add_file_headers(request, media_type, file_size, upload_name)
 
         with open(file_path, "rb") as f:
-            await ThreadedFileSender(request.reactor).beginFileTransfer(f, request)
+            await make_deferred_yieldable(FileSender().beginFileTransfer(f, request))
 
         finish_request(request)
     else:
@@ -607,132 +601,3 @@ def _parseparam(s: bytes) -> Generator[bytes, None, None]:
         f = s[:end]
         yield f.strip()
         s = s[end:]
-
-
-@implementer(interfaces.IPushProducer)
-class ThreadedFileSender:
-    """
-    A producer that sends the contents of a file to a consumer, reading from the
-    file on a thread.
-
-    This works by spawning a loop in a threadpool that repeatedly reads from the
-    file and sends it to the consumer. The main thread communicates with the
-    loop via two `threading.Event`, which controls when to start/pause reading
-    and when to terminate.
-    """
-
-    # How much data to read in one go.
-    CHUNK_SIZE = 2**14
-
-    # How long we wait for the consumer to be ready again before aborting the
-    # read.
-    TIMEOUT_SECONDS = 90.0
-
-    def __init__(self, reactor: ISynapseReactor) -> None:
-        self.reactor = reactor
-
-        self.file: Optional[BinaryIO] = None
-        self.deferred: "Deferred[None]" = Deferred()
-        self.consumer: Optional[interfaces.IConsumer] = None
-
-        # Signals if the thread should keep reading/sending data. Set means
-        # continue, clear means pause.
-        self.wakeup_event = threading.Event()
-
-        # Signals if the thread should terminate, e.g. because the consumer has
-        # gone away. Both this and `wakeup_event` should be set to terminate the
-        # loop (otherwise the thread will block on `wakeup_event`).
-        self.stop_event = threading.Event()
-
-    def beginFileTransfer(
-        self, file: BinaryIO, consumer: interfaces.IConsumer
-    ) -> "Deferred[None]":
-        """
-        Begin transferring a file
-        """
-        self.file = file
-        self.consumer = consumer
-
-        self.consumer.registerProducer(self, True)
-
-        # We set the wakeup signal as we should start producing immediately.
-        self.wakeup_event.set()
-        defer_to_thread(self.reactor, self._on_thread_read_loop)
-
-        return make_deferred_yieldable(self.deferred)
-
-    def resumeProducing(self) -> None:
-        """interfaces.IPushProducer"""
-        self.wakeup_event.set()
-
-    def pauseProducing(self) -> None:
-        """interfaces.IPushProducer"""
-        self.wakeup_event.clear()
-
-    def stopProducing(self) -> None:
-        """interfaces.IPushProducer"""
-
-        # Terminate the thread loop.
-        self.wakeup_event.set()
-        self.stop_event.set()
-
-        if not self.deferred.called:
-            self.deferred.errback(Exception("Consumer asked us to stop producing"))
-
-    def _on_thread_read_loop(self) -> None:
-        """This is the loop that happens on a thread."""
-
-        try:
-            while not self.stop_event.is_set():
-                # We wait for the producer to signal that the consumer wants
-                # more data (or we should abort)
-                if not self.wakeup_event.is_set():
-                    ret = self.wakeup_event.wait(self.TIMEOUT_SECONDS)
-                    if not ret:
-                        raise Exception("Timed out waiting to resume")
-
-                # Check if we were woken up so that we abort the download
-                if self.stop_event.is_set():
-                    return
-
-                # The file should always have been set before we get here.
-                assert self.file is not None
-
-                chunk = self.file.read(self.CHUNK_SIZE)
-                if not chunk:
-                    return
-
-                self.reactor.callFromThread(self._write, chunk)
-
-        except Exception:
-            self.reactor.callFromThread(self._error, Failure())
-        finally:
-            self.reactor.callFromThread(self._finish)
-
-    def _write(self, chunk: bytes) -> None:
-        """Called from the thread to write a chunk of data"""
-        if self.consumer:
-            self.consumer.write(chunk)
-
-    def _error(self, failure: Failure) -> None:
-        """Called from the thread when there was a fatal error"""
-        if self.consumer:
-            self.consumer.unregisterProducer()
-            self.consumer = None
-
-        if not self.deferred.called:
-            self.deferred.errback(failure)
-
-    def _finish(self) -> None:
-        """Called from the thread when it finishes (either on success or
-        failure)."""
-        if self.file:
-            self.file.close()
-            self.file = None
-
-        if self.consumer:
-            self.consumer.unregisterProducer()
-            self.consumer = None
-
-        if not self.deferred.called:
-            self.deferred.callback(None)

synapse/media/media_repository.py

@@ -430,7 +430,6 @@ class MediaRepository:
         media_id: str,
         name: Optional[str],
         max_timeout_ms: int,
-        allow_authenticated: bool = True,
         federation: bool = False,
     ) -> None:
         """Responds to requests for local media, if exists, or returns 404.
@@ -443,7 +442,6 @@ class MediaRepository:
                 the filename in the Content-Disposition header of the response.
             max_timeout_ms: the maximum number of milliseconds to wait for the
                 media to be uploaded.
-            allow_authenticated: whether media marked as authenticated may be served to this request
             federation: whether the local media being fetched is for a federation request
 
         Returns:
@@ -453,10 +451,6 @@ class MediaRepository:
         if not media_info:
             return
 
-        if self.hs.config.media.enable_authenticated_media and not allow_authenticated:
-            if media_info.authenticated:
-                raise NotFoundError()
-
         self.mark_recently_accessed(None, media_id)
 
         media_type = media_info.media_type
@@ -487,7 +481,6 @@ class MediaRepository:
         max_timeout_ms: int,
         ip_address: str,
         use_federation_endpoint: bool,
-        allow_authenticated: bool = True,
     ) -> None:
         """Respond to requests for remote media.
 
@@ -502,8 +495,6 @@ class MediaRepository:
             ip_address: the IP address of the requester
             use_federation_endpoint: whether to request the remote media over the new
                 federation `/download` endpoint
-            allow_authenticated: whether media marked as authenticated may be served to this
-                request
 
         Returns:
             Resolves once a response has successfully been written to request
@@ -535,7 +526,6 @@ class MediaRepository:
                 self.download_ratelimiter,
                 ip_address,
                 use_federation_endpoint,
-                allow_authenticated,
             )
 
         # We deliberately stream the file outside the lock
@@ -558,7 +548,6 @@ class MediaRepository:
         max_timeout_ms: int,
         ip_address: str,
         use_federation: bool,
-        allow_authenticated: bool,
     ) -> RemoteMedia:
         """Gets the media info associated with the remote file, downloading
         if necessary.
@@ -571,8 +560,6 @@ class MediaRepository:
             ip_address: IP address of the requester
             use_federation: if a download is necessary, whether to request the remote file
                 over the federation `/download` endpoint
-            allow_authenticated: whether media marked as authenticated may be served to this
-                request
 
         Returns:
             The media info of the file
@@ -594,7 +581,6 @@ class MediaRepository:
                 self.download_ratelimiter,
                 ip_address,
                 use_federation,
-                allow_authenticated,
             )
 
         # Ensure we actually use the responder so that it releases resources
@@ -612,7 +598,6 @@ class MediaRepository:
         download_ratelimiter: Ratelimiter,
         ip_address: str,
         use_federation_endpoint: bool,
-        allow_authenticated: bool,
     ) -> Tuple[Optional[Responder], RemoteMedia]:
         """Looks for media in local cache, if not there then attempt to
         download from remote server.
@@ -634,11 +619,6 @@ class MediaRepository:
         """
         media_info = await self.store.get_cached_remote_media(server_name, media_id)
 
-        if self.hs.config.media.enable_authenticated_media and not allow_authenticated:
-            # if it isn't cached then don't fetch it or if it's authenticated then don't serve it
-            if not media_info or media_info.authenticated:
-                raise NotFoundError()
-
         # file_id is the ID we use to track the file locally. If we've already
         # seen the file then reuse the existing ID, otherwise generate a new
         # one.
@@ -812,11 +792,6 @@ class MediaRepository:
 
         logger.info("Stored remote media in file %r", fname)
 
-        if self.hs.config.media.enable_authenticated_media:
-            authenticated = True
-        else:
-            authenticated = False
-
         return RemoteMedia(
             media_origin=server_name,
             media_id=media_id,
@@ -827,7 +802,6 @@ class MediaRepository:
             filesystem_id=file_id,
             last_access_ts=time_now_ms,
             quarantined_by=None,
-            authenticated=authenticated,
         )
 
     async def _federation_download_remote_file(
@@ -941,11 +915,6 @@ class MediaRepository:
 
         logger.debug("Stored remote media in file %r", fname)
 
-        if self.hs.config.media.enable_authenticated_media:
-            authenticated = True
-        else:
-            authenticated = False
-
         return RemoteMedia(
             media_origin=server_name,
             media_id=media_id,
@@ -956,7 +925,6 @@ class MediaRepository:
             filesystem_id=file_id,
             last_access_ts=time_now_ms,
             quarantined_by=None,
-            authenticated=authenticated,
         )
 
     def _get_thumbnail_requirements(
@@ -1062,12 +1030,7 @@ class MediaRepository:
             t_len = os.path.getsize(output_path)
 
             await self.store.store_local_thumbnail(
-                media_id,
-                t_width,
-                t_height,
-                t_type,
-                t_method,
-                t_len,
+                media_id, t_width, t_height, t_type, t_method, t_len
             )
 
             return output_path

synapse/media/media_storage.py

@@ -49,15 +49,19 @@ from zope.interface import implementer
 from twisted.internet import interfaces
 from twisted.internet.defer import Deferred
 from twisted.internet.interfaces import IConsumer
+from twisted.protocols.basic import FileSender
 
 from synapse.api.errors import NotFoundError
-from synapse.logging.context import defer_to_thread, run_in_background
+from synapse.logging.context import (
+    defer_to_thread,
+    make_deferred_yieldable,
+    run_in_background,
+)
 from synapse.logging.opentracing import start_active_span, trace, trace_with_opname
-from synapse.media._base import ThreadedFileSender
 from synapse.util import Clock
 from synapse.util.file_consumer import BackgroundFileConsumer
 
-from ..types import ISynapseReactor, JsonDict
+from ..types import JsonDict
 from ._base import FileInfo, Responder
 from .filepath import MediaFilePaths
 
@@ -209,7 +213,7 @@ class MediaStorage:
             local_path = os.path.join(self.local_media_directory, path)
             if os.path.exists(local_path):
                 logger.debug("responding with local file %s", local_path)
-                return FileResponder(self.reactor, open(local_path, "rb"))
+                return FileResponder(open(local_path, "rb"))
             logger.debug("local file %s did not exist", local_path)
 
         for provider in self.storage_providers:
@@ -332,13 +336,12 @@ class FileResponder(Responder):
             is closed when finished streaming.
     """
 
-    def __init__(self, reactor: ISynapseReactor, open_file: BinaryIO):
-        self.reactor = reactor
+    def __init__(self, open_file: IO):
         self.open_file = open_file
 
     def write_to_consumer(self, consumer: IConsumer) -> Deferred:
-        return ThreadedFileSender(self.reactor).beginFileTransfer(
-            self.open_file, consumer
+        return make_deferred_yieldable(
+            FileSender().beginFileTransfer(self.open_file, consumer)
         )
 
     def __exit__(

synapse/media/storage_provider.py

@@ -145,7 +145,6 @@ class FileStorageProviderBackend(StorageProvider):
 
     def __init__(self, hs: "HomeServer", config: str):
         self.hs = hs
-        self.reactor = hs.get_reactor()
         self.cache_directory = hs.config.media.media_store_path
         self.base_directory = config
 
@@ -166,7 +165,7 @@ class FileStorageProviderBackend(StorageProvider):
         shutil_copyfile: Callable[[str, str], str] = shutil.copyfile
         with start_active_span("shutil_copyfile"):
             await defer_to_thread(
-                self.reactor,
+                self.hs.get_reactor(),
                 shutil_copyfile,
                 primary_fname,
                 backup_fname,
@@ -178,7 +177,7 @@ class FileStorageProviderBackend(StorageProvider):
 
         backup_fname = os.path.join(self.base_directory, path)
         if os.path.isfile(backup_fname):
-            return FileResponder(self.reactor, open(backup_fname, "rb"))
+            return FileResponder(open(backup_fname, "rb"))
 
         return None
 

synapse/media/thumbnailer.py

@@ -26,7 +26,7 @@ from typing import TYPE_CHECKING, List, Optional, Tuple, Type
 
 from PIL import Image
 
-from synapse.api.errors import Codes, NotFoundError, SynapseError, cs_error
+from synapse.api.errors import Codes, SynapseError, cs_error
 from synapse.config.repository import THUMBNAIL_SUPPORTED_MEDIA_FORMAT_MAP
 from synapse.http.server import respond_with_json
 from synapse.http.site import SynapseRequest
@@ -259,7 +259,6 @@ class ThumbnailProvider:
         media_storage: MediaStorage,
     ):
         self.hs = hs
-        self.reactor = hs.get_reactor()
         self.media_repo = media_repo
         self.media_storage = media_storage
         self.store = hs.get_datastores().main
@@ -275,7 +274,6 @@ class ThumbnailProvider:
         m_type: str,
         max_timeout_ms: int,
         for_federation: bool,
-        allow_authenticated: bool = True,
     ) -> None:
         media_info = await self.media_repo.get_local_media_info(
             request, media_id, max_timeout_ms
@@ -283,12 +281,6 @@ class ThumbnailProvider:
         if not media_info:
             return
 
-        # if the media the thumbnail is generated from is authenticated, don't serve the
-        # thumbnail over an unauthenticated endpoint
-        if self.hs.config.media.enable_authenticated_media and not allow_authenticated:
-            if media_info.authenticated:
-                raise NotFoundError()
-
         thumbnail_infos = await self.store.get_local_media_thumbnails(media_id)
         await self._select_and_respond_with_thumbnail(
             request,
@@ -315,20 +307,14 @@ class ThumbnailProvider:
         desired_type: str,
         max_timeout_ms: int,
         for_federation: bool,
-        allow_authenticated: bool = True,
     ) -> None:
         media_info = await self.media_repo.get_local_media_info(
             request, media_id, max_timeout_ms
         )
+
         if not media_info:
             return
 
-        # if the media the thumbnail is generated from is authenticated, don't serve the
-        # thumbnail over an unauthenticated endpoint
-        if self.hs.config.media.enable_authenticated_media and not allow_authenticated:
-            if media_info.authenticated:
-                raise NotFoundError()
-
         thumbnail_infos = await self.store.get_local_media_thumbnails(media_id)
         for info in thumbnail_infos:
             t_w = info.width == desired_width
@@ -374,7 +360,7 @@ class ThumbnailProvider:
                 await respond_with_multipart_responder(
                     self.hs.get_clock(),
                     request,
-                    FileResponder(self.reactor, open(file_path, "rb")),
+                    FileResponder(open(file_path, "rb")),
                     media_info,
                 )
             else:
@@ -395,27 +381,14 @@ class ThumbnailProvider:
         max_timeout_ms: int,
         ip_address: str,
         use_federation: bool,
-        allow_authenticated: bool = True,
     ) -> None:
         media_info = await self.media_repo.get_remote_media_info(
-            server_name,
-            media_id,
-            max_timeout_ms,
-            ip_address,
-            use_federation,
-            allow_authenticated,
+            server_name, media_id, max_timeout_ms, ip_address, use_federation
         )
         if not media_info:
             respond_404(request)
             return
 
-        # if the media the thumbnail is generated from is authenticated, don't serve the
-        # thumbnail over an unauthenticated endpoint
-        if self.hs.config.media.enable_authenticated_media and not allow_authenticated:
-            if media_info.authenticated:
-                respond_404(request)
-                return
-
         thumbnail_infos = await self.store.get_remote_media_thumbnails(
             server_name, media_id
         )
@@ -473,28 +446,16 @@ class ThumbnailProvider:
         max_timeout_ms: int,
         ip_address: str,
         use_federation: bool,
-        allow_authenticated: bool = True,
     ) -> None:
         # TODO: Don't download the whole remote file
         # We should proxy the thumbnail from the remote server instead of
         # downloading the remote file and generating our own thumbnails.
         media_info = await self.media_repo.get_remote_media_info(
-            server_name,
-            media_id,
-            max_timeout_ms,
-            ip_address,
-            use_federation,
-            allow_authenticated,
+            server_name, media_id, max_timeout_ms, ip_address, use_federation
         )
         if not media_info:
             return
 
-        # if the media the thumbnail is generated from is authenticated, don't serve the
-        # thumbnail over an unauthenticated endpoint
-        if self.hs.config.media.enable_authenticated_media and not allow_authenticated:
-            if media_info.authenticated:
-                raise NotFoundError()
-
         thumbnail_infos = await self.store.get_remote_media_thumbnails(
             server_name, media_id
         )
@@ -524,8 +485,8 @@ class ThumbnailProvider:
         file_id: str,
         url_cache: bool,
         for_federation: bool,
-        media_info: Optional[LocalMedia] = None,
         server_name: Optional[str] = None,
+        media_info: Optional[LocalMedia] = None,
     ) -> None:
         """
         Respond to a request with an appropriate thumbnail from the previously generated thumbnails.

synapse/notifier.py

@@ -773,7 +773,6 @@ class Notifier:
         stream_token = await self.event_sources.bound_future_token(stream_token)
 
         start = self.clock.time_msec()
-        logged = False
         while True:
             current_token = self.event_sources.get_current_token()
             if stream_token.is_before_or_eq(current_token):
@@ -784,13 +783,11 @@ class Notifier:
             if now - start > 10_000:
                 return False
 
-            if not logged:
-                logger.info(
-                    "Waiting for current token to reach %s; currently at %s",
-                    stream_token,
-                    current_token,
-                )
-                logged = True
+            logger.info(
+                "Waiting for current token to reach %s; currently at %s",
+                stream_token,
+                current_token,
+            )
 
             # TODO: be better
             await self.clock.sleep(0.5)

synapse/rest/__init__.py

@@ -18,8 +18,7 @@
 # [This file includes modifications made by New Vector Limited]
 #
 #
-import logging
-from typing import TYPE_CHECKING, Callable, Dict, Iterable, List, Optional, Tuple
+from typing import TYPE_CHECKING, Callable
 
 from synapse.http.server import HttpServer, JsonResource
 from synapse.rest import admin
@@ -68,64 +67,11 @@ from synapse.rest.client import (
     voip,
 )
 
-logger = logging.getLogger(__name__)
-
 if TYPE_CHECKING:
     from synapse.server import HomeServer
 
 RegisterServletsFunc = Callable[["HomeServer", HttpServer], None]
 
-CLIENT_SERVLET_FUNCTIONS: Tuple[RegisterServletsFunc, ...] = (
-    versions.register_servlets,
-    initial_sync.register_servlets,
-    room.register_deprecated_servlets,
-    events.register_servlets,
-    room.register_servlets,
-    login.register_servlets,
-    profile.register_servlets,
-    presence.register_servlets,
-    directory.register_servlets,
-    voip.register_servlets,
-    pusher.register_servlets,
-    push_rule.register_servlets,
-    logout.register_servlets,
-    sync.register_servlets,
-    filter.register_servlets,
-    account.register_servlets,
-    register.register_servlets,
-    auth.register_servlets,
-    receipts.register_servlets,
-    read_marker.register_servlets,
-    room_keys.register_servlets,
-    keys.register_servlets,
-    tokenrefresh.register_servlets,
-    tags.register_servlets,
-    account_data.register_servlets,
-    reporting.register_servlets,
-    openid.register_servlets,
-    notifications.register_servlets,
-    devices.register_servlets,
-    thirdparty.register_servlets,
-    sendtodevice.register_servlets,
-    user_directory.register_servlets,
-    room_upgrade_rest_servlet.register_servlets,
-    capabilities.register_servlets,
-    account_validity.register_servlets,
-    relations.register_servlets,
-    password_policy.register_servlets,
-    knock.register_servlets,
-    appservice_ping.register_servlets,
-    admin.register_servlets_for_client_rest_resource,
-    mutual_rooms.register_servlets,
-    login_token_request.register_servlets,
-    rendezvous.register_servlets,
-    auth_issuer.register_servlets,
-)
-
-SERVLET_GROUPS: Dict[str, Iterable[RegisterServletsFunc]] = {
-    "client": CLIENT_SERVLET_FUNCTIONS,
-}
-
 
 class ClientRestResource(JsonResource):
     """Matrix Client API REST resource.
@@ -137,56 +83,80 @@ class ClientRestResource(JsonResource):
        * etc
     """
 
-    def __init__(self, hs: "HomeServer", servlet_groups: Optional[List[str]] = None):
+    def __init__(self, hs: "HomeServer"):
         JsonResource.__init__(self, hs, canonical_json=False)
-        if hs.config.media.can_load_media_repo:
-            # This import is here to prevent a circular import failure
-            from synapse.rest.client import media
-
-            SERVLET_GROUPS["media"] = (media.register_servlets,)
-        self.register_servlets(self, hs, servlet_groups)
+        self.register_servlets(self, hs)
 
     @staticmethod
-    def register_servlets(
-        client_resource: HttpServer,
-        hs: "HomeServer",
-        servlet_groups: Optional[Iterable[str]] = None,
-    ) -> None:
+    def register_servlets(client_resource: HttpServer, hs: "HomeServer") -> None:
         # Some servlets are only registered on the main process (and not worker
         # processes).
         is_main_process = hs.config.worker.worker_app is None
 
-        if not servlet_groups:
-            servlet_groups = SERVLET_GROUPS.keys()
+        versions.register_servlets(hs, client_resource)
 
-        for servlet_group in servlet_groups:
-            # Fail on unknown servlet groups.
-            if servlet_group not in SERVLET_GROUPS:
-                if servlet_group == "media":
-                    logger.warn(
-                        "media.can_load_media_repo needs to be configured for the media servlet to be available"
-                    )
-                raise RuntimeError(
-                    f"Attempting to register unknown client servlet: '{servlet_group}'"
-                )
+        # Deprecated in r0
+        initial_sync.register_servlets(hs, client_resource)
+        room.register_deprecated_servlets(hs, client_resource)
 
-            for servletfunc in SERVLET_GROUPS[servlet_group]:
-                if not is_main_process and servletfunc in [
-                    pusher.register_servlets,
-                    logout.register_servlets,
-                    auth.register_servlets,
-                    tokenrefresh.register_servlets,
-                    reporting.register_servlets,
-                    openid.register_servlets,
-                    thirdparty.register_servlets,
-                    room_upgrade_rest_servlet.register_servlets,
-                    account_validity.register_servlets,
-                    admin.register_servlets_for_client_rest_resource,
-                    mutual_rooms.register_servlets,
-                    login_token_request.register_servlets,
-                    rendezvous.register_servlets,
-                    auth_issuer.register_servlets,
-                ]:
-                    continue
+        # Partially deprecated in r0
+        events.register_servlets(hs, client_resource)
 
-                servletfunc(hs, client_resource)
+        room.register_servlets(hs, client_resource)
+        login.register_servlets(hs, client_resource)
+        profile.register_servlets(hs, client_resource)
+        presence.register_servlets(hs, client_resource)
+        directory.register_servlets(hs, client_resource)
+        voip.register_servlets(hs, client_resource)
+        if is_main_process:
+            pusher.register_servlets(hs, client_resource)
+        push_rule.register_servlets(hs, client_resource)
+        if is_main_process:
+            logout.register_servlets(hs, client_resource)
+        sync.register_servlets(hs, client_resource)
+        filter.register_servlets(hs, client_resource)
+        account.register_servlets(hs, client_resource)
+        register.register_servlets(hs, client_resource)
+        if is_main_process:
+            auth.register_servlets(hs, client_resource)
+        receipts.register_servlets(hs, client_resource)
+        read_marker.register_servlets(hs, client_resource)
+        room_keys.register_servlets(hs, client_resource)
+        keys.register_servlets(hs, client_resource)
+        if is_main_process:
+            tokenrefresh.register_servlets(hs, client_resource)
+        tags.register_servlets(hs, client_resource)
+        account_data.register_servlets(hs, client_resource)
+        if is_main_process:
+            reporting.register_servlets(hs, client_resource)
+            openid.register_servlets(hs, client_resource)
+        notifications.register_servlets(hs, client_resource)
+        devices.register_servlets(hs, client_resource)
+        if is_main_process:
+            thirdparty.register_servlets(hs, client_resource)
+        sendtodevice.register_servlets(hs, client_resource)
+        user_directory.register_servlets(hs, client_resource)
+        if is_main_process:
+            room_upgrade_rest_servlet.register_servlets(hs, client_resource)
+        capabilities.register_servlets(hs, client_resource)
+        if is_main_process:
+            account_validity.register_servlets(hs, client_resource)
+        relations.register_servlets(hs, client_resource)
+        password_policy.register_servlets(hs, client_resource)
+        knock.register_servlets(hs, client_resource)
+        appservice_ping.register_servlets(hs, client_resource)
+        if hs.config.server.enable_media_repo:
+            from synapse.rest.client import media
+
+            media.register_servlets(hs, client_resource)
+
+        # moving to /_synapse/admin
+        if is_main_process:
+            admin.register_servlets_for_client_rest_resource(hs, client_resource)
+
+        # unstable
+        if is_main_process:
+            mutual_rooms.register_servlets(hs, client_resource)
+            login_token_request.register_servlets(hs, client_resource)
+            rendezvous.register_servlets(hs, client_resource)
+            auth_issuer.register_servlets(hs, client_resource)

synapse/rest/client/keys.py

@@ -256,15 +256,9 @@ class KeyChangesServlet(RestServlet):
 
         user_id = requester.user.to_string()
 
-        device_list_updates = await self.device_handler.get_user_ids_changed(
-            user_id, from_token
-        )
+        results = await self.device_handler.get_user_ids_changed(user_id, from_token)
 
-        response: JsonDict = {}
-        response["changed"] = list(device_list_updates.changed)
-        response["left"] = list(device_list_updates.left)
-
-        return 200, response
+        return 200, results
 
 
 class OneTimeKeyServlet(RestServlet):

synapse/rest/client/sync.py

@@ -52,9 +52,9 @@ from synapse.http.servlet import (
     parse_string,
 )
 from synapse.http.site import SynapseRequest
-from synapse.logging.opentracing import log_kv, set_tag, trace_with_opname
+from synapse.logging.opentracing import trace_with_opname
 from synapse.rest.admin.experimental_features import ExperimentalFeature
-from synapse.types import JsonDict, Requester, SlidingSyncStreamToken, StreamToken
+from synapse.types import JsonDict, Requester, StreamToken
 from synapse.types.rest.client import SlidingSyncBody
 from synapse.util import json_decoder
 from synapse.util.caches.lrucache import LruCache
@@ -881,6 +881,7 @@ class SlidingSyncRestServlet(RestServlet):
         )
 
         user = requester.user
+        device_id = requester.device_id
 
         timeout = parse_integer(request, "timeout", default=0)
         # Position in the stream
@@ -888,50 +889,22 @@ class SlidingSyncRestServlet(RestServlet):
 
         from_token = None
         if from_token_string is not None:
-            from_token = await SlidingSyncStreamToken.from_string(
-                self.store, from_token_string
-            )
+            from_token = await StreamToken.from_string(self.store, from_token_string)
 
         # TODO: We currently don't know whether we're going to use sticky params or
         # maybe some filters like sync v2  where they are built up once and referenced
         # by filter ID. For now, we will just prototype with always passing everything
         # in.
         body = parse_and_validate_json_object_from_request(request, SlidingSyncBody)
-
-        # Tag and log useful data to differentiate requests.
-        set_tag(
-            "sliding_sync.sync_type", "initial" if from_token is None else "incremental"
-        )
-        set_tag("sliding_sync.conn_id", body.conn_id or "")
-        log_kv(
-            {
-                "sliding_sync.lists": {
-                    list_name: {
-                        "ranges": list_config.ranges,
-                        "timeline_limit": list_config.timeline_limit,
-                    }
-                    for list_name, list_config in (body.lists or {}).items()
-                },
-                "sliding_sync.room_subscriptions": list(
-                    (body.room_subscriptions or {}).keys()
-                ),
-                # We also include the number of room subscriptions because logs are
-                # limited to 1024 characters and the large room ID list above can be cut
-                # off.
-                "sliding_sync.num_room_subscriptions": len(
-                    (body.room_subscriptions or {}).keys()
-                ),
-            }
-        )
+        logger.info("Sliding sync request: %r", body)
 
         sync_config = SlidingSyncConfig(
             user=user,
-            requester=requester,
+            device_id=device_id,
             # FIXME: Currently, we're just manually copying the fields from the
-            # `SlidingSyncBody` into the config. How can we guarantee into the future
+            # `SlidingSyncBody` into the config. How can we gurantee into the future
             # that we don't forget any? I would like something more structured like
             # `copy_attributes(from=body, to=config)`
-            conn_id=body.conn_id,
             lists=body.lists,
             room_subscriptions=body.room_subscriptions,
             extensions=body.extensions,
@@ -954,6 +927,7 @@ class SlidingSyncRestServlet(RestServlet):
 
         return 200, response_content
 
+    # TODO: Is there a better way to encode things?
     async def encode_response(
         self,
         requester: Requester,
@@ -968,9 +942,7 @@ class SlidingSyncRestServlet(RestServlet):
         response["rooms"] = await self.encode_rooms(
             requester, sliding_sync_result.rooms
         )
-        response["extensions"] = await self.encode_extensions(
-            requester, sliding_sync_result.extensions
-        )
+        response["extensions"] = {}  # TODO: sliding_sync_result.extensions
 
         return response
 
@@ -1023,21 +995,8 @@ class SlidingSyncRestServlet(RestServlet):
             if room_result.avatar:
                 serialized_rooms[room_id]["avatar"] = room_result.avatar
 
-            if room_result.heroes is not None and len(room_result.heroes) > 0:
-                serialized_heroes = []
-                for hero in room_result.heroes:
-                    serialized_hero = {
-                        "user_id": hero.user_id,
-                    }
-                    if hero.display_name is not None:
-                        # Not a typo, just how "displayname" is spelled in the spec
-                        serialized_hero["displayname"] = hero.display_name
-
-                    if hero.avatar_url is not None:
-                        serialized_hero["avatar_url"] = hero.avatar_url
-
-                    serialized_heroes.append(serialized_hero)
-                serialized_rooms[room_id]["heroes"] = serialized_heroes
+            if room_result.heroes:
+                serialized_rooms[room_id]["heroes"] = room_result.heroes
 
             # We should only include the `initial` key if it's `True` to save bandwidth.
             # The absense of this flag means `False`.
@@ -1045,10 +1004,7 @@ class SlidingSyncRestServlet(RestServlet):
                 serialized_rooms[room_id]["initial"] = room_result.initial
 
             # This will be omitted for invite/knock rooms with `stripped_state`
-            if (
-                room_result.required_state is not None
-                and len(room_result.required_state) > 0
-            ):
+            if room_result.required_state is not None:
                 serialized_required_state = (
                     await self.event_serializer.serialize_events(
                         room_result.required_state,
@@ -1059,10 +1015,7 @@ class SlidingSyncRestServlet(RestServlet):
                 serialized_rooms[room_id]["required_state"] = serialized_required_state
 
             # This will be omitted for invite/knock rooms with `stripped_state`
-            if (
-                room_result.timeline_events is not None
-                and len(room_result.timeline_events) > 0
-            ):
+            if room_result.timeline_events is not None:
                 serialized_timeline = await self.event_serializer.serialize_events(
                     room_result.timeline_events,
                     time_now,
@@ -1090,10 +1043,7 @@ class SlidingSyncRestServlet(RestServlet):
                 serialized_rooms[room_id]["is_dm"] = room_result.is_dm
 
             # Stripped state only applies to invite/knock rooms
-            if (
-                room_result.stripped_state is not None
-                and len(room_result.stripped_state) > 0
-            ):
+            if room_result.stripped_state is not None:
                 # TODO: `knocked_state` but that isn't specced yet.
                 #
                 # TODO: Instead of adding `knocked_state`, it would be good to rename
@@ -1104,73 +1054,6 @@ class SlidingSyncRestServlet(RestServlet):
 
         return serialized_rooms
 
-    async def encode_extensions(
-        self, requester: Requester, extensions: SlidingSyncResult.Extensions
-    ) -> JsonDict:
-        serialized_extensions: JsonDict = {}
-
-        if extensions.to_device is not None:
-            serialized_extensions["to_device"] = {
-                "next_batch": extensions.to_device.next_batch,
-                "events": extensions.to_device.events,
-            }
-
-        if extensions.e2ee is not None:
-            serialized_extensions["e2ee"] = {
-                # We always include this because
-                # https://github.com/vector-im/element-android/issues/3725. The spec
-                # isn't terribly clear on when this can be omitted and how a client
-                # would tell the difference between "no keys present" and "nothing
-                # changed" in terms of whole field absent / individual key type entry
-                # absent Corresponding synapse issue:
-                # https://github.com/matrix-org/synapse/issues/10456
-                "device_one_time_keys_count": extensions.e2ee.device_one_time_keys_count,
-                # https://github.com/matrix-org/matrix-doc/blob/54255851f642f84a4f1aaf7bc063eebe3d76752b/proposals/2732-olm-fallback-keys.md
-                # states that this field should always be included, as long as the
-                # server supports the feature.
-                "device_unused_fallback_key_types": extensions.e2ee.device_unused_fallback_key_types,
-            }
-
-            if extensions.e2ee.device_list_updates is not None:
-                serialized_extensions["e2ee"]["device_lists"] = {}
-
-                serialized_extensions["e2ee"]["device_lists"]["changed"] = list(
-                    extensions.e2ee.device_list_updates.changed
-                )
-                serialized_extensions["e2ee"]["device_lists"]["left"] = list(
-                    extensions.e2ee.device_list_updates.left
-                )
-
-        if extensions.account_data is not None:
-            serialized_extensions["account_data"] = {
-                # Same as the the top-level `account_data.events` field in Sync v2.
-                "global": [
-                    {"type": account_data_type, "content": content}
-                    for account_data_type, content in extensions.account_data.global_account_data_map.items()
-                ],
-                # Same as the joined room's account_data field in Sync v2, e.g the path
-                # `rooms.join["!foo:bar"].account_data.events`.
-                "rooms": {
-                    room_id: [
-                        {"type": account_data_type, "content": content}
-                        for account_data_type, content in event_map.items()
-                    ]
-                    for room_id, event_map in extensions.account_data.account_data_by_room_map.items()
-                },
-            }
-
-        if extensions.receipts is not None:
-            serialized_extensions["receipts"] = {
-                "rooms": extensions.receipts.room_id_to_receipt_map,
-            }
-
-        if extensions.typing is not None:
-            serialized_extensions["typing"] = {
-                "rooms": extensions.typing.room_id_to_typing_map,
-            }
-
-        return serialized_extensions
-
 
 def register_servlets(hs: "HomeServer", http_server: HttpServer) -> None:
     SyncRestServlet(hs).register(http_server)

synapse/rest/media/download_resource.py

@@ -84,7 +84,7 @@ class DownloadResource(RestServlet):
 
         if self._is_mine_server_name(server_name):
             await self.media_repo.get_local_media(
-                request, media_id, file_name, max_timeout_ms, allow_authenticated=False
+                request, media_id, file_name, max_timeout_ms
             )
         else:
             allow_remote = parse_boolean(request, "allow_remote", default=True)
@@ -106,5 +106,4 @@ class DownloadResource(RestServlet):
                 max_timeout_ms,
                 ip_address,
                 False,
-                allow_authenticated=False,
             )

synapse/rest/media/thumbnail_resource.py

@@ -96,7 +96,6 @@ class ThumbnailResource(RestServlet):
                     m_type,
                     max_timeout_ms,
                     False,
-                    allow_authenticated=False,
                 )
             else:
                 await self.thumbnail_provider.respond_local_thumbnail(
@@ -108,7 +107,6 @@ class ThumbnailResource(RestServlet):
                     m_type,
                     max_timeout_ms,
                     False,
-                    allow_authenticated=False,
                 )
             self.media_repo.mark_recently_accessed(None, media_id)
         else:
@@ -136,7 +134,6 @@ class ThumbnailResource(RestServlet):
                 m_type,
                 max_timeout_ms,
                 ip_address,
-                use_federation=False,
-                allow_authenticated=False,
+                False,
             )
             self.media_repo.mark_recently_accessed(server_name, media_id)

synapse/server.py

@@ -559,7 +559,6 @@ class HomeServer(metaclass=abc.ABCMeta):
     def get_sync_handler(self) -> SyncHandler:
         return SyncHandler(self)
 
-    @cache_in_self
     def get_sliding_sync_handler(self) -> SlidingSyncHandler:
         return SlidingSyncHandler(self)
 

synapse/storage/_base.py

@@ -120,15 +120,10 @@ class SQLBaseStore(metaclass=ABCMeta):
                 "get_user_in_room_with_profile", (room_id, user_id)
             )
             self._attempt_to_invalidate_cache("get_rooms_for_user", (user_id,))
-            self._attempt_to_invalidate_cache(
-                "_get_rooms_for_local_user_where_membership_is_inner", (user_id,)
-            )
 
         # Purge other caches based on room state.
         self._attempt_to_invalidate_cache("get_room_summary", (room_id,))
         self._attempt_to_invalidate_cache("get_partial_current_state_ids", (room_id,))
-        self._attempt_to_invalidate_cache("get_room_type", (room_id,))
-        self._attempt_to_invalidate_cache("get_room_encryption", (room_id,))
 
     def _invalidate_state_caches_all(self, room_id: str) -> None:
         """Invalidates caches that are based on the current state, but does
@@ -151,12 +146,7 @@ class SQLBaseStore(metaclass=ABCMeta):
         self._attempt_to_invalidate_cache("does_pair_of_users_share_a_room", None)
         self._attempt_to_invalidate_cache("get_user_in_room_with_profile", None)
         self._attempt_to_invalidate_cache("get_rooms_for_user", None)
-        self._attempt_to_invalidate_cache(
-            "_get_rooms_for_local_user_where_membership_is_inner", None
-        )
         self._attempt_to_invalidate_cache("get_room_summary", (room_id,))
-        self._attempt_to_invalidate_cache("get_room_type", (room_id,))
-        self._attempt_to_invalidate_cache("get_room_encryption", (room_id,))
 
     def _attempt_to_invalidate_cache(
         self, cache_name: str, key: Optional[Collection[Any]]

synapse/storage/databases/main/cache.py

@@ -268,23 +268,13 @@ class CacheInvalidationWorkerStore(SQLBaseStore):
             self._curr_state_delta_stream_cache.entity_has_changed(data.room_id, token)  # type: ignore[attr-defined]
 
             if data.type == EventTypes.Member:
-                self._attempt_to_invalidate_cache(
-                    "get_rooms_for_user", (data.state_key,)
-                )
-            elif data.type == EventTypes.RoomEncryption:
-                self._attempt_to_invalidate_cache(
-                    "get_room_encryption", (data.room_id,)
-                )
-            elif data.type == EventTypes.Create:
-                self._attempt_to_invalidate_cache("get_room_type", (data.room_id,))
+                self.get_rooms_for_user.invalidate((data.state_key,))  # type: ignore[attr-defined]
         elif row.type == EventsStreamAllStateRow.TypeId:
             assert isinstance(data, EventsStreamAllStateRow)
             # Similar to the above, but the entire caches are invalidated. This is
             # unfortunate for the membership caches, but should recover quickly.
             self._curr_state_delta_stream_cache.entity_has_changed(data.room_id, token)  # type: ignore[attr-defined]
-            self._attempt_to_invalidate_cache("get_rooms_for_user", None)
-            self._attempt_to_invalidate_cache("get_room_type", (data.room_id,))
-            self._attempt_to_invalidate_cache("get_room_encryption", (data.room_id,))
+            self.get_rooms_for_user.invalidate_all()  # type: ignore[attr-defined]
         else:
             raise Exception("Unknown events stream row type %s" % (row.type,))
 
@@ -341,9 +331,6 @@ class CacheInvalidationWorkerStore(SQLBaseStore):
                 "get_invited_rooms_for_local_user", (state_key,)
             )
             self._attempt_to_invalidate_cache("get_rooms_for_user", (state_key,))
-            self._attempt_to_invalidate_cache(
-                "_get_rooms_for_local_user_where_membership_is_inner", (state_key,)
-            )
 
             self._attempt_to_invalidate_cache(
                 "did_forget",
@@ -355,10 +342,6 @@ class CacheInvalidationWorkerStore(SQLBaseStore):
             self._attempt_to_invalidate_cache(
                 "get_forgotten_rooms_for_user", (state_key,)
             )
-        elif etype == EventTypes.Create:
-            self._attempt_to_invalidate_cache("get_room_type", (room_id,))
-        elif etype == EventTypes.RoomEncryption:
-            self._attempt_to_invalidate_cache("get_room_encryption", (room_id,))
 
         if relates_to:
             self._attempt_to_invalidate_cache(
@@ -410,17 +393,12 @@ class CacheInvalidationWorkerStore(SQLBaseStore):
         self._attempt_to_invalidate_cache("get_thread_id_for_receipts", None)
         self._attempt_to_invalidate_cache("get_invited_rooms_for_local_user", None)
         self._attempt_to_invalidate_cache("get_rooms_for_user", None)
-        self._attempt_to_invalidate_cache(
-            "_get_rooms_for_local_user_where_membership_is_inner", None
-        )
         self._attempt_to_invalidate_cache("did_forget", None)
         self._attempt_to_invalidate_cache("get_forgotten_rooms_for_user", None)
         self._attempt_to_invalidate_cache("get_references_for_event", None)
         self._attempt_to_invalidate_cache("get_thread_summary", None)
         self._attempt_to_invalidate_cache("get_thread_participated", None)
         self._attempt_to_invalidate_cache("get_threads", (room_id,))
-        self._attempt_to_invalidate_cache("get_room_type", (room_id,))
-        self._attempt_to_invalidate_cache("get_room_encryption", (room_id,))
 
         self._attempt_to_invalidate_cache("_get_state_group_for_event", None)
 
@@ -473,8 +451,6 @@ class CacheInvalidationWorkerStore(SQLBaseStore):
         self._attempt_to_invalidate_cache("get_forgotten_rooms_for_user", None)
         self._attempt_to_invalidate_cache("_get_membership_from_event_id", None)
         self._attempt_to_invalidate_cache("get_room_version_id", (room_id,))
-        self._attempt_to_invalidate_cache("get_room_type", (room_id,))
-        self._attempt_to_invalidate_cache("get_room_encryption", (room_id,))
 
         # And delete state caches.
 

synapse/storage/databases/main/event_federation.py

@@ -1313,11 +1313,6 @@ class EventFederationWorkerStore(SignatureWorkerStore, EventsWorkerStore, SQLBas
         # We want to make the cache more effective, so we clamp to the last
         # change before the given ordering.
         last_change = self._events_stream_cache.get_max_pos_of_last_change(room_id)  # type: ignore[attr-defined]
-        if last_change is None:
-            # If the room isn't in the cache we know that the last change was
-            # somewhere before the earliest known position of the cache, so we
-            # can clamp to that.
-            last_change = self._events_stream_cache.get_earliest_known_position()  # type: ignore[attr-defined]
 
         # We don't always have a full stream_to_exterm_id table, e.g. after
         # the upgrade that introduced it, so we make sure we never ask for a

synapse/storage/databases/main/media_repository.py

@@ -64,7 +64,6 @@ class LocalMedia:
     quarantined_by: Optional[str]
     safe_from_quarantine: bool
     user_id: Optional[str]
-    authenticated: Optional[bool]
 
 
 @attr.s(slots=True, frozen=True, auto_attribs=True)
@@ -78,7 +77,6 @@ class RemoteMedia:
     created_ts: int
     last_access_ts: int
     quarantined_by: Optional[str]
-    authenticated: Optional[bool]
 
 
 @attr.s(slots=True, frozen=True, auto_attribs=True)
@@ -220,7 +218,6 @@ class MediaRepositoryStore(MediaRepositoryBackgroundUpdateStore):
                 "last_access_ts",
                 "safe_from_quarantine",
                 "user_id",
-                "authenticated",
             ),
             allow_none=True,
             desc="get_local_media",
@@ -238,7 +235,6 @@ class MediaRepositoryStore(MediaRepositoryBackgroundUpdateStore):
             last_access_ts=row[6],
             safe_from_quarantine=row[7],
             user_id=row[8],
-            authenticated=row[9],
         )
 
     async def get_local_media_by_user_paginate(
@@ -294,8 +290,7 @@ class MediaRepositoryStore(MediaRepositoryBackgroundUpdateStore):
                     last_access_ts,
                     quarantined_by,
                     safe_from_quarantine,
-                    user_id,
-                    authenticated
+                    user_id
                 FROM local_media_repository
                 WHERE user_id = ?
                 ORDER BY {order_by_column} {order}, media_id ASC
@@ -319,7 +314,6 @@ class MediaRepositoryStore(MediaRepositoryBackgroundUpdateStore):
                     quarantined_by=row[7],
                     safe_from_quarantine=bool(row[8]),
                     user_id=row[9],
-                    authenticated=row[10],
                 )
                 for row in txn
             ]
@@ -423,18 +417,12 @@ class MediaRepositoryStore(MediaRepositoryBackgroundUpdateStore):
         time_now_ms: int,
         user_id: UserID,
     ) -> None:
-        if self.hs.config.media.enable_authenticated_media:
-            authenticated = True
-        else:
-            authenticated = False
-
         await self.db_pool.simple_insert(
             "local_media_repository",
             {
                 "media_id": media_id,
                 "created_ts": time_now_ms,
                 "user_id": user_id.to_string(),
-                "authenticated": authenticated,
             },
             desc="store_local_media_id",
         )
@@ -450,11 +438,6 @@ class MediaRepositoryStore(MediaRepositoryBackgroundUpdateStore):
         user_id: UserID,
         url_cache: Optional[str] = None,
     ) -> None:
-        if self.hs.config.media.enable_authenticated_media:
-            authenticated = True
-        else:
-            authenticated = False
-
         await self.db_pool.simple_insert(
             "local_media_repository",
             {
@@ -465,7 +448,6 @@ class MediaRepositoryStore(MediaRepositoryBackgroundUpdateStore):
                 "media_length": media_length,
                 "user_id": user_id.to_string(),
                 "url_cache": url_cache,
-                "authenticated": authenticated,
             },
             desc="store_local_media",
         )
@@ -656,7 +638,6 @@ class MediaRepositoryStore(MediaRepositoryBackgroundUpdateStore):
                 "filesystem_id",
                 "last_access_ts",
                 "quarantined_by",
-                "authenticated",
             ),
             allow_none=True,
             desc="get_cached_remote_media",
@@ -673,7 +654,6 @@ class MediaRepositoryStore(MediaRepositoryBackgroundUpdateStore):
             filesystem_id=row[4],
             last_access_ts=row[5],
             quarantined_by=row[6],
-            authenticated=row[7],
         )
 
     async def store_cached_remote_media(
@@ -686,11 +666,6 @@ class MediaRepositoryStore(MediaRepositoryBackgroundUpdateStore):
         upload_name: Optional[str],
         filesystem_id: str,
     ) -> None:
-        if self.hs.config.media.enable_authenticated_media:
-            authenticated = True
-        else:
-            authenticated = False
-
         await self.db_pool.simple_insert(
             "remote_media_cache",
             {
@@ -702,7 +677,6 @@ class MediaRepositoryStore(MediaRepositoryBackgroundUpdateStore):
                 "upload_name": upload_name,
                 "filesystem_id": filesystem_id,
                 "last_access_ts": time_now_ms,
-                "authenticated": authenticated,
             },
             desc="store_cached_remote_media",
         )

synapse/storage/databases/main/roommember.py

@@ -39,7 +39,6 @@ from typing import (
 import attr
 
 from synapse.api.constants import EventTypes, Membership
-from synapse.logging.opentracing import trace
 from synapse.metrics import LaterGauge
 from synapse.metrics.background_process_metrics import wrap_as_background_process
 from synapse.storage._base import SQLBaseStore, db_to_json, make_in_list_sql_clause
@@ -280,19 +279,8 @@ class RoomMemberWorkerStore(EventsWorkerStore, CacheInvalidationWorkerStore):
 
     @cached(max_entries=100000)  # type: ignore[synapse-@cached-mutable]
     async def get_room_summary(self, room_id: str) -> Mapping[str, MemberSummary]:
-        """
-        Get the details of a room roughly suitable for use by the room
+        """Get the details of a room roughly suitable for use by the room
         summary extension to /sync. Useful when lazy loading room members.
-
-        Returns the total count of members in the room by membership type, and a
-        truncated list of members (the heroes). This will be the first 6 members of the
-        room:
-        - We want 5 heroes plus 1, in case one of them is the
-        calling user.
-        - They are ordered by `stream_ordering`, which are joined or
-        invited. When no joined or invited members are available, this also includes
-        banned and left users.
-
         Args:
             room_id: The room ID to query
         Returns:
@@ -320,36 +308,23 @@ class RoomMemberWorkerStore(EventsWorkerStore, CacheInvalidationWorkerStore):
             for count, membership in txn:
                 res.setdefault(membership, MemberSummary([], count))
 
-            # Order by membership (joins -> invites -> leave (former insiders) ->
-            # everything else (outsiders like bans/knocks), then by `stream_ordering` so
-            # the first members in the room show up first and to make the sort stable
-            # (consistent heroes).
-            #
-            # Note: rejected events will have a null membership field, so we we manually
-            # filter them out.
+            # we order by membership and then fairly arbitrarily by event_id so
+            # heroes are consistent
+            # Note, rejected events will have a null membership field, so
+            # we we manually filter them out.
             sql = """
                 SELECT state_key, membership, event_id
                 FROM current_state_events
                 WHERE type = 'm.room.member' AND room_id = ?
                     AND membership IS NOT NULL
                 ORDER BY
-                    CASE membership WHEN ? THEN 1 WHEN ? THEN 2 WHEN ? THEN 3 ELSE 4 END ASC,
-                    event_stream_ordering ASC
+                    CASE membership WHEN ? THEN 1 WHEN ? THEN 2 ELSE 3 END ASC,
+                    event_id ASC
                 LIMIT ?
             """
 
-            txn.execute(
-                sql,
-                (
-                    room_id,
-                    # Sort order
-                    Membership.JOIN,
-                    Membership.INVITE,
-                    Membership.LEAVE,
-                    # 6 is 5 (number of heroes) plus 1, in case one of them is the calling user.
-                    6,
-                ),
-            )
+            # 6 is 5 (number of heroes) plus 1, in case one of them is the calling user.
+            txn.execute(sql, (room_id, Membership.JOIN, Membership.INVITE, 6))
             for user_id, membership, event_id in txn:
                 summary = res[membership]
                 # we will always have a summary for this membership type at this
@@ -423,7 +398,6 @@ class RoomMemberWorkerStore(EventsWorkerStore, CacheInvalidationWorkerStore):
                 return invite
         return None
 
-    @trace
     async def get_rooms_for_local_user_where_membership_is(
         self,
         user_id: str,
@@ -447,11 +421,9 @@ class RoomMemberWorkerStore(EventsWorkerStore, CacheInvalidationWorkerStore):
         if not membership_list:
             return []
 
-        # Convert membership list to frozen set as a) it needs to be hashable,
-        # and b) we don't care about the order.
-        membership_list = frozenset(membership_list)
-
-        rooms = await self._get_rooms_for_local_user_where_membership_is_inner(
+        rooms = await self.db_pool.runInteraction(
+            "get_rooms_for_local_user_where_membership_is",
+            self._get_rooms_for_local_user_where_membership_is_txn,
             user_id,
             membership_list,
         )
@@ -470,24 +442,6 @@ class RoomMemberWorkerStore(EventsWorkerStore, CacheInvalidationWorkerStore):
 
         return [room for room in rooms if room.room_id not in rooms_to_exclude]
 
-    @cached(max_entries=1000, tree=True)
-    async def _get_rooms_for_local_user_where_membership_is_inner(
-        self,
-        user_id: str,
-        membership_list: Collection[str],
-    ) -> Sequence[RoomsForUser]:
-        if not membership_list:
-            return []
-
-        rooms = await self.db_pool.runInteraction(
-            "get_rooms_for_local_user_where_membership_is",
-            self._get_rooms_for_local_user_where_membership_is_txn,
-            user_id,
-            membership_list,
-        )
-
-        return rooms
-
     def _get_rooms_for_local_user_where_membership_is_txn(
         self,
         txn: LoggingTransaction,
@@ -1555,19 +1509,10 @@ def extract_heroes_from_room_summary(
 ) -> List[str]:
     """Determine the users that represent a room, from the perspective of the `me` user.
 
-    This function expects `MemberSummary.members` to already be sorted by
-    `stream_ordering` like the results from `get_room_summary(...)`.
-
     The rules which say which users we select are specified in the "Room Summary"
     section of
     https://spec.matrix.org/v1.4/client-server-api/#get_matrixclientv3sync
 
-
-    Args:
-        details: Mapping from membership type to member summary. We expect
-            `MemberSummary.members` to already be sorted by `stream_ordering`.
-        me: The user for whom we are determining the heroes for.
-
     Returns a list (possibly empty) of heroes' mxids.
     """
     empty_ms = MemberSummary([], 0)
@@ -1582,11 +1527,11 @@ def extract_heroes_from_room_summary(
         r[0] for r in details.get(Membership.LEAVE, empty_ms).members if r[0] != me
     ] + [r[0] for r in details.get(Membership.BAN, empty_ms).members if r[0] != me]
 
-    # We expect `MemberSummary.members` to already be sorted by `stream_ordering`
+    # FIXME: order by stream ordering rather than as returned by SQL
     if joined_user_ids or invited_user_ids:
-        return (joined_user_ids + invited_user_ids)[0:5]
+        return sorted(joined_user_ids + invited_user_ids)[0:5]
     else:
-        return gone_user_ids[0:5]
+        return sorted(gone_user_ids)[0:5]
 
 
 @attr.s(slots=True, auto_attribs=True)

synapse/storage/databases/main/state.py

@@ -30,7 +30,6 @@ from typing import (
     Iterable,
     List,
     Mapping,
-    MutableMapping,
     Optional,
     Set,
     Tuple,
@@ -42,7 +41,7 @@ from typing import (
 
 import attr
 
-from synapse.api.constants import EventContentFields, EventTypes, Membership
+from synapse.api.constants import EventTypes, Membership
 from synapse.api.errors import NotFoundError, UnsupportedRoomVersionError
 from synapse.api.room_versions import KNOWN_ROOM_VERSIONS, RoomVersion
 from synapse.events import EventBase
@@ -73,18 +72,10 @@ logger = logging.getLogger(__name__)
 
 _T = TypeVar("_T")
 
+
 MAX_STATE_DELTA_HOPS = 100
 
 
-# Freeze so it's immutable and we can use it as a cache value
-@attr.s(slots=True, frozen=True, auto_attribs=True)
-class Sentinel:
-    pass
-
-
-ROOM_UNKNOWN_SENTINEL = Sentinel()
-
-
 @attr.s(slots=True, frozen=True, auto_attribs=True)
 class EventMetadata:
     """Returned by `get_metadata_for_events`"""
@@ -307,194 +298,6 @@ class StateGroupWorkerStore(EventsWorkerStore, SQLBaseStore):
         create_event = await self.get_event(create_id)
         return create_event
 
-    @cached(max_entries=10000)
-    async def get_room_type(self, room_id: str) -> Optional[str]:
-        raise NotImplementedError()
-
-    @cachedList(cached_method_name="get_room_type", list_name="room_ids")
-    async def bulk_get_room_type(
-        self, room_ids: Set[str]
-    ) -> Mapping[str, Union[Optional[str], Sentinel]]:
-        """
-        Bulk fetch room types for the given rooms (via current state).
-
-        Since this function is cached, any missing values would be cached as `None`. In
-        order to distinguish between an unencrypted room that has `None` encryption and
-        a room that is unknown to the server where we might want to omit the value
-        (which would make it cached as `None`), instead we use the sentinel value
-        `ROOM_UNKNOWN_SENTINEL`.
-
-        Returns:
-            A mapping from room ID to the room's type (`None` is a valid room type).
-            Rooms unknown to this server will return `ROOM_UNKNOWN_SENTINEL`.
-        """
-
-        def txn(
-            txn: LoggingTransaction,
-        ) -> MutableMapping[str, Union[Optional[str], Sentinel]]:
-            clause, args = make_in_list_sql_clause(
-                txn.database_engine, "room_id", room_ids
-            )
-
-            # We can't rely on `room_stats_state.room_type` if the server has left the
-            # room because the `room_id` will still be in the table but everything will
-            # be set to `None` but `None` is a valid room type value. We join against
-            # the `room_stats_current` table which keeps track of the
-            # `current_state_events` count (and a proxy value `local_users_in_room`
-            # which can used to assume the server is participating in the room and has
-            # current state) to ensure that the data in `room_stats_state` is up-to-date
-            # with the current state.
-            #
-            # FIXME: Use `room_stats_current.current_state_events` instead of
-            # `room_stats_current.local_users_in_room` once
-            # https://github.com/element-hq/synapse/issues/17457 is fixed.
-            sql = f"""
-                SELECT room_id, room_type
-                FROM room_stats_state
-                INNER JOIN room_stats_current USING (room_id)
-                WHERE
-                    {clause}
-                    AND local_users_in_room > 0
-            """
-
-            txn.execute(sql, args)
-
-            room_id_to_type_map = {}
-            for row in txn:
-                room_id_to_type_map[row[0]] = row[1]
-
-            return room_id_to_type_map
-
-        results = await self.db_pool.runInteraction(
-            "bulk_get_room_type",
-            txn,
-        )
-
-        # If we haven't updated `room_stats_state` with the room yet, query the
-        # create events directly. This should happen only rarely so we don't
-        # mind if we do this in a loop.
-        for room_id in room_ids - results.keys():
-            try:
-                create_event = await self.get_create_event_for_room(room_id)
-                room_type = create_event.content.get(EventContentFields.ROOM_TYPE)
-                results[room_id] = room_type
-            except NotFoundError:
-                # We use the sentinel value to distinguish between `None` which is a
-                # valid room type and a room that is unknown to the server so the value
-                # is just unset.
-                results[room_id] = ROOM_UNKNOWN_SENTINEL
-
-        return results
-
-    @cached(max_entries=10000)
-    async def get_room_encryption(self, room_id: str) -> Optional[str]:
-        raise NotImplementedError()
-
-    @cachedList(cached_method_name="get_room_encryption", list_name="room_ids")
-    async def bulk_get_room_encryption(
-        self, room_ids: Set[str]
-    ) -> Mapping[str, Union[Optional[str], Sentinel]]:
-        """
-        Bulk fetch room encryption for the given rooms (via current state).
-
-        Since this function is cached, any missing values would be cached as `None`. In
-        order to distinguish between an unencrypted room that has `None` encryption and
-        a room that is unknown to the server where we might want to omit the value
-        (which would make it cached as `None`), instead we use the sentinel value
-        `ROOM_UNKNOWN_SENTINEL`.
-
-        Returns:
-            A mapping from room ID to the room's encryption algorithm if the room is
-            encrypted, otherwise `None`. Rooms unknown to this server will return
-            `ROOM_UNKNOWN_SENTINEL`.
-        """
-
-        def txn(
-            txn: LoggingTransaction,
-        ) -> MutableMapping[str, Union[Optional[str], Sentinel]]:
-            clause, args = make_in_list_sql_clause(
-                txn.database_engine, "room_id", room_ids
-            )
-
-            # We can't rely on `room_stats_state.encryption` if the server has left the
-            # room because the `room_id` will still be in the table but everything will
-            # be set to `None` but `None` is a valid encryption value. We join against
-            # the `room_stats_current` table which keeps track of the
-            # `current_state_events` count (and a proxy value `local_users_in_room`
-            # which can used to assume the server is participating in the room and has
-            # current state) to ensure that the data in `room_stats_state` is up-to-date
-            # with the current state.
-            #
-            # FIXME: Use `room_stats_current.current_state_events` instead of
-            # `room_stats_current.local_users_in_room` once
-            # https://github.com/element-hq/synapse/issues/17457 is fixed.
-            sql = f"""
-                SELECT room_id, encryption
-                FROM room_stats_state
-                INNER JOIN room_stats_current USING (room_id)
-                WHERE
-                    {clause}
-                    AND local_users_in_room > 0
-            """
-
-            txn.execute(sql, args)
-
-            room_id_to_encryption_map = {}
-            for row in txn:
-                room_id_to_encryption_map[row[0]] = row[1]
-
-            return room_id_to_encryption_map
-
-        results = await self.db_pool.runInteraction(
-            "bulk_get_room_encryption",
-            txn,
-        )
-
-        # If we haven't updated `room_stats_state` with the room yet, query the state
-        # directly. This should happen only rarely so we don't mind if we do this in a
-        # loop.
-        encryption_event_ids: List[str] = []
-        for room_id in room_ids - results.keys():
-            state_map = await self.get_partial_filtered_current_state_ids(
-                room_id,
-                state_filter=StateFilter.from_types(
-                    [
-                        (EventTypes.Create, ""),
-                        (EventTypes.RoomEncryption, ""),
-                    ]
-                ),
-            )
-            # We can use the create event as a canary to tell whether the server has
-            # seen the room before
-            create_event_id = state_map.get((EventTypes.Create, ""))
-            encryption_event_id = state_map.get((EventTypes.RoomEncryption, ""))
-
-            if create_event_id is None:
-                # We use the sentinel value to distinguish between `None` which is a
-                # valid room type and a room that is unknown to the server so the value
-                # is just unset.
-                results[room_id] = ROOM_UNKNOWN_SENTINEL
-                continue
-
-            if encryption_event_id is None:
-                results[room_id] = None
-            else:
-                encryption_event_ids.append(encryption_event_id)
-
-        encryption_event_map = await self.get_events(encryption_event_ids)
-
-        for encryption_event_id in encryption_event_ids:
-            encryption_event = encryption_event_map.get(encryption_event_id)
-            # If the curent state says there is an encryption event, we should have it
-            # in the database.
-            assert encryption_event is not None
-
-            results[encryption_event.room_id] = encryption_event.content.get(
-                EventContentFields.ENCRYPTION_ALGORITHM
-            )
-
-        return results
-
     @cached(max_entries=100000, iterable=True)
     async def get_partial_current_state_ids(self, room_id: str) -> StateMap[str]:
         """Get the current state event ids for a room based on the

synapse/storage/databases/main/state_deltas.py

@@ -24,11 +24,8 @@ from typing import List, Optional, Tuple
 
 import attr
 
-from synapse.logging.opentracing import trace
 from synapse.storage._base import SQLBaseStore
 from synapse.storage.database import LoggingTransaction
-from synapse.storage.databases.main.stream import _filter_results_by_stream
-from synapse.types import RoomStreamToken
 from synapse.util.caches.stream_change_cache import StreamChangeCache
 
 logger = logging.getLogger(__name__)
@@ -159,44 +156,3 @@ class StateDeltasStore(SQLBaseStore):
             "get_max_stream_id_in_current_state_deltas",
             self._get_max_stream_id_in_current_state_deltas_txn,
         )
-
-    @trace
-    async def get_current_state_deltas_for_room(
-        self, room_id: str, from_token: RoomStreamToken, to_token: RoomStreamToken
-    ) -> List[StateDelta]:
-        """Get the state deltas between two tokens."""
-
-        if not self._curr_state_delta_stream_cache.has_entity_changed(
-            room_id, from_token.stream
-        ):
-            return []
-
-        def get_current_state_deltas_for_room_txn(
-            txn: LoggingTransaction,
-        ) -> List[StateDelta]:
-            sql = """
-                SELECT instance_name, stream_id, type, state_key, event_id, prev_event_id
-                FROM current_state_delta_stream
-                WHERE room_id = ? AND ? < stream_id AND stream_id <= ?
-                ORDER BY stream_id ASC
-            """
-            txn.execute(
-                sql, (room_id, from_token.stream, to_token.get_max_stream_pos())
-            )
-
-            return [
-                StateDelta(
-                    stream_id=row[1],
-                    room_id=room_id,
-                    event_type=row[2],
-                    state_key=row[3],
-                    event_id=row[4],
-                    prev_event_id=row[5],
-                )
-                for row in txn
-                if _filter_results_by_stream(from_token, to_token, row[0], row[1])
-            ]
-
-        return await self.db_pool.runInteraction(
-            "get_current_state_deltas_for_room", get_current_state_deltas_for_room_txn
-        )

synapse/storage/databases/main/stream.py

@@ -51,7 +51,6 @@ from typing import (
     Iterable,
     List,
     Optional,
-    Protocol,
     Set,
     Tuple,
     cast,
@@ -60,7 +59,7 @@ from typing import (
 
 import attr
 from immutabledict import immutabledict
-from typing_extensions import Literal, assert_never
+from typing_extensions import Literal
 
 from twisted.internet import defer
 
@@ -68,7 +67,7 @@ from synapse.api.constants import Direction, EventTypes, Membership
 from synapse.api.filtering import Filter
 from synapse.events import EventBase
 from synapse.logging.context import make_deferred_yieldable, run_in_background
-from synapse.logging.opentracing import tag_args, trace
+from synapse.logging.opentracing import trace
 from synapse.storage._base import SQLBaseStore
 from synapse.storage.database import (
     DatabasePool,
@@ -79,11 +78,10 @@ from synapse.storage.database import (
 from synapse.storage.databases.main.events_worker import EventsWorkerStore
 from synapse.storage.engines import BaseDatabaseEngine, PostgresEngine, Sqlite3Engine
 from synapse.storage.util.id_generators import MultiWriterIdGenerator
-from synapse.types import PersistedEventPosition, RoomStreamToken, StrCollection
+from synapse.types import PersistedEventPosition, RoomStreamToken
 from synapse.util.caches.descriptors import cached
 from synapse.util.caches.stream_change_cache import StreamChangeCache
 from synapse.util.cancellation import cancellable
-from synapse.util.iterutils import batch_iter
 
 if TYPE_CHECKING:
     from synapse.server import HomeServer
@@ -98,18 +96,6 @@ _STREAM_TOKEN = "stream"
 _TOPOLOGICAL_TOKEN = "topological"
 
 
-class PaginateFunction(Protocol):
-    async def __call__(
-        self,
-        *,
-        room_id: str,
-        from_key: RoomStreamToken,
-        to_key: Optional[RoomStreamToken] = None,
-        direction: Direction = Direction.BACKWARDS,
-        limit: int = 0,
-    ) -> Tuple[List[EventBase], RoomStreamToken]: ...
-
-
 # Used as return values for pagination APIs
 @attr.s(slots=True, frozen=True, auto_attribs=True)
 class _EventDictReturn:
@@ -293,7 +279,7 @@ def generate_pagination_bounds(
 
 
 def generate_next_token(
-    direction: Direction, last_topo_ordering: Optional[int], last_stream_ordering: int
+    direction: Direction, last_topo_ordering: int, last_stream_ordering: int
 ) -> RoomStreamToken:
     """
     Generate the next room stream token based on the currently returned data.
@@ -460,6 +446,7 @@ def _filter_results_by_stream(
     The `instance_name` arg is optional to handle historic rows, and is
     interpreted as if it was "master".
     """
+
     if instance_name is None:
         instance_name = "master"
 
@@ -672,43 +659,33 @@ class StreamWorkerStore(EventsWorkerStore, SQLBaseStore):
 
     async def get_room_events_stream_for_rooms(
         self,
-        *,
         room_ids: Collection[str],
         from_key: RoomStreamToken,
-        to_key: Optional[RoomStreamToken] = None,
-        direction: Direction = Direction.BACKWARDS,
+        to_key: RoomStreamToken,
         limit: int = 0,
+        order: str = "DESC",
     ) -> Dict[str, Tuple[List[EventBase], RoomStreamToken]]:
         """Get new room events in stream ordering since `from_key`.
 
         Args:
             room_ids
-            from_key: The token to stream from (starting point and heading in the given
-                direction)
-            to_key: The token representing the end stream position (end point)
+            from_key: Token from which no events are returned before
+            to_key: Token from which no events are returned after. (This
+                is typically the current stream token)
             limit: Maximum number of events to return
-            direction: Indicates whether we are paginating forwards or backwards
-                from `from_key`.
+            order: Either "DESC" or "ASC". Determines which events are
+                returned when the result is limited. If "DESC" then the most
+                recent `limit` events are returned, otherwise returns the
+                oldest `limit` events.
 
         Returns:
             A map from room id to a tuple containing:
                 - list of recent events in the room
                 - stream ordering key for the start of the chunk of events returned.
-
-            When Direction.FORWARDS: from_key < x <= to_key, (ascending order)
-            When Direction.BACKWARDS: from_key >= x > to_key, (descending order)
         """
-        if direction == Direction.FORWARDS:
-            room_ids = self._events_stream_cache.get_entities_changed(
-                room_ids, from_key.stream
-            )
-        elif direction == Direction.BACKWARDS:
-            if to_key is not None:
-                room_ids = self._events_stream_cache.get_entities_changed(
-                    room_ids, to_key.stream
-                )
-        else:
-            assert_never(direction)
+        room_ids = self._events_stream_cache.get_entities_changed(
+            room_ids, from_key.stream
+        )
 
         if not room_ids:
             return {}
@@ -720,12 +697,12 @@ class StreamWorkerStore(EventsWorkerStore, SQLBaseStore):
                 defer.gatherResults(
                     [
                         run_in_background(
-                            self.paginate_room_events_by_stream_ordering,
-                            room_id=room_id,
-                            from_key=from_key,
-                            to_key=to_key,
-                            direction=direction,
-                            limit=limit,
+                            self.get_room_events_stream_for_room,
+                            room_id,
+                            from_key,
+                            to_key,
+                            limit,
+                            order=order,
                         )
                         for room_id in rm_ids
                     ],
@@ -749,122 +726,69 @@ class StreamWorkerStore(EventsWorkerStore, SQLBaseStore):
             if self._events_stream_cache.has_entity_changed(room_id, from_id)
         }
 
-    async def paginate_room_events_by_stream_ordering(
+    async def get_room_events_stream_for_room(
         self,
-        *,
         room_id: str,
         from_key: RoomStreamToken,
-        to_key: Optional[RoomStreamToken] = None,
-        direction: Direction = Direction.BACKWARDS,
+        to_key: RoomStreamToken,
         limit: int = 0,
+        order: str = "DESC",
     ) -> Tuple[List[EventBase], RoomStreamToken]:
-        """
-        Paginate events by `stream_ordering` in the room from the `from_key` in the
-        given `direction` to the `to_key` or `limit`.
+        """Get new room events in stream ordering since `from_key`.
 
         Args:
             room_id
-            from_key: The token to stream from (starting point and heading in the given
-                direction)
-            to_key: The token representing the end stream position (end point)
-            direction: Indicates whether we are paginating forwards or backwards
-                from `from_key`.
+            from_key: Token from which no events are returned before
+            to_key: Token from which no events are returned after. (This
+                is typically the current stream token)
             limit: Maximum number of events to return
+            order: Either "DESC" or "ASC". Determines which events are
+                returned when the result is limited. If "DESC" then the most
+                recent `limit` events are returned, otherwise returns the
+                oldest `limit` events.
 
         Returns:
-            The results as a list of events and a token that points to the end
-            of the result set. If no events are returned then the end of the
-            stream has been reached (i.e. there are no events between `from_key`
-            and `to_key`).
-
-            When Direction.FORWARDS: from_key < x <= to_key, (ascending order)
-            When Direction.BACKWARDS: from_key >= x > to_key, (descending order)
+            The list of events (in ascending stream order) and the token from the start
+            of the chunk of events returned.
         """
+        if from_key == to_key:
+            return [], from_key
 
-        # FIXME: When going forwards, we should enforce that the `to_key` is not `None`
-        # because we always need an upper bound when querying the events stream (as
-        # otherwise we'll potentially pick up events that are not fully persisted).
-
-        # We should only be working with `stream_ordering` tokens here
-        assert from_key is None or from_key.topological is None
-        assert to_key is None or to_key.topological is None
-
-        # We can bail early if we're looking forwards, and our `to_key` is already
-        # before our `from_key`.
-        if (
-            direction == Direction.FORWARDS
-            and to_key is not None
-            and to_key.is_before_or_eq(from_key)
-        ):
-            # Token selection matches what we do below if there are no rows
-            return [], to_key if to_key else from_key
-        # Or vice-versa, if we're looking backwards and our `from_key` is already before
-        # our `to_key`.
-        elif (
-            direction == Direction.BACKWARDS
-            and to_key is not None
-            and from_key.is_before_or_eq(to_key)
-        ):
-            # Token selection matches what we do below if there are no rows
-            return [], to_key if to_key else from_key
-
-        # We can do a quick sanity check to see if any events have been sent in the room
-        # since the earlier token.
-        has_changed = True
-        if direction == Direction.FORWARDS:
-            has_changed = self._events_stream_cache.has_entity_changed(
-                room_id, from_key.stream
-            )
-        elif direction == Direction.BACKWARDS:
-            if to_key is not None:
-                has_changed = self._events_stream_cache.has_entity_changed(
-                    room_id, to_key.stream
-                )
-        else:
-            assert_never(direction)
+        has_changed = self._events_stream_cache.has_entity_changed(
+            room_id, from_key.stream
+        )
 
         if not has_changed:
-            # Token selection matches what we do below if there are no rows
-            return [], to_key if to_key else from_key
-
-        order, from_bound, to_bound = generate_pagination_bounds(
-            direction, from_key, to_key
-        )
-
-        bounds = generate_pagination_where_clause(
-            direction=direction,
-            # The empty string will shortcut downstream code to only use the
-            # `stream_ordering` column
-            column_names=("", "stream_ordering"),
-            from_token=from_bound,
-            to_token=to_bound,
-            engine=self.database_engine,
-        )
+            return [], from_key
 
         def f(txn: LoggingTransaction) -> List[_EventDictReturn]:
-            sql = f"""
-                SELECT event_id, instance_name, stream_ordering
+            # To handle tokens with a non-empty instance_map we fetch more
+            # results than necessary and then filter down
+            min_from_id = from_key.stream
+            max_to_id = to_key.get_max_stream_pos()
+
+            sql = """
+                SELECT event_id, instance_name, topological_ordering, stream_ordering
                 FROM events
                 WHERE
                     room_id = ?
                     AND not outlier
-                    AND {bounds}
-                ORDER BY stream_ordering {order} LIMIT ?
-            """
-            txn.execute(sql, (room_id, 2 * limit))
+                    AND stream_ordering > ? AND stream_ordering <= ?
+                ORDER BY stream_ordering %s LIMIT ?
+            """ % (
+                order,
+            )
+            txn.execute(sql, (room_id, min_from_id, max_to_id, 2 * limit))
 
             rows = [
                 _EventDictReturn(event_id, None, stream_ordering)
-                for event_id, instance_name, stream_ordering in txn
-                if _filter_results_by_stream(
-                    lower_token=(
-                        to_key if direction == Direction.BACKWARDS else from_key
-                    ),
-                    upper_token=(
-                        from_key if direction == Direction.BACKWARDS else to_key
-                    ),
-                    instance_name=instance_name,
-                    stream_ordering=stream_ordering,
+                for event_id, instance_name, topological_ordering, stream_ordering in txn
+                if _filter_results(
+                    from_key,
+                    to_key,
+                    instance_name,
+                    topological_ordering,
+                    stream_ordering,
                 )
             ][:limit]
             return rows
@@ -875,20 +799,18 @@ class StreamWorkerStore(EventsWorkerStore, SQLBaseStore):
             [r.event_id for r in rows], get_prev_content=True
         )
 
+        if order.lower() == "desc":
+            ret.reverse()
+
         if rows:
-            next_key = generate_next_token(
-                direction=direction,
-                last_topo_ordering=None,
-                last_stream_ordering=rows[-1].stream_ordering,
-            )
+            key = RoomStreamToken(stream=min(r.stream_ordering for r in rows))
         else:
-            # TODO (erikj): We should work out what to do here instead. (same as
-            # `_paginate_room_events_by_topological_ordering_txn(...)`)
-            next_key = to_key if to_key else from_key
+            # Assume we didn't get anything because there was nothing to
+            # get.
+            key = from_key
 
-        return ret, next_key
+        return ret, key
 
-    @trace
     async def get_current_state_delta_membership_changes_for_user(
         self,
         user_id: str,
@@ -1194,7 +1116,7 @@ class StreamWorkerStore(EventsWorkerStore, SQLBaseStore):
 
         rows, token = await self.db_pool.runInteraction(
             "get_recent_event_ids_for_room",
-            self._paginate_room_events_by_topological_ordering_txn,
+            self._paginate_room_events_txn,
             room_id,
             from_token=end_token,
             limit=limit,
@@ -1263,7 +1185,6 @@ class StreamWorkerStore(EventsWorkerStore, SQLBaseStore):
 
         return None
 
-    @trace
     async def get_last_event_pos_in_room_before_stream_ordering(
         self,
         room_id: str,
@@ -1372,126 +1293,6 @@ class StreamWorkerStore(EventsWorkerStore, SQLBaseStore):
             get_last_event_pos_in_room_before_stream_ordering_txn,
         )
 
-    async def bulk_get_last_event_pos_in_room_before_stream_ordering(
-        self,
-        room_ids: StrCollection,
-        end_token: RoomStreamToken,
-    ) -> Dict[str, int]:
-        """Bulk fetch the stream position of the latest events in the given
-        rooms
-        """
-
-        min_token = end_token.stream
-        max_token = end_token.get_max_stream_pos()
-        results: Dict[str, int] = {}
-
-        # First, we check for the rooms in the stream change cache to see if we
-        # can just use the latest position from it.
-        missing_room_ids: Set[str] = set()
-        for room_id in room_ids:
-            stream_pos = self._events_stream_cache.get_max_pos_of_last_change(room_id)
-            if stream_pos and stream_pos <= min_token:
-                results[room_id] = stream_pos
-            else:
-                missing_room_ids.add(room_id)
-
-        # Next, we query the stream position from the DB. At first we fetch all
-        # positions less than the *max* stream pos in the token, then filter
-        # them down. We do this as a) this is a cheaper query, and b) the vast
-        # majority of rooms will have a latest token from before the min stream
-        # pos.
-
-        def bulk_get_last_event_pos_txn(
-            txn: LoggingTransaction, batch_room_ids: StrCollection
-        ) -> Dict[str, int]:
-            # This query fetches the latest stream position in the rooms before
-            # the given max position.
-            clause, args = make_in_list_sql_clause(
-                self.database_engine, "room_id", batch_room_ids
-            )
-            sql = f"""
-                SELECT room_id, (
-                    SELECT stream_ordering FROM events AS e
-                    LEFT JOIN rejections USING (event_id)
-                    WHERE e.room_id = r.room_id
-                        AND stream_ordering <= ?
-                        AND NOT outlier
-                        AND rejection_reason IS NULL
-                    ORDER BY stream_ordering DESC
-                    LIMIT 1
-                )
-                FROM rooms AS r
-                WHERE {clause}
-            """
-            txn.execute(sql, [max_token] + args)
-            return {row[0]: row[1] for row in txn}
-
-        recheck_rooms: Set[str] = set()
-        for batched in batch_iter(missing_room_ids, 1000):
-            result = await self.db_pool.runInteraction(
-                "bulk_get_last_event_pos_in_room_before_stream_ordering",
-                bulk_get_last_event_pos_txn,
-                batched,
-            )
-
-            # Check that the stream position for the rooms are from before the
-            # minimum position of the token. If not then we need to fetch more
-            # rows.
-            for room_id, stream in result.items():
-                if stream <= min_token:
-                    results[room_id] = stream
-                else:
-                    recheck_rooms.add(room_id)
-
-        if not recheck_rooms:
-            return results
-
-        # For the remaining rooms we need to fetch all rows between the min and
-        # max stream positions in the end token, and filter out the rows that
-        # are after the end token.
-        #
-        # This query should be fast as the range between the min and max should
-        # be small.
-
-        def bulk_get_last_event_pos_recheck_txn(
-            txn: LoggingTransaction, batch_room_ids: StrCollection
-        ) -> Dict[str, int]:
-            clause, args = make_in_list_sql_clause(
-                self.database_engine, "room_id", batch_room_ids
-            )
-            sql = f"""
-                SELECT room_id, instance_name, stream_ordering
-                FROM events
-                WHERE ? < stream_ordering AND stream_ordering <= ?
-                    AND NOT outlier
-                    AND rejection_reason IS NULL
-                    AND {clause}
-                ORDER BY stream_ordering ASC
-            """
-            txn.execute(sql, [min_token, max_token] + args)
-
-            # We take the max stream ordering that is less than the token. Since
-            # we ordered by stream ordering we just need to iterate through and
-            # take the last matching stream ordering.
-            txn_results: Dict[str, int] = {}
-            for row in txn:
-                room_id = row[0]
-                event_pos = PersistedEventPosition(row[1], row[2])
-                if not event_pos.persisted_after(end_token):
-                    txn_results[room_id] = event_pos.stream
-
-            return txn_results
-
-        for batched in batch_iter(recheck_rooms, 1000):
-            recheck_result = await self.db_pool.runInteraction(
-                "bulk_get_last_event_pos_in_room_before_stream_ordering_recheck",
-                bulk_get_last_event_pos_recheck_txn,
-                batched,
-            )
-            results.update(recheck_result)
-
-        return results
-
     async def get_current_room_stream_token_for_room_id(
         self, room_id: str
     ) -> RoomStreamToken:
@@ -1700,7 +1501,7 @@ class StreamWorkerStore(EventsWorkerStore, SQLBaseStore):
             topological=topological_ordering, stream=stream_ordering
         )
 
-        rows, start_token = self._paginate_room_events_by_topological_ordering_txn(
+        rows, start_token = self._paginate_room_events_txn(
             txn,
             room_id,
             before_token,
@@ -1710,7 +1511,7 @@ class StreamWorkerStore(EventsWorkerStore, SQLBaseStore):
         )
         events_before = [r.event_id for r in rows]
 
-        rows, end_token = self._paginate_room_events_by_topological_ordering_txn(
+        rows, end_token = self._paginate_room_events_txn(
             txn,
             room_id,
             after_token,
@@ -1873,14 +1674,14 @@ class StreamWorkerStore(EventsWorkerStore, SQLBaseStore):
     def has_room_changed_since(self, room_id: str, stream_id: int) -> bool:
         return self._events_stream_cache.has_entity_changed(room_id, stream_id)
 
-    def _paginate_room_events_by_topological_ordering_txn(
+    def _paginate_room_events_txn(
         self,
         txn: LoggingTransaction,
         room_id: str,
         from_token: RoomStreamToken,
         to_token: Optional[RoomStreamToken] = None,
         direction: Direction = Direction.BACKWARDS,
-        limit: int = 0,
+        limit: int = -1,
         event_filter: Optional[Filter] = None,
     ) -> Tuple[List[_EventDictReturn], RoomStreamToken]:
         """Returns list of events before or after a given token.
@@ -1902,24 +1703,6 @@ class StreamWorkerStore(EventsWorkerStore, SQLBaseStore):
             been reached (i.e. there are no events between `from_token` and
             `to_token`), or `limit` is zero.
         """
-        # We can bail early if we're looking forwards, and our `to_key` is already
-        # before our `from_token`.
-        if (
-            direction == Direction.FORWARDS
-            and to_token is not None
-            and to_token.is_before_or_eq(from_token)
-        ):
-            # Token selection matches what we do below if there are no rows
-            return [], to_token if to_token else from_token
-        # Or vice-versa, if we're looking backwards and our `from_token` is already before
-        # our `to_token`.
-        elif (
-            direction == Direction.BACKWARDS
-            and to_token is not None
-            and from_token.is_before_or_eq(to_token)
-        ):
-            # Token selection matches what we do below if there are no rows
-            return [], to_token if to_token else from_token
 
         args: List[Any] = [room_id]
 
@@ -2004,6 +1787,7 @@ class StreamWorkerStore(EventsWorkerStore, SQLBaseStore):
             "bounds": bounds,
             "order": order,
         }
+
         txn.execute(sql, args)
 
         # Filter the result set.
@@ -2035,30 +1819,27 @@ class StreamWorkerStore(EventsWorkerStore, SQLBaseStore):
         return rows, next_token
 
     @trace
-    @tag_args
-    async def paginate_room_events_by_topological_ordering(
+    async def paginate_room_events(
         self,
-        *,
         room_id: str,
         from_key: RoomStreamToken,
         to_key: Optional[RoomStreamToken] = None,
         direction: Direction = Direction.BACKWARDS,
-        limit: int = 0,
+        limit: int = -1,
         event_filter: Optional[Filter] = None,
     ) -> Tuple[List[EventBase], RoomStreamToken]:
-        """
-        Paginate events by `topological_ordering` (tie-break with `stream_ordering`) in
-        the room from the `from_key` in the given `direction` to the `to_key` or
-        `limit`.
+        """Returns list of events before or after a given token.
+
+        When Direction.FORWARDS: from_key < x <= to_key
+        When Direction.BACKWARDS: from_key >= x > to_key
 
         Args:
             room_id
-            from_key: The token to stream from (starting point and heading in the given
-                direction)
-            to_key: The token representing the end stream position (end point)
+            from_key: The token used to stream from
+            to_key: A token which if given limits the results to only those before
             direction: Indicates whether we are paginating forwards or backwards
                 from `from_key`.
-            limit: Maximum number of events to return
+            limit: The maximum number of events to return.
             event_filter: If provided filters the events to those that match the filter.
 
         Returns:
@@ -2066,18 +1847,8 @@ class StreamWorkerStore(EventsWorkerStore, SQLBaseStore):
             of the result set. If no events are returned then the end of the
             stream has been reached (i.e. there are no events between `from_key`
             and `to_key`).
-
-            When Direction.FORWARDS: from_key < x <= to_key, (ascending order)
-            When Direction.BACKWARDS: from_key >= x > to_key, (descending order)
         """
 
-        # FIXME: When going forwards, we should enforce that the `to_key` is not `None`
-        # because we always need an upper bound when querying the events stream (as
-        # otherwise we'll potentially pick up events that are not fully persisted).
-
-        # We have these checks outside of the transaction function (txn) to save getting
-        # a DB connection and switching threads if we don't need to.
-        #
         # We can bail early if we're looking forwards, and our `to_key` is already
         # before our `from_key`.
         if (
@@ -2100,8 +1871,8 @@ class StreamWorkerStore(EventsWorkerStore, SQLBaseStore):
             return [], to_key if to_key else from_key
 
         rows, token = await self.db_pool.runInteraction(
-            "paginate_room_events_by_topological_ordering",
-            self._paginate_room_events_by_topological_ordering_txn,
+            "paginate_room_events",
+            self._paginate_room_events_txn,
             room_id,
             from_key,
             to_key,
@@ -2212,14 +1983,3 @@ class StreamWorkerStore(EventsWorkerStore, SQLBaseStore):
             return RoomStreamToken(stream=last_position.stream - 1)
 
         return None
-
-    @trace
-    def get_rooms_that_might_have_updates(
-        self, room_ids: StrCollection, from_token: RoomStreamToken
-    ) -> StrCollection:
-        """Filters given room IDs down to those that might have updates, i.e.
-        removes rooms that definitely do not have updates.
-        """
-        return self._events_stream_cache.get_entities_changed(
-            room_ids, from_token.stream
-        )

synapse/storage/schema/__init__.py

@@ -19,7 +19,7 @@
 #
 #
 
-SCHEMA_VERSION = 86  # remember to update the list below when updating
+SCHEMA_VERSION = 85  # remember to update the list below when updating
 """Represents the expectations made by the codebase about the database schema
 
 This should be incremented whenever the codebase changes its requirements on the
@@ -139,9 +139,6 @@ Changes in SCHEMA_VERSION = 84
 
 Changes in SCHEMA_VERSION = 85
     - Add a column `suspended` to the `users` table
-
-Changes in SCHEMA_VERSION = 86
-    - Add a column `authenticated` to the tables `local_media_repository` and `remote_media_cache`
 """
 
 

synapse/storage/schema/main/delta/86/01_authenticate_media.sql

@@ -1,15 +0,0 @@
---
--- This file is licensed under the Affero General Public License (AGPL) version 3.
---
--- Copyright (C) 2024 New Vector, Ltd
---
--- This program is free software: you can redistribute it and/or modify
--- it under the terms of the GNU Affero General Public License as
--- published by the Free Software Foundation, either version 3 of the
--- License, or (at your option) any later version.
---
--- See the GNU Affero General Public License for more details:
--- <https://www.gnu.org/licenses/agpl-3.0.html>.
-
-ALTER TABLE remote_media_cache ADD COLUMN authenticated BOOLEAN DEFAULT FALSE NOT NULL;
-ALTER TABLE local_media_repository ADD COLUMN authenticated BOOLEAN DEFAULT FALSE NOT NULL;

synapse/types/__init__.py

@@ -20,7 +20,6 @@
 #
 #
 import abc
-import logging
 import re
 import string
 from enum import Enum
@@ -75,9 +74,6 @@ if TYPE_CHECKING:
     from synapse.storage.databases.main import DataStore, PurgeEventsStore
     from synapse.storage.databases.main.appservice import ApplicationServiceWorkerStore
 
-
-logger = logging.getLogger(__name__)
-
 # Define a state map type from type/state_key to T (usually an event ID or
 # event)
 T = TypeVar("T")
@@ -458,8 +454,6 @@ class AbstractMultiWriterStreamToken(metaclass=abc.ABCMeta):
     represented by a default `stream` attribute and a map of instance name to
     stream position of any writers that are ahead of the default stream
     position.
-
-    The values in `instance_map` must be greater than the `stream` attribute.
     """
 
     stream: int = attr.ib(validator=attr.validators.instance_of(int), kw_only=True)
@@ -474,15 +468,6 @@ class AbstractMultiWriterStreamToken(metaclass=abc.ABCMeta):
         kw_only=True,
     )
 
-    def __attrs_post_init__(self) -> None:
-        # Enforce that all instances have a value greater than the min stream
-        # position.
-        for i, v in self.instance_map.items():
-            if v <= self.stream:
-                raise ValueError(
-                    f"'instance_map' includes a stream position before the main 'stream' attribute. Instance: {i}"
-                )
-
     @classmethod
     @abc.abstractmethod
     async def parse(cls, store: "DataStore", string: str) -> "Self":
@@ -509,9 +494,6 @@ class AbstractMultiWriterStreamToken(metaclass=abc.ABCMeta):
             for instance in set(self.instance_map).union(other.instance_map)
         }
 
-        # Filter out any redundant entries.
-        instance_map = {i: s for i, s in instance_map.items() if s > max_stream}
-
         return attr.evolve(
             self, stream=max_stream, instance_map=immutabledict(instance_map)
         )
@@ -557,15 +539,10 @@ class AbstractMultiWriterStreamToken(metaclass=abc.ABCMeta):
     def bound_stream_token(self, max_stream: int) -> "Self":
         """Bound the stream positions to a maximum value"""
 
-        min_pos = min(self.stream, max_stream)
         return type(self)(
-            stream=min_pos,
+            stream=min(self.stream, max_stream),
             instance_map=immutabledict(
-                {
-                    k: min(s, max_stream)
-                    for k, s in self.instance_map.items()
-                    if min(s, max_stream) > min_pos
-                }
+                {k: min(s, max_stream) for k, s in self.instance_map.items()}
             ),
         )
 
@@ -660,8 +637,6 @@ class RoomStreamToken(AbstractMultiWriterStreamToken):
                 "Cannot set both 'topological' and 'instance_map' on 'RoomStreamToken'."
             )
 
-        super().__attrs_post_init__()
-
     @classmethod
     async def parse(cls, store: "PurgeEventsStore", string: str) -> "RoomStreamToken":
         try:
@@ -676,11 +651,6 @@ class RoomStreamToken(AbstractMultiWriterStreamToken):
 
                 instance_map = {}
                 for part in parts[1:]:
-                    if not part:
-                        # Handle tokens of the form `m5~`, which were created by
-                        # a bug
-                        continue
-
                     key, value = part.split(".")
                     instance_id = int(key)
                     pos = int(value)
@@ -696,10 +666,7 @@ class RoomStreamToken(AbstractMultiWriterStreamToken):
         except CancelledError:
             raise
         except Exception:
-            # We log an exception here as even though this *might* be a client
-            # handing a bad token, its more likely that Synapse returned a bad
-            # token (and we really want to catch those!).
-            logger.exception("Failed to parse stream token: %r", string)
+            pass
         raise SynapseError(400, "Invalid room stream token %r" % (string,))
 
     @classmethod
@@ -746,8 +713,6 @@ class RoomStreamToken(AbstractMultiWriterStreamToken):
         return self.instance_map.get(instance_name, self.stream)
 
     async def to_string(self, store: "DataStore") -> str:
-        """See class level docstring for information about the format."""
-
         if self.topological is not None:
             return "t%d-%d" % (self.topological, self.stream)
         elif self.instance_map:
@@ -762,10 +727,8 @@ class RoomStreamToken(AbstractMultiWriterStreamToken):
                 instance_id = await store.get_id_for_instance(name)
                 entries.append(f"{instance_id}.{pos}")
 
-            if entries:
-                encoded_map = "~".join(entries)
-                return f"m{self.stream}~{encoded_map}"
-            return f"s{self.stream}"
+            encoded_map = "~".join(entries)
+            return f"m{self.stream}~{encoded_map}"
         else:
             return "s%d" % (self.stream,)
 
@@ -777,13 +740,6 @@ class RoomStreamToken(AbstractMultiWriterStreamToken):
 
         return super().bound_stream_token(max_stream)
 
-    def __str__(self) -> str:
-        instances = ", ".join(f"{k}: {v}" for k, v in sorted(self.instance_map.items()))
-        return (
-            f"RoomStreamToken(stream: {self.stream}, topological: {self.topological}, "
-            f"instances: {{{instances}}})"
-        )
-
 
 @attr.s(frozen=True, slots=True, order=False)
 class MultiWriterStreamToken(AbstractMultiWriterStreamToken):
@@ -800,11 +756,6 @@ class MultiWriterStreamToken(AbstractMultiWriterStreamToken):
 
                 instance_map = {}
                 for part in parts[1:]:
-                    if not part:
-                        # Handle tokens of the form `m5~`, which were created by
-                        # a bug
-                        continue
-
                     key, value = part.split(".")
                     instance_id = int(key)
                     pos = int(value)
@@ -819,15 +770,10 @@ class MultiWriterStreamToken(AbstractMultiWriterStreamToken):
         except CancelledError:
             raise
         except Exception:
-            # We log an exception here as even though this *might* be a client
-            # handing a bad token, its more likely that Synapse returned a bad
-            # token (and we really want to catch those!).
-            logger.exception("Failed to parse stream token: %r", string)
+            pass
         raise SynapseError(400, "Invalid stream token %r" % (string,))
 
     async def to_string(self, store: "DataStore") -> str:
-        """See class level docstring for information about the format."""
-
         if self.instance_map:
             entries = []
             for name, pos in self.instance_map.items():
@@ -840,10 +786,8 @@ class MultiWriterStreamToken(AbstractMultiWriterStreamToken):
                 instance_id = await store.get_id_for_instance(name)
                 entries.append(f"{instance_id}.{pos}")
 
-            if entries:
-                encoded_map = "~".join(entries)
-                return f"m{self.stream}~{encoded_map}"
-            return str(self.stream)
+            encoded_map = "~".join(entries)
+            return f"m{self.stream}~{encoded_map}"
         else:
             return str(self.stream)
 
@@ -880,13 +824,6 @@ class MultiWriterStreamToken(AbstractMultiWriterStreamToken):
 
         return True
 
-    def __str__(self) -> str:
-        instances = ", ".join(f"{k}: {v}" for k, v in sorted(self.instance_map.items()))
-        return (
-            f"MultiWriterStreamToken(stream: {self.stream}, "
-            f"instances: {{{instances}}})"
-        )
-
 
 class StreamKeyType(Enum):
     """Known stream types.
@@ -1145,64 +1082,12 @@ class StreamToken:
 
         return True
 
-    def __str__(self) -> str:
-        return (
-            f"StreamToken(room: {self.room_key}, presence: {self.presence_key}, "
-            f"typing: {self.typing_key}, receipt: {self.receipt_key}, "
-            f"account_data: {self.account_data_key}, push_rules: {self.push_rules_key}, "
-            f"to_device: {self.to_device_key}, device_list: {self.device_list_key}, "
-            f"groups: {self.groups_key}, un_partial_stated_rooms: {self.un_partial_stated_rooms_key})"
-        )
-
 
 StreamToken.START = StreamToken(
     RoomStreamToken(stream=0), 0, 0, MultiWriterStreamToken(stream=0), 0, 0, 0, 0, 0, 0
 )
 
 
-@attr.s(slots=True, frozen=True, auto_attribs=True)
-class SlidingSyncStreamToken:
-    """The same as a `StreamToken`, but includes an extra field at the start for
-    the sliding sync connection token (separated by a '/'). This is used to
-    store per-connection state.
-
-    This then looks something like:
-        5/s2633508_17_338_6732159_1082514_541479_274711_265584_1_379
-
-    Attributes:
-        stream_token: Token representing the position of all the standard
-            streams.
-        connection_position: Token used by sliding sync to track updates to any
-            per-connection state stored by Synapse.
-    """
-
-    stream_token: StreamToken
-    connection_position: int
-
-    @staticmethod
-    @cancellable
-    async def from_string(store: "DataStore", string: str) -> "SlidingSyncStreamToken":
-        """Creates a SlidingSyncStreamToken from its textual representation."""
-        try:
-            connection_position_str, stream_token_str = string.split("/", 1)
-            connection_position = int(connection_position_str)
-            stream_token = await StreamToken.from_string(store, stream_token_str)
-
-            return SlidingSyncStreamToken(
-                stream_token=stream_token,
-                connection_position=connection_position,
-            )
-        except CancelledError:
-            raise
-        except Exception:
-            raise SynapseError(400, "Invalid stream token")
-
-    async def to_string(self, store: "DataStore") -> str:
-        """Serializes the token to a string"""
-        stream_token_str = await self.stream_token.to_string(store)
-        return f"{self.connection_position}/{stream_token_str}"
-
-
 @attr.s(slots=True, frozen=True, auto_attribs=True)
 class PersistedPosition:
     """Position of a newly persisted row with instance that persisted it."""
@@ -1285,12 +1170,11 @@ class ReadReceipt:
 @attr.s(slots=True, frozen=True, auto_attribs=True)
 class DeviceListUpdates:
     """
-    An object containing a diff of information regarding other users' device lists,
-    intended for a recipient to carry out device list tracking.
+    An object containing a diff of information regarding other users' device lists, intended for
+    a recipient to carry out device list tracking.
 
     Attributes:
-        changed: A set of users who have updated their device identity or
-            cross-signing keys, or who now share an encrypted room with.
+        changed: A set of users whose device lists have changed recently.
         left: A set of users who the recipient no longer needs to track the device lists of.
             Typically when those users no longer share any end-to-end encryption enabled rooms.
     """

synapse/types/handlers/__init__.py

@@ -18,7 +18,7 @@
 #
 #
 from enum import Enum
-from typing import TYPE_CHECKING, Dict, Final, List, Mapping, Optional, Sequence, Tuple
+from typing import TYPE_CHECKING, Dict, Final, List, Optional, Tuple
 
 import attr
 from typing_extensions import TypedDict
@@ -31,15 +31,7 @@ else:
     from pydantic import Extra
 
 from synapse.events import EventBase
-from synapse.types import (
-    DeviceListUpdates,
-    JsonDict,
-    JsonMapping,
-    Requester,
-    SlidingSyncStreamToken,
-    StreamToken,
-    UserID,
-)
+from synapse.types import JsonDict, JsonMapping, StreamToken, UserID
 from synapse.types.rest.client import SlidingSyncBody
 
 if TYPE_CHECKING:
@@ -110,7 +102,7 @@ class SlidingSyncConfig(SlidingSyncBody):
     """
 
     user: UserID
-    requester: Requester
+    device_id: Optional[str]
 
     # Pydantic config
     class Config:
@@ -152,7 +144,7 @@ class SlidingSyncResult:
     Attributes:
         next_pos: The next position token in the sliding window to request (next_batch).
         lists: Sliding window API. A map of list key to list results.
-        rooms: Room subscription API. A map of room ID to room results.
+        rooms: Room subscription API. A map of room ID to room subscription to room results.
         extensions: Extensions API. A map of extension key to extension results.
     """
 
@@ -182,8 +174,8 @@ class SlidingSyncResult:
                 absent on joined/left rooms
             prev_batch: A token that can be passed as a start parameter to the
                 `/rooms/<room_id>/messages` API to retrieve earlier messages.
-            limited: True if there are more events than `timeline_limit` looking
-                backwards from the `response.pos` to the `request.pos`.
+            limited: True if their are more events than fit between the given position and now.
+                Sync again to get more.
             num_live: The number of timeline events which have just occurred and are not historical.
                 The last N events are 'live' and should be treated as such. This is mostly
                 useful to determine whether a given @mention event should make a noise or not.
@@ -208,24 +200,18 @@ class SlidingSyncResult:
                 flag set. (same as sync v2)
         """
 
-        @attr.s(slots=True, frozen=True, auto_attribs=True)
-        class StrippedHero:
-            user_id: str
-            display_name: Optional[str]
-            avatar_url: Optional[str]
-
         name: Optional[str]
         avatar: Optional[str]
-        heroes: Optional[List[StrippedHero]]
+        heroes: Optional[List[EventBase]]
         is_dm: bool
         initial: bool
-        # Should be empty for invite/knock rooms with `stripped_state`
-        required_state: List[EventBase]
-        # Should be empty for invite/knock rooms with `stripped_state`
-        timeline_events: List[EventBase]
+        # Only optional because it won't be included for invite/knock rooms with `stripped_state`
+        required_state: Optional[List[EventBase]]
+        # Only optional because it won't be included for invite/knock rooms with `stripped_state`
+        timeline_events: Optional[List[EventBase]]
         bundled_aggregations: Optional[Dict[str, "BundledAggregations"]]
         # Optional because it's only relevant to invite/knock rooms
-        stripped_state: List[JsonDict]
+        stripped_state: Optional[List[JsonDict]]
         # Only optional because it won't be included for invite/knock rooms with `stripped_state`
         prev_batch: Optional[StreamToken]
         # Only optional because it won't be included for invite/knock rooms with `stripped_state`
@@ -238,17 +224,6 @@ class SlidingSyncResult:
         notification_count: int
         highlight_count: int
 
-        def __bool__(self) -> bool:
-            return (
-                # If this is the first time the client is seeing the room, we should not filter it out
-                # under any circumstance.
-                self.initial
-                # We need to let the client know if there are any new events
-                or bool(self.required_state)
-                or bool(self.timeline_events)
-                or bool(self.stripped_state)
-            )
-
     @attr.s(slots=True, frozen=True, auto_attribs=True)
     class SlidingWindowList:
         """
@@ -277,155 +252,24 @@ class SlidingSyncResult:
         count: int
         ops: List[Operation]
 
-    @attr.s(slots=True, frozen=True, auto_attribs=True)
-    class Extensions:
-        """Responses for extensions
-
-        Attributes:
-            to_device: The to-device extension (MSC3885)
-            e2ee: The E2EE device extension (MSC3884)
-        """
-
-        @attr.s(slots=True, frozen=True, auto_attribs=True)
-        class ToDeviceExtension:
-            """The to-device extension (MSC3885)
-
-            Attributes:
-                next_batch: The to-device stream token the client should use
-                    to get more results
-                events: A list of to-device messages for the client
-            """
-
-            next_batch: str
-            events: Sequence[JsonMapping]
-
-            def __bool__(self) -> bool:
-                return bool(self.events)
-
-        @attr.s(slots=True, frozen=True, auto_attribs=True)
-        class E2eeExtension:
-            """The E2EE device extension (MSC3884)
-
-            Attributes:
-                device_list_updates: List of user_ids whose devices have changed or left (only
-                    present on incremental syncs).
-                device_one_time_keys_count: Map from key algorithm to the number of
-                    unclaimed one-time keys currently held on the server for this device. If
-                    an algorithm is unlisted, the count for that algorithm is assumed to be
-                    zero. If this entire parameter is missing, the count for all algorithms
-                    is assumed to be zero.
-                device_unused_fallback_key_types: List of unused fallback key algorithms
-                    for this device.
-            """
-
-            # Only present on incremental syncs
-            device_list_updates: Optional[DeviceListUpdates]
-            device_one_time_keys_count: Mapping[str, int]
-            device_unused_fallback_key_types: Sequence[str]
-
-            def __bool__(self) -> bool:
-                # Note that "signed_curve25519" is always returned in key count responses
-                # regardless of whether we uploaded any keys for it. This is necessary until
-                # https://github.com/matrix-org/matrix-doc/issues/3298 is fixed.
-                #
-                # Also related:
-                # https://github.com/element-hq/element-android/issues/3725 and
-                # https://github.com/matrix-org/synapse/issues/10456
-                default_otk = self.device_one_time_keys_count.get("signed_curve25519")
-                more_than_default_otk = len(self.device_one_time_keys_count) > 1 or (
-                    default_otk is not None and default_otk > 0
-                )
-
-                return bool(
-                    more_than_default_otk
-                    or self.device_list_updates
-                    or self.device_unused_fallback_key_types
-                )
-
-        @attr.s(slots=True, frozen=True, auto_attribs=True)
-        class AccountDataExtension:
-            """The Account Data extension (MSC3959)
-
-            Attributes:
-                global_account_data_map: Mapping from `type` to `content` of global account
-                    data events.
-                account_data_by_room_map: Mapping from room_id to mapping of `type` to
-                    `content` of room account data events.
-            """
-
-            global_account_data_map: Mapping[str, JsonMapping]
-            account_data_by_room_map: Mapping[str, Mapping[str, JsonMapping]]
-
-            def __bool__(self) -> bool:
-                return bool(
-                    self.global_account_data_map or self.account_data_by_room_map
-                )
-
-        @attr.s(slots=True, frozen=True, auto_attribs=True)
-        class ReceiptsExtension:
-            """The Receipts extension (MSC3960)
-
-            Attributes:
-                room_id_to_receipt_map: Mapping from room_id to `m.receipt` ephemeral
-                    event (type, content)
-            """
-
-            room_id_to_receipt_map: Mapping[str, JsonMapping]
-
-            def __bool__(self) -> bool:
-                return bool(self.room_id_to_receipt_map)
-
-        @attr.s(slots=True, frozen=True, auto_attribs=True)
-        class TypingExtension:
-            """The Typing Notification extension (MSC3961)
-
-            Attributes:
-                room_id_to_typing_map: Mapping from room_id to `m.typing` ephemeral
-                    event (type, content)
-            """
-
-            room_id_to_typing_map: Mapping[str, JsonMapping]
-
-            def __bool__(self) -> bool:
-                return bool(self.room_id_to_typing_map)
-
-        to_device: Optional[ToDeviceExtension] = None
-        e2ee: Optional[E2eeExtension] = None
-        account_data: Optional[AccountDataExtension] = None
-        receipts: Optional[ReceiptsExtension] = None
-        typing: Optional[TypingExtension] = None
-
-        def __bool__(self) -> bool:
-            return bool(
-                self.to_device
-                or self.e2ee
-                or self.account_data
-                or self.receipts
-                or self.typing
-            )
-
-    next_pos: SlidingSyncStreamToken
+    next_pos: StreamToken
     lists: Dict[str, SlidingWindowList]
     rooms: Dict[str, RoomResult]
-    extensions: Extensions
+    extensions: JsonMapping
 
     def __bool__(self) -> bool:
         """Make the result appear empty if there are no updates. This is used
         to tell if the notifier needs to wait for more events when polling for
         events.
         """
-        # We don't include `self.lists` here, as a) `lists` is always non-empty even if
-        # there are no changes, and b) since we're sorting rooms by `stream_ordering` of
-        # the latest activity, anything that would cause the order to change would end
-        # up in `self.rooms` and cause us to send down the change.
-        return bool(self.rooms or self.extensions)
+        return bool(self.lists or self.rooms or self.extensions)
 
     @staticmethod
-    def empty(next_pos: SlidingSyncStreamToken) -> "SlidingSyncResult":
+    def empty(next_pos: StreamToken) -> "SlidingSyncResult":
         "Return a new empty result"
         return SlidingSyncResult(
             next_pos=next_pos,
             lists={},
             rooms={},
-            extensions=SlidingSyncResult.Extensions(),
+            extensions={},
         )

synapse/types/rest/client/__init__.py

@@ -120,9 +120,6 @@ class SlidingSyncBody(RequestBodyModel):
     Sliding Sync API request body.
 
     Attributes:
-        conn_id: An optional string to identify this connection to the server.
-            Only one sliding sync connection is allowed per given conn_id (empty
-            or not).
         lists: Sliding window API. A map of list key to list information
             (:class:`SlidingSyncList`). Max lists: 100. The list keys should be
             arbitrary strings which the client is using to refer to the list. Keep this
@@ -203,6 +200,9 @@ class SlidingSyncBody(RequestBodyModel):
                     }
 
             timeline_limit: The maximum number of timeline events to return per response.
+            include_heroes: Return a stripped variant of membership events (containing
+                `user_id` and optionally `avatar_url` and `displayname`) for the users used
+                to calculate the room name.
             filters: Filters to apply to the list before sorting.
         """
 
@@ -270,119 +270,16 @@ class SlidingSyncBody(RequestBodyModel):
         else:
             ranges: Optional[List[Tuple[conint(ge=0, strict=True), conint(ge=0, strict=True)]]] = None  # type: ignore[valid-type]
         slow_get_all_rooms: Optional[StrictBool] = False
+        include_heroes: Optional[StrictBool] = False
         filters: Optional[Filters] = None
 
     class RoomSubscription(CommonRoomParameters):
         pass
 
-    class Extensions(RequestBodyModel):
-        """The extensions section of the request.
-
-        Extensions MUST have an `enabled` flag which defaults to `false`. If a client
-        sends an unknown extension name, the server MUST ignore it (or else backwards
-        compatibility between clients and servers is broken when a newer client tries to
-        communicate with an older server).
-        """
-
-        class ToDeviceExtension(RequestBodyModel):
-            """The to-device extension (MSC3885)
-
-            Attributes:
-                enabled
-                limit: Maximum number of to-device messages to return
-                since: The `next_batch` from the previous sync response
-            """
-
-            enabled: Optional[StrictBool] = False
-            limit: StrictInt = 100
-            since: Optional[StrictStr] = None
-
-            @validator("since")
-            def since_token_check(
-                cls, value: Optional[StrictStr]
-            ) -> Optional[StrictStr]:
-                # `since` comes in as an opaque string token but we know that it's just
-                # an integer representing the position in the device inbox stream. We
-                # want to pre-validate it to make sure it works fine in downstream code.
-                if value is None:
-                    return value
-
-                try:
-                    int(value)
-                except ValueError:
-                    raise ValueError(
-                        "'extensions.to_device.since' is invalid (should look like an int)"
-                    )
-
-                return value
-
-        class E2eeExtension(RequestBodyModel):
-            """The E2EE device extension (MSC3884)
-
-            Attributes:
-                enabled
-            """
-
-            enabled: Optional[StrictBool] = False
-
-        class AccountDataExtension(RequestBodyModel):
-            """The Account Data extension (MSC3959)
-
-            Attributes:
-                enabled
-                lists: List of list keys (from the Sliding Window API) to apply this
-                    extension to.
-                rooms: List of room IDs (from the Room Subscription API) to apply this
-                    extension to.
-            """
-
-            enabled: Optional[StrictBool] = False
-            # Process all lists defined in the Sliding Window API. (This is the default.)
-            lists: Optional[List[StrictStr]] = ["*"]
-            # Process all room subscriptions defined in the Room Subscription API. (This is the default.)
-            rooms: Optional[List[StrictStr]] = ["*"]
-
-        class ReceiptsExtension(RequestBodyModel):
-            """The Receipts extension (MSC3960)
-
-            Attributes:
-                enabled
-                lists: List of list keys (from the Sliding Window API) to apply this
-                    extension to.
-                rooms: List of room IDs (from the Room Subscription API) to apply this
-                    extension to.
-            """
-
-            enabled: Optional[StrictBool] = False
-            # Process all lists defined in the Sliding Window API. (This is the default.)
-            lists: Optional[List[StrictStr]] = ["*"]
-            # Process all room subscriptions defined in the Room Subscription API. (This is the default.)
-            rooms: Optional[List[StrictStr]] = ["*"]
-
-        class TypingExtension(RequestBodyModel):
-            """The Typing Notification extension (MSC3961)
-
-            Attributes:
-                enabled
-                lists: List of list keys (from the Sliding Window API) to apply this
-                    extension to.
-                rooms: List of room IDs (from the Room Subscription API) to apply this
-                    extension to.
-            """
-
-            enabled: Optional[StrictBool] = False
-            # Process all lists defined in the Sliding Window API. (This is the default.)
-            lists: Optional[List[StrictStr]] = ["*"]
-            # Process all room subscriptions defined in the Room Subscription API. (This is the default.)
-            rooms: Optional[List[StrictStr]] = ["*"]
-
-        to_device: Optional[ToDeviceExtension] = None
-        e2ee: Optional[E2eeExtension] = None
-        account_data: Optional[AccountDataExtension] = None
-        receipts: Optional[ReceiptsExtension] = None
-        typing: Optional[TypingExtension] = None
-
-    conn_id: Optional[str]
+    class Extension(RequestBodyModel):
+        enabled: Optional[StrictBool] = False
+        lists: Optional[List[StrictStr]] = None
+        rooms: Optional[List[StrictStr]] = None
 
     # mypy workaround via https://github.com/pydantic/pydantic/issues/156#issuecomment-1130883884
     if TYPE_CHECKING:
@@ -390,7 +287,7 @@ class SlidingSyncBody(RequestBodyModel):
     else:
         lists: Optional[Dict[constr(max_length=64, strict=True), SlidingSyncList]] = None  # type: ignore[valid-type]
     room_subscriptions: Optional[Dict[StrictStr, RoomSubscription]] = None
-    extensions: Optional[Extensions] = None
+    extensions: Optional[Dict[StrictStr, Extension]] = None
 
     @validator("lists")
     def lists_length_check(

synapse/util/caches/stream_change_cache.py

@@ -327,7 +327,7 @@ class StreamChangeCache:
             for entity in r:
                 self._entity_to_key.pop(entity, None)
 
-    def get_max_pos_of_last_change(self, entity: EntityType) -> Optional[int]:
+    def get_max_pos_of_last_change(self, entity: EntityType) -> int:
         """Returns an upper bound of the stream id of the last change to an
         entity.
 
@@ -335,11 +335,7 @@ class StreamChangeCache:
             entity: The entity to check.
 
         Return:
-            The stream position of the latest change for the given entity, if
-            known
+            The stream position of the latest change for the given entity or
+            the earliest known stream position if the entitiy is unknown.
         """
-        return self._entity_to_key.get(entity)
-
-    def get_earliest_known_position(self) -> int:
-        """Returns the earliest position in the cache."""
-        return self._earliest_known_stream_pos
+        return self._entity_to_key.get(entity, self._earliest_known_stream_pos)

tests/handlers/test_e2e_keys.py

@@ -43,7 +43,9 @@ from tests.unittest import override_config
 class E2eKeysHandlerTestCase(unittest.HomeserverTestCase):
     def make_homeserver(self, reactor: MemoryReactor, clock: Clock) -> HomeServer:
         self.appservice_api = mock.AsyncMock()
-        return self.setup_test_homeserver(application_service_api=self.appservice_api)
+        return self.setup_test_homeserver(
+            federation_client=mock.Mock(), application_service_api=self.appservice_api
+        )
 
     def prepare(self, reactor: MemoryReactor, clock: Clock, hs: HomeServer) -> None:
         self.handler = hs.get_e2e_keys_handler()
@@ -1222,61 +1224,6 @@ class E2eKeysHandlerTestCase(unittest.HomeserverTestCase):
             },
         )
 
-    def test_query_devices_remote_down(self) -> None:
-        """Tests that querying keys for a remote user on an unreachable server returns
-        results in the "failures" property
-        """
-
-        remote_user_id = "@test:other"
-        local_user_id = "@test:test"
-
-        # The backoff code treats time zero as special
-        self.reactor.advance(5)
-
-        self.hs.get_federation_http_client().agent.request = mock.AsyncMock(  # type: ignore[method-assign]
-            side_effect=Exception("boop")
-        )
-
-        e2e_handler = self.hs.get_e2e_keys_handler()
-
-        query_result = self.get_success(
-            e2e_handler.query_devices(
-                {
-                    "device_keys": {remote_user_id: []},
-                },
-                timeout=10,
-                from_user_id=local_user_id,
-                from_device_id="some_device_id",
-            )
-        )
-
-        self.assertEqual(
-            query_result["failures"],
-            {
-                "other": {
-                    "message": "Failed to send request: Exception: boop",
-                    "status": 503,
-                }
-            },
-        )
-
-        # Do it again: we should hit the backoff
-        query_result = self.get_success(
-            e2e_handler.query_devices(
-                {
-                    "device_keys": {remote_user_id: []},
-                },
-                timeout=10,
-                from_user_id=local_user_id,
-                from_device_id="some_device_id",
-            )
-        )
-
-        self.assertEqual(
-            query_result["failures"],
-            {"other": {"message": "Not ready for retry", "status": 503}},
-        )
-
     @parameterized.expand(
         [
             # The remote homeserver's response indicates that this user has 0/1/2 devices.

tests/handlers/test_sync.py

@@ -211,7 +211,6 @@ class SyncTestCase(tests.unittest.HomeserverTestCase):
 
         # Blow away caches (supported room versions can only change due to a restart).
         self.store.get_rooms_for_user.invalidate_all()
-        self.store._get_rooms_for_local_user_where_membership_is_inner.invalidate_all()
         self.store._get_event_cache.clear()
         self.store._event_ref.clear()
 

tests/media/test_media_storage.py

@@ -1057,15 +1057,13 @@ class RemoteDownloadLimiterTestCase(unittest.HomeserverTestCase):
         )
         assert channel.code == 200
 
-    @override_config({"remote_media_download_burst_count": "87M"})
     @patch(
         "synapse.http.matrixfederationclient.read_body_with_max_size",
         read_body_with_max_size_30MiB,
     )
-    def test_download_ratelimit_unknown_length(self) -> None:
+    def test_download_ratelimit_max_size_sub(self) -> None:
         """
-        Test that if no content-length is provided, ratelimit will still be applied after
-        download once length is known
+        Test that if no content-length is provided, the default max size is applied instead
         """
 
         # mock out actually sending the request
@@ -1079,48 +1077,19 @@ class RemoteDownloadLimiterTestCase(unittest.HomeserverTestCase):
 
         self.client._send_request = _send_request  # type: ignore
 
-        # 3 requests should go through (note 3rd one would technically violate ratelimit but
-        # is applied *after* download - the next one will be ratelimited)
-        for i in range(3):
-            channel = self.make_request(
+        # ten requests should go through using the max size (500MB/50MB)
+        for i in range(10):
+            channel2 = self.make_request(
                 "GET",
                 f"/_matrix/media/v3/download/remote.org/abcdefghijklmnopqrstuvwxy{i}",
                 shorthand=False,
             )
-            assert channel.code == 200
+            assert channel2.code == 200
 
-        # 4th will hit ratelimit
-        channel2 = self.make_request(
+        # eleventh will hit ratelimit
+        channel3 = self.make_request(
             "GET",
             "/_matrix/media/v3/download/remote.org/abcdefghijklmnopqrstuvwxyx",
             shorthand=False,
         )
-        assert channel2.code == 429
-
-    @override_config({"max_upload_size": "29M"})
-    @patch(
-        "synapse.http.matrixfederationclient.read_body_with_max_size",
-        read_body_with_max_size_30MiB,
-    )
-    def test_max_download_respected(self) -> None:
-        """
-        Test that the max download size is enforced - note that max download size is determined
-        by the max_upload_size
-        """
-
-        # mock out actually sending the request
-        async def _send_request(*args: Any, **kwargs: Any) -> IResponse:
-            resp = MagicMock(spec=IResponse)
-            resp.code = 200
-            resp.length = 31457280
-            resp.headers = Headers({"Content-Type": ["application/octet-stream"]})
-            resp.phrase = b"OK"
-            return resp
-
-        self.client._send_request = _send_request  # type: ignore
-
-        channel = self.make_request(
-            "GET", "/_matrix/media/v3/download/remote.org/abcd", shorthand=False
-        )
-        assert channel.code == 502
-        assert channel.json_body["errcode"] == "M_TOO_LARGE"
+        assert channel3.code == 429

tests/rest/client/sliding_sync/__init__.py

@@ -1,13 +0,0 @@
-#
-# This file is licensed under the Affero General Public License (AGPL) version 3.
-#
-# Copyright (C) 2024 New Vector, Ltd
-#
-# This program is free software: you can redistribute it and/or modify
-# it under the terms of the GNU Affero General Public License as
-# published by the Free Software Foundation, either version 3 of the
-# License, or (at your option) any later version.
-#
-# See the GNU Affero General Public License for more details:
-# <https://www.gnu.org/licenses/agpl-3.0.html>.
-#

tests/rest/client/sliding_sync/test_connection_tracking.py

@@ -1,381 +0,0 @@
-#
-# This file is licensed under the Affero General Public License (AGPL) version 3.
-#
-# Copyright (C) 2024 New Vector, Ltd
-#
-# This program is free software: you can redistribute it and/or modify
-# it under the terms of the GNU Affero General Public License as
-# published by the Free Software Foundation, either version 3 of the
-# License, or (at your option) any later version.
-#
-# See the GNU Affero General Public License for more details:
-# <https://www.gnu.org/licenses/agpl-3.0.html>.
-#
-import logging
-
-from parameterized import parameterized
-
-from twisted.test.proto_helpers import MemoryReactor
-
-import synapse.rest.admin
-from synapse.api.constants import EventTypes
-from synapse.rest.client import login, room, sync
-from synapse.server import HomeServer
-from synapse.util import Clock
-
-from tests.rest.client.sliding_sync.test_sliding_sync import SlidingSyncBase
-
-logger = logging.getLogger(__name__)
-
-
-class SlidingSyncConnectionTrackingTestCase(SlidingSyncBase):
-    """
-    Test connection tracking in the Sliding Sync API.
-    """
-
-    servlets = [
-        synapse.rest.admin.register_servlets,
-        login.register_servlets,
-        room.register_servlets,
-        sync.register_servlets,
-    ]
-
-    def prepare(self, reactor: MemoryReactor, clock: Clock, hs: HomeServer) -> None:
-        self.store = hs.get_datastores().main
-        self.storage_controllers = hs.get_storage_controllers()
-
-    def test_rooms_required_state_incremental_sync_LIVE(self) -> None:
-        """Test that we only get state updates in incremental sync for rooms
-        we've already seen (LIVE).
-        """
-
-        user1_id = self.register_user("user1", "pass")
-        user1_tok = self.login(user1_id, "pass")
-        user2_id = self.register_user("user2", "pass")
-        user2_tok = self.login(user2_id, "pass")
-
-        room_id1 = self.helper.create_room_as(user2_id, tok=user2_tok)
-        self.helper.join(room_id1, user1_id, tok=user1_tok)
-
-        # Make the Sliding Sync request
-        sync_body = {
-            "lists": {
-                "foo-list": {
-                    "ranges": [[0, 1]],
-                    "required_state": [
-                        [EventTypes.Create, ""],
-                        [EventTypes.RoomHistoryVisibility, ""],
-                        # This one doesn't exist in the room
-                        [EventTypes.Name, ""],
-                    ],
-                    "timeline_limit": 0,
-                }
-            }
-        }
-
-        response_body, from_token = self.do_sync(sync_body, tok=user1_tok)
-
-        state_map = self.get_success(
-            self.storage_controllers.state.get_current_state(room_id1)
-        )
-
-        self._assertRequiredStateIncludes(
-            response_body["rooms"][room_id1]["required_state"],
-            {
-                state_map[(EventTypes.Create, "")],
-                state_map[(EventTypes.RoomHistoryVisibility, "")],
-            },
-            exact=True,
-        )
-
-        # Send a state event
-        self.helper.send_state(
-            room_id1, EventTypes.Name, body={"name": "foo"}, tok=user2_tok
-        )
-
-        response_body, _ = self.do_sync(sync_body, since=from_token, tok=user1_tok)
-
-        state_map = self.get_success(
-            self.storage_controllers.state.get_current_state(room_id1)
-        )
-
-        self.assertNotIn("initial", response_body["rooms"][room_id1])
-        self._assertRequiredStateIncludes(
-            response_body["rooms"][room_id1]["required_state"],
-            {
-                state_map[(EventTypes.Name, "")],
-            },
-            exact=True,
-        )
-
-    @parameterized.expand([(False,), (True,)])
-    def test_rooms_timeline_incremental_sync_PREVIOUSLY(self, limited: bool) -> None:
-        """
-        Test getting room data where we have previously sent down the room, but
-        we missed sending down some timeline events previously and so its status
-        is considered PREVIOUSLY.
-
-        There are two versions of this test, one where there are more messages
-        than the timeline limit, and one where there isn't.
-        """
-
-        user1_id = self.register_user("user1", "pass")
-        user1_tok = self.login(user1_id, "pass")
-
-        room_id1 = self.helper.create_room_as(user1_id, tok=user1_tok)
-        room_id2 = self.helper.create_room_as(user1_id, tok=user1_tok)
-
-        self.helper.send(room_id1, "msg", tok=user1_tok)
-
-        timeline_limit = 5
-        sync_body = {
-            "lists": {
-                "foo-list": {
-                    "ranges": [[0, 0]],
-                    "required_state": [],
-                    "timeline_limit": timeline_limit,
-                }
-            },
-            "conn_id": "conn_id",
-        }
-
-        # The first room gets sent down the initial sync
-        response_body, initial_from_token = self.do_sync(sync_body, tok=user1_tok)
-        self.assertCountEqual(
-            response_body["rooms"].keys(), {room_id1}, response_body["rooms"]
-        )
-
-        # We now send down some events in room1 (depending on the test param).
-        expected_events = []  # The set of events in the timeline
-        if limited:
-            for _ in range(10):
-                resp = self.helper.send(room_id1, "msg1", tok=user1_tok)
-                expected_events.append(resp["event_id"])
-        else:
-            resp = self.helper.send(room_id1, "msg1", tok=user1_tok)
-            expected_events.append(resp["event_id"])
-
-        # A second messages happens in the other room, so room1 won't get sent down.
-        self.helper.send(room_id2, "msg", tok=user1_tok)
-
-        # Only the second room gets sent down sync.
-        response_body, from_token = self.do_sync(
-            sync_body, since=initial_from_token, tok=user1_tok
-        )
-
-        self.assertCountEqual(
-            response_body["rooms"].keys(), {room_id2}, response_body["rooms"]
-        )
-
-        # We now send another event to room1, so we should sync all the missing events.
-        resp = self.helper.send(room_id1, "msg2", tok=user1_tok)
-        expected_events.append(resp["event_id"])
-
-        # This sync should contain the messages from room1 not yet sent down.
-        response_body, _ = self.do_sync(sync_body, since=from_token, tok=user1_tok)
-
-        self.assertCountEqual(
-            response_body["rooms"].keys(), {room_id1}, response_body["rooms"]
-        )
-        self.assertNotIn("initial", response_body["rooms"][room_id1])
-
-        self.assertEqual(
-            [ev["event_id"] for ev in response_body["rooms"][room_id1]["timeline"]],
-            expected_events[-timeline_limit:],
-        )
-        self.assertEqual(response_body["rooms"][room_id1]["limited"], limited)
-        self.assertEqual(response_body["rooms"][room_id1].get("required_state"), None)
-
-    def test_rooms_required_state_incremental_sync_PREVIOUSLY(self) -> None:
-        """
-        Test getting room data where we have previously sent down the room, but
-        we missed sending down some state previously and so its status is
-        considered PREVIOUSLY.
-        """
-
-        user1_id = self.register_user("user1", "pass")
-        user1_tok = self.login(user1_id, "pass")
-
-        room_id1 = self.helper.create_room_as(user1_id, tok=user1_tok)
-        room_id2 = self.helper.create_room_as(user1_id, tok=user1_tok)
-
-        self.helper.send(room_id1, "msg", tok=user1_tok)
-
-        sync_body = {
-            "lists": {
-                "foo-list": {
-                    "ranges": [[0, 0]],
-                    "required_state": [
-                        [EventTypes.Create, ""],
-                        [EventTypes.RoomHistoryVisibility, ""],
-                        # This one doesn't exist in the room
-                        [EventTypes.Name, ""],
-                    ],
-                    "timeline_limit": 0,
-                }
-            },
-            "conn_id": "conn_id",
-        }
-
-        # The first room gets sent down the initial sync
-        response_body, initial_from_token = self.do_sync(sync_body, tok=user1_tok)
-        self.assertCountEqual(
-            response_body["rooms"].keys(), {room_id1}, response_body["rooms"]
-        )
-
-        # We now send down some state in room1
-        resp = self.helper.send_state(
-            room_id1, EventTypes.Name, {"name": "foo"}, tok=user1_tok
-        )
-        name_change_id = resp["event_id"]
-
-        # A second messages happens in the other room, so room1 won't get sent down.
-        self.helper.send(room_id2, "msg", tok=user1_tok)
-
-        # Only the second room gets sent down sync.
-        response_body, from_token = self.do_sync(
-            sync_body, since=initial_from_token, tok=user1_tok
-        )
-
-        self.assertCountEqual(
-            response_body["rooms"].keys(), {room_id2}, response_body["rooms"]
-        )
-
-        # We now send another event to room1, so we should sync all the missing state.
-        self.helper.send(room_id1, "msg", tok=user1_tok)
-
-        # This sync should contain the state changes from room1.
-        response_body, _ = self.do_sync(sync_body, since=from_token, tok=user1_tok)
-
-        self.assertCountEqual(
-            response_body["rooms"].keys(), {room_id1}, response_body["rooms"]
-        )
-        self.assertNotIn("initial", response_body["rooms"][room_id1])
-
-        # We should only see the name change.
-        self.assertEqual(
-            [
-                ev["event_id"]
-                for ev in response_body["rooms"][room_id1]["required_state"]
-            ],
-            [name_change_id],
-        )
-
-    def test_rooms_required_state_incremental_sync_NEVER(self) -> None:
-        """
-        Test getting `required_state` where we have NEVER sent down the room before
-        """
-
-        user1_id = self.register_user("user1", "pass")
-        user1_tok = self.login(user1_id, "pass")
-
-        room_id1 = self.helper.create_room_as(user1_id, tok=user1_tok)
-        room_id2 = self.helper.create_room_as(user1_id, tok=user1_tok)
-
-        self.helper.send(room_id1, "msg", tok=user1_tok)
-
-        sync_body = {
-            "lists": {
-                "foo-list": {
-                    "ranges": [[0, 0]],
-                    "required_state": [
-                        [EventTypes.Create, ""],
-                        [EventTypes.RoomHistoryVisibility, ""],
-                        # This one doesn't exist in the room
-                        [EventTypes.Name, ""],
-                    ],
-                    "timeline_limit": 1,
-                }
-            },
-        }
-
-        # A message happens in the other room, so room1 won't get sent down.
-        self.helper.send(room_id2, "msg", tok=user1_tok)
-
-        # Only the second room gets sent down sync.
-        response_body, from_token = self.do_sync(sync_body, tok=user1_tok)
-
-        self.assertCountEqual(
-            response_body["rooms"].keys(), {room_id2}, response_body["rooms"]
-        )
-
-        # We now send another event to room1, so we should send down the full
-        # room.
-        self.helper.send(room_id1, "msg2", tok=user1_tok)
-
-        # This sync should contain the messages from room1 not yet sent down.
-        response_body, _ = self.do_sync(sync_body, since=from_token, tok=user1_tok)
-
-        self.assertCountEqual(
-            response_body["rooms"].keys(), {room_id1}, response_body["rooms"]
-        )
-
-        self.assertEqual(response_body["rooms"][room_id1]["initial"], True)
-
-        state_map = self.get_success(
-            self.storage_controllers.state.get_current_state(room_id1)
-        )
-
-        self._assertRequiredStateIncludes(
-            response_body["rooms"][room_id1]["required_state"],
-            {
-                state_map[(EventTypes.Create, "")],
-                state_map[(EventTypes.RoomHistoryVisibility, "")],
-            },
-            exact=True,
-        )
-
-    def test_rooms_timeline_incremental_sync_NEVER(self) -> None:
-        """
-        Test getting timeline room data where we have NEVER sent down the room
-        before
-        """
-
-        user1_id = self.register_user("user1", "pass")
-        user1_tok = self.login(user1_id, "pass")
-
-        room_id1 = self.helper.create_room_as(user1_id, tok=user1_tok)
-        room_id2 = self.helper.create_room_as(user1_id, tok=user1_tok)
-
-        sync_body = {
-            "lists": {
-                "foo-list": {
-                    "ranges": [[0, 0]],
-                    "required_state": [],
-                    "timeline_limit": 5,
-                }
-            },
-        }
-
-        expected_events = []
-        for _ in range(4):
-            resp = self.helper.send(room_id1, "msg", tok=user1_tok)
-            expected_events.append(resp["event_id"])
-
-        # A message happens in the other room, so room1 won't get sent down.
-        self.helper.send(room_id2, "msg", tok=user1_tok)
-
-        # Only the second room gets sent down sync.
-        response_body, from_token = self.do_sync(sync_body, tok=user1_tok)
-
-        self.assertCountEqual(
-            response_body["rooms"].keys(), {room_id2}, response_body["rooms"]
-        )
-
-        # We now send another event to room1 so it comes down sync
-        resp = self.helper.send(room_id1, "msg2", tok=user1_tok)
-        expected_events.append(resp["event_id"])
-
-        # This sync should contain the messages from room1 not yet sent down.
-        response_body, _ = self.do_sync(sync_body, since=from_token, tok=user1_tok)
-
-        self.assertCountEqual(
-            response_body["rooms"].keys(), {room_id1}, response_body["rooms"]
-        )
-
-        self.assertEqual(
-            [ev["event_id"] for ev in response_body["rooms"][room_id1]["timeline"]],
-            expected_events,
-        )
-        self.assertEqual(response_body["rooms"][room_id1]["limited"], True)
-        self.assertEqual(response_body["rooms"][room_id1]["initial"], True)

tests/rest/client/sliding_sync/test_extension_account_data.py

@@ -1,495 +0,0 @@
-#
-# This file is licensed under the Affero General Public License (AGPL) version 3.
-#
-# Copyright (C) 2024 New Vector, Ltd
-#
-# This program is free software: you can redistribute it and/or modify
-# it under the terms of the GNU Affero General Public License as
-# published by the Free Software Foundation, either version 3 of the
-# License, or (at your option) any later version.
-#
-# See the GNU Affero General Public License for more details:
-# <https://www.gnu.org/licenses/agpl-3.0.html>.
-#
-import logging
-
-from twisted.test.proto_helpers import MemoryReactor
-
-import synapse.rest.admin
-from synapse.api.constants import AccountDataTypes
-from synapse.rest.client import login, room, sendtodevice, sync
-from synapse.server import HomeServer
-from synapse.types import StreamKeyType
-from synapse.util import Clock
-
-from tests.rest.client.sliding_sync.test_sliding_sync import SlidingSyncBase
-from tests.server import TimedOutException
-
-logger = logging.getLogger(__name__)
-
-
-class SlidingSyncAccountDataExtensionTestCase(SlidingSyncBase):
-    """Tests for the account_data sliding sync extension"""
-
-    servlets = [
-        synapse.rest.admin.register_servlets,
-        login.register_servlets,
-        room.register_servlets,
-        sync.register_servlets,
-        sendtodevice.register_servlets,
-    ]
-
-    def prepare(self, reactor: MemoryReactor, clock: Clock, hs: HomeServer) -> None:
-        self.store = hs.get_datastores().main
-        self.account_data_handler = hs.get_account_data_handler()
-
-    def test_no_data_initial_sync(self) -> None:
-        """
-        Test that enabling the account_data extension works during an intitial sync,
-        even if there is no-data.
-        """
-        user1_id = self.register_user("user1", "pass")
-        user1_tok = self.login(user1_id, "pass")
-
-        # Make an initial Sliding Sync request with the account_data extension enabled
-        sync_body = {
-            "lists": {},
-            "extensions": {
-                "account_data": {
-                    "enabled": True,
-                }
-            },
-        }
-        response_body, _ = self.do_sync(sync_body, tok=user1_tok)
-
-        self.assertIncludes(
-            {
-                global_event["type"]
-                for global_event in response_body["extensions"]["account_data"].get(
-                    "global"
-                )
-            },
-            # Even though we don't have any global account data set, Synapse saves some
-            # default push rules for us.
-            {AccountDataTypes.PUSH_RULES},
-            exact=True,
-        )
-        self.assertIncludes(
-            response_body["extensions"]["account_data"].get("rooms").keys(),
-            set(),
-            exact=True,
-        )
-
-    def test_no_data_incremental_sync(self) -> None:
-        """
-        Test that enabling account_data extension works during an incremental sync, even
-        if there is no-data.
-        """
-        user1_id = self.register_user("user1", "pass")
-        user1_tok = self.login(user1_id, "pass")
-
-        sync_body = {
-            "lists": {},
-            "extensions": {
-                "account_data": {
-                    "enabled": True,
-                }
-            },
-        }
-        _, from_token = self.do_sync(sync_body, tok=user1_tok)
-
-        # Make an incremental Sliding Sync request with the account_data extension enabled
-        response_body, _ = self.do_sync(sync_body, since=from_token, tok=user1_tok)
-
-        # There has been no account data changes since the `from_token` so we shouldn't
-        # see any account data here.
-        self.assertIncludes(
-            {
-                global_event["type"]
-                for global_event in response_body["extensions"]["account_data"].get(
-                    "global"
-                )
-            },
-            set(),
-            exact=True,
-        )
-        self.assertIncludes(
-            response_body["extensions"]["account_data"].get("rooms").keys(),
-            set(),
-            exact=True,
-        )
-
-    def test_global_account_data_initial_sync(self) -> None:
-        """
-        On initial sync, we should return all global account data on initial sync.
-        """
-        user1_id = self.register_user("user1", "pass")
-        user1_tok = self.login(user1_id, "pass")
-
-        # Update the global account data
-        self.get_success(
-            self.account_data_handler.add_account_data_for_user(
-                user_id=user1_id,
-                account_data_type="org.matrix.foobarbaz",
-                content={"foo": "bar"},
-            )
-        )
-
-        # Make an initial Sliding Sync request with the account_data extension enabled
-        sync_body = {
-            "lists": {},
-            "extensions": {
-                "account_data": {
-                    "enabled": True,
-                }
-            },
-        }
-        response_body, _ = self.do_sync(sync_body, tok=user1_tok)
-
-        # It should show us all of the global account data
-        self.assertIncludes(
-            {
-                global_event["type"]
-                for global_event in response_body["extensions"]["account_data"].get(
-                    "global"
-                )
-            },
-            {AccountDataTypes.PUSH_RULES, "org.matrix.foobarbaz"},
-            exact=True,
-        )
-        self.assertIncludes(
-            response_body["extensions"]["account_data"].get("rooms").keys(),
-            set(),
-            exact=True,
-        )
-
-    def test_global_account_data_incremental_sync(self) -> None:
-        """
-        On incremental sync, we should only account data that has changed since the
-        `from_token`.
-        """
-        user1_id = self.register_user("user1", "pass")
-        user1_tok = self.login(user1_id, "pass")
-
-        # Add some global account data
-        self.get_success(
-            self.account_data_handler.add_account_data_for_user(
-                user_id=user1_id,
-                account_data_type="org.matrix.foobarbaz",
-                content={"foo": "bar"},
-            )
-        )
-
-        sync_body = {
-            "lists": {},
-            "extensions": {
-                "account_data": {
-                    "enabled": True,
-                }
-            },
-        }
-        _, from_token = self.do_sync(sync_body, tok=user1_tok)
-
-        # Add some other global account data
-        self.get_success(
-            self.account_data_handler.add_account_data_for_user(
-                user_id=user1_id,
-                account_data_type="org.matrix.doodardaz",
-                content={"doo": "dar"},
-            )
-        )
-
-        # Make an incremental Sliding Sync request with the account_data extension enabled
-        response_body, _ = self.do_sync(sync_body, since=from_token, tok=user1_tok)
-
-        self.assertIncludes(
-            {
-                global_event["type"]
-                for global_event in response_body["extensions"]["account_data"].get(
-                    "global"
-                )
-            },
-            # We should only see the new global account data that happened after the `from_token`
-            {"org.matrix.doodardaz"},
-            exact=True,
-        )
-        self.assertIncludes(
-            response_body["extensions"]["account_data"].get("rooms").keys(),
-            set(),
-            exact=True,
-        )
-
-    def test_room_account_data_initial_sync(self) -> None:
-        """
-        On initial sync, we return all account data for a given room but only for
-        rooms that we request and are being returned in the Sliding Sync response.
-        """
-        user1_id = self.register_user("user1", "pass")
-        user1_tok = self.login(user1_id, "pass")
-
-        # Create a room and add some room account data
-        room_id1 = self.helper.create_room_as(user1_id, tok=user1_tok)
-        self.get_success(
-            self.account_data_handler.add_account_data_to_room(
-                user_id=user1_id,
-                room_id=room_id1,
-                account_data_type="org.matrix.roorarraz",
-                content={"roo": "rar"},
-            )
-        )
-
-        # Create another room with some room account data
-        room_id2 = self.helper.create_room_as(user1_id, tok=user1_tok)
-        self.get_success(
-            self.account_data_handler.add_account_data_to_room(
-                user_id=user1_id,
-                room_id=room_id2,
-                account_data_type="org.matrix.roorarraz",
-                content={"roo": "rar"},
-            )
-        )
-
-        # Make an initial Sliding Sync request with the account_data extension enabled
-        sync_body = {
-            "lists": {},
-            "room_subscriptions": {
-                room_id1: {
-                    "required_state": [],
-                    "timeline_limit": 0,
-                }
-            },
-            "extensions": {
-                "account_data": {
-                    "enabled": True,
-                    "rooms": [room_id1, room_id2],
-                }
-            },
-        }
-        response_body, _ = self.do_sync(sync_body, tok=user1_tok)
-
-        self.assertIsNotNone(response_body["extensions"]["account_data"].get("global"))
-        # Even though we requested room2, we only expect room1 to show up because that's
-        # the only room in the Sliding Sync response (room2 is not one of our room
-        # subscriptions or in a sliding window list).
-        self.assertIncludes(
-            response_body["extensions"]["account_data"].get("rooms").keys(),
-            {room_id1},
-            exact=True,
-        )
-        self.assertIncludes(
-            {
-                event["type"]
-                for event in response_body["extensions"]["account_data"]
-                .get("rooms")
-                .get(room_id1)
-            },
-            {"org.matrix.roorarraz"},
-            exact=True,
-        )
-
-    def test_room_account_data_incremental_sync(self) -> None:
-        """
-        On incremental sync, we return all account data for a given room but only for
-        rooms that we request and are being returned in the Sliding Sync response.
-        """
-        user1_id = self.register_user("user1", "pass")
-        user1_tok = self.login(user1_id, "pass")
-
-        # Create a room and add some room account data
-        room_id1 = self.helper.create_room_as(user1_id, tok=user1_tok)
-        self.get_success(
-            self.account_data_handler.add_account_data_to_room(
-                user_id=user1_id,
-                room_id=room_id1,
-                account_data_type="org.matrix.roorarraz",
-                content={"roo": "rar"},
-            )
-        )
-
-        # Create another room with some room account data
-        room_id2 = self.helper.create_room_as(user1_id, tok=user1_tok)
-        self.get_success(
-            self.account_data_handler.add_account_data_to_room(
-                user_id=user1_id,
-                room_id=room_id2,
-                account_data_type="org.matrix.roorarraz",
-                content={"roo": "rar"},
-            )
-        )
-
-        sync_body = {
-            "lists": {},
-            "room_subscriptions": {
-                room_id1: {
-                    "required_state": [],
-                    "timeline_limit": 0,
-                }
-            },
-            "extensions": {
-                "account_data": {
-                    "enabled": True,
-                    "rooms": [room_id1, room_id2],
-                }
-            },
-        }
-        _, from_token = self.do_sync(sync_body, tok=user1_tok)
-
-        # Add some other room account data
-        self.get_success(
-            self.account_data_handler.add_account_data_to_room(
-                user_id=user1_id,
-                room_id=room_id1,
-                account_data_type="org.matrix.roorarraz2",
-                content={"roo": "rar"},
-            )
-        )
-        self.get_success(
-            self.account_data_handler.add_account_data_to_room(
-                user_id=user1_id,
-                room_id=room_id2,
-                account_data_type="org.matrix.roorarraz2",
-                content={"roo": "rar"},
-            )
-        )
-
-        # Make an incremental Sliding Sync request with the account_data extension enabled
-        response_body, _ = self.do_sync(sync_body, since=from_token, tok=user1_tok)
-
-        self.assertIsNotNone(response_body["extensions"]["account_data"].get("global"))
-        # Even though we requested room2, we only expect room1 to show up because that's
-        # the only room in the Sliding Sync response (room2 is not one of our room
-        # subscriptions or in a sliding window list).
-        self.assertIncludes(
-            response_body["extensions"]["account_data"].get("rooms").keys(),
-            {room_id1},
-            exact=True,
-        )
-        # We should only see the new room account data that happened after the `from_token`
-        self.assertIncludes(
-            {
-                event["type"]
-                for event in response_body["extensions"]["account_data"]
-                .get("rooms")
-                .get(room_id1)
-            },
-            {"org.matrix.roorarraz2"},
-            exact=True,
-        )
-
-    def test_wait_for_new_data(self) -> None:
-        """
-        Test to make sure that the Sliding Sync request waits for new data to arrive.
-
-        (Only applies to incremental syncs with a `timeout` specified)
-        """
-        user1_id = self.register_user("user1", "pass")
-        user1_tok = self.login(user1_id, "pass")
-        user2_id = self.register_user("user2", "pass")
-        user2_tok = self.login(user2_id, "pass")
-
-        room_id = self.helper.create_room_as(user2_id, tok=user2_tok)
-        self.helper.join(room_id, user1_id, tok=user1_tok)
-
-        sync_body = {
-            "lists": {},
-            "extensions": {
-                "account_data": {
-                    "enabled": True,
-                }
-            },
-        }
-        _, from_token = self.do_sync(sync_body, tok=user1_tok)
-
-        # Make an incremental Sliding Sync request with the account_data extension enabled
-        channel = self.make_request(
-            "POST",
-            self.sync_endpoint + f"?timeout=10000&pos={from_token}",
-            content=sync_body,
-            access_token=user1_tok,
-            await_result=False,
-        )
-        # Block for 5 seconds to make sure we are `notifier.wait_for_events(...)`
-        with self.assertRaises(TimedOutException):
-            channel.await_result(timeout_ms=5000)
-        # Bump the global account data to trigger new results
-        self.get_success(
-            self.account_data_handler.add_account_data_for_user(
-                user1_id,
-                "org.matrix.foobarbaz",
-                {"foo": "bar"},
-            )
-        )
-        # Should respond before the 10 second timeout
-        channel.await_result(timeout_ms=3000)
-        self.assertEqual(channel.code, 200, channel.json_body)
-
-        # We should see the global account data update
-        self.assertIncludes(
-            {
-                global_event["type"]
-                for global_event in channel.json_body["extensions"]["account_data"].get(
-                    "global"
-                )
-            },
-            {"org.matrix.foobarbaz"},
-            exact=True,
-        )
-        self.assertIncludes(
-            channel.json_body["extensions"]["account_data"].get("rooms").keys(),
-            set(),
-            exact=True,
-        )
-
-    def test_wait_for_new_data_timeout(self) -> None:
-        """
-        Test to make sure that the Sliding Sync request waits for new data to arrive but
-        no data ever arrives so we timeout. We're also making sure that the default data
-        from the account_data extension doesn't trigger a false-positive for new data.
-        """
-        user1_id = self.register_user("user1", "pass")
-        user1_tok = self.login(user1_id, "pass")
-
-        sync_body = {
-            "lists": {},
-            "extensions": {
-                "account_data": {
-                    "enabled": True,
-                }
-            },
-        }
-        _, from_token = self.do_sync(sync_body, tok=user1_tok)
-
-        # Make the Sliding Sync request
-        channel = self.make_request(
-            "POST",
-            self.sync_endpoint + f"?timeout=10000&pos={from_token}",
-            content=sync_body,
-            access_token=user1_tok,
-            await_result=False,
-        )
-        # Block for 5 seconds to make sure we are `notifier.wait_for_events(...)`
-        with self.assertRaises(TimedOutException):
-            channel.await_result(timeout_ms=5000)
-        # Wake-up `notifier.wait_for_events(...)` that will cause us test
-        # `SlidingSyncResult.__bool__` for new results.
-        self._bump_notifier_wait_for_events(
-            user1_id,
-            # We choose `StreamKeyType.PRESENCE` because we're testing for account data
-            # and don't want to contaminate the account data results using
-            # `StreamKeyType.ACCOUNT_DATA`.
-            wake_stream_key=StreamKeyType.PRESENCE,
-        )
-        # Block for a little bit more to ensure we don't see any new results.
-        with self.assertRaises(TimedOutException):
-            channel.await_result(timeout_ms=4000)
-        # Wait for the sync to complete (wait for the rest of the 10 second timeout,
-        # 5000 + 4000 + 1200 > 10000)
-        channel.await_result(timeout_ms=1200)
-        self.assertEqual(channel.code, 200, channel.json_body)
-
-        self.assertIsNotNone(
-            channel.json_body["extensions"]["account_data"].get("global")
-        )
-        self.assertIsNotNone(
-            channel.json_body["extensions"]["account_data"].get("rooms")
-        )

tests/rest/client/sliding_sync/test_extension_e2ee.py

@@ -1,441 +0,0 @@
-#
-# This file is licensed under the Affero General Public License (AGPL) version 3.
-#
-# Copyright (C) 2024 New Vector, Ltd
-#
-# This program is free software: you can redistribute it and/or modify
-# it under the terms of the GNU Affero General Public License as
-# published by the Free Software Foundation, either version 3 of the
-# License, or (at your option) any later version.
-#
-# See the GNU Affero General Public License for more details:
-# <https://www.gnu.org/licenses/agpl-3.0.html>.
-#
-import logging
-
-from twisted.test.proto_helpers import MemoryReactor
-
-import synapse.rest.admin
-from synapse.rest.client import devices, login, room, sync
-from synapse.server import HomeServer
-from synapse.types import JsonDict, StreamKeyType
-from synapse.util import Clock
-
-from tests.rest.client.sliding_sync.test_sliding_sync import SlidingSyncBase
-from tests.server import TimedOutException
-
-logger = logging.getLogger(__name__)
-
-
-class SlidingSyncE2eeExtensionTestCase(SlidingSyncBase):
-    """Tests for the e2ee sliding sync extension"""
-
-    servlets = [
-        synapse.rest.admin.register_servlets,
-        login.register_servlets,
-        room.register_servlets,
-        sync.register_servlets,
-        devices.register_servlets,
-    ]
-
-    def prepare(self, reactor: MemoryReactor, clock: Clock, hs: HomeServer) -> None:
-        self.store = hs.get_datastores().main
-        self.e2e_keys_handler = hs.get_e2e_keys_handler()
-
-    def test_no_data_initial_sync(self) -> None:
-        """
-        Test that enabling e2ee extension works during an intitial sync, even if there
-        is no-data
-        """
-        user1_id = self.register_user("user1", "pass")
-        user1_tok = self.login(user1_id, "pass")
-
-        # Make an initial Sliding Sync request with the e2ee extension enabled
-        sync_body = {
-            "lists": {},
-            "extensions": {
-                "e2ee": {
-                    "enabled": True,
-                }
-            },
-        }
-        response_body, _ = self.do_sync(sync_body, tok=user1_tok)
-
-        # Device list updates are only present for incremental syncs
-        self.assertIsNone(response_body["extensions"]["e2ee"].get("device_lists"))
-
-        # Both of these should be present even when empty
-        self.assertEqual(
-            response_body["extensions"]["e2ee"]["device_one_time_keys_count"],
-            {
-                # This is always present because of
-                # https://github.com/element-hq/element-android/issues/3725 and
-                # https://github.com/matrix-org/synapse/issues/10456
-                "signed_curve25519": 0
-            },
-        )
-        self.assertEqual(
-            response_body["extensions"]["e2ee"]["device_unused_fallback_key_types"],
-            [],
-        )
-
-    def test_no_data_incremental_sync(self) -> None:
-        """
-        Test that enabling e2ee extension works during an incremental sync, even if
-        there is no-data
-        """
-        user1_id = self.register_user("user1", "pass")
-        user1_tok = self.login(user1_id, "pass")
-
-        sync_body = {
-            "lists": {},
-            "extensions": {
-                "e2ee": {
-                    "enabled": True,
-                }
-            },
-        }
-        _, from_token = self.do_sync(sync_body, tok=user1_tok)
-
-        # Make an incremental Sliding Sync request with the e2ee extension enabled
-        response_body, _ = self.do_sync(sync_body, since=from_token, tok=user1_tok)
-
-        # Device list shows up for incremental syncs
-        self.assertEqual(
-            response_body["extensions"]["e2ee"].get("device_lists", {}).get("changed"),
-            [],
-        )
-        self.assertEqual(
-            response_body["extensions"]["e2ee"].get("device_lists", {}).get("left"),
-            [],
-        )
-
-        # Both of these should be present even when empty
-        self.assertEqual(
-            response_body["extensions"]["e2ee"]["device_one_time_keys_count"],
-            {
-                # Note that "signed_curve25519" is always returned in key count responses
-                # regardless of whether we uploaded any keys for it. This is necessary until
-                # https://github.com/matrix-org/matrix-doc/issues/3298 is fixed.
-                #
-                # Also related:
-                # https://github.com/element-hq/element-android/issues/3725 and
-                # https://github.com/matrix-org/synapse/issues/10456
-                "signed_curve25519": 0
-            },
-        )
-        self.assertEqual(
-            response_body["extensions"]["e2ee"]["device_unused_fallback_key_types"],
-            [],
-        )
-
-    def test_wait_for_new_data(self) -> None:
-        """
-        Test to make sure that the Sliding Sync request waits for new data to arrive.
-
-        (Only applies to incremental syncs with a `timeout` specified)
-        """
-        user1_id = self.register_user("user1", "pass")
-        user1_tok = self.login(user1_id, "pass")
-        user2_id = self.register_user("user2", "pass")
-        user2_tok = self.login(user2_id, "pass")
-        test_device_id = "TESTDEVICE"
-        user3_id = self.register_user("user3", "pass")
-        user3_tok = self.login(user3_id, "pass", device_id=test_device_id)
-
-        room_id = self.helper.create_room_as(user2_id, tok=user2_tok)
-        self.helper.join(room_id, user1_id, tok=user1_tok)
-        self.helper.join(room_id, user3_id, tok=user3_tok)
-
-        sync_body = {
-            "lists": {},
-            "extensions": {
-                "e2ee": {
-                    "enabled": True,
-                }
-            },
-        }
-        _, from_token = self.do_sync(sync_body, tok=user1_tok)
-
-        # Make the Sliding Sync request
-        channel = self.make_request(
-            "POST",
-            self.sync_endpoint + "?timeout=10000" + f"&pos={from_token}",
-            content=sync_body,
-            access_token=user1_tok,
-            await_result=False,
-        )
-        # Block for 5 seconds to make sure we are `notifier.wait_for_events(...)`
-        with self.assertRaises(TimedOutException):
-            channel.await_result(timeout_ms=5000)
-        # Bump the device lists to trigger new results
-        # Have user3 update their device list
-        device_update_channel = self.make_request(
-            "PUT",
-            f"/devices/{test_device_id}",
-            {
-                "display_name": "New Device Name",
-            },
-            access_token=user3_tok,
-        )
-        self.assertEqual(
-            device_update_channel.code, 200, device_update_channel.json_body
-        )
-        # Should respond before the 10 second timeout
-        channel.await_result(timeout_ms=3000)
-        self.assertEqual(channel.code, 200, channel.json_body)
-
-        # We should see the device list update
-        self.assertEqual(
-            channel.json_body["extensions"]["e2ee"]
-            .get("device_lists", {})
-            .get("changed"),
-            [user3_id],
-        )
-        self.assertEqual(
-            channel.json_body["extensions"]["e2ee"].get("device_lists", {}).get("left"),
-            [],
-        )
-
-    def test_wait_for_new_data_timeout(self) -> None:
-        """
-        Test to make sure that the Sliding Sync request waits for new data to arrive but
-        no data ever arrives so we timeout. We're also making sure that the default data
-        from the E2EE extension doesn't trigger a false-positive for new data (see
-        `device_one_time_keys_count.signed_curve25519`).
-        """
-        user1_id = self.register_user("user1", "pass")
-        user1_tok = self.login(user1_id, "pass")
-
-        sync_body = {
-            "lists": {},
-            "extensions": {
-                "e2ee": {
-                    "enabled": True,
-                }
-            },
-        }
-        _, from_token = self.do_sync(sync_body, tok=user1_tok)
-
-        # Make the Sliding Sync request
-        channel = self.make_request(
-            "POST",
-            self.sync_endpoint + f"?timeout=10000&pos={from_token}",
-            content=sync_body,
-            access_token=user1_tok,
-            await_result=False,
-        )
-        # Block for 5 seconds to make sure we are `notifier.wait_for_events(...)`
-        with self.assertRaises(TimedOutException):
-            channel.await_result(timeout_ms=5000)
-        # Wake-up `notifier.wait_for_events(...)` that will cause us test
-        # `SlidingSyncResult.__bool__` for new results.
-        self._bump_notifier_wait_for_events(
-            user1_id, wake_stream_key=StreamKeyType.ACCOUNT_DATA
-        )
-        # Block for a little bit more to ensure we don't see any new results.
-        with self.assertRaises(TimedOutException):
-            channel.await_result(timeout_ms=4000)
-        # Wait for the sync to complete (wait for the rest of the 10 second timeout,
-        # 5000 + 4000 + 1200 > 10000)
-        channel.await_result(timeout_ms=1200)
-        self.assertEqual(channel.code, 200, channel.json_body)
-
-        # Device lists are present for incremental syncs but empty because no device changes
-        self.assertEqual(
-            channel.json_body["extensions"]["e2ee"]
-            .get("device_lists", {})
-            .get("changed"),
-            [],
-        )
-        self.assertEqual(
-            channel.json_body["extensions"]["e2ee"].get("device_lists", {}).get("left"),
-            [],
-        )
-
-        # Both of these should be present even when empty
-        self.assertEqual(
-            channel.json_body["extensions"]["e2ee"]["device_one_time_keys_count"],
-            {
-                # Note that "signed_curve25519" is always returned in key count responses
-                # regardless of whether we uploaded any keys for it. This is necessary until
-                # https://github.com/matrix-org/matrix-doc/issues/3298 is fixed.
-                #
-                # Also related:
-                # https://github.com/element-hq/element-android/issues/3725 and
-                # https://github.com/matrix-org/synapse/issues/10456
-                "signed_curve25519": 0
-            },
-        )
-        self.assertEqual(
-            channel.json_body["extensions"]["e2ee"]["device_unused_fallback_key_types"],
-            [],
-        )
-
-    def test_device_lists(self) -> None:
-        """
-        Test that device list updates are included in the response
-        """
-        user1_id = self.register_user("user1", "pass")
-        user1_tok = self.login(user1_id, "pass")
-        user2_id = self.register_user("user2", "pass")
-        user2_tok = self.login(user2_id, "pass")
-
-        test_device_id = "TESTDEVICE"
-        user3_id = self.register_user("user3", "pass")
-        user3_tok = self.login(user3_id, "pass", device_id=test_device_id)
-
-        user4_id = self.register_user("user4", "pass")
-        user4_tok = self.login(user4_id, "pass")
-
-        room_id = self.helper.create_room_as(user2_id, tok=user2_tok)
-        self.helper.join(room_id, user1_id, tok=user1_tok)
-        self.helper.join(room_id, user3_id, tok=user3_tok)
-        self.helper.join(room_id, user4_id, tok=user4_tok)
-
-        sync_body = {
-            "lists": {},
-            "extensions": {
-                "e2ee": {
-                    "enabled": True,
-                }
-            },
-        }
-        _, from_token = self.do_sync(sync_body, tok=user1_tok)
-
-        # Have user3 update their device list
-        channel = self.make_request(
-            "PUT",
-            f"/devices/{test_device_id}",
-            {
-                "display_name": "New Device Name",
-            },
-            access_token=user3_tok,
-        )
-        self.assertEqual(channel.code, 200, channel.json_body)
-
-        # User4 leaves the room
-        self.helper.leave(room_id, user4_id, tok=user4_tok)
-
-        # Make an incremental Sliding Sync request with the e2ee extension enabled
-        response_body, _ = self.do_sync(sync_body, since=from_token, tok=user1_tok)
-
-        # Device list updates show up
-        self.assertEqual(
-            response_body["extensions"]["e2ee"].get("device_lists", {}).get("changed"),
-            [user3_id],
-        )
-        self.assertEqual(
-            response_body["extensions"]["e2ee"].get("device_lists", {}).get("left"),
-            [user4_id],
-        )
-
-    def test_device_one_time_keys_count(self) -> None:
-        """
-        Test that `device_one_time_keys_count` are included in the response
-        """
-        test_device_id = "TESTDEVICE"
-        user1_id = self.register_user("user1", "pass")
-        user1_tok = self.login(user1_id, "pass", device_id=test_device_id)
-
-        # Upload one time keys for the user/device
-        keys: JsonDict = {
-            "alg1:k1": "key1",
-            "alg2:k2": {"key": "key2", "signatures": {"k1": "sig1"}},
-            "alg2:k3": {"key": "key3"},
-        }
-        upload_keys_response = self.get_success(
-            self.e2e_keys_handler.upload_keys_for_user(
-                user1_id, test_device_id, {"one_time_keys": keys}
-            )
-        )
-        self.assertDictEqual(
-            upload_keys_response,
-            {
-                "one_time_key_counts": {
-                    "alg1": 1,
-                    "alg2": 2,
-                    # Note that "signed_curve25519" is always returned in key count responses
-                    # regardless of whether we uploaded any keys for it. This is necessary until
-                    # https://github.com/matrix-org/matrix-doc/issues/3298 is fixed.
-                    #
-                    # Also related:
-                    # https://github.com/element-hq/element-android/issues/3725 and
-                    # https://github.com/matrix-org/synapse/issues/10456
-                    "signed_curve25519": 0,
-                }
-            },
-        )
-
-        # Make a Sliding Sync request with the e2ee extension enabled
-        sync_body = {
-            "lists": {},
-            "extensions": {
-                "e2ee": {
-                    "enabled": True,
-                }
-            },
-        }
-        response_body, _ = self.do_sync(sync_body, tok=user1_tok)
-
-        # Check for those one time key counts
-        self.assertEqual(
-            response_body["extensions"]["e2ee"].get("device_one_time_keys_count"),
-            {
-                "alg1": 1,
-                "alg2": 2,
-                # Note that "signed_curve25519" is always returned in key count responses
-                # regardless of whether we uploaded any keys for it. This is necessary until
-                # https://github.com/matrix-org/matrix-doc/issues/3298 is fixed.
-                #
-                # Also related:
-                # https://github.com/element-hq/element-android/issues/3725 and
-                # https://github.com/matrix-org/synapse/issues/10456
-                "signed_curve25519": 0,
-            },
-        )
-
-    def test_device_unused_fallback_key_types(self) -> None:
-        """
-        Test that `device_unused_fallback_key_types` are included in the response
-        """
-        test_device_id = "TESTDEVICE"
-        user1_id = self.register_user("user1", "pass")
-        user1_tok = self.login(user1_id, "pass", device_id=test_device_id)
-
-        # We shouldn't have any unused fallback keys yet
-        res = self.get_success(
-            self.store.get_e2e_unused_fallback_key_types(user1_id, test_device_id)
-        )
-        self.assertEqual(res, [])
-
-        # Upload a fallback key for the user/device
-        self.get_success(
-            self.e2e_keys_handler.upload_keys_for_user(
-                user1_id,
-                test_device_id,
-                {"fallback_keys": {"alg1:k1": "fallback_key1"}},
-            )
-        )
-        # We should now have an unused alg1 key
-        fallback_res = self.get_success(
-            self.store.get_e2e_unused_fallback_key_types(user1_id, test_device_id)
-        )
-        self.assertEqual(fallback_res, ["alg1"], fallback_res)
-
-        # Make a Sliding Sync request with the e2ee extension enabled
-        sync_body = {
-            "lists": {},
-            "extensions": {
-                "e2ee": {
-                    "enabled": True,
-                }
-            },
-        }
-        response_body, _ = self.do_sync(sync_body, tok=user1_tok)
-
-        # Check for the unused fallback key types
-        self.assertListEqual(
-            response_body["extensions"]["e2ee"].get("device_unused_fallback_key_types"),
-            ["alg1"],
-        )

tests/rest/client/sliding_sync/test_extension_receipts.py

@@ -1,679 +0,0 @@
-#
-# This file is licensed under the Affero General Public License (AGPL) version 3.
-#
-# Copyright (C) 2024 New Vector, Ltd
-#
-# This program is free software: you can redistribute it and/or modify
-# it under the terms of the GNU Affero General Public License as
-# published by the Free Software Foundation, either version 3 of the
-# License, or (at your option) any later version.
-#
-# See the GNU Affero General Public License for more details:
-# <https://www.gnu.org/licenses/agpl-3.0.html>.
-#
-import logging
-
-from twisted.test.proto_helpers import MemoryReactor
-
-import synapse.rest.admin
-from synapse.api.constants import EduTypes, ReceiptTypes
-from synapse.rest.client import login, receipts, room, sync
-from synapse.server import HomeServer
-from synapse.types import StreamKeyType
-from synapse.util import Clock
-
-from tests.rest.client.sliding_sync.test_sliding_sync import SlidingSyncBase
-from tests.server import TimedOutException
-
-logger = logging.getLogger(__name__)
-
-
-class SlidingSyncReceiptsExtensionTestCase(SlidingSyncBase):
-    """Tests for the receipts sliding sync extension"""
-
-    servlets = [
-        synapse.rest.admin.register_servlets,
-        login.register_servlets,
-        room.register_servlets,
-        sync.register_servlets,
-        receipts.register_servlets,
-    ]
-
-    def prepare(self, reactor: MemoryReactor, clock: Clock, hs: HomeServer) -> None:
-        self.store = hs.get_datastores().main
-
-    def test_no_data_initial_sync(self) -> None:
-        """
-        Test that enabling the receipts extension works during an intitial sync,
-        even if there is no-data.
-        """
-        user1_id = self.register_user("user1", "pass")
-        user1_tok = self.login(user1_id, "pass")
-
-        # Make an initial Sliding Sync request with the receipts extension enabled
-        sync_body = {
-            "lists": {},
-            "extensions": {
-                "receipts": {
-                    "enabled": True,
-                }
-            },
-        }
-        response_body, _ = self.do_sync(sync_body, tok=user1_tok)
-
-        self.assertIncludes(
-            response_body["extensions"]["receipts"].get("rooms").keys(),
-            set(),
-            exact=True,
-        )
-
-    def test_no_data_incremental_sync(self) -> None:
-        """
-        Test that enabling receipts extension works during an incremental sync, even
-        if there is no-data.
-        """
-        user1_id = self.register_user("user1", "pass")
-        user1_tok = self.login(user1_id, "pass")
-
-        sync_body = {
-            "lists": {},
-            "extensions": {
-                "receipts": {
-                    "enabled": True,
-                }
-            },
-        }
-        _, from_token = self.do_sync(sync_body, tok=user1_tok)
-
-        # Make an incremental Sliding Sync request with the receipts extension enabled
-        response_body, _ = self.do_sync(sync_body, since=from_token, tok=user1_tok)
-
-        self.assertIncludes(
-            response_body["extensions"]["receipts"].get("rooms").keys(),
-            set(),
-            exact=True,
-        )
-
-    def test_receipts_initial_sync_with_timeline(self) -> None:
-        """
-        On initial sync, we only return receipts for events in a given room's timeline.
-
-        We also make sure that we only return receipts for rooms that we request and are
-        already being returned in the Sliding Sync response.
-        """
-        user1_id = self.register_user("user1", "pass")
-        user1_tok = self.login(user1_id, "pass")
-        user2_id = self.register_user("user2", "pass")
-        user2_tok = self.login(user2_id, "pass")
-        user3_id = self.register_user("user3", "pass")
-        user3_tok = self.login(user3_id, "pass")
-        user4_id = self.register_user("user4", "pass")
-        user4_tok = self.login(user4_id, "pass")
-
-        # Create a room
-        room_id1 = self.helper.create_room_as(user2_id, tok=user2_tok)
-        self.helper.join(room_id1, user1_id, tok=user1_tok)
-        self.helper.join(room_id1, user3_id, tok=user3_tok)
-        self.helper.join(room_id1, user4_id, tok=user4_tok)
-        room1_event_response1 = self.helper.send(
-            room_id1, body="new event1", tok=user2_tok
-        )
-        room1_event_response2 = self.helper.send(
-            room_id1, body="new event2", tok=user2_tok
-        )
-        # User1 reads the last event
-        channel = self.make_request(
-            "POST",
-            f"/rooms/{room_id1}/receipt/{ReceiptTypes.READ}/{room1_event_response2['event_id']}",
-            {},
-            access_token=user1_tok,
-        )
-        self.assertEqual(channel.code, 200, channel.json_body)
-        # User2 reads the last event
-        channel = self.make_request(
-            "POST",
-            f"/rooms/{room_id1}/receipt/{ReceiptTypes.READ}/{room1_event_response2['event_id']}",
-            {},
-            access_token=user2_tok,
-        )
-        self.assertEqual(channel.code, 200, channel.json_body)
-        # User3 reads the first event
-        channel = self.make_request(
-            "POST",
-            f"/rooms/{room_id1}/receipt/{ReceiptTypes.READ}/{room1_event_response1['event_id']}",
-            {},
-            access_token=user3_tok,
-        )
-        self.assertEqual(channel.code, 200, channel.json_body)
-        # User4 privately reads the last event (make sure this doesn't leak to the other users)
-        channel = self.make_request(
-            "POST",
-            f"/rooms/{room_id1}/receipt/{ReceiptTypes.READ_PRIVATE}/{room1_event_response2['event_id']}",
-            {},
-            access_token=user4_tok,
-        )
-        self.assertEqual(channel.code, 200, channel.json_body)
-
-        # Create another room
-        room_id2 = self.helper.create_room_as(user2_id, tok=user2_tok)
-        self.helper.join(room_id2, user1_id, tok=user1_tok)
-        self.helper.join(room_id2, user3_id, tok=user3_tok)
-        self.helper.join(room_id2, user4_id, tok=user4_tok)
-        room2_event_response1 = self.helper.send(
-            room_id2, body="new event2", tok=user2_tok
-        )
-        # User1 reads the last event
-        channel = self.make_request(
-            "POST",
-            f"/rooms/{room_id2}/receipt/{ReceiptTypes.READ}/{room2_event_response1['event_id']}",
-            {},
-            access_token=user1_tok,
-        )
-        self.assertEqual(channel.code, 200, channel.json_body)
-        # User2 reads the last event
-        channel = self.make_request(
-            "POST",
-            f"/rooms/{room_id2}/receipt/{ReceiptTypes.READ}/{room2_event_response1['event_id']}",
-            {},
-            access_token=user2_tok,
-        )
-        self.assertEqual(channel.code, 200, channel.json_body)
-        # User4 privately reads the last event (make sure this doesn't leak to the other users)
-        channel = self.make_request(
-            "POST",
-            f"/rooms/{room_id2}/receipt/{ReceiptTypes.READ_PRIVATE}/{room2_event_response1['event_id']}",
-            {},
-            access_token=user4_tok,
-        )
-        self.assertEqual(channel.code, 200, channel.json_body)
-
-        # Make an initial Sliding Sync request with the receipts extension enabled
-        sync_body = {
-            "lists": {},
-            "room_subscriptions": {
-                room_id1: {
-                    "required_state": [],
-                    # On initial sync, we only have receipts for events in the timeline
-                    "timeline_limit": 1,
-                }
-            },
-            "extensions": {
-                "receipts": {
-                    "enabled": True,
-                    "rooms": [room_id1, room_id2],
-                }
-            },
-        }
-        response_body, _ = self.do_sync(sync_body, tok=user1_tok)
-
-        # Only the latest event in the room is in the timelie because the `timeline_limit` is 1
-        self.assertIncludes(
-            {
-                event["event_id"]
-                for event in response_body["rooms"][room_id1].get("timeline", [])
-            },
-            {room1_event_response2["event_id"]},
-            exact=True,
-            message=str(response_body["rooms"][room_id1]),
-        )
-
-        # Even though we requested room2, we only expect room1 to show up because that's
-        # the only room in the Sliding Sync response (room2 is not one of our room
-        # subscriptions or in a sliding window list).
-        self.assertIncludes(
-            response_body["extensions"]["receipts"].get("rooms").keys(),
-            {room_id1},
-            exact=True,
-        )
-        # Sanity check that it's the correct ephemeral event type
-        self.assertEqual(
-            response_body["extensions"]["receipts"]["rooms"][room_id1]["type"],
-            EduTypes.RECEIPT,
-        )
-        # We can see user1 and user2 read receipts
-        self.assertIncludes(
-            response_body["extensions"]["receipts"]["rooms"][room_id1]["content"][
-                room1_event_response2["event_id"]
-            ][ReceiptTypes.READ].keys(),
-            {user1_id, user2_id},
-            exact=True,
-        )
-        # User1 did not have a private read receipt and we shouldn't leak others'
-        # private read receipts
-        self.assertIncludes(
-            response_body["extensions"]["receipts"]["rooms"][room_id1]["content"][
-                room1_event_response2["event_id"]
-            ]
-            .get(ReceiptTypes.READ_PRIVATE, {})
-            .keys(),
-            set(),
-            exact=True,
-        )
-
-        # We shouldn't see receipts for event2 since it wasn't in the timeline and this is an initial sync
-        self.assertIsNone(
-            response_body["extensions"]["receipts"]["rooms"][room_id1]["content"].get(
-                room1_event_response1["event_id"]
-            )
-        )
-
-    def test_receipts_incremental_sync(self) -> None:
-        """
-        On incremental sync, we return all receipts in the token range for a given room
-        but only for rooms that we request and are being returned in the Sliding Sync
-        response.
-        """
-
-        user1_id = self.register_user("user1", "pass")
-        user1_tok = self.login(user1_id, "pass")
-        user2_id = self.register_user("user2", "pass")
-        user2_tok = self.login(user2_id, "pass")
-        user3_id = self.register_user("user3", "pass")
-        user3_tok = self.login(user3_id, "pass")
-
-        # Create room1
-        room_id1 = self.helper.create_room_as(user2_id, tok=user2_tok)
-        self.helper.join(room_id1, user1_id, tok=user1_tok)
-        self.helper.join(room_id1, user3_id, tok=user3_tok)
-        room1_event_response1 = self.helper.send(
-            room_id1, body="new event2", tok=user2_tok
-        )
-        # User2 reads the last event (before the `from_token`)
-        channel = self.make_request(
-            "POST",
-            f"/rooms/{room_id1}/receipt/{ReceiptTypes.READ}/{room1_event_response1['event_id']}",
-            {},
-            access_token=user2_tok,
-        )
-        self.assertEqual(channel.code, 200, channel.json_body)
-
-        # Create room2
-        room_id2 = self.helper.create_room_as(user2_id, tok=user2_tok)
-        self.helper.join(room_id2, user1_id, tok=user1_tok)
-        room2_event_response1 = self.helper.send(
-            room_id2, body="new event2", tok=user2_tok
-        )
-        # User1 reads the last event (before the `from_token`)
-        channel = self.make_request(
-            "POST",
-            f"/rooms/{room_id2}/receipt/{ReceiptTypes.READ}/{room2_event_response1['event_id']}",
-            {},
-            access_token=user1_tok,
-        )
-        self.assertEqual(channel.code, 200, channel.json_body)
-
-        # Create room3
-        room_id3 = self.helper.create_room_as(user2_id, tok=user2_tok)
-        self.helper.join(room_id3, user1_id, tok=user1_tok)
-        self.helper.join(room_id3, user3_id, tok=user3_tok)
-        room3_event_response1 = self.helper.send(
-            room_id3, body="new event", tok=user2_tok
-        )
-
-        # Create room4
-        room_id4 = self.helper.create_room_as(user2_id, tok=user2_tok)
-        self.helper.join(room_id4, user1_id, tok=user1_tok)
-        self.helper.join(room_id4, user3_id, tok=user3_tok)
-        event_response4 = self.helper.send(room_id4, body="new event", tok=user2_tok)
-        # User1 reads the last event (before the `from_token`)
-        channel = self.make_request(
-            "POST",
-            f"/rooms/{room_id4}/receipt/{ReceiptTypes.READ}/{event_response4['event_id']}",
-            {},
-            access_token=user1_tok,
-        )
-        self.assertEqual(channel.code, 200, channel.json_body)
-
-        sync_body = {
-            "lists": {},
-            "room_subscriptions": {
-                room_id1: {
-                    "required_state": [],
-                    "timeline_limit": 0,
-                },
-                room_id3: {
-                    "required_state": [],
-                    "timeline_limit": 0,
-                },
-                room_id4: {
-                    "required_state": [],
-                    "timeline_limit": 0,
-                },
-            },
-            "extensions": {
-                "receipts": {
-                    "enabled": True,
-                    "rooms": [room_id1, room_id2, room_id3, room_id4],
-                }
-            },
-        }
-        _, from_token = self.do_sync(sync_body, tok=user1_tok)
-
-        # Add some more read receipts after the `from_token`
-        #
-        # User1 reads room1
-        channel = self.make_request(
-            "POST",
-            f"/rooms/{room_id1}/receipt/{ReceiptTypes.READ}/{room1_event_response1['event_id']}",
-            {},
-            access_token=user1_tok,
-        )
-        self.assertEqual(channel.code, 200, channel.json_body)
-        # User1 privately reads room2
-        channel = self.make_request(
-            "POST",
-            f"/rooms/{room_id2}/receipt/{ReceiptTypes.READ_PRIVATE}/{room2_event_response1['event_id']}",
-            {},
-            access_token=user1_tok,
-        )
-        self.assertEqual(channel.code, 200, channel.json_body)
-        # User3 reads room3
-        channel = self.make_request(
-            "POST",
-            f"/rooms/{room_id3}/receipt/{ReceiptTypes.READ}/{room3_event_response1['event_id']}",
-            {},
-            access_token=user3_tok,
-        )
-        self.assertEqual(channel.code, 200, channel.json_body)
-        # No activity for room4 after the `from_token`
-
-        # Make an incremental Sliding Sync request with the receipts extension enabled
-        response_body, _ = self.do_sync(sync_body, since=from_token, tok=user1_tok)
-
-        # Even though we requested room2, we only expect rooms to show up if they are
-        # already in the Sliding Sync response. room4 doesn't show up because there is
-        # no activity after the `from_token`.
-        self.assertIncludes(
-            response_body["extensions"]["receipts"].get("rooms").keys(),
-            {room_id1, room_id3},
-            exact=True,
-        )
-
-        # Check room1:
-        #
-        # Sanity check that it's the correct ephemeral event type
-        self.assertEqual(
-            response_body["extensions"]["receipts"]["rooms"][room_id1]["type"],
-            EduTypes.RECEIPT,
-        )
-        # We only see that user1 has read something in room1 since the `from_token`
-        self.assertIncludes(
-            response_body["extensions"]["receipts"]["rooms"][room_id1]["content"][
-                room1_event_response1["event_id"]
-            ][ReceiptTypes.READ].keys(),
-            {user1_id},
-            exact=True,
-        )
-        # User1 did not send a private read receipt in this room and we shouldn't leak
-        # others' private read receipts
-        self.assertIncludes(
-            response_body["extensions"]["receipts"]["rooms"][room_id1]["content"][
-                room1_event_response1["event_id"]
-            ]
-            .get(ReceiptTypes.READ_PRIVATE, {})
-            .keys(),
-            set(),
-            exact=True,
-        )
-        # No events in the timeline since they were sent before the `from_token`
-        self.assertNotIn(room_id1, response_body["rooms"])
-
-        # Check room3:
-        #
-        # Sanity check that it's the correct ephemeral event type
-        self.assertEqual(
-            response_body["extensions"]["receipts"]["rooms"][room_id3]["type"],
-            EduTypes.RECEIPT,
-        )
-        # We only see that user3 has read something in room1 since the `from_token`
-        self.assertIncludes(
-            response_body["extensions"]["receipts"]["rooms"][room_id3]["content"][
-                room3_event_response1["event_id"]
-            ][ReceiptTypes.READ].keys(),
-            {user3_id},
-            exact=True,
-        )
-        # User1 did not send a private read receipt in this room and we shouldn't leak
-        # others' private read receipts
-        self.assertIncludes(
-            response_body["extensions"]["receipts"]["rooms"][room_id3]["content"][
-                room3_event_response1["event_id"]
-            ]
-            .get(ReceiptTypes.READ_PRIVATE, {})
-            .keys(),
-            set(),
-            exact=True,
-        )
-        # No events in the timeline since they were sent before the `from_token`
-        self.assertNotIn(room_id3, response_body["rooms"])
-
-    def test_receipts_incremental_sync_all_live_receipts(self) -> None:
-        """
-        On incremental sync, we return all receipts in the token range for a given room
-        even if they are not in the timeline.
-        """
-
-        user1_id = self.register_user("user1", "pass")
-        user1_tok = self.login(user1_id, "pass")
-        user2_id = self.register_user("user2", "pass")
-        user2_tok = self.login(user2_id, "pass")
-
-        # Create room1
-        room_id1 = self.helper.create_room_as(user2_id, tok=user2_tok)
-        self.helper.join(room_id1, user1_id, tok=user1_tok)
-
-        sync_body = {
-            "lists": {},
-            "room_subscriptions": {
-                room_id1: {
-                    "required_state": [],
-                    # The timeline will only include event2
-                    "timeline_limit": 1,
-                },
-            },
-            "extensions": {
-                "receipts": {
-                    "enabled": True,
-                    "rooms": [room_id1],
-                }
-            },
-        }
-        _, from_token = self.do_sync(sync_body, tok=user1_tok)
-
-        room1_event_response1 = self.helper.send(
-            room_id1, body="new event1", tok=user2_tok
-        )
-        room1_event_response2 = self.helper.send(
-            room_id1, body="new event2", tok=user2_tok
-        )
-
-        # User1 reads event1
-        channel = self.make_request(
-            "POST",
-            f"/rooms/{room_id1}/receipt/{ReceiptTypes.READ}/{room1_event_response1['event_id']}",
-            {},
-            access_token=user1_tok,
-        )
-        self.assertEqual(channel.code, 200, channel.json_body)
-        # User2 reads event2
-        channel = self.make_request(
-            "POST",
-            f"/rooms/{room_id1}/receipt/{ReceiptTypes.READ}/{room1_event_response2['event_id']}",
-            {},
-            access_token=user2_tok,
-        )
-        self.assertEqual(channel.code, 200, channel.json_body)
-
-        # Make an incremental Sliding Sync request with the receipts extension enabled
-        response_body, _ = self.do_sync(sync_body, since=from_token, tok=user1_tok)
-
-        # We should see room1 because it has receipts in the token range
-        self.assertIncludes(
-            response_body["extensions"]["receipts"].get("rooms").keys(),
-            {room_id1},
-            exact=True,
-        )
-        # Sanity check that it's the correct ephemeral event type
-        self.assertEqual(
-            response_body["extensions"]["receipts"]["rooms"][room_id1]["type"],
-            EduTypes.RECEIPT,
-        )
-        # We should see all receipts in the token range regardless of whether the events
-        # are in the timeline
-        self.assertIncludes(
-            response_body["extensions"]["receipts"]["rooms"][room_id1]["content"][
-                room1_event_response1["event_id"]
-            ][ReceiptTypes.READ].keys(),
-            {user1_id},
-            exact=True,
-        )
-        self.assertIncludes(
-            response_body["extensions"]["receipts"]["rooms"][room_id1]["content"][
-                room1_event_response2["event_id"]
-            ][ReceiptTypes.READ].keys(),
-            {user2_id},
-            exact=True,
-        )
-        # Only the latest event in the timeline because the `timeline_limit` is 1
-        self.assertIncludes(
-            {
-                event["event_id"]
-                for event in response_body["rooms"][room_id1].get("timeline", [])
-            },
-            {room1_event_response2["event_id"]},
-            exact=True,
-            message=str(response_body["rooms"][room_id1]),
-        )
-
-    def test_wait_for_new_data(self) -> None:
-        """
-        Test to make sure that the Sliding Sync request waits for new data to arrive.
-
-        (Only applies to incremental syncs with a `timeout` specified)
-        """
-        user1_id = self.register_user("user1", "pass")
-        user1_tok = self.login(user1_id, "pass")
-        user2_id = self.register_user("user2", "pass")
-        user2_tok = self.login(user2_id, "pass")
-
-        room_id = self.helper.create_room_as(user2_id, tok=user2_tok)
-        self.helper.join(room_id, user1_id, tok=user1_tok)
-        event_response = self.helper.send(room_id, body="new event", tok=user2_tok)
-
-        sync_body = {
-            "lists": {},
-            "room_subscriptions": {
-                room_id: {
-                    "required_state": [],
-                    "timeline_limit": 0,
-                },
-            },
-            "extensions": {
-                "receipts": {
-                    "enabled": True,
-                    "rooms": [room_id],
-                }
-            },
-        }
-        _, from_token = self.do_sync(sync_body, tok=user1_tok)
-
-        # Make an incremental Sliding Sync request with the receipts extension enabled
-        channel = self.make_request(
-            "POST",
-            self.sync_endpoint + f"?timeout=10000&pos={from_token}",
-            content=sync_body,
-            access_token=user1_tok,
-            await_result=False,
-        )
-        # Block for 5 seconds to make sure we are `notifier.wait_for_events(...)`
-        with self.assertRaises(TimedOutException):
-            channel.await_result(timeout_ms=5000)
-        # Bump the receipts to trigger new results
-        receipt_channel = self.make_request(
-            "POST",
-            f"/rooms/{room_id}/receipt/{ReceiptTypes.READ}/{event_response['event_id']}",
-            {},
-            access_token=user2_tok,
-        )
-        self.assertEqual(receipt_channel.code, 200, receipt_channel.json_body)
-        # Should respond before the 10 second timeout
-        channel.await_result(timeout_ms=3000)
-        self.assertEqual(channel.code, 200, channel.json_body)
-
-        # We should see the new receipt
-        self.assertIncludes(
-            channel.json_body.get("extensions", {})
-            .get("receipts", {})
-            .get("rooms", {})
-            .keys(),
-            {room_id},
-            exact=True,
-            message=str(channel.json_body),
-        )
-        self.assertIncludes(
-            channel.json_body["extensions"]["receipts"]["rooms"][room_id]["content"][
-                event_response["event_id"]
-            ][ReceiptTypes.READ].keys(),
-            {user2_id},
-            exact=True,
-        )
-        # User1 did not send a private read receipt in this room and we shouldn't leak
-        # others' private read receipts
-        self.assertIncludes(
-            channel.json_body["extensions"]["receipts"]["rooms"][room_id]["content"][
-                event_response["event_id"]
-            ]
-            .get(ReceiptTypes.READ_PRIVATE, {})
-            .keys(),
-            set(),
-            exact=True,
-        )
-
-    def test_wait_for_new_data_timeout(self) -> None:
-        """
-        Test to make sure that the Sliding Sync request waits for new data to arrive but
-        no data ever arrives so we timeout. We're also making sure that the default data
-        from the receipts extension doesn't trigger a false-positive for new data.
-        """
-        user1_id = self.register_user("user1", "pass")
-        user1_tok = self.login(user1_id, "pass")
-
-        sync_body = {
-            "lists": {},
-            "extensions": {
-                "receipts": {
-                    "enabled": True,
-                }
-            },
-        }
-        _, from_token = self.do_sync(sync_body, tok=user1_tok)
-
-        # Make the Sliding Sync request
-        channel = self.make_request(
-            "POST",
-            self.sync_endpoint + f"?timeout=10000&pos={from_token}",
-            content=sync_body,
-            access_token=user1_tok,
-            await_result=False,
-        )
-        # Block for 5 seconds to make sure we are `notifier.wait_for_events(...)`
-        with self.assertRaises(TimedOutException):
-            channel.await_result(timeout_ms=5000)
-        # Wake-up `notifier.wait_for_events(...)` that will cause us test
-        # `SlidingSyncResult.__bool__` for new results.
-        self._bump_notifier_wait_for_events(
-            user1_id, wake_stream_key=StreamKeyType.ACCOUNT_DATA
-        )
-        # Block for a little bit more to ensure we don't see any new results.
-        with self.assertRaises(TimedOutException):
-            channel.await_result(timeout_ms=4000)
-        # Wait for the sync to complete (wait for the rest of the 10 second timeout,
-        # 5000 + 4000 + 1200 > 10000)
-        channel.await_result(timeout_ms=1200)
-        self.assertEqual(channel.code, 200, channel.json_body)
-
-        self.assertIncludes(
-            channel.json_body["extensions"]["receipts"].get("rooms").keys(),
-            set(),
-            exact=True,
-        )

tests/rest/client/sliding_sync/test_extension_to_device.py

@@ -1,278 +0,0 @@
-#
-# This file is licensed under the Affero General Public License (AGPL) version 3.
-#
-# Copyright (C) 2024 New Vector, Ltd
-#
-# This program is free software: you can redistribute it and/or modify
-# it under the terms of the GNU Affero General Public License as
-# published by the Free Software Foundation, either version 3 of the
-# License, or (at your option) any later version.
-#
-# See the GNU Affero General Public License for more details:
-# <https://www.gnu.org/licenses/agpl-3.0.html>.
-#
-import logging
-from typing import List
-
-from twisted.test.proto_helpers import MemoryReactor
-
-import synapse.rest.admin
-from synapse.rest.client import login, sendtodevice, sync
-from synapse.server import HomeServer
-from synapse.types import JsonDict, StreamKeyType
-from synapse.util import Clock
-
-from tests.rest.client.sliding_sync.test_sliding_sync import SlidingSyncBase
-from tests.server import TimedOutException
-
-logger = logging.getLogger(__name__)
-
-
-class SlidingSyncToDeviceExtensionTestCase(SlidingSyncBase):
-    """Tests for the to-device sliding sync extension"""
-
-    servlets = [
-        synapse.rest.admin.register_servlets,
-        login.register_servlets,
-        sync.register_servlets,
-        sendtodevice.register_servlets,
-    ]
-
-    def prepare(self, reactor: MemoryReactor, clock: Clock, hs: HomeServer) -> None:
-        self.store = hs.get_datastores().main
-
-    def _assert_to_device_response(
-        self, response_body: JsonDict, expected_messages: List[JsonDict]
-    ) -> str:
-        """Assert the sliding sync response was successful and has the expected
-        to-device messages.
-
-        Returns the next_batch token from the to-device section.
-        """
-        extensions = response_body["extensions"]
-        to_device = extensions["to_device"]
-        self.assertIsInstance(to_device["next_batch"], str)
-        self.assertEqual(to_device["events"], expected_messages)
-
-        return to_device["next_batch"]
-
-    def test_no_data(self) -> None:
-        """Test that enabling to-device extension works, even if there is
-        no-data
-        """
-        user1_id = self.register_user("user1", "pass")
-        user1_tok = self.login(user1_id, "pass")
-
-        sync_body = {
-            "lists": {},
-            "extensions": {
-                "to_device": {
-                    "enabled": True,
-                }
-            },
-        }
-        response_body, _ = self.do_sync(sync_body, tok=user1_tok)
-
-        # We expect no to-device messages
-        self._assert_to_device_response(response_body, [])
-
-    def test_data_initial_sync(self) -> None:
-        """Test that we get to-device messages when we don't specify a since
-        token"""
-
-        user1_id = self.register_user("user1", "pass")
-        user1_tok = self.login(user1_id, "pass", "d1")
-        user2_id = self.register_user("u2", "pass")
-        user2_tok = self.login(user2_id, "pass", "d2")
-
-        # Send the to-device message
-        test_msg = {"foo": "bar"}
-        chan = self.make_request(
-            "PUT",
-            "/_matrix/client/r0/sendToDevice/m.test/1234",
-            content={"messages": {user1_id: {"d1": test_msg}}},
-            access_token=user2_tok,
-        )
-        self.assertEqual(chan.code, 200, chan.result)
-
-        sync_body = {
-            "lists": {},
-            "extensions": {
-                "to_device": {
-                    "enabled": True,
-                }
-            },
-        }
-        response_body, _ = self.do_sync(sync_body, tok=user1_tok)
-        self._assert_to_device_response(
-            response_body,
-            [{"content": test_msg, "sender": user2_id, "type": "m.test"}],
-        )
-
-    def test_data_incremental_sync(self) -> None:
-        """Test that we get to-device messages over incremental syncs"""
-
-        user1_id = self.register_user("user1", "pass")
-        user1_tok = self.login(user1_id, "pass", "d1")
-        user2_id = self.register_user("u2", "pass")
-        user2_tok = self.login(user2_id, "pass", "d2")
-
-        sync_body: JsonDict = {
-            "lists": {},
-            "extensions": {
-                "to_device": {
-                    "enabled": True,
-                }
-            },
-        }
-        response_body, _ = self.do_sync(sync_body, tok=user1_tok)
-        # No to-device messages yet.
-        next_batch = self._assert_to_device_response(response_body, [])
-
-        test_msg = {"foo": "bar"}
-        chan = self.make_request(
-            "PUT",
-            "/_matrix/client/r0/sendToDevice/m.test/1234",
-            content={"messages": {user1_id: {"d1": test_msg}}},
-            access_token=user2_tok,
-        )
-        self.assertEqual(chan.code, 200, chan.result)
-
-        sync_body = {
-            "lists": {},
-            "extensions": {
-                "to_device": {
-                    "enabled": True,
-                    "since": next_batch,
-                }
-            },
-        }
-        response_body, _ = self.do_sync(sync_body, tok=user1_tok)
-        next_batch = self._assert_to_device_response(
-            response_body,
-            [{"content": test_msg, "sender": user2_id, "type": "m.test"}],
-        )
-
-        # The next sliding sync request should not include the to-device
-        # message.
-        sync_body = {
-            "lists": {},
-            "extensions": {
-                "to_device": {
-                    "enabled": True,
-                    "since": next_batch,
-                }
-            },
-        }
-        response_body, _ = self.do_sync(sync_body, tok=user1_tok)
-        self._assert_to_device_response(response_body, [])
-
-        # An initial sliding sync request should not include the to-device
-        # message, as it should have been deleted
-        sync_body = {
-            "lists": {},
-            "extensions": {
-                "to_device": {
-                    "enabled": True,
-                }
-            },
-        }
-        response_body, _ = self.do_sync(sync_body, tok=user1_tok)
-        self._assert_to_device_response(response_body, [])
-
-    def test_wait_for_new_data(self) -> None:
-        """
-        Test to make sure that the Sliding Sync request waits for new data to arrive.
-
-        (Only applies to incremental syncs with a `timeout` specified)
-        """
-        user1_id = self.register_user("user1", "pass")
-        user1_tok = self.login(user1_id, "pass", "d1")
-        user2_id = self.register_user("u2", "pass")
-        user2_tok = self.login(user2_id, "pass", "d2")
-
-        sync_body = {
-            "lists": {},
-            "extensions": {
-                "to_device": {
-                    "enabled": True,
-                }
-            },
-        }
-        _, from_token = self.do_sync(sync_body, tok=user1_tok)
-
-        # Make the Sliding Sync request
-        channel = self.make_request(
-            "POST",
-            self.sync_endpoint + "?timeout=10000" + f"&pos={from_token}",
-            content=sync_body,
-            access_token=user1_tok,
-            await_result=False,
-        )
-        # Block for 5 seconds to make sure we are `notifier.wait_for_events(...)`
-        with self.assertRaises(TimedOutException):
-            channel.await_result(timeout_ms=5000)
-        # Bump the to-device messages to trigger new results
-        test_msg = {"foo": "bar"}
-        send_to_device_channel = self.make_request(
-            "PUT",
-            "/_matrix/client/r0/sendToDevice/m.test/1234",
-            content={"messages": {user1_id: {"d1": test_msg}}},
-            access_token=user2_tok,
-        )
-        self.assertEqual(
-            send_to_device_channel.code, 200, send_to_device_channel.result
-        )
-        # Should respond before the 10 second timeout
-        channel.await_result(timeout_ms=3000)
-        self.assertEqual(channel.code, 200, channel.json_body)
-
-        self._assert_to_device_response(
-            channel.json_body,
-            [{"content": test_msg, "sender": user2_id, "type": "m.test"}],
-        )
-
-    def test_wait_for_new_data_timeout(self) -> None:
-        """
-        Test to make sure that the Sliding Sync request waits for new data to arrive but
-        no data ever arrives so we timeout. We're also making sure that the default data
-        from the To-Device extension doesn't trigger a false-positive for new data.
-        """
-        user1_id = self.register_user("user1", "pass")
-        user1_tok = self.login(user1_id, "pass")
-
-        sync_body = {
-            "lists": {},
-            "extensions": {
-                "to_device": {
-                    "enabled": True,
-                }
-            },
-        }
-        _, from_token = self.do_sync(sync_body, tok=user1_tok)
-
-        # Make the Sliding Sync request
-        channel = self.make_request(
-            "POST",
-            self.sync_endpoint + "?timeout=10000" + f"&pos={from_token}",
-            content=sync_body,
-            access_token=user1_tok,
-            await_result=False,
-        )
-        # Block for 5 seconds to make sure we are `notifier.wait_for_events(...)`
-        with self.assertRaises(TimedOutException):
-            channel.await_result(timeout_ms=5000)
-        # Wake-up `notifier.wait_for_events(...)` that will cause us test
-        # `SlidingSyncResult.__bool__` for new results.
-        self._bump_notifier_wait_for_events(
-            user1_id, wake_stream_key=StreamKeyType.ACCOUNT_DATA
-        )
-        # Block for a little bit more to ensure we don't see any new results.
-        with self.assertRaises(TimedOutException):
-            channel.await_result(timeout_ms=4000)
-        # Wait for the sync to complete (wait for the rest of the 10 second timeout,
-        # 5000 + 4000 + 1200 > 10000)
-        channel.await_result(timeout_ms=1200)
-        self.assertEqual(channel.code, 200, channel.json_body)
-
-        self._assert_to_device_response(channel.json_body, [])

tests/rest/client/sliding_sync/test_extension_typing.py

@@ -1,482 +0,0 @@
-#
-# This file is licensed under the Affero General Public License (AGPL) version 3.
-#
-# Copyright (C) 2024 New Vector, Ltd
-#
-# This program is free software: you can redistribute it and/or modify
-# it under the terms of the GNU Affero General Public License as
-# published by the Free Software Foundation, either version 3 of the
-# License, or (at your option) any later version.
-#
-# See the GNU Affero General Public License for more details:
-# <https://www.gnu.org/licenses/agpl-3.0.html>.
-#
-import logging
-
-from twisted.test.proto_helpers import MemoryReactor
-
-import synapse.rest.admin
-from synapse.api.constants import EduTypes
-from synapse.rest.client import login, room, sync
-from synapse.server import HomeServer
-from synapse.types import StreamKeyType
-from synapse.util import Clock
-
-from tests.rest.client.sliding_sync.test_sliding_sync import SlidingSyncBase
-from tests.server import TimedOutException
-
-logger = logging.getLogger(__name__)
-
-
-class SlidingSyncTypingExtensionTestCase(SlidingSyncBase):
-    """Tests for the typing notification sliding sync extension"""
-
-    servlets = [
-        synapse.rest.admin.register_servlets,
-        login.register_servlets,
-        room.register_servlets,
-        sync.register_servlets,
-    ]
-
-    def prepare(self, reactor: MemoryReactor, clock: Clock, hs: HomeServer) -> None:
-        self.store = hs.get_datastores().main
-
-    def test_no_data_initial_sync(self) -> None:
-        """
-        Test that enabling the typing extension works during an intitial sync,
-        even if there is no-data.
-        """
-        user1_id = self.register_user("user1", "pass")
-        user1_tok = self.login(user1_id, "pass")
-
-        # Make an initial Sliding Sync request with the typing extension enabled
-        sync_body = {
-            "lists": {},
-            "extensions": {
-                "typing": {
-                    "enabled": True,
-                }
-            },
-        }
-        response_body, _ = self.do_sync(sync_body, tok=user1_tok)
-
-        self.assertIncludes(
-            response_body["extensions"]["typing"].get("rooms").keys(),
-            set(),
-            exact=True,
-        )
-
-    def test_no_data_incremental_sync(self) -> None:
-        """
-        Test that enabling typing extension works during an incremental sync, even
-        if there is no-data.
-        """
-        user1_id = self.register_user("user1", "pass")
-        user1_tok = self.login(user1_id, "pass")
-
-        sync_body = {
-            "lists": {},
-            "extensions": {
-                "typing": {
-                    "enabled": True,
-                }
-            },
-        }
-        _, from_token = self.do_sync(sync_body, tok=user1_tok)
-
-        # Make an incremental Sliding Sync request with the typing extension enabled
-        response_body, _ = self.do_sync(sync_body, since=from_token, tok=user1_tok)
-
-        self.assertIncludes(
-            response_body["extensions"]["typing"].get("rooms").keys(),
-            set(),
-            exact=True,
-        )
-
-    def test_typing_initial_sync(self) -> None:
-        """
-        On initial sync, we return all typing notifications for rooms that we request
-        and are being returned in the Sliding Sync response.
-        """
-        user1_id = self.register_user("user1", "pass")
-        user1_tok = self.login(user1_id, "pass")
-        user2_id = self.register_user("user2", "pass")
-        user2_tok = self.login(user2_id, "pass")
-        user3_id = self.register_user("user3", "pass")
-        user3_tok = self.login(user3_id, "pass")
-        user4_id = self.register_user("user4", "pass")
-        user4_tok = self.login(user4_id, "pass")
-
-        # Create a room
-        room_id1 = self.helper.create_room_as(user2_id, tok=user2_tok)
-        self.helper.join(room_id1, user1_id, tok=user1_tok)
-        self.helper.join(room_id1, user3_id, tok=user3_tok)
-        self.helper.join(room_id1, user4_id, tok=user4_tok)
-        # User1 starts typing in room1
-        channel = self.make_request(
-            "PUT",
-            f"/rooms/{room_id1}/typing/{user1_id}",
-            b'{"typing": true, "timeout": 30000}',
-            access_token=user1_tok,
-        )
-        self.assertEqual(channel.code, 200, channel.json_body)
-        # User2 starts typing in room1
-        channel = self.make_request(
-            "PUT",
-            f"/rooms/{room_id1}/typing/{user2_id}",
-            b'{"typing": true, "timeout": 30000}',
-            access_token=user2_tok,
-        )
-        self.assertEqual(channel.code, 200, channel.json_body)
-
-        # Create another room
-        room_id2 = self.helper.create_room_as(user2_id, tok=user2_tok)
-        self.helper.join(room_id2, user1_id, tok=user1_tok)
-        self.helper.join(room_id2, user3_id, tok=user3_tok)
-        self.helper.join(room_id2, user4_id, tok=user4_tok)
-        # User1 starts typing in room2
-        channel = self.make_request(
-            "PUT",
-            f"/rooms/{room_id2}/typing/{user1_id}",
-            b'{"typing": true, "timeout": 30000}',
-            access_token=user1_tok,
-        )
-        self.assertEqual(channel.code, 200, channel.json_body)
-        # User2 starts typing in room2
-        channel = self.make_request(
-            "PUT",
-            f"/rooms/{room_id2}/typing/{user2_id}",
-            b'{"typing": true, "timeout": 30000}',
-            access_token=user2_tok,
-        )
-        self.assertEqual(channel.code, 200, channel.json_body)
-
-        # Make an initial Sliding Sync request with the typing extension enabled
-        sync_body = {
-            "lists": {},
-            "room_subscriptions": {
-                room_id1: {
-                    "required_state": [],
-                    "timeline_limit": 0,
-                }
-            },
-            "extensions": {
-                "typing": {
-                    "enabled": True,
-                    "rooms": [room_id1, room_id2],
-                }
-            },
-        }
-        response_body, _ = self.do_sync(sync_body, tok=user1_tok)
-
-        # Even though we requested room2, we only expect room1 to show up because that's
-        # the only room in the Sliding Sync response (room2 is not one of our room
-        # subscriptions or in a sliding window list).
-        self.assertIncludes(
-            response_body["extensions"]["typing"].get("rooms").keys(),
-            {room_id1},
-            exact=True,
-        )
-        # Sanity check that it's the correct ephemeral event type
-        self.assertEqual(
-            response_body["extensions"]["typing"]["rooms"][room_id1]["type"],
-            EduTypes.TYPING,
-        )
-        # We can see user1 and user2 typing
-        self.assertIncludes(
-            set(
-                response_body["extensions"]["typing"]["rooms"][room_id1]["content"][
-                    "user_ids"
-                ]
-            ),
-            {user1_id, user2_id},
-            exact=True,
-        )
-
-    def test_typing_incremental_sync(self) -> None:
-        """
-        On incremental sync, we return all typing notifications in the token range for a
-        given room but only for rooms that we request and are being returned in the
-        Sliding Sync response.
-        """
-
-        user1_id = self.register_user("user1", "pass")
-        user1_tok = self.login(user1_id, "pass")
-        user2_id = self.register_user("user2", "pass")
-        user2_tok = self.login(user2_id, "pass")
-        user3_id = self.register_user("user3", "pass")
-        user3_tok = self.login(user3_id, "pass")
-
-        # Create room1
-        room_id1 = self.helper.create_room_as(user2_id, tok=user2_tok)
-        self.helper.join(room_id1, user1_id, tok=user1_tok)
-        self.helper.join(room_id1, user3_id, tok=user3_tok)
-        # User2 starts typing in room1
-        channel = self.make_request(
-            "PUT",
-            f"/rooms/{room_id1}/typing/{user2_id}",
-            b'{"typing": true, "timeout": 30000}',
-            access_token=user2_tok,
-        )
-        self.assertEqual(channel.code, 200, channel.json_body)
-
-        # Create room2
-        room_id2 = self.helper.create_room_as(user2_id, tok=user2_tok)
-        self.helper.join(room_id2, user1_id, tok=user1_tok)
-        # User1 starts typing in room2 (before the `from_token`)
-        channel = self.make_request(
-            "PUT",
-            f"/rooms/{room_id2}/typing/{user1_id}",
-            b'{"typing": true, "timeout": 30000}',
-            access_token=user1_tok,
-        )
-        self.assertEqual(channel.code, 200, channel.json_body)
-
-        # Create room3
-        room_id3 = self.helper.create_room_as(user2_id, tok=user2_tok)
-        self.helper.join(room_id3, user1_id, tok=user1_tok)
-        self.helper.join(room_id3, user3_id, tok=user3_tok)
-
-        # Create room4
-        room_id4 = self.helper.create_room_as(user2_id, tok=user2_tok)
-        self.helper.join(room_id4, user1_id, tok=user1_tok)
-        self.helper.join(room_id4, user3_id, tok=user3_tok)
-        # User1 starts typing in room4 (before the `from_token`)
-        channel = self.make_request(
-            "PUT",
-            f"/rooms/{room_id4}/typing/{user1_id}",
-            b'{"typing": true, "timeout": 30000}',
-            access_token=user1_tok,
-        )
-        self.assertEqual(channel.code, 200, channel.json_body)
-
-        # Advance time so all of the typing notifications timeout before we make our
-        # Sliding Sync requests. Even though these are sent before the `from_token`, the
-        # typing code only keeps track of stream position of the latest typing
-        # notification so "old" typing notifications that are still "alive" (haven't
-        # timed out) can appear in the response.
-        self.reactor.advance(36)
-
-        sync_body = {
-            "lists": {},
-            "room_subscriptions": {
-                room_id1: {
-                    "required_state": [],
-                    "timeline_limit": 0,
-                },
-                room_id3: {
-                    "required_state": [],
-                    "timeline_limit": 0,
-                },
-                room_id4: {
-                    "required_state": [],
-                    "timeline_limit": 0,
-                },
-            },
-            "extensions": {
-                "typing": {
-                    "enabled": True,
-                    "rooms": [room_id1, room_id2, room_id3, room_id4],
-                }
-            },
-        }
-        _, from_token = self.do_sync(sync_body, tok=user1_tok)
-
-        # Add some more typing notifications after the `from_token`
-        #
-        # User1 starts typing in room1
-        channel = self.make_request(
-            "PUT",
-            f"/rooms/{room_id1}/typing/{user1_id}",
-            b'{"typing": true, "timeout": 30000}',
-            access_token=user1_tok,
-        )
-        self.assertEqual(channel.code, 200, channel.json_body)
-        # User1 starts typing in room2
-        channel = self.make_request(
-            "PUT",
-            f"/rooms/{room_id2}/typing/{user1_id}",
-            b'{"typing": true, "timeout": 30000}',
-            access_token=user1_tok,
-        )
-        self.assertEqual(channel.code, 200, channel.json_body)
-        # User3 starts typing in room3
-        channel = self.make_request(
-            "PUT",
-            f"/rooms/{room_id3}/typing/{user3_id}",
-            b'{"typing": true, "timeout": 30000}',
-            access_token=user3_tok,
-        )
-        self.assertEqual(channel.code, 200, channel.json_body)
-        # No activity for room4 after the `from_token`
-
-        # Make an incremental Sliding Sync request with the typing extension enabled
-        response_body, _ = self.do_sync(sync_body, since=from_token, tok=user1_tok)
-
-        # Even though we requested room2, we only expect rooms to show up if they are
-        # already in the Sliding Sync response. room4 doesn't show up because there is
-        # no activity after the `from_token`.
-        self.assertIncludes(
-            response_body["extensions"]["typing"].get("rooms").keys(),
-            {room_id1, room_id3},
-            exact=True,
-        )
-
-        # Check room1:
-        #
-        # Sanity check that it's the correct ephemeral event type
-        self.assertEqual(
-            response_body["extensions"]["typing"]["rooms"][room_id1]["type"],
-            EduTypes.TYPING,
-        )
-        # We only see that user1 is typing in room1 since the `from_token`
-        self.assertIncludes(
-            set(
-                response_body["extensions"]["typing"]["rooms"][room_id1]["content"][
-                    "user_ids"
-                ]
-            ),
-            {user1_id},
-            exact=True,
-        )
-
-        # Check room3:
-        #
-        # Sanity check that it's the correct ephemeral event type
-        self.assertEqual(
-            response_body["extensions"]["typing"]["rooms"][room_id3]["type"],
-            EduTypes.TYPING,
-        )
-        # We only see that user3 is typing in room1 since the `from_token`
-        self.assertIncludes(
-            set(
-                response_body["extensions"]["typing"]["rooms"][room_id3]["content"][
-                    "user_ids"
-                ]
-            ),
-            {user3_id},
-            exact=True,
-        )
-
-    def test_wait_for_new_data(self) -> None:
-        """
-        Test to make sure that the Sliding Sync request waits for new data to arrive.
-
-        (Only applies to incremental syncs with a `timeout` specified)
-        """
-        user1_id = self.register_user("user1", "pass")
-        user1_tok = self.login(user1_id, "pass")
-        user2_id = self.register_user("user2", "pass")
-        user2_tok = self.login(user2_id, "pass")
-
-        room_id = self.helper.create_room_as(user2_id, tok=user2_tok)
-        self.helper.join(room_id, user1_id, tok=user1_tok)
-
-        sync_body = {
-            "lists": {},
-            "room_subscriptions": {
-                room_id: {
-                    "required_state": [],
-                    "timeline_limit": 0,
-                },
-            },
-            "extensions": {
-                "typing": {
-                    "enabled": True,
-                    "rooms": [room_id],
-                }
-            },
-        }
-        _, from_token = self.do_sync(sync_body, tok=user1_tok)
-
-        # Make an incremental Sliding Sync request with the typing extension enabled
-        channel = self.make_request(
-            "POST",
-            self.sync_endpoint + f"?timeout=10000&pos={from_token}",
-            content=sync_body,
-            access_token=user1_tok,
-            await_result=False,
-        )
-        # Block for 5 seconds to make sure we are `notifier.wait_for_events(...)`
-        with self.assertRaises(TimedOutException):
-            channel.await_result(timeout_ms=5000)
-        # Bump the typing status to trigger new results
-        typing_channel = self.make_request(
-            "PUT",
-            f"/rooms/{room_id}/typing/{user2_id}",
-            b'{"typing": true, "timeout": 30000}',
-            access_token=user2_tok,
-        )
-        self.assertEqual(typing_channel.code, 200, typing_channel.json_body)
-        # Should respond before the 10 second timeout
-        channel.await_result(timeout_ms=3000)
-        self.assertEqual(channel.code, 200, channel.json_body)
-
-        # We should see the new typing notification
-        self.assertIncludes(
-            channel.json_body.get("extensions", {})
-            .get("typing", {})
-            .get("rooms", {})
-            .keys(),
-            {room_id},
-            exact=True,
-            message=str(channel.json_body),
-        )
-        self.assertIncludes(
-            set(
-                channel.json_body["extensions"]["typing"]["rooms"][room_id]["content"][
-                    "user_ids"
-                ]
-            ),
-            {user2_id},
-            exact=True,
-        )
-
-    def test_wait_for_new_data_timeout(self) -> None:
-        """
-        Test to make sure that the Sliding Sync request waits for new data to arrive but
-        no data ever arrives so we timeout. We're also making sure that the default data
-        from the typing extension doesn't trigger a false-positive for new data.
-        """
-        user1_id = self.register_user("user1", "pass")
-        user1_tok = self.login(user1_id, "pass")
-
-        sync_body = {
-            "lists": {},
-            "extensions": {
-                "typing": {
-                    "enabled": True,
-                }
-            },
-        }
-        _, from_token = self.do_sync(sync_body, tok=user1_tok)
-
-        # Make the Sliding Sync request
-        channel = self.make_request(
-            "POST",
-            self.sync_endpoint + f"?timeout=10000&pos={from_token}",
-            content=sync_body,
-            access_token=user1_tok,
-            await_result=False,
-        )
-        # Block for 5 seconds to make sure we are `notifier.wait_for_events(...)`
-        with self.assertRaises(TimedOutException):
-            channel.await_result(timeout_ms=5000)
-        # Wake-up `notifier.wait_for_events(...)` that will cause us test
-        # `SlidingSyncResult.__bool__` for new results.
-        self._bump_notifier_wait_for_events(
-            user1_id, wake_stream_key=StreamKeyType.ACCOUNT_DATA
-        )
-        # Block for a little bit more to ensure we don't see any new results.
-        with self.assertRaises(TimedOutException):
-            channel.await_result(timeout_ms=4000)
-        # Wait for the sync to complete (wait for the rest of the 10 second timeout,
-        # 5000 + 4000 + 1200 > 10000)
-        channel.await_result(timeout_ms=1200)
-        self.assertEqual(channel.code, 200, channel.json_body)
-
-        self.assertIncludes(
-            channel.json_body["extensions"]["typing"].get("rooms").keys(),
-            set(),
-            exact=True,
-        )

tests/rest/client/sliding_sync/test_extensions.py

@@ -1,283 +0,0 @@
-#
-# This file is licensed under the Affero General Public License (AGPL) version 3.
-#
-# Copyright (C) 2024 New Vector, Ltd
-#
-# This program is free software: you can redistribute it and/or modify
-# it under the terms of the GNU Affero General Public License as
-# published by the Free Software Foundation, either version 3 of the
-# License, or (at your option) any later version.
-#
-# See the GNU Affero General Public License for more details:
-# <https://www.gnu.org/licenses/agpl-3.0.html>.
-#
-import logging
-from typing import Literal
-
-from parameterized import parameterized
-from typing_extensions import assert_never
-
-from twisted.test.proto_helpers import MemoryReactor
-
-import synapse.rest.admin
-from synapse.api.constants import ReceiptTypes
-from synapse.rest.client import login, receipts, room, sync
-from synapse.server import HomeServer
-from synapse.util import Clock
-
-from tests.rest.client.sliding_sync.test_sliding_sync import SlidingSyncBase
-
-logger = logging.getLogger(__name__)
-
-
-class SlidingSyncExtensionsTestCase(SlidingSyncBase):
-    """
-    Test general extensions behavior in the Sliding Sync API. Each extension has their
-    own suite of tests in their own file as well.
-    """
-
-    servlets = [
-        synapse.rest.admin.register_servlets,
-        login.register_servlets,
-        room.register_servlets,
-        sync.register_servlets,
-        receipts.register_servlets,
-    ]
-
-    def prepare(self, reactor: MemoryReactor, clock: Clock, hs: HomeServer) -> None:
-        self.store = hs.get_datastores().main
-        self.storage_controllers = hs.get_storage_controllers()
-        self.account_data_handler = hs.get_account_data_handler()
-
-    # Any extensions that use `lists`/`rooms` should be tested here
-    @parameterized.expand([("account_data",), ("receipts",), ("typing",)])
-    def test_extensions_lists_rooms_relevant_rooms(
-        self,
-        extension_name: Literal["account_data", "receipts", "typing"],
-    ) -> None:
-        """
-        With various extensions, test out requesting different variations of
-        `lists`/`rooms`.
-
-        Stresses `SlidingSyncHandler.find_relevant_room_ids_for_extension(...)`
-        """
-        user1_id = self.register_user("user1", "pass")
-        user1_tok = self.login(user1_id, "pass")
-
-        # Create some rooms
-        room_id1 = self.helper.create_room_as(user1_id, tok=user1_tok)
-        room_id2 = self.helper.create_room_as(user1_id, tok=user1_tok)
-        room_id3 = self.helper.create_room_as(user1_id, tok=user1_tok)
-        room_id4 = self.helper.create_room_as(user1_id, tok=user1_tok)
-        room_id5 = self.helper.create_room_as(user1_id, tok=user1_tok)
-
-        room_id_to_human_name_map = {
-            room_id1: "room1",
-            room_id2: "room2",
-            room_id3: "room3",
-            room_id4: "room4",
-            room_id5: "room5",
-        }
-
-        for room_id in room_id_to_human_name_map.keys():
-            if extension_name == "account_data":
-                # Add some account data to each room
-                self.get_success(
-                    self.account_data_handler.add_account_data_to_room(
-                        user_id=user1_id,
-                        room_id=room_id,
-                        account_data_type="org.matrix.roorarraz",
-                        content={"roo": "rar"},
-                    )
-                )
-            elif extension_name == "receipts":
-                event_response = self.helper.send(
-                    room_id, body="new event", tok=user1_tok
-                )
-                # Read last event
-                channel = self.make_request(
-                    "POST",
-                    f"/rooms/{room_id}/receipt/{ReceiptTypes.READ}/{event_response['event_id']}",
-                    {},
-                    access_token=user1_tok,
-                )
-                self.assertEqual(channel.code, 200, channel.json_body)
-            elif extension_name == "typing":
-                # Start a typing notification
-                channel = self.make_request(
-                    "PUT",
-                    f"/rooms/{room_id}/typing/{user1_id}",
-                    b'{"typing": true, "timeout": 30000}',
-                    access_token=user1_tok,
-                )
-                self.assertEqual(channel.code, 200, channel.json_body)
-            else:
-                assert_never(extension_name)
-
-        main_sync_body = {
-            "lists": {
-                # We expect this list range to include room5 and room4
-                "foo-list": {
-                    "ranges": [[0, 1]],
-                    "required_state": [],
-                    "timeline_limit": 0,
-                },
-                # We expect this list range to include room5, room4, room3
-                "bar-list": {
-                    "ranges": [[0, 2]],
-                    "required_state": [],
-                    "timeline_limit": 0,
-                },
-            },
-            "room_subscriptions": {
-                room_id1: {
-                    "required_state": [],
-                    "timeline_limit": 0,
-                }
-            },
-        }
-
-        # Mix lists and rooms
-        sync_body = {
-            **main_sync_body,
-            "extensions": {
-                extension_name: {
-                    "enabled": True,
-                    "lists": ["foo-list", "non-existent-list"],
-                    "rooms": [room_id1, room_id2, "!non-existent-room"],
-                }
-            },
-        }
-        response_body, _ = self.do_sync(sync_body, tok=user1_tok)
-
-        # room1: ✅ Requested via `rooms` and a room subscription exists
-        # room2: ❌ Requested via `rooms` but not in the response (from lists or room subscriptions)
-        # room3: ❌ Not requested
-        # room4: ✅ Shows up because requested via `lists` and list exists in the response
-        # room5: ✅ Shows up because requested via `lists` and list exists in the response
-        self.assertIncludes(
-            {
-                room_id_to_human_name_map[room_id]
-                for room_id in response_body["extensions"][extension_name]
-                .get("rooms")
-                .keys()
-            },
-            {"room1", "room4", "room5"},
-            exact=True,
-        )
-
-        # Try wildcards (this is the default)
-        sync_body = {
-            **main_sync_body,
-            "extensions": {
-                extension_name: {
-                    "enabled": True,
-                    # "lists": ["*"],
-                    # "rooms": ["*"],
-                }
-            },
-        }
-        response_body, _ = self.do_sync(sync_body, tok=user1_tok)
-
-        # room1: ✅ Shows up because of default `rooms` wildcard and is in one of the room subscriptions
-        # room2: ❌ Not requested
-        # room3: ✅ Shows up because of default `lists` wildcard and is in a list
-        # room4: ✅ Shows up because of default `lists` wildcard and is in a list
-        # room5: ✅ Shows up because of default `lists` wildcard and is in a list
-        self.assertIncludes(
-            {
-                room_id_to_human_name_map[room_id]
-                for room_id in response_body["extensions"][extension_name]
-                .get("rooms")
-                .keys()
-            },
-            {"room1", "room3", "room4", "room5"},
-            exact=True,
-        )
-
-        # Empty list will return nothing
-        sync_body = {
-            **main_sync_body,
-            "extensions": {
-                extension_name: {
-                    "enabled": True,
-                    "lists": [],
-                    "rooms": [],
-                }
-            },
-        }
-        response_body, _ = self.do_sync(sync_body, tok=user1_tok)
-
-        # room1: ❌ Not requested
-        # room2: ❌ Not requested
-        # room3: ❌ Not requested
-        # room4: ❌ Not requested
-        # room5: ❌ Not requested
-        self.assertIncludes(
-            {
-                room_id_to_human_name_map[room_id]
-                for room_id in response_body["extensions"][extension_name]
-                .get("rooms")
-                .keys()
-            },
-            set(),
-            exact=True,
-        )
-
-        # Try wildcard and none
-        sync_body = {
-            **main_sync_body,
-            "extensions": {
-                extension_name: {
-                    "enabled": True,
-                    "lists": ["*"],
-                    "rooms": [],
-                }
-            },
-        }
-        response_body, _ = self.do_sync(sync_body, tok=user1_tok)
-
-        # room1: ❌ Not requested
-        # room2: ❌ Not requested
-        # room3: ✅ Shows up because of default `lists` wildcard and is in a list
-        # room4: ✅ Shows up because of default `lists` wildcard and is in a list
-        # room5: ✅ Shows up because of default `lists` wildcard and is in a list
-        self.assertIncludes(
-            {
-                room_id_to_human_name_map[room_id]
-                for room_id in response_body["extensions"][extension_name]
-                .get("rooms")
-                .keys()
-            },
-            {"room3", "room4", "room5"},
-            exact=True,
-        )
-
-        # Try requesting a room that is only in a list
-        sync_body = {
-            **main_sync_body,
-            "extensions": {
-                extension_name: {
-                    "enabled": True,
-                    "lists": [],
-                    "rooms": [room_id5],
-                }
-            },
-        }
-        response_body, _ = self.do_sync(sync_body, tok=user1_tok)
-
-        # room1: ❌ Not requested
-        # room2: ❌ Not requested
-        # room3: ❌ Not requested
-        # room4: ❌ Not requested
-        # room5: ✅ Requested via `rooms` and is in a list
-        self.assertIncludes(
-            {
-                room_id_to_human_name_map[room_id]
-                for room_id in response_body["extensions"][extension_name]
-                .get("rooms")
-                .keys()
-            },
-            {"room5"},
-            exact=True,
-        )

tests/rest/client/sliding_sync/test_room_subscriptions.py

@@ -1,285 +0,0 @@
-#
-# This file is licensed under the Affero General Public License (AGPL) version 3.
-#
-# Copyright (C) 2024 New Vector, Ltd
-#
-# This program is free software: you can redistribute it and/or modify
-# it under the terms of the GNU Affero General Public License as
-# published by the Free Software Foundation, either version 3 of the
-# License, or (at your option) any later version.
-#
-# See the GNU Affero General Public License for more details:
-# <https://www.gnu.org/licenses/agpl-3.0.html>.
-#
-import logging
-from http import HTTPStatus
-
-from twisted.test.proto_helpers import MemoryReactor
-
-import synapse.rest.admin
-from synapse.api.constants import EventTypes, HistoryVisibility
-from synapse.rest.client import login, room, sync
-from synapse.server import HomeServer
-from synapse.util import Clock
-
-from tests.rest.client.sliding_sync.test_sliding_sync import SlidingSyncBase
-
-logger = logging.getLogger(__name__)
-
-
-class SlidingSyncRoomSubscriptionsTestCase(SlidingSyncBase):
-    """
-    Test `room_subscriptions` in the Sliding Sync API.
-    """
-
-    servlets = [
-        synapse.rest.admin.register_servlets,
-        login.register_servlets,
-        room.register_servlets,
-        sync.register_servlets,
-    ]
-
-    def prepare(self, reactor: MemoryReactor, clock: Clock, hs: HomeServer) -> None:
-        self.store = hs.get_datastores().main
-        self.storage_controllers = hs.get_storage_controllers()
-
-    def test_room_subscriptions_with_join_membership(self) -> None:
-        """
-        Test `room_subscriptions` with a joined room should give us timeline and current
-        state events.
-        """
-        user1_id = self.register_user("user1", "pass")
-        user1_tok = self.login(user1_id, "pass")
-        user2_id = self.register_user("user2", "pass")
-        user2_tok = self.login(user2_id, "pass")
-
-        room_id1 = self.helper.create_room_as(user2_id, tok=user2_tok)
-        join_response = self.helper.join(room_id1, user1_id, tok=user1_tok)
-
-        # Make the Sliding Sync request with just the room subscription
-        sync_body = {
-            "room_subscriptions": {
-                room_id1: {
-                    "required_state": [
-                        [EventTypes.Create, ""],
-                    ],
-                    "timeline_limit": 1,
-                }
-            },
-        }
-        response_body, _ = self.do_sync(sync_body, tok=user1_tok)
-
-        state_map = self.get_success(
-            self.storage_controllers.state.get_current_state(room_id1)
-        )
-
-        # We should see some state
-        self._assertRequiredStateIncludes(
-            response_body["rooms"][room_id1]["required_state"],
-            {
-                state_map[(EventTypes.Create, "")],
-            },
-            exact=True,
-        )
-        self.assertIsNone(response_body["rooms"][room_id1].get("invite_state"))
-
-        # We should see some events
-        self.assertEqual(
-            [
-                event["event_id"]
-                for event in response_body["rooms"][room_id1]["timeline"]
-            ],
-            [
-                join_response["event_id"],
-            ],
-            response_body["rooms"][room_id1]["timeline"],
-        )
-        # No "live" events in an initial sync (no `from_token` to define the "live"
-        # range)
-        self.assertEqual(
-            response_body["rooms"][room_id1]["num_live"],
-            0,
-            response_body["rooms"][room_id1],
-        )
-        # There are more events to paginate to
-        self.assertEqual(
-            response_body["rooms"][room_id1]["limited"],
-            True,
-            response_body["rooms"][room_id1],
-        )
-
-    def test_room_subscriptions_with_leave_membership(self) -> None:
-        """
-        Test `room_subscriptions` with a leave room should give us timeline and state
-        events up to the leave event.
-        """
-        user1_id = self.register_user("user1", "pass")
-        user1_tok = self.login(user1_id, "pass")
-        user2_id = self.register_user("user2", "pass")
-        user2_tok = self.login(user2_id, "pass")
-
-        room_id1 = self.helper.create_room_as(user2_id, tok=user2_tok)
-        self.helper.send_state(
-            room_id1,
-            event_type="org.matrix.foo_state",
-            state_key="",
-            body={"foo": "bar"},
-            tok=user2_tok,
-        )
-
-        join_response = self.helper.join(room_id1, user1_id, tok=user1_tok)
-        leave_response = self.helper.leave(room_id1, user1_id, tok=user1_tok)
-
-        state_map = self.get_success(
-            self.storage_controllers.state.get_current_state(room_id1)
-        )
-
-        # Send some events after user1 leaves
-        self.helper.send(room_id1, "activity after leave", tok=user2_tok)
-        # Update state after user1 leaves
-        self.helper.send_state(
-            room_id1,
-            event_type="org.matrix.foo_state",
-            state_key="",
-            body={"foo": "qux"},
-            tok=user2_tok,
-        )
-
-        # Make the Sliding Sync request with just the room subscription
-        sync_body = {
-            "room_subscriptions": {
-                room_id1: {
-                    "required_state": [
-                        ["org.matrix.foo_state", ""],
-                    ],
-                    "timeline_limit": 2,
-                }
-            },
-        }
-        response_body, _ = self.do_sync(sync_body, tok=user1_tok)
-
-        # We should see the state at the time of the leave
-        self._assertRequiredStateIncludes(
-            response_body["rooms"][room_id1]["required_state"],
-            {
-                state_map[("org.matrix.foo_state", "")],
-            },
-            exact=True,
-        )
-        self.assertIsNone(response_body["rooms"][room_id1].get("invite_state"))
-
-        # We should see some before we left (nothing after)
-        self.assertEqual(
-            [
-                event["event_id"]
-                for event in response_body["rooms"][room_id1]["timeline"]
-            ],
-            [
-                join_response["event_id"],
-                leave_response["event_id"],
-            ],
-            response_body["rooms"][room_id1]["timeline"],
-        )
-        # No "live" events in an initial sync (no `from_token` to define the "live"
-        # range)
-        self.assertEqual(
-            response_body["rooms"][room_id1]["num_live"],
-            0,
-            response_body["rooms"][room_id1],
-        )
-        # There are more events to paginate to
-        self.assertEqual(
-            response_body["rooms"][room_id1]["limited"],
-            True,
-            response_body["rooms"][room_id1],
-        )
-
-    def test_room_subscriptions_no_leak_private_room(self) -> None:
-        """
-        Test `room_subscriptions` with a private room we have never been in should not
-        leak any data to the user.
-        """
-        user1_id = self.register_user("user1", "pass")
-        user1_tok = self.login(user1_id, "pass")
-        user2_id = self.register_user("user2", "pass")
-        user2_tok = self.login(user2_id, "pass")
-
-        room_id1 = self.helper.create_room_as(user2_id, tok=user2_tok, is_public=False)
-
-        # We should not be able to join the private room
-        self.helper.join(
-            room_id1, user1_id, tok=user1_tok, expect_code=HTTPStatus.FORBIDDEN
-        )
-
-        # Make the Sliding Sync request with just the room subscription
-        sync_body = {
-            "room_subscriptions": {
-                room_id1: {
-                    "required_state": [
-                        [EventTypes.Create, ""],
-                    ],
-                    "timeline_limit": 1,
-                }
-            },
-        }
-        response_body, _ = self.do_sync(sync_body, tok=user1_tok)
-
-        # We should not see the room at all (we're not in it)
-        self.assertIsNone(response_body["rooms"].get(room_id1), response_body["rooms"])
-
-    def test_room_subscriptions_world_readable(self) -> None:
-        """
-        Test `room_subscriptions` with a room that has `world_readable` history visibility
-
-        FIXME: We should be able to see the room timeline and state
-        """
-        user1_id = self.register_user("user1", "pass")
-        user1_tok = self.login(user1_id, "pass")
-        user2_id = self.register_user("user2", "pass")
-        user2_tok = self.login(user2_id, "pass")
-
-        # Create a room with `world_readable` history visibility
-        room_id1 = self.helper.create_room_as(
-            user2_id,
-            tok=user2_tok,
-            extra_content={
-                "preset": "public_chat",
-                "initial_state": [
-                    {
-                        "content": {
-                            "history_visibility": HistoryVisibility.WORLD_READABLE
-                        },
-                        "state_key": "",
-                        "type": EventTypes.RoomHistoryVisibility,
-                    }
-                ],
-            },
-        )
-        # Ensure we're testing with a room with `world_readable` history visibility
-        # which means events are visible to anyone even without membership.
-        history_visibility_response = self.helper.get_state(
-            room_id1, EventTypes.RoomHistoryVisibility, tok=user2_tok
-        )
-        self.assertEqual(
-            history_visibility_response.get("history_visibility"),
-            HistoryVisibility.WORLD_READABLE,
-        )
-
-        # Note: We never join the room
-
-        # Make the Sliding Sync request with just the room subscription
-        sync_body = {
-            "room_subscriptions": {
-                room_id1: {
-                    "required_state": [
-                        [EventTypes.Create, ""],
-                    ],
-                    "timeline_limit": 1,
-                }
-            },
-        }
-        response_body, _ = self.do_sync(sync_body, tok=user1_tok)
-
-        # FIXME: In the future, we should be able to see the room because it's
-        # `world_readable` but currently we don't support this.
-        self.assertIsNone(response_body["rooms"].get(room_id1), response_body["rooms"])

tests/rest/client/sliding_sync/test_rooms_invites.py

@@ -1,510 +0,0 @@
-#
-# This file is licensed under the Affero General Public License (AGPL) version 3.
-#
-# Copyright (C) 2024 New Vector, Ltd
-#
-# This program is free software: you can redistribute it and/or modify
-# it under the terms of the GNU Affero General Public License as
-# published by the Free Software Foundation, either version 3 of the
-# License, or (at your option) any later version.
-#
-# See the GNU Affero General Public License for more details:
-# <https://www.gnu.org/licenses/agpl-3.0.html>.
-#
-import logging
-
-from twisted.test.proto_helpers import MemoryReactor
-
-import synapse.rest.admin
-from synapse.api.constants import EventTypes, HistoryVisibility
-from synapse.rest.client import login, room, sync
-from synapse.server import HomeServer
-from synapse.types import UserID
-from synapse.util import Clock
-
-from tests.rest.client.sliding_sync.test_sliding_sync import SlidingSyncBase
-
-logger = logging.getLogger(__name__)
-
-
-class SlidingSyncRoomsInvitesTestCase(SlidingSyncBase):
-    """
-    Test to make sure the `rooms` response looks good for invites in the Sliding Sync API.
-
-    Invites behave a lot different than other rooms because we don't include the
-    `timeline` (`num_live`, `limited`, `prev_batch`) or `required_state` in favor of
-    some stripped state under the `invite_state` key.
-
-    Knocks probably have the same behavior but the spec doesn't mention knocks yet.
-    """
-
-    servlets = [
-        synapse.rest.admin.register_servlets,
-        login.register_servlets,
-        room.register_servlets,
-        sync.register_servlets,
-    ]
-
-    def prepare(self, reactor: MemoryReactor, clock: Clock, hs: HomeServer) -> None:
-        self.store = hs.get_datastores().main
-        self.storage_controllers = hs.get_storage_controllers()
-
-    def test_rooms_invite_shared_history_initial_sync(self) -> None:
-        """
-        Test that `rooms` we are invited to have some stripped `invite_state` during an
-        initial sync.
-
-        This is an `invite` room so we should only have `stripped_state` (no `timeline`)
-        but we also shouldn't see any timeline events because the history visiblity is
-        `shared` and we haven't joined the room yet.
-        """
-        user1_id = self.register_user("user1", "pass")
-        user1_tok = self.login(user1_id, "pass")
-        user1 = UserID.from_string(user1_id)
-        user2_id = self.register_user("user2", "pass")
-        user2_tok = self.login(user2_id, "pass")
-        user2 = UserID.from_string(user2_id)
-
-        room_id1 = self.helper.create_room_as(user2_id, tok=user2_tok)
-        # Ensure we're testing with a room with `shared` history visibility which means
-        # history visible until you actually join the room.
-        history_visibility_response = self.helper.get_state(
-            room_id1, EventTypes.RoomHistoryVisibility, tok=user2_tok
-        )
-        self.assertEqual(
-            history_visibility_response.get("history_visibility"),
-            HistoryVisibility.SHARED,
-        )
-
-        self.helper.send(room_id1, "activity before1", tok=user2_tok)
-        self.helper.send(room_id1, "activity before2", tok=user2_tok)
-        self.helper.invite(room_id1, src=user2_id, targ=user1_id, tok=user2_tok)
-        self.helper.send(room_id1, "activity after3", tok=user2_tok)
-        self.helper.send(room_id1, "activity after4", tok=user2_tok)
-
-        # Make the Sliding Sync request
-        sync_body = {
-            "lists": {
-                "foo-list": {
-                    "ranges": [[0, 1]],
-                    "required_state": [],
-                    "timeline_limit": 3,
-                }
-            }
-        }
-        response_body, _ = self.do_sync(sync_body, tok=user1_tok)
-
-        # `timeline` is omitted for `invite` rooms with `stripped_state`
-        self.assertIsNone(
-            response_body["rooms"][room_id1].get("timeline"),
-            response_body["rooms"][room_id1],
-        )
-        # `num_live` is omitted for `invite` rooms with `stripped_state` (no timeline anyway)
-        self.assertIsNone(
-            response_body["rooms"][room_id1].get("num_live"),
-            response_body["rooms"][room_id1],
-        )
-        # `limited` is omitted for `invite` rooms with `stripped_state` (no timeline anyway)
-        self.assertIsNone(
-            response_body["rooms"][room_id1].get("limited"),
-            response_body["rooms"][room_id1],
-        )
-        # `prev_batch` is omitted for `invite` rooms with `stripped_state` (no timeline anyway)
-        self.assertIsNone(
-            response_body["rooms"][room_id1].get("prev_batch"),
-            response_body["rooms"][room_id1],
-        )
-        # `required_state` is omitted for `invite` rooms with `stripped_state`
-        self.assertIsNone(
-            response_body["rooms"][room_id1].get("required_state"),
-            response_body["rooms"][room_id1],
-        )
-        # We should have some `stripped_state` so the potential joiner can identify the
-        # room (we don't care about the order).
-        self.assertCountEqual(
-            response_body["rooms"][room_id1]["invite_state"],
-            [
-                {
-                    "content": {"creator": user2_id, "room_version": "10"},
-                    "sender": user2_id,
-                    "state_key": "",
-                    "type": "m.room.create",
-                },
-                {
-                    "content": {"join_rule": "public"},
-                    "sender": user2_id,
-                    "state_key": "",
-                    "type": "m.room.join_rules",
-                },
-                {
-                    "content": {"displayname": user2.localpart, "membership": "join"},
-                    "sender": user2_id,
-                    "state_key": user2_id,
-                    "type": "m.room.member",
-                },
-                {
-                    "content": {"displayname": user1.localpart, "membership": "invite"},
-                    "sender": user2_id,
-                    "state_key": user1_id,
-                    "type": "m.room.member",
-                },
-            ],
-            response_body["rooms"][room_id1]["invite_state"],
-        )
-
-    def test_rooms_invite_shared_history_incremental_sync(self) -> None:
-        """
-        Test that `rooms` we are invited to have some stripped `invite_state` during an
-        incremental sync.
-
-        This is an `invite` room so we should only have `stripped_state` (no `timeline`)
-        but we also shouldn't see any timeline events because the history visiblity is
-        `shared` and we haven't joined the room yet.
-        """
-        user1_id = self.register_user("user1", "pass")
-        user1_tok = self.login(user1_id, "pass")
-        user1 = UserID.from_string(user1_id)
-        user2_id = self.register_user("user2", "pass")
-        user2_tok = self.login(user2_id, "pass")
-        user2 = UserID.from_string(user2_id)
-
-        room_id1 = self.helper.create_room_as(user2_id, tok=user2_tok)
-        # Ensure we're testing with a room with `shared` history visibility which means
-        # history visible until you actually join the room.
-        history_visibility_response = self.helper.get_state(
-            room_id1, EventTypes.RoomHistoryVisibility, tok=user2_tok
-        )
-        self.assertEqual(
-            history_visibility_response.get("history_visibility"),
-            HistoryVisibility.SHARED,
-        )
-
-        self.helper.send(room_id1, "activity before invite1", tok=user2_tok)
-        self.helper.send(room_id1, "activity before invite2", tok=user2_tok)
-        self.helper.invite(room_id1, src=user2_id, targ=user1_id, tok=user2_tok)
-        self.helper.send(room_id1, "activity after invite3", tok=user2_tok)
-        self.helper.send(room_id1, "activity after invite4", tok=user2_tok)
-
-        sync_body = {
-            "lists": {
-                "foo-list": {
-                    "ranges": [[0, 1]],
-                    "required_state": [],
-                    "timeline_limit": 3,
-                }
-            }
-        }
-        _, from_token = self.do_sync(sync_body, tok=user1_tok)
-
-        self.helper.send(room_id1, "activity after token5", tok=user2_tok)
-        self.helper.send(room_id1, "activity after toekn6", tok=user2_tok)
-
-        # Make the Sliding Sync request
-        response_body, from_token = self.do_sync(
-            sync_body, since=from_token, tok=user1_tok
-        )
-
-        # `timeline` is omitted for `invite` rooms with `stripped_state`
-        self.assertIsNone(
-            response_body["rooms"][room_id1].get("timeline"),
-            response_body["rooms"][room_id1],
-        )
-        # `num_live` is omitted for `invite` rooms with `stripped_state` (no timeline anyway)
-        self.assertIsNone(
-            response_body["rooms"][room_id1].get("num_live"),
-            response_body["rooms"][room_id1],
-        )
-        # `limited` is omitted for `invite` rooms with `stripped_state` (no timeline anyway)
-        self.assertIsNone(
-            response_body["rooms"][room_id1].get("limited"),
-            response_body["rooms"][room_id1],
-        )
-        # `prev_batch` is omitted for `invite` rooms with `stripped_state` (no timeline anyway)
-        self.assertIsNone(
-            response_body["rooms"][room_id1].get("prev_batch"),
-            response_body["rooms"][room_id1],
-        )
-        # `required_state` is omitted for `invite` rooms with `stripped_state`
-        self.assertIsNone(
-            response_body["rooms"][room_id1].get("required_state"),
-            response_body["rooms"][room_id1],
-        )
-        # We should have some `stripped_state` so the potential joiner can identify the
-        # room (we don't care about the order).
-        self.assertCountEqual(
-            response_body["rooms"][room_id1]["invite_state"],
-            [
-                {
-                    "content": {"creator": user2_id, "room_version": "10"},
-                    "sender": user2_id,
-                    "state_key": "",
-                    "type": "m.room.create",
-                },
-                {
-                    "content": {"join_rule": "public"},
-                    "sender": user2_id,
-                    "state_key": "",
-                    "type": "m.room.join_rules",
-                },
-                {
-                    "content": {"displayname": user2.localpart, "membership": "join"},
-                    "sender": user2_id,
-                    "state_key": user2_id,
-                    "type": "m.room.member",
-                },
-                {
-                    "content": {"displayname": user1.localpart, "membership": "invite"},
-                    "sender": user2_id,
-                    "state_key": user1_id,
-                    "type": "m.room.member",
-                },
-            ],
-            response_body["rooms"][room_id1]["invite_state"],
-        )
-
-    def test_rooms_invite_world_readable_history_initial_sync(self) -> None:
-        """
-        Test that `rooms` we are invited to have some stripped `invite_state` during an
-        initial sync.
-
-        This is an `invite` room so we should only have `stripped_state` (no `timeline`)
-        but depending on the semantics we decide, we could potentially see some
-        historical events before/after the `from_token` because the history is
-        `world_readable`. Same situation for events after the `from_token` if the
-        history visibility was set to `invited`.
-        """
-        user1_id = self.register_user("user1", "pass")
-        user1_tok = self.login(user1_id, "pass")
-        user1 = UserID.from_string(user1_id)
-        user2_id = self.register_user("user2", "pass")
-        user2_tok = self.login(user2_id, "pass")
-        user2 = UserID.from_string(user2_id)
-
-        room_id1 = self.helper.create_room_as(
-            user2_id,
-            tok=user2_tok,
-            extra_content={
-                "preset": "public_chat",
-                "initial_state": [
-                    {
-                        "content": {
-                            "history_visibility": HistoryVisibility.WORLD_READABLE
-                        },
-                        "state_key": "",
-                        "type": EventTypes.RoomHistoryVisibility,
-                    }
-                ],
-            },
-        )
-        # Ensure we're testing with a room with `world_readable` history visibility
-        # which means events are visible to anyone even without membership.
-        history_visibility_response = self.helper.get_state(
-            room_id1, EventTypes.RoomHistoryVisibility, tok=user2_tok
-        )
-        self.assertEqual(
-            history_visibility_response.get("history_visibility"),
-            HistoryVisibility.WORLD_READABLE,
-        )
-
-        self.helper.send(room_id1, "activity before1", tok=user2_tok)
-        self.helper.send(room_id1, "activity before2", tok=user2_tok)
-        self.helper.invite(room_id1, src=user2_id, targ=user1_id, tok=user2_tok)
-        self.helper.send(room_id1, "activity after3", tok=user2_tok)
-        self.helper.send(room_id1, "activity after4", tok=user2_tok)
-
-        # Make the Sliding Sync request
-        sync_body = {
-            "lists": {
-                "foo-list": {
-                    "ranges": [[0, 1]],
-                    "required_state": [],
-                    # Large enough to see the latest events and before the invite
-                    "timeline_limit": 4,
-                }
-            }
-        }
-        response_body, _ = self.do_sync(sync_body, tok=user1_tok)
-
-        # `timeline` is omitted for `invite` rooms with `stripped_state`
-        self.assertIsNone(
-            response_body["rooms"][room_id1].get("timeline"),
-            response_body["rooms"][room_id1],
-        )
-        # `num_live` is omitted for `invite` rooms with `stripped_state` (no timeline anyway)
-        self.assertIsNone(
-            response_body["rooms"][room_id1].get("num_live"),
-            response_body["rooms"][room_id1],
-        )
-        # `limited` is omitted for `invite` rooms with `stripped_state` (no timeline anyway)
-        self.assertIsNone(
-            response_body["rooms"][room_id1].get("limited"),
-            response_body["rooms"][room_id1],
-        )
-        # `prev_batch` is omitted for `invite` rooms with `stripped_state` (no timeline anyway)
-        self.assertIsNone(
-            response_body["rooms"][room_id1].get("prev_batch"),
-            response_body["rooms"][room_id1],
-        )
-        # `required_state` is omitted for `invite` rooms with `stripped_state`
-        self.assertIsNone(
-            response_body["rooms"][room_id1].get("required_state"),
-            response_body["rooms"][room_id1],
-        )
-        # We should have some `stripped_state` so the potential joiner can identify the
-        # room (we don't care about the order).
-        self.assertCountEqual(
-            response_body["rooms"][room_id1]["invite_state"],
-            [
-                {
-                    "content": {"creator": user2_id, "room_version": "10"},
-                    "sender": user2_id,
-                    "state_key": "",
-                    "type": "m.room.create",
-                },
-                {
-                    "content": {"join_rule": "public"},
-                    "sender": user2_id,
-                    "state_key": "",
-                    "type": "m.room.join_rules",
-                },
-                {
-                    "content": {"displayname": user2.localpart, "membership": "join"},
-                    "sender": user2_id,
-                    "state_key": user2_id,
-                    "type": "m.room.member",
-                },
-                {
-                    "content": {"displayname": user1.localpart, "membership": "invite"},
-                    "sender": user2_id,
-                    "state_key": user1_id,
-                    "type": "m.room.member",
-                },
-            ],
-            response_body["rooms"][room_id1]["invite_state"],
-        )
-
-    def test_rooms_invite_world_readable_history_incremental_sync(self) -> None:
-        """
-        Test that `rooms` we are invited to have some stripped `invite_state` during an
-        incremental sync.
-
-        This is an `invite` room so we should only have `stripped_state` (no `timeline`)
-        but depending on the semantics we decide, we could potentially see some
-        historical events before/after the `from_token` because the history is
-        `world_readable`. Same situation for events after the `from_token` if the
-        history visibility was set to `invited`.
-        """
-        user1_id = self.register_user("user1", "pass")
-        user1_tok = self.login(user1_id, "pass")
-        user1 = UserID.from_string(user1_id)
-        user2_id = self.register_user("user2", "pass")
-        user2_tok = self.login(user2_id, "pass")
-        user2 = UserID.from_string(user2_id)
-
-        room_id1 = self.helper.create_room_as(
-            user2_id,
-            tok=user2_tok,
-            extra_content={
-                "preset": "public_chat",
-                "initial_state": [
-                    {
-                        "content": {
-                            "history_visibility": HistoryVisibility.WORLD_READABLE
-                        },
-                        "state_key": "",
-                        "type": EventTypes.RoomHistoryVisibility,
-                    }
-                ],
-            },
-        )
-        # Ensure we're testing with a room with `world_readable` history visibility
-        # which means events are visible to anyone even without membership.
-        history_visibility_response = self.helper.get_state(
-            room_id1, EventTypes.RoomHistoryVisibility, tok=user2_tok
-        )
-        self.assertEqual(
-            history_visibility_response.get("history_visibility"),
-            HistoryVisibility.WORLD_READABLE,
-        )
-
-        self.helper.send(room_id1, "activity before invite1", tok=user2_tok)
-        self.helper.send(room_id1, "activity before invite2", tok=user2_tok)
-        self.helper.invite(room_id1, src=user2_id, targ=user1_id, tok=user2_tok)
-        self.helper.send(room_id1, "activity after invite3", tok=user2_tok)
-        self.helper.send(room_id1, "activity after invite4", tok=user2_tok)
-
-        sync_body = {
-            "lists": {
-                "foo-list": {
-                    "ranges": [[0, 1]],
-                    "required_state": [],
-                    # Large enough to see the latest events and before the invite
-                    "timeline_limit": 4,
-                }
-            }
-        }
-        _, from_token = self.do_sync(sync_body, tok=user1_tok)
-
-        self.helper.send(room_id1, "activity after token5", tok=user2_tok)
-        self.helper.send(room_id1, "activity after toekn6", tok=user2_tok)
-
-        # Make the incremental Sliding Sync request
-        response_body, _ = self.do_sync(sync_body, since=from_token, tok=user1_tok)
-
-        # `timeline` is omitted for `invite` rooms with `stripped_state`
-        self.assertIsNone(
-            response_body["rooms"][room_id1].get("timeline"),
-            response_body["rooms"][room_id1],
-        )
-        # `num_live` is omitted for `invite` rooms with `stripped_state` (no timeline anyway)
-        self.assertIsNone(
-            response_body["rooms"][room_id1].get("num_live"),
-            response_body["rooms"][room_id1],
-        )
-        # `limited` is omitted for `invite` rooms with `stripped_state` (no timeline anyway)
-        self.assertIsNone(
-            response_body["rooms"][room_id1].get("limited"),
-            response_body["rooms"][room_id1],
-        )
-        # `prev_batch` is omitted for `invite` rooms with `stripped_state` (no timeline anyway)
-        self.assertIsNone(
-            response_body["rooms"][room_id1].get("prev_batch"),
-            response_body["rooms"][room_id1],
-        )
-        # `required_state` is omitted for `invite` rooms with `stripped_state`
-        self.assertIsNone(
-            response_body["rooms"][room_id1].get("required_state"),
-            response_body["rooms"][room_id1],
-        )
-        # We should have some `stripped_state` so the potential joiner can identify the
-        # room (we don't care about the order).
-        self.assertCountEqual(
-            response_body["rooms"][room_id1]["invite_state"],
-            [
-                {
-                    "content": {"creator": user2_id, "room_version": "10"},
-                    "sender": user2_id,
-                    "state_key": "",
-                    "type": "m.room.create",
-                },
-                {
-                    "content": {"join_rule": "public"},
-                    "sender": user2_id,
-                    "state_key": "",
-                    "type": "m.room.join_rules",
-                },
-                {
-                    "content": {"displayname": user2.localpart, "membership": "join"},
-                    "sender": user2_id,
-                    "state_key": user2_id,
-                    "type": "m.room.member",
-                },
-                {
-                    "content": {"displayname": user1.localpart, "membership": "invite"},
-                    "sender": user2_id,
-                    "state_key": user1_id,
-                    "type": "m.room.member",
-                },
-            ],
-            response_body["rooms"][room_id1]["invite_state"],
-        )

tests/rest/client/sliding_sync/test_rooms_meta.py

@@ -1,710 +0,0 @@
-#
-# This file is licensed under the Affero General Public License (AGPL) version 3.
-#
-# Copyright (C) 2024 New Vector, Ltd
-#
-# This program is free software: you can redistribute it and/or modify
-# it under the terms of the GNU Affero General Public License as
-# published by the Free Software Foundation, either version 3 of the
-# License, or (at your option) any later version.
-#
-# See the GNU Affero General Public License for more details:
-# <https://www.gnu.org/licenses/agpl-3.0.html>.
-#
-import logging
-
-from twisted.test.proto_helpers import MemoryReactor
-
-import synapse.rest.admin
-from synapse.api.constants import EventTypes, Membership
-from synapse.api.room_versions import RoomVersions
-from synapse.rest.client import login, room, sync
-from synapse.server import HomeServer
-from synapse.util import Clock
-
-from tests.rest.client.sliding_sync.test_sliding_sync import SlidingSyncBase
-from tests.test_utils.event_injection import create_event
-
-logger = logging.getLogger(__name__)
-
-
-class SlidingSyncRoomsMetaTestCase(SlidingSyncBase):
-    """
-    Test rooms meta info like name, avatar, joined_count, invited_count, is_dm,
-    bump_stamp in the Sliding Sync API.
-    """
-
-    servlets = [
-        synapse.rest.admin.register_servlets,
-        login.register_servlets,
-        room.register_servlets,
-        sync.register_servlets,
-    ]
-
-    def prepare(self, reactor: MemoryReactor, clock: Clock, hs: HomeServer) -> None:
-        self.store = hs.get_datastores().main
-        self.storage_controllers = hs.get_storage_controllers()
-
-    def test_rooms_meta_when_joined(self) -> None:
-        """
-        Test that the `rooms` `name` and `avatar` are included in the response and
-        reflect the current state of the room when the user is joined to the room.
-        """
-        user1_id = self.register_user("user1", "pass")
-        user1_tok = self.login(user1_id, "pass")
-        user2_id = self.register_user("user2", "pass")
-        user2_tok = self.login(user2_id, "pass")
-
-        room_id1 = self.helper.create_room_as(
-            user2_id,
-            tok=user2_tok,
-            extra_content={
-                "name": "my super room",
-            },
-        )
-        # Set the room avatar URL
-        self.helper.send_state(
-            room_id1,
-            EventTypes.RoomAvatar,
-            {"url": "mxc://DUMMY_MEDIA_ID"},
-            tok=user2_tok,
-        )
-
-        self.helper.join(room_id1, user1_id, tok=user1_tok)
-
-        # Make the Sliding Sync request
-        sync_body = {
-            "lists": {
-                "foo-list": {
-                    "ranges": [[0, 1]],
-                    "required_state": [],
-                    "timeline_limit": 0,
-                }
-            }
-        }
-        response_body, _ = self.do_sync(sync_body, tok=user1_tok)
-
-        # Reflect the current state of the room
-        self.assertEqual(
-            response_body["rooms"][room_id1]["name"],
-            "my super room",
-            response_body["rooms"][room_id1],
-        )
-        self.assertEqual(
-            response_body["rooms"][room_id1]["avatar"],
-            "mxc://DUMMY_MEDIA_ID",
-            response_body["rooms"][room_id1],
-        )
-        self.assertEqual(
-            response_body["rooms"][room_id1]["joined_count"],
-            2,
-        )
-        self.assertEqual(
-            response_body["rooms"][room_id1]["invited_count"],
-            0,
-        )
-        self.assertIsNone(
-            response_body["rooms"][room_id1].get("is_dm"),
-        )
-
-    def test_rooms_meta_when_invited(self) -> None:
-        """
-        Test that the `rooms` `name` and `avatar` are included in the response and
-        reflect the current state of the room when the user is invited to the room.
-        """
-        user1_id = self.register_user("user1", "pass")
-        user1_tok = self.login(user1_id, "pass")
-        user2_id = self.register_user("user2", "pass")
-        user2_tok = self.login(user2_id, "pass")
-
-        room_id1 = self.helper.create_room_as(
-            user2_id,
-            tok=user2_tok,
-            extra_content={
-                "name": "my super room",
-            },
-        )
-        # Set the room avatar URL
-        self.helper.send_state(
-            room_id1,
-            EventTypes.RoomAvatar,
-            {"url": "mxc://DUMMY_MEDIA_ID"},
-            tok=user2_tok,
-        )
-
-        # User1 is invited to the room
-        self.helper.invite(room_id1, src=user2_id, targ=user1_id, tok=user2_tok)
-
-        # Update the room name after user1 has left
-        self.helper.send_state(
-            room_id1,
-            EventTypes.Name,
-            {"name": "my super duper room"},
-            tok=user2_tok,
-        )
-        # Update the room avatar URL after user1 has left
-        self.helper.send_state(
-            room_id1,
-            EventTypes.RoomAvatar,
-            {"url": "mxc://UPDATED_DUMMY_MEDIA_ID"},
-            tok=user2_tok,
-        )
-
-        # Make the Sliding Sync request
-        sync_body = {
-            "lists": {
-                "foo-list": {
-                    "ranges": [[0, 1]],
-                    "required_state": [],
-                    "timeline_limit": 0,
-                }
-            }
-        }
-        response_body, _ = self.do_sync(sync_body, tok=user1_tok)
-
-        # This should still reflect the current state of the room even when the user is
-        # invited.
-        self.assertEqual(
-            response_body["rooms"][room_id1]["name"],
-            "my super duper room",
-            response_body["rooms"][room_id1],
-        )
-        self.assertEqual(
-            response_body["rooms"][room_id1]["avatar"],
-            "mxc://UPDATED_DUMMY_MEDIA_ID",
-            response_body["rooms"][room_id1],
-        )
-        self.assertEqual(
-            response_body["rooms"][room_id1]["joined_count"],
-            1,
-        )
-        self.assertEqual(
-            response_body["rooms"][room_id1]["invited_count"],
-            1,
-        )
-        self.assertIsNone(
-            response_body["rooms"][room_id1].get("is_dm"),
-        )
-
-    def test_rooms_meta_when_banned(self) -> None:
-        """
-        Test that the `rooms` `name` and `avatar` reflect the state of the room when the
-        user was banned (do not leak current state).
-        """
-        user1_id = self.register_user("user1", "pass")
-        user1_tok = self.login(user1_id, "pass")
-        user2_id = self.register_user("user2", "pass")
-        user2_tok = self.login(user2_id, "pass")
-
-        room_id1 = self.helper.create_room_as(
-            user2_id,
-            tok=user2_tok,
-            extra_content={
-                "name": "my super room",
-            },
-        )
-        # Set the room avatar URL
-        self.helper.send_state(
-            room_id1,
-            EventTypes.RoomAvatar,
-            {"url": "mxc://DUMMY_MEDIA_ID"},
-            tok=user2_tok,
-        )
-
-        self.helper.join(room_id1, user1_id, tok=user1_tok)
-        self.helper.ban(room_id1, src=user2_id, targ=user1_id, tok=user2_tok)
-
-        # Update the room name after user1 has left
-        self.helper.send_state(
-            room_id1,
-            EventTypes.Name,
-            {"name": "my super duper room"},
-            tok=user2_tok,
-        )
-        # Update the room avatar URL after user1 has left
-        self.helper.send_state(
-            room_id1,
-            EventTypes.RoomAvatar,
-            {"url": "mxc://UPDATED_DUMMY_MEDIA_ID"},
-            tok=user2_tok,
-        )
-
-        # Make the Sliding Sync request
-        sync_body = {
-            "lists": {
-                "foo-list": {
-                    "ranges": [[0, 1]],
-                    "required_state": [],
-                    "timeline_limit": 0,
-                }
-            }
-        }
-        response_body, _ = self.do_sync(sync_body, tok=user1_tok)
-
-        # Reflect the state of the room at the time of leaving
-        self.assertEqual(
-            response_body["rooms"][room_id1]["name"],
-            "my super room",
-            response_body["rooms"][room_id1],
-        )
-        self.assertEqual(
-            response_body["rooms"][room_id1]["avatar"],
-            "mxc://DUMMY_MEDIA_ID",
-            response_body["rooms"][room_id1],
-        )
-        self.assertEqual(
-            response_body["rooms"][room_id1]["joined_count"],
-            # FIXME: The actual number should be "1" (user2) but we currently don't
-            # support this for rooms where the user has left/been banned.
-            0,
-        )
-        self.assertEqual(
-            response_body["rooms"][room_id1]["invited_count"],
-            0,
-        )
-        self.assertIsNone(
-            response_body["rooms"][room_id1].get("is_dm"),
-        )
-
-    def test_rooms_meta_heroes(self) -> None:
-        """
-        Test that the `rooms` `heroes` are included in the response when the room
-        doesn't have a room name set.
-        """
-        user1_id = self.register_user("user1", "pass")
-        user1_tok = self.login(user1_id, "pass")
-        user2_id = self.register_user("user2", "pass")
-        user2_tok = self.login(user2_id, "pass")
-        user3_id = self.register_user("user3", "pass")
-        _user3_tok = self.login(user3_id, "pass")
-
-        room_id1 = self.helper.create_room_as(
-            user2_id,
-            tok=user2_tok,
-            extra_content={
-                "name": "my super room",
-            },
-        )
-        self.helper.join(room_id1, user1_id, tok=user1_tok)
-        # User3 is invited
-        self.helper.invite(room_id1, src=user2_id, targ=user3_id, tok=user2_tok)
-
-        room_id2 = self.helper.create_room_as(
-            user2_id,
-            tok=user2_tok,
-            extra_content={
-                # No room name set so that `heroes` is populated
-                #
-                # "name": "my super room2",
-            },
-        )
-        self.helper.join(room_id2, user1_id, tok=user1_tok)
-        # User3 is invited
-        self.helper.invite(room_id2, src=user2_id, targ=user3_id, tok=user2_tok)
-
-        # Make the Sliding Sync request
-        sync_body = {
-            "lists": {
-                "foo-list": {
-                    "ranges": [[0, 1]],
-                    "required_state": [],
-                    "timeline_limit": 0,
-                }
-            }
-        }
-        response_body, _ = self.do_sync(sync_body, tok=user1_tok)
-
-        # Room1 has a name so we shouldn't see any `heroes` which the client would use
-        # the calculate the room name themselves.
-        self.assertEqual(
-            response_body["rooms"][room_id1]["name"],
-            "my super room",
-            response_body["rooms"][room_id1],
-        )
-        self.assertIsNone(response_body["rooms"][room_id1].get("heroes"))
-        self.assertEqual(
-            response_body["rooms"][room_id1]["joined_count"],
-            2,
-        )
-        self.assertEqual(
-            response_body["rooms"][room_id1]["invited_count"],
-            1,
-        )
-
-        # Room2 doesn't have a name so we should see `heroes` populated
-        self.assertIsNone(response_body["rooms"][room_id2].get("name"))
-        self.assertCountEqual(
-            [
-                hero["user_id"]
-                for hero in response_body["rooms"][room_id2].get("heroes", [])
-            ],
-            # Heroes shouldn't include the user themselves (we shouldn't see user1)
-            [user2_id, user3_id],
-        )
-        self.assertEqual(
-            response_body["rooms"][room_id2]["joined_count"],
-            2,
-        )
-        self.assertEqual(
-            response_body["rooms"][room_id2]["invited_count"],
-            1,
-        )
-
-        # We didn't request any state so we shouldn't see any `required_state`
-        self.assertIsNone(response_body["rooms"][room_id1].get("required_state"))
-        self.assertIsNone(response_body["rooms"][room_id2].get("required_state"))
-
-    def test_rooms_meta_heroes_max(self) -> None:
-        """
-        Test that the `rooms` `heroes` only includes the first 5 users (not including
-        yourself).
-        """
-        user1_id = self.register_user("user1", "pass")
-        user1_tok = self.login(user1_id, "pass")
-        user2_id = self.register_user("user2", "pass")
-        user2_tok = self.login(user2_id, "pass")
-        user3_id = self.register_user("user3", "pass")
-        user3_tok = self.login(user3_id, "pass")
-        user4_id = self.register_user("user4", "pass")
-        user4_tok = self.login(user4_id, "pass")
-        user5_id = self.register_user("user5", "pass")
-        user5_tok = self.login(user5_id, "pass")
-        user6_id = self.register_user("user6", "pass")
-        user6_tok = self.login(user6_id, "pass")
-        user7_id = self.register_user("user7", "pass")
-        user7_tok = self.login(user7_id, "pass")
-
-        room_id1 = self.helper.create_room_as(
-            user2_id,
-            tok=user2_tok,
-            extra_content={
-                # No room name set so that `heroes` is populated
-                #
-                # "name": "my super room",
-            },
-        )
-        self.helper.join(room_id1, user1_id, tok=user1_tok)
-        self.helper.join(room_id1, user3_id, tok=user3_tok)
-        self.helper.join(room_id1, user4_id, tok=user4_tok)
-        self.helper.join(room_id1, user5_id, tok=user5_tok)
-        self.helper.join(room_id1, user6_id, tok=user6_tok)
-        self.helper.join(room_id1, user7_id, tok=user7_tok)
-
-        # Make the Sliding Sync request
-        sync_body = {
-            "lists": {
-                "foo-list": {
-                    "ranges": [[0, 1]],
-                    "required_state": [],
-                    "timeline_limit": 0,
-                }
-            }
-        }
-        response_body, _ = self.do_sync(sync_body, tok=user1_tok)
-
-        # Room2 doesn't have a name so we should see `heroes` populated
-        self.assertIsNone(response_body["rooms"][room_id1].get("name"))
-        self.assertCountEqual(
-            [
-                hero["user_id"]
-                for hero in response_body["rooms"][room_id1].get("heroes", [])
-            ],
-            # Heroes should be the first 5 users in the room (excluding the user
-            # themselves, we shouldn't see `user1`)
-            [user2_id, user3_id, user4_id, user5_id, user6_id],
-        )
-        self.assertEqual(
-            response_body["rooms"][room_id1]["joined_count"],
-            7,
-        )
-        self.assertEqual(
-            response_body["rooms"][room_id1]["invited_count"],
-            0,
-        )
-
-        # We didn't request any state so we shouldn't see any `required_state`
-        self.assertIsNone(response_body["rooms"][room_id1].get("required_state"))
-
-    def test_rooms_meta_heroes_when_banned(self) -> None:
-        """
-        Test that the `rooms` `heroes` are included in the response when the room
-        doesn't have a room name set but doesn't leak information past their ban.
-        """
-        user1_id = self.register_user("user1", "pass")
-        user1_tok = self.login(user1_id, "pass")
-        user2_id = self.register_user("user2", "pass")
-        user2_tok = self.login(user2_id, "pass")
-        user3_id = self.register_user("user3", "pass")
-        _user3_tok = self.login(user3_id, "pass")
-        user4_id = self.register_user("user4", "pass")
-        user4_tok = self.login(user4_id, "pass")
-        user5_id = self.register_user("user5", "pass")
-        _user5_tok = self.login(user5_id, "pass")
-
-        room_id1 = self.helper.create_room_as(
-            user2_id,
-            tok=user2_tok,
-            extra_content={
-                # No room name set so that `heroes` is populated
-                #
-                # "name": "my super room",
-            },
-        )
-        # User1 joins the room
-        self.helper.join(room_id1, user1_id, tok=user1_tok)
-        # User3 is invited
-        self.helper.invite(room_id1, src=user2_id, targ=user3_id, tok=user2_tok)
-
-        # User1 is banned from the room
-        self.helper.ban(room_id1, src=user2_id, targ=user1_id, tok=user2_tok)
-
-        # User4 joins the room after user1 is banned
-        self.helper.join(room_id1, user4_id, tok=user4_tok)
-        # User5 is invited after user1 is banned
-        self.helper.invite(room_id1, src=user2_id, targ=user5_id, tok=user2_tok)
-
-        # Make the Sliding Sync request
-        sync_body = {
-            "lists": {
-                "foo-list": {
-                    "ranges": [[0, 1]],
-                    "required_state": [],
-                    "timeline_limit": 0,
-                }
-            }
-        }
-        response_body, _ = self.do_sync(sync_body, tok=user1_tok)
-
-        # Room2 doesn't have a name so we should see `heroes` populated
-        self.assertIsNone(response_body["rooms"][room_id1].get("name"))
-        self.assertCountEqual(
-            [
-                hero["user_id"]
-                for hero in response_body["rooms"][room_id1].get("heroes", [])
-            ],
-            # Heroes shouldn't include the user themselves (we shouldn't see user1). We
-            # also shouldn't see user4 since they joined after user1 was banned.
-            #
-            # FIXME: The actual result should be `[user2_id, user3_id]` but we currently
-            # don't support this for rooms where the user has left/been banned.
-            [],
-        )
-
-        self.assertEqual(
-            response_body["rooms"][room_id1]["joined_count"],
-            # FIXME: The actual number should be "1" (user2) but we currently don't
-            # support this for rooms where the user has left/been banned.
-            0,
-        )
-        self.assertEqual(
-            response_body["rooms"][room_id1]["invited_count"],
-            # We shouldn't see user5 since they were invited after user1 was banned.
-            #
-            # FIXME: The actual number should be "1" (user3) but we currently don't
-            # support this for rooms where the user has left/been banned.
-            0,
-        )
-
-    def test_rooms_bump_stamp(self) -> None:
-        """
-        Test that `bump_stamp` is present and pointing to relevant events.
-        """
-        user1_id = self.register_user("user1", "pass")
-        user1_tok = self.login(user1_id, "pass")
-
-        room_id1 = self.helper.create_room_as(
-            user1_id,
-            tok=user1_tok,
-        )
-        event_response1 = message_response = self.helper.send(
-            room_id1, "message in room1", tok=user1_tok
-        )
-        event_pos1 = self.get_success(
-            self.store.get_position_for_event(event_response1["event_id"])
-        )
-        room_id2 = self.helper.create_room_as(
-            user1_id,
-            tok=user1_tok,
-        )
-        send_response2 = self.helper.send(room_id2, "message in room2", tok=user1_tok)
-        event_pos2 = self.get_success(
-            self.store.get_position_for_event(send_response2["event_id"])
-        )
-
-        # Send a reaction in room1 but it shouldn't affect the `bump_stamp`
-        # because reactions are not part of the `DEFAULT_BUMP_EVENT_TYPES`
-        self.helper.send_event(
-            room_id1,
-            type=EventTypes.Reaction,
-            content={
-                "m.relates_to": {
-                    "event_id": message_response["event_id"],
-                    "key": "👍",
-                    "rel_type": "m.annotation",
-                }
-            },
-            tok=user1_tok,
-        )
-
-        # Make the Sliding Sync request
-        sync_body = {
-            "lists": {
-                "foo-list": {
-                    "ranges": [[0, 1]],
-                    "required_state": [],
-                    "timeline_limit": 100,
-                }
-            }
-        }
-        response_body, _ = self.do_sync(sync_body, tok=user1_tok)
-
-        # Make sure it has the foo-list we requested
-        self.assertListEqual(
-            list(response_body["lists"].keys()),
-            ["foo-list"],
-            response_body["lists"].keys(),
-        )
-
-        # Make sure the list includes the rooms in the right order
-        self.assertListEqual(
-            list(response_body["lists"]["foo-list"]["ops"]),
-            [
-                {
-                    "op": "SYNC",
-                    "range": [0, 1],
-                    # room1 sorts before room2 because it has the latest event (the
-                    # reaction)
-                    "room_ids": [room_id1, room_id2],
-                }
-            ],
-            response_body["lists"]["foo-list"],
-        )
-
-        # The `bump_stamp` for room1 should point at the latest message (not the
-        # reaction since it's not one of the `DEFAULT_BUMP_EVENT_TYPES`)
-        self.assertEqual(
-            response_body["rooms"][room_id1]["bump_stamp"],
-            event_pos1.stream,
-            response_body["rooms"][room_id1],
-        )
-
-        # The `bump_stamp` for room2 should point at the latest message
-        self.assertEqual(
-            response_body["rooms"][room_id2]["bump_stamp"],
-            event_pos2.stream,
-            response_body["rooms"][room_id2],
-        )
-
-    def test_rooms_bump_stamp_backfill(self) -> None:
-        """
-        Test that `bump_stamp` ignores backfilled events, i.e. events with a
-        negative stream ordering.
-        """
-
-        user1_id = self.register_user("user1", "pass")
-        user1_tok = self.login(user1_id, "pass")
-
-        # Create a remote room
-        creator = "@user:other"
-        room_id = "!foo:other"
-        shared_kwargs = {
-            "room_id": room_id,
-            "room_version": "10",
-        }
-
-        create_tuple = self.get_success(
-            create_event(
-                self.hs,
-                prev_event_ids=[],
-                type=EventTypes.Create,
-                state_key="",
-                sender=creator,
-                **shared_kwargs,
-            )
-        )
-        creator_tuple = self.get_success(
-            create_event(
-                self.hs,
-                prev_event_ids=[create_tuple[0].event_id],
-                auth_event_ids=[create_tuple[0].event_id],
-                type=EventTypes.Member,
-                state_key=creator,
-                content={"membership": Membership.JOIN},
-                sender=creator,
-                **shared_kwargs,
-            )
-        )
-        # We add a message event as a valid "bump type"
-        msg_tuple = self.get_success(
-            create_event(
-                self.hs,
-                prev_event_ids=[creator_tuple[0].event_id],
-                auth_event_ids=[create_tuple[0].event_id],
-                type=EventTypes.Message,
-                content={"body": "foo", "msgtype": "m.text"},
-                sender=creator,
-                **shared_kwargs,
-            )
-        )
-        invite_tuple = self.get_success(
-            create_event(
-                self.hs,
-                prev_event_ids=[msg_tuple[0].event_id],
-                auth_event_ids=[create_tuple[0].event_id, creator_tuple[0].event_id],
-                type=EventTypes.Member,
-                state_key=user1_id,
-                content={"membership": Membership.INVITE},
-                sender=creator,
-                **shared_kwargs,
-            )
-        )
-
-        remote_events_and_contexts = [
-            create_tuple,
-            creator_tuple,
-            msg_tuple,
-            invite_tuple,
-        ]
-
-        # Ensure the local HS knows the room version
-        self.get_success(
-            self.store.store_room(room_id, creator, False, RoomVersions.V10)
-        )
-
-        # Persist these events as backfilled events.
-        persistence = self.hs.get_storage_controllers().persistence
-        assert persistence is not None
-
-        for event, context in remote_events_and_contexts:
-            self.get_success(persistence.persist_event(event, context, backfilled=True))
-
-        # Now we join the local user to the room
-        join_tuple = self.get_success(
-            create_event(
-                self.hs,
-                prev_event_ids=[invite_tuple[0].event_id],
-                auth_event_ids=[create_tuple[0].event_id, invite_tuple[0].event_id],
-                type=EventTypes.Member,
-                state_key=user1_id,
-                content={"membership": Membership.JOIN},
-                sender=user1_id,
-                **shared_kwargs,
-            )
-        )
-        self.get_success(persistence.persist_event(*join_tuple))
-
-        # Doing an SS request should return a positive `bump_stamp`, even though
-        # the only event that matches the bump types has as negative stream
-        # ordering.
-        sync_body = {
-            "lists": {
-                "foo-list": {
-                    "ranges": [[0, 1]],
-                    "required_state": [],
-                    "timeline_limit": 5,
-                }
-            }
-        }
-        response_body, _ = self.do_sync(sync_body, tok=user1_tok)
-
-        self.assertGreater(response_body["rooms"][room_id]["bump_stamp"], 0)