Add security release notes

This PR reverts https://github.com/element-hq/synapse/pull/18751

### Why revert?

@reivilibre
[found](https://matrix.to/#/!vcyiEtMVHIhWXcJAfl:sw1v.org/$u9OEmMxaFYUzWHhCk1A_r50Y0aGrtKEhepF7WxWJkUA?via=matrix.org&via=node.marinchik.ink&via=element.io)
that our CI was failing in bizarre ways (thanks for stepping up to dive
into this 🙇). Examples:

- `twisted.internet.error.ProcessTerminated: A process has ended with a
probable error condition: process ended by signal 9.`
- `twisted.internet.error.ProcessTerminated: A process has ended with a
probable error condition: process ended by signal 15.`

<details>
<summary>More detailed part of the log</summary>


https://github.com/element-hq/synapse/actions/runs/16758038107/job/47500520633#step:9:6809
```
tests.util.test_wheel_timer.WheelTimerTestCase.test_single_insert_fetch
===============================================================================
Error: 
Traceback (most recent call last):
  File "/home/runner/.cache/pypoetry/virtualenvs/matrix-synapse-pswDeSvb-py3.9/lib/python3.9/site-packages/twisted/trial/_dist/disttrial.py", line 371, in task
    await worker.run(case, result)
  File "/home/runner/.cache/pypoetry/virtualenvs/matrix-synapse-pswDeSvb-py3.9/lib/python3.9/site-packages/twisted/trial/_dist/worker.py", line 305, in run
    return await self.callRemote(workercommands.Run, testCase=testCaseId)  # type: ignore[no-any-return]
  File "/home/runner/.cache/pypoetry/virtualenvs/matrix-synapse-pswDeSvb-py3.9/lib/python3.9/site-packages/twisted/internet/defer.py", line 1187, in __iter__
    yield self
  File "/home/runner/.cache/pypoetry/virtualenvs/matrix-synapse-pswDeSvb-py3.9/lib/python3.9/site-packages/twisted/internet/defer.py", line 1092, in _runCallbacks
    current.result = callback(  # type: ignore[misc]
  File "/home/runner/.cache/pypoetry/virtualenvs/matrix-synapse-pswDeSvb-py3.9/lib/python3.9/site-packages/twisted/protocols/amp.py", line 1968, in _massageError
    error.trap(RemoteAmpError)
  File "/home/runner/.cache/pypoetry/virtualenvs/matrix-synapse-pswDeSvb-py3.9/lib/python3.9/site-packages/twisted/python/failure.py", line 431, in trap
    self.raiseException()
  File "/home/runner/.cache/pypoetry/virtualenvs/matrix-synapse-pswDeSvb-py3.9/lib/python3.9/site-packages/twisted/python/failure.py", line 455, in raiseException
    raise self.value.with_traceback(self.tb)
twisted.internet.error.ProcessTerminated: A process has ended with a probable error condition: process ended by signal 9.

tests.util.test_macaroons.MacaroonGeneratorTestCase.test_guest_access_token
-------------------------------------------------------------------------------
Ran 4325 tests in 669.321s

FAILED (skips=159, errors=62, successes=4108)
while calling from thread
Traceback (most recent call last):
  File "/home/runner/.cache/pypoetry/virtualenvs/matrix-synapse-pswDeSvb-py3.9/lib/python3.9/site-packages/twisted/internet/base.py", line 1064, in runUntilCurrent
    f(*a, **kw)
  File "/home/runner/.cache/pypoetry/virtualenvs/matrix-synapse-pswDeSvb-py3.9/lib/python3.9/site-packages/twisted/internet/base.py", line 790, in stop
    raise error.ReactorNotRunning("Can't stop reactor that isn't running.")
twisted.internet.error.ReactorNotRunning: Can't stop reactor that isn't running.

joining disttrial worker #0 failed
Traceback (most recent call last):
  File "/home/runner/.cache/pypoetry/virtualenvs/matrix-synapse-pswDeSvb-py3.9/lib/python3.9/site-packages/twisted/internet/defer.py", line 1853, in _inlineCallbacks
    result = context.run(
  File "/home/runner/.cache/pypoetry/virtualenvs/matrix-synapse-pswDeSvb-py3.9/lib/python3.9/site-packages/twisted/python/failure.py", line 467, in throwExceptionIntoGenerator
    return g.throw(self.value.with_traceback(self.tb))
  File "/home/runner/.cache/pypoetry/virtualenvs/matrix-synapse-pswDeSvb-py3.9/lib/python3.9/site-packages/twisted/trial/_dist/worker.py", line 406, in exit
    await endDeferred
  File "/home/runner/.cache/pypoetry/virtualenvs/matrix-synapse-pswDeSvb-py3.9/lib/python3.9/site-packages/twisted/internet/defer.py", line 1187, in __iter__
    yield self
twisted.internet.error.ProcessTerminated: A process has ended with a probable error condition: process ended by signal 15.
```

</details>


With more debugging (thanks @devonh for also stepping in as maintainer),
we were finding that the CI was consistently failing at
`test_exposed_to_prometheus` which was a bit of smoke because of all of
the [metrics
changes](https://github.com/element-hq/synapse/issues/18592) that were
merged recently.

Locally, although I wasn't able to reproduce the bizarre errors, I could
easily see increased memory usage (~20GB vs ~2GB) and the
`test_exposed_to_prometheus` test taking a while to complete when
running a full test run (`SYNAPSE_TEST_LOG_LEVEL=INFO poetry run trial
tests`).

<img width="1485" height="78" alt="Lots of memory usage"
src="https://github.com/user-attachments/assets/811e2a96-75e5-4a3c-966c-00dc0512cea9"
/>

After updating `test_exposed_to_prometheus` to dump the
`latest_metrics_response = generate_latest(REGISTRY)`, I could see that
it's a massive 3.2GB response. Inspecting the contents, we can see 4.1M
(4,137,123) entries for just
`synapse_background_update_status{server_name="test"} 3.0` which is a
`LaterGauge`. I don't think we have 4.1M test cases so it's also unclear
why we end up with so many samples but it does make sense that we do see
a lot of duplicates because each `HomeserverTestCase` will create a
homeserver for each test case that will `LaterGauge.register_hook(...)`
(part of the https://github.com/element-hq/synapse/pull/18751 changes).

`tests/storage/databases/main/test_metrics.py`
```python
        latest_metrics_response = generate_latest(REGISTRY)
        with open("/tmp/synapse-test-metrics", "wb") as f:
            f.write(latest_metrics_response)
```

After reverting the https://github.com/element-hq/synapse/pull/18751
changes, running the full test suite locally doesn't result in memory
spikes and seems to run normally.



### Dev notes

Discussion in the
[`#synapse-dev:matrix.org`](https://matrix.to/#/!vcyiEtMVHIhWXcJAfl:sw1v.org/$vkMATs04yqZggVVd6Noop5nU8M2DVoTkrAWshw7u1-w?via=matrix.org&via=node.marinchik.ink&via=element.io)
room.

### Pull Request Checklist

<!-- Please read
https://element-hq.github.io/synapse/latest/development/contributing_guide.html
before submitting your pull request -->

* [x] Pull request is based on the develop branch
* [ ] Pull request includes a [changelog
file](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#changelog).
The entry should:
- Be a short description of your change which makes sense to users.
"Fixed a bug that prevented receiving messages from other servers."
instead of "Moved X method from `EventStore` to `EventWorkerStore`.".
  - Use markdown where necessary, mostly for `code blocks`.
  - End with either a period (.) or an exclamation mark (!).
  - Start with a capital letter.
- Feel free to credit yourself, by adding a sentence "Contributed by
@github_username." or "Contributed by [Your Name]." to the end of the
entry.
* [ ] [Code
style](https://element-hq.github.io/synapse/latest/code_style.html) is
correct (run the
[linters](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#run-the-linters))

.github/workflows/triage_labelled.yml

@@ -16,6 +16,10 @@ jobs:
         with:
           project-url: "https://github.com/orgs/matrix-org/projects/67"
           github-token: ${{ secrets.ELEMENT_BOT_TOKEN }}
+        # This action will error if the issue already exists on the project. Which is
+        # common as `X-Needs-Info` will often be added to issues that are already in
+        # the triage queue. Prevent the whole job from failing in this case.
+        continue-on-error: true
       - name: Set status
         env:
           GITHUB_TOKEN: ${{ secrets.ELEMENT_BOT_TOKEN }}

CHANGES.md

@@ -1,3 +1,110 @@
+# Synapse 1.136.0 (2025-08-12)
+
+Note: This release includes the security fixes from `1.135.2` and `1.136.0rc2`, detailed below.
+
+### Bugfixes
+
+- Fix bug introduced in 1.135.2 and 1.136.0rc2 where the [Make Room Admin API](https://element-hq.github.io/synapse/latest/admin_api/rooms.html#make-room-admin-api) would not treat a room v12's creator power level as the highest in room. ([\#18805](https://github.com/element-hq/synapse/issues/18805))
+
+
+
+
+# Synapse 1.136.0rc2 (2025-08-11)
+
+This is the Synapse portion of the [Matrix coordinated security release](https://matrix.org/blog/2025/07/security-predisclosure/). This release includes support for [room version](https://spec.matrix.org/v1.15/rooms/) 12 which fixes a number of security vulnerabilities, including [CVE-2025-49090](https://www.cve.org/CVERecord?id=CVE-2025-49090).
+
+The default room version is not changed. Not all clients will support room version 12 immediately, and not all users will be using the latest version of their clients. Large, public rooms are advised to wait a few weeks before upgrading to room version 12 to allow users throughout the Matrix ecosystem to update their clients.
+
+Note: release 1.135.1 was skipped due to issues discovered during the release process.
+
+Two patched Synapse releases are now available:
+
+* `1.135.2`: stable release comprised of `1.135.0` + security patches
+    * Upgrade to this release **if you are currently running 1.135.0 or below**.
+* `1.136.0rc2`: unstable release candidate comprised of `1.136.0rc1` + security patches.
+    * Upgrade to this release **only if you are on 1.136.0rc1**.
+
+### Bugfixes
+
+- Update MSC4293 redaction logic for room v12. ([\#80](https://github.com/element-hq/synapse/issues/80))
+
+### Internal Changes
+
+- Add a parameter to `upgrade_rooms(..)` to allow auto join local users. ([\#83](https://github.com/element-hq/synapse/issues/83))
+
+
+
+
+# Synapse 1.136.0rc1 (2025-08-05)
+
+Please check [the relevant section in the upgrade notes](https://github.com/element-hq/synapse/blob/develop/docs/upgrade.md#upgrading-to-v11360) as this release contains changes to MAS support, metrics labels and the module API which may require your attention when upgrading.
+
+### Features
+
+- Add configurable rate limiting for the creation of rooms. ([\#18514](https://github.com/element-hq/synapse/issues/18514))
+- Add support for [MSC4293](https://github.com/matrix-org/matrix-spec-proposals/pull/4293) - Redact on Kick/Ban. ([\#18540](https://github.com/element-hq/synapse/issues/18540))
+- When admins enable themselves to see soft-failed events, they will also see if the cause is due to the policy server flagging them as spam via `unsigned`. ([\#18585](https://github.com/element-hq/synapse/issues/18585))
+- Add ability to configure forward/outbound proxy via homeserver config instead of environment variables. See `http_proxy`, `https_proxy`, `no_proxy_hosts`. ([\#18686](https://github.com/element-hq/synapse/issues/18686))
+- Advertise experimental support for [MSC4306](https://github.com/matrix-org/matrix-spec-proposals/pull/4306) (Thread Subscriptions) through `/_matrix/clients/versions` if enabled. ([\#18722](https://github.com/element-hq/synapse/issues/18722))
+- Stabilise support for delegating authentication to [Matrix Authentication Service](https://github.com/element-hq/matrix-authentication-service/). ([\#18759](https://github.com/element-hq/synapse/issues/18759))
+- Implement the push rules for experimental [MSC4306: Thread Subscriptions](https://github.com/matrix-org/matrix-doc/issues/4306). ([\#18762](https://github.com/element-hq/synapse/issues/18762))
+
+### Bugfixes
+
+- Allow return code 403 (allowed by C2S Spec since v1.2) when fetching profiles via federation. ([\#18696](https://github.com/element-hq/synapse/issues/18696))
+- Register the MSC4306 (Thread Subscriptions) endpoints in the CS API when the experimental feature is enabled. ([\#18726](https://github.com/element-hq/synapse/issues/18726))
+- Fix a long-standing bug where suspended users could not have server notices sent to them (a 403 was returned to the admin). ([\#18750](https://github.com/element-hq/synapse/issues/18750))
+- Fix an issue that could cause logcontexts to be lost on rate-limited requests. Found by @realtyem. ([\#18763](https://github.com/element-hq/synapse/issues/18763))
+- Fix invalidation of storage cache that was broken in 1.135.0. ([\#18786](https://github.com/element-hq/synapse/issues/18786))
+
+### Improved Documentation
+
+- Minor improvements to README. ([\#18700](https://github.com/element-hq/synapse/issues/18700))
+- Document that there can be multiple workers handling the `receipts` stream. ([\#18760](https://github.com/element-hq/synapse/issues/18760))
+- Improve worker documentation for some device paths. ([\#18761](https://github.com/element-hq/synapse/issues/18761))
+
+### Deprecations and Removals
+
+- Deprecate `run_as_background_process` exported as part of the module API interface in favor of `ModuleApi.run_as_background_process`. See [the relevant section in the upgrade notes](https://github.com/element-hq/synapse/blob/develop/docs/upgrade.md#upgrading-to-v11360) for more information. ([\#18737](https://github.com/element-hq/synapse/issues/18737))
+
+### Internal Changes
+
+- Add debug logging for HMAC digest verification failures when using the admin API to register users. ([\#18474](https://github.com/element-hq/synapse/issues/18474))
+- Speed up upgrading a room with large numbers of banned users. ([\#18574](https://github.com/element-hq/synapse/issues/18574))
+- Fix config documentation generation script on Windows by enforcing UTF-8. ([\#18580](https://github.com/element-hq/synapse/issues/18580))
+- Refactor cache, background process, `Counter`, `LaterGauge`, `GaugeBucketCollector`, `Histogram`, and `Gauge` metrics to be homeserver-scoped. ([\#18656](https://github.com/element-hq/synapse/issues/18656), [\#18714](https://github.com/element-hq/synapse/issues/18714), [\#18715](https://github.com/element-hq/synapse/issues/18715), [\#18724](https://github.com/element-hq/synapse/issues/18724), [\#18753](https://github.com/element-hq/synapse/issues/18753), [\#18725](https://github.com/element-hq/synapse/issues/18725), [\#18670](https://github.com/element-hq/synapse/issues/18670), [\#18748](https://github.com/element-hq/synapse/issues/18748), [\#18751](https://github.com/element-hq/synapse/issues/18751))
+- Reduce database usage in Sliding Sync by not querying for background update completion after the update is known to be complete. ([\#18718](https://github.com/element-hq/synapse/issues/18718))
+- Improve order of validation and ratelimiting in room creation. ([\#18723](https://github.com/element-hq/synapse/issues/18723))
+- Bump minimum version bound on Twisted to 21.2.0. ([\#18727](https://github.com/element-hq/synapse/issues/18727), [\#18729](https://github.com/element-hq/synapse/issues/18729))
+- Use `twisted.internet.testing` module in tests instead of deprecated `twisted.test.proto_helpers`. ([\#18728](https://github.com/element-hq/synapse/issues/18728))
+- Remove obsolete `/send_event` replication endpoint. ([\#18730](https://github.com/element-hq/synapse/issues/18730))
+- Update metrics linting to be able to handle custom metrics. ([\#18733](https://github.com/element-hq/synapse/issues/18733))
+- Work around `twisted.protocols.amp.TooLong` error by reducing logging in some tests. ([\#18736](https://github.com/element-hq/synapse/issues/18736))
+- Prevent "Move labelled issues to correct projects" GitHub Actions workflow from failing when an issue is already on the project board. ([\#18755](https://github.com/element-hq/synapse/issues/18755))
+- Bump minimum supported Rust version (MSRV) to 1.82.0. Missed in [#18553](https://github.com/element-hq/synapse/pull/18553) (released in Synapse 1.134.0). ([\#18757](https://github.com/element-hq/synapse/issues/18757))
+- Make `Clock.sleep(...)` return a coroutine, so that mypy can catch places where we don't await on it. ([\#18772](https://github.com/element-hq/synapse/issues/18772))
+- Update implementation of [MSC4306: Thread Subscriptions](https://github.com/matrix-org/matrix-doc/issues/4306) to include automatic subscription conflict prevention as introduced in later drafts. ([\#18756](https://github.com/element-hq/synapse/issues/18756))
+
+
+
+### Updates to locked dependencies
+
+* Bump gitpython from 3.1.44 to 3.1.45. ([\#18743](https://github.com/element-hq/synapse/issues/18743))
+* Bump mypy-zope from 1.0.12 to 1.0.13. ([\#18744](https://github.com/element-hq/synapse/issues/18744))
+* Bump phonenumbers from 9.0.9 to 9.0.10. ([\#18741](https://github.com/element-hq/synapse/issues/18741))
+* Bump ruff from 0.12.4 to 0.12.5. ([\#18742](https://github.com/element-hq/synapse/issues/18742))
+* Bump sentry-sdk from 2.32.0 to 2.33.2. ([\#18745](https://github.com/element-hq/synapse/issues/18745))
+* Bump tokio from 1.46.1 to 1.47.0. ([\#18740](https://github.com/element-hq/synapse/issues/18740))
+* Bump types-jsonschema from 4.24.0.20250708 to 4.25.0.20250720. ([\#18703](https://github.com/element-hq/synapse/issues/18703))
+* Bump types-psycopg2 from 2.9.21.20250516 to 2.9.21.20250718. ([\#18706](https://github.com/element-hq/synapse/issues/18706))
+
+# Synapse 1.135.0 (2025-08-01)
+
+No significant changes since 1.135.0rc2.
+
+
+
+
 # Synapse 1.135.0rc2 (2025-07-30)
 
 ### Bugfixes

changelog.d/18474.misc

@@ -1 +0,0 @@
-Add debug logging for HMAC digest verification failures when using the admin API to register users.

changelog.d/18514.feature

@@ -1 +0,0 @@
-Add configurable rate limiting for the creation of rooms.

changelog.d/18540.feature

@@ -1 +0,0 @@
-Add support for [MSC4293](https://github.com/matrix-org/matrix-spec-proposals/pull/4293) - Redact on Kick/Ban.

changelog.d/18580.misc

@@ -1 +0,0 @@
-Fix config documentation generation script on Windows by enforcing UTF-8.

changelog.d/18585.feature

@@ -1 +0,0 @@
-When admins enable themselves to see soft-failed events, they will also see if the cause is due to the policy server flagging them as spam via `unsigned`.

changelog.d/18656.misc

@@ -1 +0,0 @@
-Refactor `Counter` metrics to be homeserver-scoped.

changelog.d/18670.misc

@@ -1 +0,0 @@
-Refactor background process metrics to be homeserver-scoped.

changelog.d/18686.feature

@@ -1 +0,0 @@
-Add ability to configure forward/outbound proxy via homeserver config instead of environment variables. See `http_proxy`, `https_proxy`, `no_proxy_hosts`.

changelog.d/18696.bugfix

@@ -1 +0,0 @@
-Allow return code 403 (allowed by C2S Spec since v1.2) when fetching profiles via federation.

changelog.d/18700.doc

@@ -1 +0,0 @@
-Minor improvements to README.

changelog.d/18714.misc

@@ -1 +0,0 @@
-Refactor `LaterGauge` metrics to be homeserver-scoped.

changelog.d/18715.misc

@@ -1 +0,0 @@
-Refactor `GaugeBucketCollector` metrics to be homeserver-scoped.

changelog.d/18718.misc

@@ -1 +0,0 @@
-Reduce database usage in Sliding Sync by not querying for background update completion after the update is known to be complete.

changelog.d/18722.feature

@@ -1 +0,0 @@
-Advertise experimental support for [MSC4306](https://github.com/matrix-org/matrix-spec-proposals/pull/4306) through `/_matrix/clients/versions` if enabled.

changelog.d/18724.misc

@@ -1 +0,0 @@
-Refactor `Histogram` metrics to be homeserver-scoped.

changelog.d/18725.misc

@@ -1 +0,0 @@
-Refactor `Gauge` metrics to be homeserver-scoped.

changelog.d/18726.bugfix

@@ -1 +0,0 @@
-Register the MSC4306 endpoints in the CS API when the experimental feature is enabled.

changelog.d/18727.misc

@@ -1 +0,0 @@
-Bump minimum version bound on Twisted to 21.2.0.

changelog.d/18728.misc

@@ -1 +0,0 @@
-Use `twisted.internet.testing` module in tests instead of deprecated `twisted.test.proto_helpers`.

changelog.d/18729.misc

@@ -1 +0,0 @@
-Bump minimum version bound on Twisted to 21.2.0.

changelog.d/18730.misc

@@ -1 +0,0 @@
-Remove obsolete `/send_event` replication endpoint.

changelog.d/18736.misc

@@ -1 +0,0 @@
-Work around `twisted.protocols.amp.TooLong` error by reducing logging in some tests.

changelog.d/18737.removal

@@ -1 +0,0 @@
-Deprecate `run_as_background_process` exported as part of the module API interface in favor of `ModuleApi.run_as_background_process`. See the relevant section in the upgrade notes for more information.

changelog.d/18750.bugfix

@@ -1 +0,0 @@
-Fix a long-standing bug where suspended users could not have server notices sent to them (a 403 was returned to the admin).

changelog.d/18753.misc

@@ -1 +0,0 @@
-Refactor `Histogram` metrics to be homeserver-scoped.

debian/changelog

@@ -1,3 +1,27 @@
+matrix-synapse-py3 (1.136.0) stable; urgency=medium
+
+  * New Synapse release 1.136.0.
+
+ -- Synapse Packaging team <packages@matrix.org>  Tue, 12 Aug 2025 13:18:03 +0100
+
+matrix-synapse-py3 (1.136.0~rc2) stable; urgency=medium
+
+  * New Synapse release 1.136.0rc2.
+
+ -- Synapse Packaging team <packages@matrix.org>  Mon, 11 Aug 2025 12:18:52 -0600
+
+matrix-synapse-py3 (1.136.0~rc1) stable; urgency=medium
+
+  * New Synapse release 1.136.0rc1.
+
+ -- Synapse Packaging team <packages@matrix.org>  Tue, 05 Aug 2025 08:13:30 -0600
+
+matrix-synapse-py3 (1.135.0) stable; urgency=medium
+
+  * New Synapse release 1.135.0.
+
+ -- Synapse Packaging team <packages@matrix.org>  Fri, 01 Aug 2025 13:12:28 +0100
+
 matrix-synapse-py3 (1.135.0~rc2) stable; urgency=medium
 
   * New Synapse release 1.135.0rc2.

docs/upgrade.md

@@ -164,7 +164,29 @@ The Grafana dashboard JSON in `contrib/grafana/synapse.json` has been updated to
 this change but you will need to manually update your own existing Grafana dashboards
 using these metrics.
 
+## Stable integration with Matrix Authentication Service
 
+Support for [Matrix Authentication Service (MAS)](https://github.com/element-hq/matrix-authentication-service) is now stable, with a simplified configuration.
+This stable integration requires MAS 0.20.0 or later.
+
+The existing `experimental_features.msc3861` configuration option is now deprecated and will be removed in Synapse v1.137.0.
+
+Synapse deployments already using MAS should now use the new configuration options:
+
+```yaml
+matrix_authentication_service:
+  # Enable the MAS integration
+  enabled: true
+  # The base URL where Synapse will contact MAS
+  endpoint: http://localhost:8080
+  # The shared secret used to authenticate MAS requests, must be the same as `matrix.secret` in the MAS configuration
+  # See https://element-hq.github.io/matrix-authentication-service/reference/configuration.html#matrix
+  secret: "asecurerandomsecretstring"
+```
+
+They must remove the `experimental_features.msc3861` configuration option from their configuration.
+
+They can also remove the client previously used by Synapse [in the MAS configuration](https://element-hq.github.io/matrix-authentication-service/reference/configuration.html#clients) as it is no longer in use.
 
 # Upgrading to v1.135.0
 
@@ -186,10 +208,10 @@ native ICU library on your system is no longer required.
 ## Documented endpoint which can be delegated to a federation worker
 
 The endpoint `^/_matrix/federation/v1/version$` can be delegated to a federation
-worker. This is not new behaviour, but had not been documented yet. The 
-[list of delegatable endpoints](workers.md#synapseappgeneric_worker) has 
+worker. This is not new behaviour, but had not been documented yet. The
+[list of delegatable endpoints](workers.md#synapseappgeneric_worker) has
 been updated to include it. Make sure to check your reverse proxy rules if you
-are using workers. 
+are using workers.
 
 # Upgrading to v1.126.0
 

docs/usage/configuration/config_documentation.md

@@ -643,6 +643,28 @@ no_proxy_hosts:
 - 172.30.0.0/16
 ```
 ---
+### `matrix_authentication_service`
+
+*(object)* The `matrix_authentication_service` setting configures integration with [Matrix Authentication Service (MAS)](https://github.com/element-hq/matrix-authentication-service).
+
+This setting has the following sub-options:
+
+* `enabled` (boolean): Whether or not to enable the MAS integration. If this is set to `false`, Synapse will use its legacy internal authentication API. Defaults to `false`.
+
+* `endpoint` (string): The URL where Synapse can reach MAS. This *must* have the `discovery` and `oauth` resources mounted. Defaults to `"http://localhost:8080"`.
+
+* `secret` (string|null): A shared secret that will be used to authenticate requests from and to MAS.
+
+* `secret_path` (string|null): Alternative to `secret`, reading the shared secret from a file. The file should be a plain text file, containing only the secret. Synapse reads the secret from the given file once at startup.
+
+Example configuration:
+```yaml
+matrix_authentication_service:
+  enabled: true
+  secret: someverysecuresecret
+  endpoint: http://localhost:8080
+```
+---
 ### `dummy_events_threshold`
 
 *(integer)* Forward extremities can build up in a room due to networking delays between homeservers. Once this happens in a large room, calculation of the state of that room can become quite expensive. To mitigate this, once the number of forward extremities reaches a given threshold, Synapse will send an `org.matrix.dummy_event` event, which will reduce the forward extremities in the room.
@@ -4152,7 +4174,7 @@ The default power levels for each preset are:
 "m.room.history_visibility": 100
 "m.room.canonical_alias": 50
 "m.room.avatar": 50
-"m.room.tombstone": 100
+"m.room.tombstone": 100 (150 if MSC4289 is used)
 "m.room.server_acl": 100
 "m.room.encryption": 100
 ```

docs/workers.md

@@ -260,7 +260,7 @@ information.
     ^/_matrix/client/(r0|v3|unstable)/keys/claim$
     ^/_matrix/client/(r0|v3|unstable)/room_keys/
     ^/_matrix/client/(r0|v3|unstable)/keys/upload
-    ^/_matrix/client/(api/v1|r0|v3|unstable/keys/device_signing/upload$
+    ^/_matrix/client/(api/v1|r0|v3|unstable)/keys/device_signing/upload$
     ^/_matrix/client/(api/v1|r0|v3|unstable)/keys/signatures/upload$
 
     # Registration/login requests
@@ -532,8 +532,9 @@ the stream writer for the `account_data` stream:
 
 ##### The `receipts` stream
 
-The following endpoints should be routed directly to the worker configured as
-the stream writer for the `receipts` stream:
+The `receipts` stream supports multiple writers. The following endpoints
+can be handled by any worker, but should be routed directly to one of the workers
+configured as stream writer for the `receipts` stream:
 
     ^/_matrix/client/(r0|v3|unstable)/rooms/.*/receipt
     ^/_matrix/client/(r0|v3|unstable)/rooms/.*/read_markers
@@ -555,13 +556,13 @@ the stream writer for the `push_rules` stream:
 ##### The `device_lists` stream
 
 The `device_lists` stream supports multiple writers. The following endpoints
-can be handled by any worker, but should be routed directly one of the workers
+can be handled by any worker, but should be routed directly to one of the workers
 configured as stream writer for the `device_lists` stream:
 
     ^/_matrix/client/(r0|v3)/delete_devices$
-    ^/_matrix/client/(api/v1|r0|v3|unstable)/devices/
+    ^/_matrix/client/(api/v1|r0|v3|unstable)/devices(/|$)
     ^/_matrix/client/(r0|v3|unstable)/keys/upload
-    ^/_matrix/client/(api/v1|r0|v3|unstable/keys/device_signing/upload$
+    ^/_matrix/client/(api/v1|r0|v3|unstable)/keys/device_signing/upload$
     ^/_matrix/client/(api/v1|r0|v3|unstable)/keys/signatures/upload$
 
 #### Restrict outbound federation traffic to a specific set of workers

mypy.ini

@@ -1,6 +1,17 @@
 [mypy]
 namespace_packages = True
-plugins = pydantic.mypy, mypy_zope:plugin, scripts-dev/mypy_synapse_plugin.py
+# Our custom mypy plugin should remain first in this list.
+#
+# mypy has a limitation where it only chooses the first plugin that returns a non-None
+# value for each hook (known-limitation, c.f.
+# https://github.com/python/mypy/issues/19524). We workaround this by putting our custom
+# plugin first in the plugin order and then manually calling any other conflicting
+# plugin hooks in our own plugin followed by our own checks.
+#
+# If you add a new plugin, make sure to check whether the hooks being used conflict with
+# our custom plugin hooks and if so, manually call the other plugin's hooks in our
+# custom plugin. (also applies to if the plugin is updated in the future)
+plugins = scripts-dev/mypy_synapse_plugin.py, pydantic.mypy, mypy_zope:plugin
 follow_imports = normal
 show_error_codes = True
 show_traceback = True
@@ -99,3 +110,6 @@ ignore_missing_imports = True
 
 [mypy-multipart.*]
 ignore_missing_imports = True
+
+[mypy-mypy_zope.*]
+ignore_missing_imports = True

pyproject.toml

@@ -101,7 +101,7 @@ module-name = "synapse.synapse_rust"
 
 [tool.poetry]
 name = "matrix-synapse"
-version = "1.135.0rc2"
+version = "1.136.0"
 description = "Homeserver for the Matrix decentralised comms protocol"
 authors = ["Matrix.org Team and Contributors <packages@matrix.org>"]
 license = "AGPL-3.0-or-later"

rust/Cargo.toml

@@ -7,7 +7,7 @@ name = "synapse"
 version = "0.1.0"
 
 edition = "2021"
-rust-version = "1.81.0"
+rust-version = "1.82.0"
 
 [lib]
 name = "synapse"

rust/benches/evaluator.rs

@@ -61,6 +61,7 @@ fn bench_match_exact(b: &mut Bencher) {
         vec![],
         false,
         false,
+        false,
     )
     .unwrap();
 
@@ -71,10 +72,10 @@ fn bench_match_exact(b: &mut Bencher) {
         },
     ));
 
-    let matched = eval.match_condition(&condition, None, None).unwrap();
+    let matched = eval.match_condition(&condition, None, None, None).unwrap();
     assert!(matched, "Didn't match");
 
-    b.iter(|| eval.match_condition(&condition, None, None).unwrap());
+    b.iter(|| eval.match_condition(&condition, None, None, None).unwrap());
 }
 
 #[bench]
@@ -107,6 +108,7 @@ fn bench_match_word(b: &mut Bencher) {
         vec![],
         false,
         false,
+        false,
     )
     .unwrap();
 
@@ -117,10 +119,10 @@ fn bench_match_word(b: &mut Bencher) {
         },
     ));
 
-    let matched = eval.match_condition(&condition, None, None).unwrap();
+    let matched = eval.match_condition(&condition, None, None, None).unwrap();
     assert!(matched, "Didn't match");
 
-    b.iter(|| eval.match_condition(&condition, None, None).unwrap());
+    b.iter(|| eval.match_condition(&condition, None, None, None).unwrap());
 }
 
 #[bench]
@@ -153,6 +155,7 @@ fn bench_match_word_miss(b: &mut Bencher) {
         vec![],
         false,
         false,
+        false,
     )
     .unwrap();
 
@@ -163,10 +166,10 @@ fn bench_match_word_miss(b: &mut Bencher) {
         },
     ));
 
-    let matched = eval.match_condition(&condition, None, None).unwrap();
+    let matched = eval.match_condition(&condition, None, None, None).unwrap();
     assert!(!matched, "Didn't match");
 
-    b.iter(|| eval.match_condition(&condition, None, None).unwrap());
+    b.iter(|| eval.match_condition(&condition, None, None, None).unwrap());
 }
 
 #[bench]
@@ -199,6 +202,7 @@ fn bench_eval_message(b: &mut Bencher) {
         vec![],
         false,
         false,
+        false,
     )
     .unwrap();
 
@@ -210,7 +214,8 @@ fn bench_eval_message(b: &mut Bencher) {
         false,
         false,
         false,
+        false,
     );
 
-    b.iter(|| eval.run(&rules, Some("bob"), Some("person")));
+    b.iter(|| eval.run(&rules, Some("bob"), Some("person"), None));
 }

rust/src/push/base_rules.rs

@@ -290,6 +290,26 @@ pub const BASE_APPEND_CONTENT_RULES: &[PushRule] = &[PushRule {
 }];
 
 pub const BASE_APPEND_UNDERRIDE_RULES: &[PushRule] = &[
+    PushRule {
+        rule_id: Cow::Borrowed("global/content/.io.element.msc4306.rule.unsubscribed_thread"),
+        priority_class: 1,
+        conditions: Cow::Borrowed(&[Condition::Known(
+            KnownCondition::Msc4306ThreadSubscription { subscribed: false },
+        )]),
+        actions: Cow::Borrowed(&[]),
+        default: true,
+        default_enabled: true,
+    },
+    PushRule {
+        rule_id: Cow::Borrowed("global/content/.io.element.msc4306.rule.subscribed_thread"),
+        priority_class: 1,
+        conditions: Cow::Borrowed(&[Condition::Known(
+            KnownCondition::Msc4306ThreadSubscription { subscribed: true },
+        )]),
+        actions: Cow::Borrowed(&[Action::Notify, SOUND_ACTION]),
+        default: true,
+        default_enabled: true,
+    },
     PushRule {
         rule_id: Cow::Borrowed("global/underride/.m.rule.call"),
         priority_class: 1,

rust/src/push/evaluator.rs

@@ -106,8 +106,11 @@ pub struct PushRuleEvaluator {
     /// flag as MSC1767 (extensible events core).
     msc3931_enabled: bool,
 
-    // If MSC4210 (remove legacy mentions) is enabled.
+    /// If MSC4210 (remove legacy mentions) is enabled.
     msc4210_enabled: bool,
+
+    /// If MSC4306 (thread subscriptions) is enabled.
+    msc4306_enabled: bool,
 }
 
 #[pymethods]
@@ -126,6 +129,7 @@ impl PushRuleEvaluator {
         room_version_feature_flags,
         msc3931_enabled,
         msc4210_enabled,
+        msc4306_enabled,
     ))]
     pub fn py_new(
         flattened_keys: BTreeMap<String, JsonValue>,
@@ -138,6 +142,7 @@ impl PushRuleEvaluator {
         room_version_feature_flags: Vec<String>,
         msc3931_enabled: bool,
         msc4210_enabled: bool,
+        msc4306_enabled: bool,
     ) -> Result<Self, Error> {
         let body = match flattened_keys.get("content.body") {
             Some(JsonValue::Value(SimpleJsonValue::Str(s))) => s.clone().into_owned(),
@@ -156,6 +161,7 @@ impl PushRuleEvaluator {
             room_version_feature_flags,
             msc3931_enabled,
             msc4210_enabled,
+            msc4306_enabled,
         })
     }
 
@@ -167,12 +173,19 @@ impl PushRuleEvaluator {
     ///
     /// Returns the set of actions, if any, that match (filtering out any
     /// `dont_notify` and `coalesce` actions).
-    #[pyo3(signature = (push_rules, user_id=None, display_name=None))]
+    ///
+    /// msc4306_thread_subscription_state: (Only populated if MSC4306 is enabled)
+    /// The thread subscription state corresponding to the thread containing this event.
+    /// - `None` if the event is not in a thread, or if MSC4306 is disabled.
+    /// - `Some(true)` if the event is in a thread and the user has a subscription for that thread
+    /// - `Some(false)` if the event is in a thread and the user does NOT have a subscription for that thread
+    #[pyo3(signature = (push_rules, user_id=None, display_name=None, msc4306_thread_subscription_state=None))]
     pub fn run(
         &self,
         push_rules: &FilteredPushRules,
         user_id: Option<&str>,
         display_name: Option<&str>,
+        msc4306_thread_subscription_state: Option<bool>,
     ) -> Vec<Action> {
         'outer: for (push_rule, enabled) in push_rules.iter() {
             if !enabled {
@@ -204,7 +217,12 @@ impl PushRuleEvaluator {
                     Condition::Known(KnownCondition::RoomVersionSupports { feature: _ }),
                 );
 
-                match self.match_condition(condition, user_id, display_name) {
+                match self.match_condition(
+                    condition,
+                    user_id,
+                    display_name,
+                    msc4306_thread_subscription_state,
+                ) {
                     Ok(true) => {}
                     Ok(false) => continue 'outer,
                     Err(err) => {
@@ -237,14 +255,20 @@ impl PushRuleEvaluator {
     }
 
     /// Check if the given condition matches.
-    #[pyo3(signature = (condition, user_id=None, display_name=None))]
+    #[pyo3(signature = (condition, user_id=None, display_name=None, msc4306_thread_subscription_state=None))]
     fn matches(
         &self,
         condition: Condition,
         user_id: Option<&str>,
         display_name: Option<&str>,
+        msc4306_thread_subscription_state: Option<bool>,
     ) -> bool {
-        match self.match_condition(&condition, user_id, display_name) {
+        match self.match_condition(
+            &condition,
+            user_id,
+            display_name,
+            msc4306_thread_subscription_state,
+        ) {
             Ok(true) => true,
             Ok(false) => false,
             Err(err) => {
@@ -262,6 +286,7 @@ impl PushRuleEvaluator {
         condition: &Condition,
         user_id: Option<&str>,
         display_name: Option<&str>,
+        msc4306_thread_subscription_state: Option<bool>,
     ) -> Result<bool, Error> {
         let known_condition = match condition {
             Condition::Known(known) => known,
@@ -393,6 +418,13 @@ impl PushRuleEvaluator {
                         && self.room_version_feature_flags.contains(&flag)
                 }
             }
+            KnownCondition::Msc4306ThreadSubscription { subscribed } => {
+                if !self.msc4306_enabled {
+                    false
+                } else {
+                    msc4306_thread_subscription_state == Some(*subscribed)
+                }
+            }
         };
 
         Ok(result)
@@ -536,10 +568,11 @@ fn push_rule_evaluator() {
         vec![],
         true,
         false,
+        false,
     )
     .unwrap();
 
-    let result = evaluator.run(&FilteredPushRules::default(), None, Some("bob"));
+    let result = evaluator.run(&FilteredPushRules::default(), None, Some("bob"), None);
     assert_eq!(result.len(), 3);
 }
 
@@ -566,6 +599,7 @@ fn test_requires_room_version_supports_condition() {
         flags,
         true,
         false,
+        false,
     )
     .unwrap();
 
@@ -575,6 +609,7 @@ fn test_requires_room_version_supports_condition() {
         &FilteredPushRules::default(),
         Some("@bob:example.org"),
         None,
+        None,
     );
     assert_eq!(result.len(), 3);
 
@@ -593,7 +628,17 @@ fn test_requires_room_version_supports_condition() {
     };
     let rules = PushRules::new(vec![custom_rule]);
     result = evaluator.run(
-        &FilteredPushRules::py_new(rules, BTreeMap::new(), true, false, true, false, false),
+        &FilteredPushRules::py_new(
+            rules,
+            BTreeMap::new(),
+            true,
+            false,
+            true,
+            false,
+            false,
+            false,
+        ),
+        None,
         None,
         None,
     );

rust/src/push/mod.rs

@@ -369,6 +369,10 @@ pub enum KnownCondition {
     RoomVersionSupports {
         feature: Cow<'static, str>,
     },
+    #[serde(rename = "io.element.msc4306.thread_subscription")]
+    Msc4306ThreadSubscription {
+        subscribed: bool,
+    },
 }
 
 impl<'source> IntoPyObject<'source> for Condition {
@@ -547,11 +551,13 @@ pub struct FilteredPushRules {
     msc3664_enabled: bool,
     msc4028_push_encrypted_events: bool,
     msc4210_enabled: bool,
+    msc4306_enabled: bool,
 }
 
 #[pymethods]
 impl FilteredPushRules {
     #[new]
+    #[allow(clippy::too_many_arguments)]
     pub fn py_new(
         push_rules: PushRules,
         enabled_map: BTreeMap<String, bool>,
@@ -560,6 +566,7 @@ impl FilteredPushRules {
         msc3664_enabled: bool,
         msc4028_push_encrypted_events: bool,
         msc4210_enabled: bool,
+        msc4306_enabled: bool,
     ) -> Self {
         Self {
             push_rules,
@@ -569,6 +576,7 @@ impl FilteredPushRules {
             msc3664_enabled,
             msc4028_push_encrypted_events,
             msc4210_enabled,
+            msc4306_enabled,
         }
     }
 
@@ -619,6 +627,10 @@ impl FilteredPushRules {
                     return false;
                 }
 
+                if !self.msc4306_enabled && rule.rule_id.contains("/.io.element.msc4306.rule.") {
+                    return false;
+                }
+
                 true
             })
             .map(|r| {

schema/synapse-config.schema.yaml

@@ -1,5 +1,5 @@
 $schema: https://element-hq.github.io/synapse/latest/schema/v1/meta.schema.json
-$id: https://element-hq.github.io/synapse/schema/synapse/v1.135/synapse-config.schema.json
+$id: https://element-hq.github.io/synapse/schema/synapse/v1.136/synapse-config.schema.json
 type: object
 properties:
   modules:
@@ -656,6 +656,43 @@ properties:
       - - master.hostname.example.com
         - 10.1.0.0/16
         - 172.30.0.0/16
+  matrix_authentication_service:
+    type: object
+    description: >-
+      The `matrix_authentication_service` setting configures integration with
+      [Matrix Authentication Service (MAS)](https://github.com/element-hq/matrix-authentication-service).
+    properties:
+      enabled:
+        type: boolean
+        description: >-
+          Whether or not to enable the MAS integration. If this is set to
+          `false`, Synapse will use its legacy internal authentication API.
+        default: false
+
+      endpoint:
+        type: string
+        format: uri
+        description: >-
+          The URL where Synapse can reach MAS. This *must* have the `discovery`
+          and `oauth` resources mounted.
+        default: http://localhost:8080
+
+      secret:
+        type: ["string", "null"]
+        description: >-
+          A shared secret that will be used to authenticate requests from and to MAS.
+
+      secret_path:
+        type: ["string", "null"]
+        description: >-
+          Alternative to `secret`, reading the shared secret from a file.
+          The file should be a plain text file, containing only the secret.
+          Synapse reads the secret from the given file once at startup.
+
+    examples:
+      - enabled: true
+        secret: someverysecuresecret
+        endpoint: http://localhost:8080
   dummy_events_threshold:
     type: integer
     description: >-
@@ -5147,7 +5184,7 @@ properties:
 
       "m.room.avatar": 50
 
-      "m.room.tombstone": 100
+      "m.room.tombstone": 100 (150 if MSC4289 is used)
 
       "m.room.server_acl": 100
 

scripts-dev/mypy_synapse_plugin.py

@@ -23,16 +23,21 @@
 can crop up, e.g the cache descriptors.
 """
 
-from typing import Callable, Optional, Tuple, Type, Union
+import enum
+from typing import Callable, Mapping, Optional, Tuple, Type, Union
 
+import attr
 import mypy.types
 from mypy.erasetype import remove_instance_last_known_values
 from mypy.errorcodes import ErrorCode
 from mypy.nodes import ARG_NAMED_OPT, ListExpr, NameExpr, TempNode, TupleExpr, Var
 from mypy.plugin import (
+    ClassDefContext,
+    Context,
     FunctionLike,
     FunctionSigContext,
     MethodSigContext,
+    MypyFile,
     Plugin,
 )
 from mypy.typeops import bind_self
@@ -41,12 +46,15 @@ from mypy.types import (
     CallableType,
     Instance,
     NoneType,
+    Options,
     TupleType,
     TypeAliasType,
     TypeVarType,
     UninhabitedType,
     UnionType,
 )
+from mypy_zope import plugin as mypy_zope_plugin
+from pydantic.mypy import plugin as mypy_pydantic_plugin
 
 PROMETHEUS_METRIC_MISSING_SERVER_NAME_LABEL = ErrorCode(
     "missing-server-name-label",
@@ -54,19 +62,153 @@ PROMETHEUS_METRIC_MISSING_SERVER_NAME_LABEL = ErrorCode(
     category="per-homeserver-tenant-metrics",
 )
 
+PROMETHEUS_METRIC_MISSING_FROM_LIST_TO_CHECK = ErrorCode(
+    "metric-type-missing-from-list",
+    "Every Prometheus metric type must be included in the `prometheus_metric_fullname_to_label_arg_map`.",
+    category="per-homeserver-tenant-metrics",
+)
+
+
+class Sentinel(enum.Enum):
+    # defining a sentinel in this way allows mypy to correctly handle the
+    # type of a dictionary lookup and subsequent type narrowing.
+    UNSET_SENTINEL = object()
+
+
+@attr.s(auto_attribs=True)
+class ArgLocation:
+    keyword_name: str
+    """
+    The keyword argument name for this argument
+    """
+    position: int
+    """
+    The 0-based positional index of this argument
+    """
+
+
+prometheus_metric_fullname_to_label_arg_map: Mapping[str, Optional[ArgLocation]] = {
+    # `Collector` subclasses:
+    "prometheus_client.metrics.MetricWrapperBase": ArgLocation("labelnames", 2),
+    "prometheus_client.metrics.Counter": ArgLocation("labelnames", 2),
+    "prometheus_client.metrics.Histogram": ArgLocation("labelnames", 2),
+    "prometheus_client.metrics.Gauge": ArgLocation("labelnames", 2),
+    "prometheus_client.metrics.Summary": ArgLocation("labelnames", 2),
+    "prometheus_client.metrics.Info": ArgLocation("labelnames", 2),
+    "prometheus_client.metrics.Enum": ArgLocation("labelnames", 2),
+    "synapse.metrics.LaterGauge": ArgLocation("labelnames", 2),
+    "synapse.metrics.InFlightGauge": ArgLocation("labels", 2),
+    "synapse.metrics.GaugeBucketCollector": ArgLocation("labelnames", 2),
+    "prometheus_client.registry.Collector": None,
+    "prometheus_client.registry._EmptyCollector": None,
+    "prometheus_client.registry.CollectorRegistry": None,
+    "prometheus_client.process_collector.ProcessCollector": None,
+    "prometheus_client.platform_collector.PlatformCollector": None,
+    "prometheus_client.gc_collector.GCCollector": None,
+    "synapse.metrics._gc.GCCounts": None,
+    "synapse.metrics._gc.PyPyGCStats": None,
+    "synapse.metrics._reactor_metrics.ReactorLastSeenMetric": None,
+    "synapse.metrics.CPUMetrics": None,
+    "synapse.metrics.jemalloc.JemallocCollector": None,
+    "synapse.util.metrics.DynamicCollectorRegistry": None,
+    "synapse.metrics.background_process_metrics._Collector": None,
+    #
+    # `Metric` subclasses:
+    "prometheus_client.metrics_core.Metric": None,
+    "prometheus_client.metrics_core.UnknownMetricFamily": ArgLocation("labels", 3),
+    "prometheus_client.metrics_core.CounterMetricFamily": ArgLocation("labels", 3),
+    "prometheus_client.metrics_core.GaugeMetricFamily": ArgLocation("labels", 3),
+    "prometheus_client.metrics_core.SummaryMetricFamily": ArgLocation("labels", 3),
+    "prometheus_client.metrics_core.InfoMetricFamily": ArgLocation("labels", 3),
+    "prometheus_client.metrics_core.HistogramMetricFamily": ArgLocation("labels", 3),
+    "prometheus_client.metrics_core.GaugeHistogramMetricFamily": ArgLocation(
+        "labels", 4
+    ),
+    "prometheus_client.metrics_core.StateSetMetricFamily": ArgLocation("labels", 3),
+    "synapse.metrics.GaugeHistogramMetricFamilyWithLabels": ArgLocation(
+        "labelnames", 4
+    ),
+}
+"""
+Map from the fullname of the Prometheus `Metric`/`Collector` classes to the keyword
+argument name and positional index of the label names. This map is useful because
+different metrics have different signatures for passing in label names and we just need
+to know where to look.
+
+This map should include any metrics that we collect with Prometheus. Which corresponds
+to anything that inherits from `prometheus_client.registry.Collector`
+(`synapse.metrics._types.Collector`) or `prometheus_client.metrics_core.Metric`. The
+exhaustiveness of this list is enforced by `analyze_prometheus_metric_classes`.
+
+The entries with `None` always fail the lint because they don't have a `labelnames`
+argument (therefore, no `SERVER_NAME_LABEL`), but we include them here so that people
+can notice and manually allow via a type ignore comment as the source of truth
+should be in the source code.
+"""
+
+# Unbound at this point because we don't know the mypy version yet.
+# This is set in the `plugin(...)` function below.
+MypyPydanticPluginClass: Type[Plugin]
+MypyZopePluginClass: Type[Plugin]
+
 
 class SynapsePlugin(Plugin):
+    def __init__(self, options: Options):
+        super().__init__(options)
+        self.mypy_pydantic_plugin = MypyPydanticPluginClass(options)
+        self.mypy_zope_plugin = MypyZopePluginClass(options)
+
+    def set_modules(self, modules: dict[str, MypyFile]) -> None:
+        """
+        This is called by mypy internals. We have to override this to ensure it's also
+        called for any other plugins that we're manually handling.
+
+        Here is how mypy describes it:
+
+        > [`self._modules`] can't be set in `__init__` because it is executed too soon
+        > in `build.py`. Therefore, `build.py` *must* set it later before graph processing
+        > starts by calling `set_modules()`.
+        """
+        super().set_modules(modules)
+        self.mypy_pydantic_plugin.set_modules(modules)
+        self.mypy_zope_plugin.set_modules(modules)
+
+    def get_base_class_hook(
+        self, fullname: str
+    ) -> Optional[Callable[[ClassDefContext], None]]:
+        def _get_base_class_hook(ctx: ClassDefContext) -> None:
+            # Run any `get_base_class_hook` checks from other plugins first.
+            #
+            # Unfortunately, because mypy only chooses the first plugin that returns a
+            # non-None value (known-limitation, c.f.
+            # https://github.com/python/mypy/issues/19524), we workaround this by
+            # putting our custom plugin first in the plugin order and then calling the
+            # other plugin's hook manually followed by our own checks.
+            if callback := self.mypy_pydantic_plugin.get_base_class_hook(fullname):
+                callback(ctx)
+            if callback := self.mypy_zope_plugin.get_base_class_hook(fullname):
+                callback(ctx)
+
+            # Now run our own checks
+            analyze_prometheus_metric_classes(ctx)
+
+        return _get_base_class_hook
+
     def get_function_signature_hook(
         self, fullname: str
     ) -> Optional[Callable[[FunctionSigContext], FunctionLike]]:
-        if fullname in (
-            "prometheus_client.metrics.Counter",
-            "prometheus_client.metrics.Histogram",
-            "prometheus_client.metrics.Gauge",
-            # TODO: Add other prometheus_client metrics that need checking as we
-            # refactor, see https://github.com/element-hq/synapse/issues/18592
-        ):
-            return check_prometheus_metric_instantiation
+        # Strip off the unique identifier for classes that are dynamically created inside
+        # functions. ex. `synapse.metrics.jemalloc.JemallocCollector@185` (this is the line
+        # number)
+        if "@" in fullname:
+            fullname = fullname.split("@", 1)[0]
+
+        # Look for any Prometheus metrics to make sure they have the `SERVER_NAME_LABEL`
+        # label.
+        if fullname in prometheus_metric_fullname_to_label_arg_map.keys():
+            # Because it's difficult to determine the `fullname` of the function in the
+            # callback, let's just pass it in while we have it.
+            return lambda ctx: check_prometheus_metric_instantiation(ctx, fullname)
 
         return None
 
@@ -90,7 +232,44 @@ class SynapsePlugin(Plugin):
         return None
 
 
-def check_prometheus_metric_instantiation(ctx: FunctionSigContext) -> CallableType:
+def analyze_prometheus_metric_classes(ctx: ClassDefContext) -> None:
+    """
+    Cross-check the list of Prometheus metric classes against the
+    `prometheus_metric_fullname_to_label_arg_map` to ensure the list is exhaustive and
+    up-to-date.
+    """
+
+    fullname = ctx.cls.fullname
+    # Strip off the unique identifier for classes that are dynamically created inside
+    # functions. ex. `synapse.metrics.jemalloc.JemallocCollector@185` (this is the line
+    # number)
+    if "@" in fullname:
+        fullname = fullname.split("@", 1)[0]
+
+    if any(
+        ancestor_type.fullname
+        in (
+            # All of the Prometheus metric classes inherit from the `Collector`.
+            "prometheus_client.registry.Collector",
+            "synapse.metrics._types.Collector",
+            # And custom metrics that inherit from `Metric`.
+            "prometheus_client.metrics_core.Metric",
+        )
+        for ancestor_type in ctx.cls.info.mro
+    ):
+        if fullname not in prometheus_metric_fullname_to_label_arg_map:
+            ctx.api.fail(
+                f"Expected {fullname} to be in `prometheus_metric_fullname_to_label_arg_map`, "
+                f"but it was not found. This is a problem with our custom mypy plugin. "
+                f"Please add it to the map.",
+                Context(),
+                code=PROMETHEUS_METRIC_MISSING_FROM_LIST_TO_CHECK,
+            )
+
+
+def check_prometheus_metric_instantiation(
+    ctx: FunctionSigContext, fullname: str
+) -> CallableType:
     """
     Ensure that the `prometheus_client` metrics include the `SERVER_NAME_LABEL` label
     when instantiated.
@@ -103,18 +282,49 @@ def check_prometheus_metric_instantiation(ctx: FunctionSigContext) -> CallableTy
     Python garbage collection, and Twisted reactor tick time, which shouldn't have the
     `SERVER_NAME_LABEL`. In those cases, use a type ignore comment to disable the
     check, e.g. `# type: ignore[missing-server-name-label]`.
+
+    Args:
+        ctx: The `FunctionSigContext` from mypy.
+        fullname: The fully qualified name of the function being called,
+            e.g. `"prometheus_client.metrics.Counter"`
     """
     # The true signature, this isn't being modified so this is what will be returned.
-    signature: CallableType = ctx.default_signature
+    signature = ctx.default_signature
+
+    # Find where the label names argument is in the function signature.
+    arg_location = prometheus_metric_fullname_to_label_arg_map.get(
+        fullname, Sentinel.UNSET_SENTINEL
+    )
+    assert arg_location is not Sentinel.UNSET_SENTINEL, (
+        f"Expected to find {fullname} in `prometheus_metric_fullname_to_label_arg_map`, "
+        f"but it was not found. This is a problem with our custom mypy plugin. "
+        f"Please add it to the map. Context: {ctx.context}"
+    )
+    # People should be using `# type: ignore[missing-server-name-label]` for
+    # process-level metrics that should not have the `SERVER_NAME_LABEL`.
+    if arg_location is None:
+        ctx.api.fail(
+            f"{signature.name} does not have a `labelnames`/`labels` argument "
+            "(if this is untrue, update `prometheus_metric_fullname_to_label_arg_map` "
+            "in our custom mypy plugin) and should probably have a type ignore comment, "
+            "e.g. `# type: ignore[missing-server-name-label]`. The reason we don't "
+            "automatically ignore this is the source of truth should be in the source code.",
+            ctx.context,
+            code=PROMETHEUS_METRIC_MISSING_SERVER_NAME_LABEL,
+        )
+        return signature
 
     # Sanity check the arguments are still as expected in this version of
     # `prometheus_client`. ex. `Counter(name, documentation, labelnames, ...)`
     #
     # `signature.arg_names` should be: ["name", "documentation", "labelnames", ...]
-    if len(signature.arg_names) < 3 or signature.arg_names[2] != "labelnames":
+    if (
+        len(signature.arg_names) < (arg_location.position + 1)
+        or signature.arg_names[arg_location.position] != arg_location.keyword_name
+    ):
         ctx.api.fail(
-            f"Expected the 3rd argument of {signature.name} to be 'labelnames', but got "
-            f"{signature.arg_names[2]}",
+            f"Expected argument number {arg_location.position + 1} of {signature.name} to be `labelnames`/`labels`, "
+            f"but got {signature.arg_names[arg_location.position]}",
             ctx.context,
         )
         return signature
@@ -137,7 +347,11 @@ def check_prometheus_metric_instantiation(ctx: FunctionSigContext) -> CallableTy
     #     ...
     # ]
     # ```
-    labelnames_arg_expression = ctx.args[2][0] if len(ctx.args[2]) > 0 else None
+    labelnames_arg_expression = (
+        ctx.args[arg_location.position][0]
+        if len(ctx.args[arg_location.position]) > 0
+        else None
+    )
     if isinstance(labelnames_arg_expression, (ListExpr, TupleExpr)):
         # Check if the `labelnames` argument includes the `server_name` label (`SERVER_NAME_LABEL`).
         for labelname_expression in labelnames_arg_expression.items:
@@ -476,10 +690,13 @@ def is_cacheable(
 
 
 def plugin(version: str) -> Type[SynapsePlugin]:
+    global MypyPydanticPluginClass, MypyZopePluginClass
     # This is the entry point of the plugin, and lets us deal with the fact
     # that the mypy plugin interface is *not* stable by looking at the version
     # string.
     #
     # However, since we pin the version of mypy Synapse uses in CI, we don't
     # really care.
+    MypyPydanticPluginClass = mypy_pydantic_plugin(version)
+    MypyZopePluginClass = mypy_zope_plugin(version)
     return SynapsePlugin

synapse/_pydantic_compat.py

@@ -34,9 +34,11 @@ HAS_PYDANTIC_V2: bool = Version(pydantic_version).major == 2
 
 if TYPE_CHECKING or HAS_PYDANTIC_V2:
     from pydantic.v1 import (
+        AnyHttpUrl,
         BaseModel,
         Extra,
         Field,
+        FilePath,
         MissingError,
         PydanticValueError,
         StrictBool,
@@ -55,9 +57,11 @@ if TYPE_CHECKING or HAS_PYDANTIC_V2:
     from pydantic.v1.typing import get_args
 else:
     from pydantic import (
+        AnyHttpUrl,
         BaseModel,
         Extra,
         Field,
+        FilePath,
         MissingError,
         PydanticValueError,
         StrictBool,
@@ -77,6 +81,7 @@ else:
 
 __all__ = (
     "HAS_PYDANTIC_V2",
+    "AnyHttpUrl",
     "BaseModel",
     "constr",
     "conbytes",
@@ -85,6 +90,7 @@ __all__ = (
     "ErrorWrapper",
     "Extra",
     "Field",
+    "FilePath",
     "get_args",
     "MissingError",
     "parse_obj_as",

synapse/api/auth/__init__.py

@@ -20,10 +20,13 @@
 #
 from typing import TYPE_CHECKING, Optional, Protocol, Tuple
 
+from prometheus_client import Histogram
+
 from twisted.web.server import Request
 
 from synapse.appservice import ApplicationService
 from synapse.http.site import SynapseRequest
+from synapse.metrics import SERVER_NAME_LABEL
 from synapse.types import Requester
 
 if TYPE_CHECKING:
@@ -33,6 +36,13 @@ if TYPE_CHECKING:
 GUEST_DEVICE_ID = "guest_device"
 
 
+introspection_response_timer = Histogram(
+    "synapse_api_auth_delegated_introspection_response",
+    "Time taken to get a response for an introspection request",
+    labelnames=["code", SERVER_NAME_LABEL],
+)
+
+
 class Auth(Protocol):
     """The interface that an auth provider must implement."""
 

synapse/api/auth/internal.py

@@ -296,4 +296,4 @@ class InternalAuth(BaseAuth):
         Returns:
             True if the user is an admin
         """
-        return await self.store.is_server_admin(requester.user)
+        return await self.store.is_server_admin(requester.user.to_string())

synapse/api/auth/mas.py

@@ -0,0 +1,432 @@
+#
+# This file is licensed under the Affero General Public License (AGPL) version 3.
+#
+# Copyright (C) 2025 New Vector, Ltd
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as
+# published by the Free Software Foundation, either version 3 of the
+# License, or (at your option) any later version.
+#
+# See the GNU Affero General Public License for more details:
+# <https://www.gnu.org/licenses/agpl-3.0.html>.
+#
+#
+import logging
+from typing import TYPE_CHECKING, Optional
+from urllib.parse import urlencode
+
+from synapse._pydantic_compat import (
+    BaseModel,
+    Extra,
+    StrictBool,
+    StrictInt,
+    StrictStr,
+    ValidationError,
+)
+from synapse.api.auth.base import BaseAuth
+from synapse.api.errors import (
+    AuthError,
+    HttpResponseException,
+    InvalidClientTokenError,
+    SynapseError,
+    UnrecognizedRequestError,
+)
+from synapse.http.site import SynapseRequest
+from synapse.logging.context import PreserveLoggingContext
+from synapse.logging.opentracing import (
+    active_span,
+    force_tracing,
+    inject_request_headers,
+    start_active_span,
+)
+from synapse.metrics import SERVER_NAME_LABEL
+from synapse.synapse_rust.http_client import HttpClient
+from synapse.types import JsonDict, Requester, UserID, create_requester
+from synapse.util import json_decoder
+from synapse.util.caches.cached_call import RetryOnExceptionCachedCall
+from synapse.util.caches.response_cache import ResponseCache, ResponseCacheContext
+
+from . import introspection_response_timer
+
+if TYPE_CHECKING:
+    from synapse.rest.admin.experimental_features import ExperimentalFeature
+    from synapse.server import HomeServer
+
+logger = logging.getLogger(__name__)
+
+# Scope as defined by MSC2967
+# https://github.com/matrix-org/matrix-spec-proposals/pull/2967
+SCOPE_MATRIX_API = "urn:matrix:org.matrix.msc2967.client:api:*"
+SCOPE_MATRIX_DEVICE_PREFIX = "urn:matrix:org.matrix.msc2967.client:device:"
+
+
+class ServerMetadata(BaseModel):
+    class Config:
+        extra = Extra.allow
+
+    issuer: StrictStr
+    account_management_uri: StrictStr
+
+
+class IntrospectionResponse(BaseModel):
+    retrieved_at_ms: StrictInt
+    active: StrictBool
+    scope: Optional[StrictStr]
+    username: Optional[StrictStr]
+    sub: Optional[StrictStr]
+    device_id: Optional[StrictStr]
+    expires_in: Optional[StrictInt]
+
+    class Config:
+        extra = Extra.allow
+
+    def get_scope_set(self) -> set[str]:
+        if not self.scope:
+            return set()
+
+        return {token for token in self.scope.split(" ") if token}
+
+    def is_active(self, now_ms: int) -> bool:
+        if not self.active:
+            return False
+
+        # Compatibility tokens don't expire and don't have an 'expires_in' field
+        if self.expires_in is None:
+            return True
+
+        absolute_expiry_ms = self.expires_in * 1000 + self.retrieved_at_ms
+        return now_ms < absolute_expiry_ms
+
+
+class MasDelegatedAuth(BaseAuth):
+    def __init__(self, hs: "HomeServer"):
+        super().__init__(hs)
+
+        self.server_name = hs.hostname
+        self._clock = hs.get_clock()
+        self._config = hs.config.mas
+
+        self._http_client = hs.get_proxied_http_client()
+        self._rust_http_client = HttpClient(
+            reactor=hs.get_reactor(),
+            user_agent=self._http_client.user_agent.decode("utf8"),
+        )
+        self._server_metadata = RetryOnExceptionCachedCall[ServerMetadata](
+            self._load_metadata
+        )
+        self._force_tracing_for_users = hs.config.tracing.force_tracing_for_users
+
+        # # Token Introspection Cache
+        # This remembers what users/devices are represented by which access tokens,
+        # in order to reduce overall system load:
+        # - on Synapse (as requests are relatively expensive)
+        # - on the network
+        # - on MAS
+        #
+        # Since there is no invalidation mechanism currently,
+        # the entries expire after 2 minutes.
+        # This does mean tokens can be treated as valid by Synapse
+        # for longer than reality.
+        #
+        # Ideally, tokens should logically be invalidated in the following circumstances:
+        # - If a session logout happens.
+        #   In this case, MAS will delete the device within Synapse
+        #   anyway and this is good enough as an invalidation.
+        # - If the client refreshes their token in MAS.
+        #   In this case, the device still exists and it's not the end of the world for
+        #   the old access token to continue working for a short time.
+        self._introspection_cache: ResponseCache[str] = ResponseCache(
+            clock=self._clock,
+            name="mas_token_introspection",
+            server_name=self.server_name,
+            timeout_ms=120_000,
+            # don't log because the keys are access tokens
+            enable_logging=False,
+        )
+
+    @property
+    def _metadata_url(self) -> str:
+        return f"{self._config.endpoint.rstrip('/')}/.well-known/openid-configuration"
+
+    @property
+    def _introspection_endpoint(self) -> str:
+        return f"{self._config.endpoint.rstrip('/')}/oauth2/introspect"
+
+    async def _load_metadata(self) -> ServerMetadata:
+        response = await self._http_client.get_json(self._metadata_url)
+        metadata = ServerMetadata(**response)
+        return metadata
+
+    async def issuer(self) -> str:
+        metadata = await self._server_metadata.get()
+        return metadata.issuer
+
+    async def account_management_url(self) -> str:
+        metadata = await self._server_metadata.get()
+        return metadata.account_management_uri
+
+    async def auth_metadata(self) -> JsonDict:
+        metadata = await self._server_metadata.get()
+        return metadata.dict()
+
+    def is_request_using_the_shared_secret(self, request: SynapseRequest) -> bool:
+        """
+        Check if the request is using the shared secret.
+
+        Args:
+            request: The request to check.
+
+        Returns:
+            True if the request is using the shared secret, False otherwise.
+        """
+        access_token = self.get_access_token_from_request(request)
+        shared_secret = self._config.secret()
+        if not shared_secret:
+            return False
+
+        return access_token == shared_secret
+
+    async def _introspect_token(
+        self, token: str, cache_context: ResponseCacheContext[str]
+    ) -> IntrospectionResponse:
+        """
+        Send a token to the introspection endpoint and returns the introspection response
+
+        Parameters:
+            token: The token to introspect
+
+        Raises:
+            HttpResponseException: If the introspection endpoint returns a non-2xx response
+            ValueError: If the introspection endpoint returns an invalid JSON response
+            JSONDecodeError: If the introspection endpoint returns a non-JSON response
+            Exception: If the HTTP request fails
+
+        Returns:
+            The introspection response
+        """
+
+        # By default, we shouldn't cache the result unless we know it's valid
+        cache_context.should_cache = False
+        raw_headers: dict[str, str] = {
+            "Content-Type": "application/x-www-form-urlencoded",
+            "Accept": "application/json",
+            "Authorization": f"Bearer {self._config.secret()}",
+            # Tell MAS that we support reading the device ID as an explicit
+            # value, not encoded in the scope. This is supported by MAS 0.15+
+            "X-MAS-Supports-Device-Id": "1",
+        }
+
+        args = {"token": token, "token_type_hint": "access_token"}
+        body = urlencode(args, True)
+
+        # Do the actual request
+
+        logger.debug("Fetching token from MAS")
+        start_time = self._clock.time()
+        try:
+            with start_active_span("mas-introspect-token"):
+                inject_request_headers(raw_headers)
+                with PreserveLoggingContext():
+                    resp_body = await self._rust_http_client.post(
+                        url=self._introspection_endpoint,
+                        response_limit=1 * 1024 * 1024,
+                        headers=raw_headers,
+                        request_body=body,
+                    )
+        except HttpResponseException as e:
+            end_time = self._clock.time()
+            introspection_response_timer.labels(
+                code=e.code, **{SERVER_NAME_LABEL: self.server_name}
+            ).observe(end_time - start_time)
+            raise
+        except Exception:
+            end_time = self._clock.time()
+            introspection_response_timer.labels(
+                code="ERR", **{SERVER_NAME_LABEL: self.server_name}
+            ).observe(end_time - start_time)
+            raise
+
+        logger.debug("Fetched token from MAS")
+
+        end_time = self._clock.time()
+        introspection_response_timer.labels(
+            code=200, **{SERVER_NAME_LABEL: self.server_name}
+        ).observe(end_time - start_time)
+
+        raw_response = json_decoder.decode(resp_body.decode("utf-8"))
+        try:
+            response = IntrospectionResponse(
+                retrieved_at_ms=self._clock.time_msec(),
+                **raw_response,
+            )
+        except ValidationError as e:
+            raise ValueError(
+                "The introspection endpoint returned an invalid JSON response"
+            ) from e
+
+        # We had a valid response, so we can cache it
+        cache_context.should_cache = True
+        return response
+
+    async def is_server_admin(self, requester: Requester) -> bool:
+        return "urn:synapse:admin:*" in requester.scope
+
+    async def get_user_by_req(
+        self,
+        request: SynapseRequest,
+        allow_guest: bool = False,
+        allow_expired: bool = False,
+        allow_locked: bool = False,
+    ) -> Requester:
+        parent_span = active_span()
+        with start_active_span("get_user_by_req"):
+            access_token = self.get_access_token_from_request(request)
+
+            requester = await self.get_appservice_user(request, access_token)
+            if not requester:
+                requester = await self.get_user_by_access_token(
+                    token=access_token,
+                    allow_expired=allow_expired,
+                )
+
+            await self._record_request(request, requester)
+
+            request.requester = requester
+
+            if parent_span:
+                if requester.authenticated_entity in self._force_tracing_for_users:
+                    # request tracing is enabled for this user, so we need to force it
+                    # tracing on for the parent span (which will be the servlet span).
+                    #
+                    # It's too late for the get_user_by_req span to inherit the setting,
+                    # so we also force it on for that.
+                    force_tracing()
+                    force_tracing(parent_span)
+                parent_span.set_tag(
+                    "authenticated_entity", requester.authenticated_entity
+                )
+                parent_span.set_tag("user_id", requester.user.to_string())
+                if requester.device_id is not None:
+                    parent_span.set_tag("device_id", requester.device_id)
+                if requester.app_service is not None:
+                    parent_span.set_tag("appservice_id", requester.app_service.id)
+            return requester
+
+    async def get_user_by_access_token(
+        self,
+        token: str,
+        allow_expired: bool = False,
+    ) -> Requester:
+        try:
+            introspection_result = await self._introspection_cache.wrap(
+                token, self._introspect_token, token, cache_context=True
+            )
+        except Exception:
+            logger.exception("Failed to introspect token")
+            raise SynapseError(503, "Unable to introspect the access token")
+
+        logger.debug("Introspection result: %r", introspection_result)
+        if not introspection_result.is_active(self._clock.time_msec()):
+            raise InvalidClientTokenError("Token is not active")
+
+        # Let's look at the scope
+        scope = introspection_result.get_scope_set()
+
+        # Determine type of user based on presence of particular scopes
+        if SCOPE_MATRIX_API not in scope:
+            raise InvalidClientTokenError(
+                "Token doesn't grant access to the Matrix C-S API"
+            )
+
+        if introspection_result.username is None:
+            raise AuthError(
+                500,
+                "Invalid username claim in the introspection result",
+            )
+
+        user_id = UserID(
+            localpart=introspection_result.username,
+            domain=self.server_name,
+        )
+
+        # Try to find a user from the username claim
+        user_info = await self.store.get_user_by_id(user_id=user_id.to_string())
+        if user_info is None:
+            raise AuthError(
+                500,
+                "User not found",
+            )
+
+        # MAS will give us the device ID as an explicit value for *compatibility* sessions
+        # If present, we get it from here, if not we get it in the scope for next-gen sessions
+        device_id = introspection_result.device_id
+        if device_id is None:
+            # Find device_ids in scope
+            # We only allow a single device_id in the scope, so we find them all in the
+            # scope list, and raise if there are more than one. The OIDC server should be
+            # the one enforcing valid scopes, so we raise a 500 if we find an invalid scope.
+            device_ids = [
+                tok[len(SCOPE_MATRIX_DEVICE_PREFIX) :]
+                for tok in scope
+                if tok.startswith(SCOPE_MATRIX_DEVICE_PREFIX)
+            ]
+
+            if len(device_ids) > 1:
+                raise AuthError(
+                    500,
+                    "Multiple device IDs in scope",
+                )
+
+            device_id = device_ids[0] if device_ids else None
+
+        if device_id is not None:
+            # Sanity check the device_id
+            if len(device_id) > 255 or len(device_id) < 1:
+                raise AuthError(
+                    500,
+                    "Invalid device ID in introspection result",
+                )
+
+            # Make sure the device exists. This helps with introspection cache
+            # invalidation: if we log out, the device gets deleted by MAS
+            device = await self.store.get_device(
+                user_id=user_id.to_string(),
+                device_id=device_id,
+            )
+            if device is None:
+                # Invalidate the introspection cache, the device was deleted
+                self._introspection_cache.unset(token)
+                raise InvalidClientTokenError("Token is not active")
+
+        return create_requester(
+            user_id=user_id,
+            device_id=device_id,
+            scope=scope,
+        )
+
+    async def get_user_by_req_experimental_feature(
+        self,
+        request: SynapseRequest,
+        feature: "ExperimentalFeature",
+        allow_guest: bool = False,
+        allow_expired: bool = False,
+        allow_locked: bool = False,
+    ) -> Requester:
+        try:
+            requester = await self.get_user_by_req(
+                request,
+                allow_guest=allow_guest,
+                allow_expired=allow_expired,
+                allow_locked=allow_locked,
+            )
+            if await self.store.is_feature_enabled(requester.user.to_string(), feature):
+                return requester
+
+            raise UnrecognizedRequestError(code=404)
+        except (AuthError, InvalidClientTokenError):
+            if feature.is_globally_enabled(self.hs.config):
+                # If its globally enabled then return the auth error
+                raise
+
+            raise UnrecognizedRequestError(code=404)

synapse/api/auth/msc3861_delegated.py

@@ -28,7 +28,6 @@ from authlib.oauth2.auth import encode_client_secret_basic, encode_client_secret
 from authlib.oauth2.rfc7523 import ClientSecretJWT, PrivateKeyJWT, private_key_jwt_sign
 from authlib.oauth2.rfc7662 import IntrospectionToken
 from authlib.oidc.discovery import OpenIDProviderMetadata, get_well_known_url
-from prometheus_client import Histogram
 
 from synapse.api.auth.base import BaseAuth
 from synapse.api.errors import (
@@ -54,19 +53,14 @@ from synapse.util import json_decoder
 from synapse.util.caches.cached_call import RetryOnExceptionCachedCall
 from synapse.util.caches.response_cache import ResponseCache, ResponseCacheContext
 
+from . import introspection_response_timer
+
 if TYPE_CHECKING:
     from synapse.rest.admin.experimental_features import ExperimentalFeature
     from synapse.server import HomeServer
 
 logger = logging.getLogger(__name__)
 
-introspection_response_timer = Histogram(
-    "synapse_api_auth_delegated_introspection_response",
-    "Time taken to get a response for an introspection request",
-    labelnames=["code", SERVER_NAME_LABEL],
-)
-
-
 # Scope as defined by MSC2967
 # https://github.com/matrix-org/matrix-spec-proposals/pull/2967
 SCOPE_MATRIX_API = "urn:matrix:org.matrix.msc2967.client:api:*"

synapse/api/constants.py

@@ -46,6 +46,9 @@ MAX_USERID_LENGTH = 255
 # Constant value used for the pseudo-thread which is the main timeline.
 MAIN_TIMELINE: Final = "main"
 
+# MAX_INT + 1, so it always trumps any PL in canonical JSON.
+CREATOR_POWER_LEVEL = 2**53
+
 
 class Membership:
     """Represents the membership states of a user in a room."""
@@ -235,6 +238,8 @@ class EventContentFields:
     #
     # This is deprecated in MSC2175.
     ROOM_CREATOR: Final = "creator"
+    # MSC4289
+    ADDITIONAL_CREATORS: Final = "additional_creators"
 
     # The version of the room for `m.room.create` events.
     ROOM_VERSION: Final = "room_version"

synapse/api/errors.py

@@ -140,6 +140,12 @@ class Codes(str, Enum):
     # Part of MSC4155
     INVITE_BLOCKED = "ORG.MATRIX.MSC4155.M_INVITE_BLOCKED"
 
+    # Part of MSC4306: Thread Subscriptions
+    MSC4306_CONFLICTING_UNSUBSCRIPTION = (
+        "IO.ELEMENT.MSC4306.M_CONFLICTING_UNSUBSCRIPTION"
+    )
+    MSC4306_NOT_IN_THREAD = "IO.ELEMENT.MSC4306.M_NOT_IN_THREAD"
+
 
 class CodeMessageException(RuntimeError):
     """An exception with integer code, a message string attributes and optional headers.

synapse/api/room_versions.py

@@ -36,12 +36,14 @@ class EventFormatVersions:
     ROOM_V1_V2 = 1  # $id:server event id format: used for room v1 and v2
     ROOM_V3 = 2  # MSC1659-style $hash event id format: used for room v3
     ROOM_V4_PLUS = 3  # MSC1884-style $hash format: introduced for room v4
+    ROOM_V11_HYDRA_PLUS = 4  # MSC4291 room IDs as hashes: introduced for room HydraV11
 
 
 KNOWN_EVENT_FORMAT_VERSIONS = {
     EventFormatVersions.ROOM_V1_V2,
     EventFormatVersions.ROOM_V3,
     EventFormatVersions.ROOM_V4_PLUS,
+    EventFormatVersions.ROOM_V11_HYDRA_PLUS,
 }
 
 
@@ -50,6 +52,7 @@ class StateResolutionVersions:
 
     V1 = 1  # room v1 state res
     V2 = 2  # MSC1442 state res: room v2 and later
+    V2_1 = 3  # MSC4297 state res
 
 
 class RoomDisposition:
@@ -109,6 +112,10 @@ class RoomVersion:
     msc3931_push_features: Tuple[str, ...]  # values from PushRuleRoomFlag
     # MSC3757: Restricting who can overwrite a state event
     msc3757_enabled: bool
+    # MSC4289: Creator power enabled
+    msc4289_creator_power_enabled: bool
+    # MSC4291: Room IDs as hashes of the create event
+    msc4291_room_ids_as_hashes: bool
 
 
 class RoomVersions:
@@ -131,6 +138,8 @@ class RoomVersions:
         enforce_int_power_levels=False,
         msc3931_push_features=(),
         msc3757_enabled=False,
+        msc4289_creator_power_enabled=False,
+        msc4291_room_ids_as_hashes=False,
     )
     V2 = RoomVersion(
         "2",
@@ -151,6 +160,8 @@ class RoomVersions:
         enforce_int_power_levels=False,
         msc3931_push_features=(),
         msc3757_enabled=False,
+        msc4289_creator_power_enabled=False,
+        msc4291_room_ids_as_hashes=False,
     )
     V3 = RoomVersion(
         "3",
@@ -171,6 +182,8 @@ class RoomVersions:
         enforce_int_power_levels=False,
         msc3931_push_features=(),
         msc3757_enabled=False,
+        msc4289_creator_power_enabled=False,
+        msc4291_room_ids_as_hashes=False,
     )
     V4 = RoomVersion(
         "4",
@@ -191,6 +204,8 @@ class RoomVersions:
         enforce_int_power_levels=False,
         msc3931_push_features=(),
         msc3757_enabled=False,
+        msc4289_creator_power_enabled=False,
+        msc4291_room_ids_as_hashes=False,
     )
     V5 = RoomVersion(
         "5",
@@ -211,6 +226,8 @@ class RoomVersions:
         enforce_int_power_levels=False,
         msc3931_push_features=(),
         msc3757_enabled=False,
+        msc4289_creator_power_enabled=False,
+        msc4291_room_ids_as_hashes=False,
     )
     V6 = RoomVersion(
         "6",
@@ -231,6 +248,8 @@ class RoomVersions:
         enforce_int_power_levels=False,
         msc3931_push_features=(),
         msc3757_enabled=False,
+        msc4289_creator_power_enabled=False,
+        msc4291_room_ids_as_hashes=False,
     )
     V7 = RoomVersion(
         "7",
@@ -251,6 +270,8 @@ class RoomVersions:
         enforce_int_power_levels=False,
         msc3931_push_features=(),
         msc3757_enabled=False,
+        msc4289_creator_power_enabled=False,
+        msc4291_room_ids_as_hashes=False,
     )
     V8 = RoomVersion(
         "8",
@@ -271,6 +292,8 @@ class RoomVersions:
         enforce_int_power_levels=False,
         msc3931_push_features=(),
         msc3757_enabled=False,
+        msc4289_creator_power_enabled=False,
+        msc4291_room_ids_as_hashes=False,
     )
     V9 = RoomVersion(
         "9",
@@ -291,6 +314,8 @@ class RoomVersions:
         enforce_int_power_levels=False,
         msc3931_push_features=(),
         msc3757_enabled=False,
+        msc4289_creator_power_enabled=False,
+        msc4291_room_ids_as_hashes=False,
     )
     V10 = RoomVersion(
         "10",
@@ -311,6 +336,8 @@ class RoomVersions:
         enforce_int_power_levels=True,
         msc3931_push_features=(),
         msc3757_enabled=False,
+        msc4289_creator_power_enabled=False,
+        msc4291_room_ids_as_hashes=False,
     )
     MSC1767v10 = RoomVersion(
         # MSC1767 (Extensible Events) based on room version "10"
@@ -332,6 +359,8 @@ class RoomVersions:
         enforce_int_power_levels=True,
         msc3931_push_features=(PushRuleRoomFlag.EXTENSIBLE_EVENTS,),
         msc3757_enabled=False,
+        msc4289_creator_power_enabled=False,
+        msc4291_room_ids_as_hashes=False,
     )
     MSC3757v10 = RoomVersion(
         # MSC3757 (Restricting who can overwrite a state event) based on room version "10"
@@ -353,6 +382,8 @@ class RoomVersions:
         enforce_int_power_levels=True,
         msc3931_push_features=(),
         msc3757_enabled=True,
+        msc4289_creator_power_enabled=False,
+        msc4291_room_ids_as_hashes=False,
     )
     V11 = RoomVersion(
         "11",
@@ -373,6 +404,8 @@ class RoomVersions:
         enforce_int_power_levels=True,
         msc3931_push_features=(),
         msc3757_enabled=False,
+        msc4289_creator_power_enabled=False,
+        msc4291_room_ids_as_hashes=False,
     )
     MSC3757v11 = RoomVersion(
         # MSC3757 (Restricting who can overwrite a state event) based on room version "11"
@@ -394,6 +427,52 @@ class RoomVersions:
         enforce_int_power_levels=True,
         msc3931_push_features=(),
         msc3757_enabled=True,
+        msc4289_creator_power_enabled=False,
+        msc4291_room_ids_as_hashes=False,
+    )
+    HydraV11 = RoomVersion(
+        "org.matrix.hydra.11",
+        RoomDisposition.UNSTABLE,
+        EventFormatVersions.ROOM_V11_HYDRA_PLUS,
+        StateResolutionVersions.V2_1,  # Changed from v11
+        enforce_key_validity=True,
+        special_case_aliases_auth=False,
+        strict_canonicaljson=True,
+        limit_notifications_power_levels=True,
+        implicit_room_creator=True,  # Used by MSC3820
+        updated_redaction_rules=True,  # Used by MSC3820
+        restricted_join_rule=True,
+        restricted_join_rule_fix=True,
+        knock_join_rule=True,
+        msc3389_relation_redactions=False,
+        knock_restricted_join_rule=True,
+        enforce_int_power_levels=True,
+        msc3931_push_features=(),
+        msc3757_enabled=False,
+        msc4289_creator_power_enabled=True,  # Changed from v11
+        msc4291_room_ids_as_hashes=True,  # Changed from v11
+    )
+    V12 = RoomVersion(
+        "12",
+        RoomDisposition.STABLE,
+        EventFormatVersions.ROOM_V11_HYDRA_PLUS,
+        StateResolutionVersions.V2_1,  # Changed from v11
+        enforce_key_validity=True,
+        special_case_aliases_auth=False,
+        strict_canonicaljson=True,
+        limit_notifications_power_levels=True,
+        implicit_room_creator=True,  # Used by MSC3820
+        updated_redaction_rules=True,  # Used by MSC3820
+        restricted_join_rule=True,
+        restricted_join_rule_fix=True,
+        knock_join_rule=True,
+        msc3389_relation_redactions=False,
+        knock_restricted_join_rule=True,
+        enforce_int_power_levels=True,
+        msc3931_push_features=(),
+        msc3757_enabled=False,
+        msc4289_creator_power_enabled=True,  # Changed from v11
+        msc4291_room_ids_as_hashes=True,  # Changed from v11
     )
 
 
@@ -411,8 +490,10 @@ KNOWN_ROOM_VERSIONS: Dict[str, RoomVersion] = {
         RoomVersions.V9,
         RoomVersions.V10,
         RoomVersions.V11,
+        RoomVersions.V12,
         RoomVersions.MSC3757v10,
         RoomVersions.MSC3757v11,
+        RoomVersions.HydraV11,
     )
 }
 

synapse/config/_base.pyi

@@ -36,6 +36,7 @@ from synapse.config import (  # noqa: F401
     jwt,
     key,
     logger,
+    mas,
     metrics,
     modules,
     oembed,
@@ -124,6 +125,7 @@ class RootConfig:
     background_updates: background_updates.BackgroundUpdateConfig
     auto_accept_invites: auto_accept_invites.AutoAcceptInvitesConfig
     user_types: user_types.UserTypesConfig
+    mas: mas.MasConfig
 
     config_classes: List[Type["Config"]] = ...
     config_files: List[str]

synapse/config/auth.py

@@ -36,13 +36,14 @@ class AuthConfig(Config):
         if password_config is None:
             password_config = {}
 
-        # The default value of password_config.enabled is True, unless msc3861 is enabled.
-        msc3861_enabled = (
-            (config.get("experimental_features") or {})
-            .get("msc3861", {})
-            .get("enabled", False)
-        )
-        passwords_enabled = password_config.get("enabled", not msc3861_enabled)
+        auth_delegated = (config.get("experimental_features") or {}).get(
+            "msc3861", {}
+        ).get("enabled", False) or (
+            config.get("matrix_authentication_service") or {}
+        ).get("enabled", False)
+
+        # The default value of password_config.enabled is True, unless auth is delegated
+        passwords_enabled = password_config.get("enabled", not auth_delegated)
 
         # 'only_for_reauth' allows users who have previously set a password to use it,
         # even though passwords would otherwise be disabled.

synapse/config/homeserver.py

@@ -36,6 +36,7 @@ from .federation import FederationConfig
 from .jwt import JWTConfig
 from .key import KeyConfig
 from .logger import LoggingConfig
+from .mas import MasConfig
 from .metrics import MetricsConfig
 from .modules import ModulesConfig
 from .oembed import OembedConfig
@@ -109,4 +110,6 @@ class HomeServerConfig(RootConfig):
         BackgroundUpdateConfig,
         AutoAcceptInvitesConfig,
         UserTypesConfig,
+        # This must be last, as it checks for conflicts with other config options.
+        MasConfig,
     ]

synapse/config/mas.py

@@ -0,0 +1,192 @@
+#
+# This file is licensed under the Affero General Public License (AGPL) version 3.
+#
+# Copyright (C) 2025 New Vector, Ltd
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as
+# published by the Free Software Foundation, either version 3 of the
+# License, or (at your option) any later version.
+#
+# See the GNU Affero General Public License for more details:
+# <https://www.gnu.org/licenses/agpl-3.0.html>.
+#
+#
+
+from typing import Any, Optional
+
+from synapse._pydantic_compat import (
+    AnyHttpUrl,
+    Field,
+    FilePath,
+    StrictBool,
+    StrictStr,
+    ValidationError,
+    validator,
+)
+from synapse.config.experimental import read_secret_from_file_once
+from synapse.types import JsonDict
+from synapse.util.pydantic_models import ParseModel
+
+from ._base import Config, ConfigError, RootConfig
+
+
+class MasConfigModel(ParseModel):
+    enabled: StrictBool = False
+    endpoint: AnyHttpUrl = Field(default="http://localhost:8080")
+    secret: Optional[StrictStr] = Field(default=None)
+    secret_path: Optional[FilePath] = Field(default=None)
+
+    @validator("secret")
+    def validate_secret_is_set_if_enabled(cls, v: Any, values: dict) -> Any:
+        if values.get("enabled", False) and not values.get("secret_path") and not v:
+            raise ValueError(
+                "You must set a `secret` or `secret_path` when enabling Matrix Authentication Service integration."
+            )
+
+        return v
+
+    @validator("secret_path")
+    def validate_secret_path_is_set_if_enabled(cls, v: Any, values: dict) -> Any:
+        if values.get("secret"):
+            raise ValueError(
+                "`secret` and `secret_path` cannot be set at the same time."
+            )
+
+        return v
+
+
+class MasConfig(Config):
+    section = "mas"
+
+    def read_config(
+        self, config: JsonDict, allow_secrets_in_config: bool, **kwargs: Any
+    ) -> None:
+        mas_config = config.get("matrix_authentication_service", {})
+        if mas_config is None:
+            mas_config = {}
+
+        try:
+            parsed = MasConfigModel(**mas_config)
+        except ValidationError as e:
+            raise ConfigError(
+                "Could not validate Matrix Authentication Service configuration",
+                path=("matrix_authentication_service",),
+            ) from e
+
+        if parsed.secret and not allow_secrets_in_config:
+            raise ConfigError(
+                "Config options that expect an in-line secret as value are disabled",
+                ("matrix_authentication_service", "secret"),
+            )
+
+        self.enabled = parsed.enabled
+        self.endpoint = parsed.endpoint
+        self._secret = parsed.secret
+        self._secret_path = parsed.secret_path
+
+        self.check_config_conflicts(self.root)
+
+    def check_config_conflicts(
+        self,
+        root: RootConfig,
+    ) -> None:
+        """Checks for any configuration conflicts with other parts of Synapse.
+
+        Raises:
+            ConfigError: If there are any configuration conflicts.
+        """
+
+        if not self.enabled:
+            return
+
+        if root.experimental.msc3861.enabled:
+            raise ConfigError(
+                "Experimental MSC3861 was replaced by Matrix Authentication Service."
+                "Please disable MSC3861 or disable Matrix Authentication Service.",
+                ("experimental", "msc3861"),
+            )
+
+        if (
+            root.auth.password_enabled_for_reauth
+            or root.auth.password_enabled_for_login
+        ):
+            raise ConfigError(
+                "Password auth cannot be enabled when OAuth delegation is enabled",
+                ("password_config", "enabled"),
+            )
+
+        if root.registration.enable_registration:
+            raise ConfigError(
+                "Registration cannot be enabled when OAuth delegation is enabled",
+                ("enable_registration",),
+            )
+
+        # We only need to test the user consent version, as if it must be set if the user_consent section was present in the config
+        if root.consent.user_consent_version is not None:
+            raise ConfigError(
+                "User consent cannot be enabled when OAuth delegation is enabled",
+                ("user_consent",),
+            )
+
+        if (
+            root.oidc.oidc_enabled
+            or root.saml2.saml2_enabled
+            or root.cas.cas_enabled
+            or root.jwt.jwt_enabled
+        ):
+            raise ConfigError("SSO cannot be enabled when OAuth delegation is enabled")
+
+        if bool(root.authproviders.password_providers):
+            raise ConfigError(
+                "Password auth providers cannot be enabled when OAuth delegation is enabled"
+            )
+
+        if root.captcha.enable_registration_captcha:
+            raise ConfigError(
+                "CAPTCHA cannot be enabled when OAuth delegation is enabled",
+                ("captcha", "enable_registration_captcha"),
+            )
+
+        if root.auth.login_via_existing_enabled:
+            raise ConfigError(
+                "Login via existing session cannot be enabled when OAuth delegation is enabled",
+                ("login_via_existing_session", "enabled"),
+            )
+
+        if root.registration.refresh_token_lifetime:
+            raise ConfigError(
+                "refresh_token_lifetime cannot be set when OAuth delegation is enabled",
+                ("refresh_token_lifetime",),
+            )
+
+        if root.registration.nonrefreshable_access_token_lifetime:
+            raise ConfigError(
+                "nonrefreshable_access_token_lifetime cannot be set when OAuth delegation is enabled",
+                ("nonrefreshable_access_token_lifetime",),
+            )
+
+        if root.registration.session_lifetime:
+            raise ConfigError(
+                "session_lifetime cannot be set when OAuth delegation is enabled",
+                ("session_lifetime",),
+            )
+
+        if root.registration.enable_3pid_changes:
+            raise ConfigError(
+                "enable_3pid_changes cannot be enabled when OAuth delegation is enabled",
+                ("enable_3pid_changes",),
+            )
+
+    def secret(self) -> str:
+        if self._secret is not None:
+            return self._secret
+        elif self._secret_path is not None:
+            return read_secret_from_file_once(
+                str(self._secret_path),
+                ("matrix_authentication_service", "secret_path"),
+            )
+        else:
+            raise RuntimeError(
+                "Neither `secret` nor `secret_path` are set, this is a bug.",
+            )

synapse/config/registration.py

@@ -148,15 +148,14 @@ class RegistrationConfig(Config):
         self.enable_set_displayname = config.get("enable_set_displayname", True)
         self.enable_set_avatar_url = config.get("enable_set_avatar_url", True)
 
+        auth_delegated = (config.get("experimental_features") or {}).get(
+            "msc3861", {}
+        ).get("enabled", False) or (
+            config.get("matrix_authentication_service") or {}
+        ).get("enabled", False)
+
         # The default value of enable_3pid_changes is True, unless msc3861 is enabled.
-        msc3861_enabled = (
-            (config.get("experimental_features") or {})
-            .get("msc3861", {})
-            .get("enabled", False)
-        )
-        self.enable_3pid_changes = config.get(
-            "enable_3pid_changes", not msc3861_enabled
-        )
+        self.enable_3pid_changes = config.get("enable_3pid_changes", not auth_delegated)
 
         self.disable_msisdn_registration = config.get(
             "disable_msisdn_registration", False

synapse/crypto/event_signing.py

@@ -101,6 +101,9 @@ def compute_content_hash(
     event_dict.pop("outlier", None)
     event_dict.pop("destinations", None)
 
+    # N.B. no need to pop the room_id from create events in MSC4291 rooms
+    # as they shouldn't have one.
+
     event_json_bytes = encode_canonical_json(event_dict)
 
     hashed = hash_algorithm(event_json_bytes)

synapse/event_auth.py

@@ -45,6 +45,7 @@ from signedjson.sign import SignatureVerifyException, verify_signed_json
 from unpaddedbase64 import decode_base64
 
 from synapse.api.constants import (
+    CREATOR_POWER_LEVEL,
     MAX_PDU_SIZE,
     EventContentFields,
     EventTypes,
@@ -64,6 +65,7 @@ from synapse.api.room_versions import (
     RoomVersion,
     RoomVersions,
 )
+from synapse.events import is_creator
 from synapse.state import CREATE_KEY
 from synapse.storage.databases.main.events_worker import EventRedactBehaviour
 from synapse.types import (
@@ -261,7 +263,8 @@ async def check_state_independent_auth_rules(
                 f"Event {event.event_id} has unexpected auth_event for {k}: {auth_event_id}",
             )
 
-        # We also need to check that the auth event itself is not rejected.
+        # 2.3 ... If there are entries which were themselves rejected under the checks performed on receipt
+        # of a PDU, reject.
         if auth_event.rejected_reason:
             raise AuthError(
                 403,
@@ -271,7 +274,7 @@ async def check_state_independent_auth_rules(
 
         auth_dict[k] = auth_event_id
 
-    # 3. If event does not have a m.room.create in its auth_events, reject.
+    # 2.4. If event does not have a m.room.create in its auth_events, reject.
     creation_event = auth_dict.get((EventTypes.Create, ""), None)
     if not creation_event:
         raise AuthError(403, "No create event in auth events")
@@ -311,13 +314,14 @@ def check_state_dependent_auth_rules(
 
     # Later code relies on there being a create event e.g _can_federate, _is_membership_change_allowed
     # so produce a more intelligible error if we don't have one.
-    if auth_dict.get(CREATE_KEY) is None:
+    create_event = auth_dict.get(CREATE_KEY)
+    if create_event is None:
         raise AuthError(
             403, f"Event {event.event_id} is missing a create event in auth_events."
         )
 
     # additional check for m.federate
-    creating_domain = get_domain_from_id(event.room_id)
+    creating_domain = get_domain_from_id(create_event.sender)
     originating_domain = get_domain_from_id(event.sender)
     if creating_domain != originating_domain:
         if not _can_federate(event, auth_dict):
@@ -470,12 +474,20 @@ def _check_create(event: "EventBase") -> None:
     if event.prev_event_ids():
         raise AuthError(403, "Create event has prev events")
 
-    # 1.2 If the domain of the room_id does not match the domain of the sender,
-    # reject.
-    sender_domain = get_domain_from_id(event.sender)
-    room_id_domain = get_domain_from_id(event.room_id)
-    if room_id_domain != sender_domain:
-        raise AuthError(403, "Creation event's room_id domain does not match sender's")
+    if event.room_version.msc4291_room_ids_as_hashes:
+        # 1.2 If the create event has a room_id, reject
+        if "room_id" in event:
+            raise AuthError(403, "Create event has a room_id")
+    else:
+        # 1.2 If the domain of the room_id does not match the domain of the sender,
+        # reject.
+        if not event.room_version.msc4291_room_ids_as_hashes:
+            sender_domain = get_domain_from_id(event.sender)
+            room_id_domain = get_domain_from_id(event.room_id)
+            if room_id_domain != sender_domain:
+                raise AuthError(
+                    403, "Creation event's room_id domain does not match sender's"
+                )
 
     # 1.3 If content.room_version is present and is not a recognised version, reject
     room_version_prop = event.content.get("room_version", "1")
@@ -492,6 +504,16 @@ def _check_create(event: "EventBase") -> None:
     ):
         raise AuthError(403, "Create event lacks a 'creator' property")
 
+    # 1.5 If the additional_creators field is present and is not an array of strings where each
+    # string is a valid user ID, reject.
+    if (
+        event.room_version.msc4289_creator_power_enabled
+        and EventContentFields.ADDITIONAL_CREATORS in event.content
+    ):
+        check_valid_additional_creators(
+            event.content[EventContentFields.ADDITIONAL_CREATORS]
+        )
+
 
 def _can_federate(event: "EventBase", auth_events: StateMap["EventBase"]) -> bool:
     creation_event = auth_events.get((EventTypes.Create, ""))
@@ -533,7 +555,13 @@ def _is_membership_change_allowed(
 
     target_user_id = event.state_key
 
-    creating_domain = get_domain_from_id(event.room_id)
+    # We need the create event in order to check if we can federate or not.
+    # If it's missing, yell loudly. Previously we only did this inside the
+    # _can_federate check.
+    create_event = auth_events.get((EventTypes.Create, ""))
+    if not create_event:
+        raise AuthError(403, "Create event missing from auth_events")
+    creating_domain = get_domain_from_id(create_event.sender)
     target_domain = get_domain_from_id(target_user_id)
     if creating_domain != target_domain:
         if not _can_federate(event, auth_events):
@@ -903,6 +931,32 @@ def _check_power_levels(
         except Exception:
             raise SynapseError(400, "Not a valid power level: %s" % (v,))
 
+    if room_version_obj.msc4289_creator_power_enabled:
+        # Enforce the creator does not appear in the users map
+        create_event = auth_events.get((EventTypes.Create, ""))
+        if not create_event:
+            raise SynapseError(
+                400, "Cannot check power levels without a create event in auth_events"
+            )
+        if create_event.sender in user_list:
+            raise SynapseError(
+                400,
+                "Creator user %s must not appear in content.users"
+                % (create_event.sender,),
+            )
+        additional_creators = create_event.content.get(
+            EventContentFields.ADDITIONAL_CREATORS, []
+        )
+        if additional_creators:
+            creators_in_user_list = set(additional_creators).intersection(
+                set(user_list)
+            )
+            if len(creators_in_user_list) > 0:
+                raise SynapseError(
+                    400,
+                    "Additional creators users must not appear in content.users",
+                )
+
     # Reject events with stringy power levels if required by room version
     if (
         event.type == EventTypes.PowerLevels
@@ -1028,6 +1082,9 @@ def get_user_power_level(user_id: str, auth_events: StateMap["EventBase"]) -> in
         "A create event in the auth events chain is required to calculate user power level correctly,"
         " but was not found. This indicates a bug"
     )
+    if create_event.room_version.msc4289_creator_power_enabled:
+        if is_creator(create_event, user_id):
+            return CREATOR_POWER_LEVEL
     power_level_event = get_power_level_event(auth_events)
     if power_level_event:
         level = power_level_event.content.get("users", {}).get(user_id)
@@ -1188,3 +1245,26 @@ def auth_types_for_event(
                 auth_types.add(key)
 
     return auth_types
+
+
+def check_valid_additional_creators(additional_creators: Any) -> None:
+    """Check if the additional_creators provided is valid according to MSC4289.
+
+    The additional_creators can be supplied from an m.room.create event or from an /upgrade request.
+
+    Raises:
+        AuthError if the additional_creators is invalid for some reason.
+    """
+    if type(additional_creators) is not list:
+        raise AuthError(400, "additional_creators must be an array")
+    for entry in additional_creators:
+        if type(entry) is not str:
+            raise AuthError(400, "entry in additional_creators is not a string")
+        if not UserID.is_valid(entry):
+            raise AuthError(400, "entry in additional_creators is not a valid user ID")
+        # UserID.is_valid doesn't actually validate everything, so check the rest manually.
+        if len(entry) > 255 or len(entry.encode("utf-8")) > 255:
+            raise AuthError(
+                400,
+                "entry in additional_creators too long",
+            )

synapse/events/__init__.py

@@ -41,10 +41,13 @@ from typing import (
 import attr
 from unpaddedbase64 import encode_base64
 
-from synapse.api.constants import EventTypes, RelationTypes
+from synapse.api.constants import EventContentFields, EventTypes, RelationTypes
 from synapse.api.room_versions import EventFormatVersions, RoomVersion, RoomVersions
 from synapse.synapse_rust.events import EventInternalMetadata
-from synapse.types import JsonDict, StrCollection
+from synapse.types import (
+    JsonDict,
+    StrCollection,
+)
 from synapse.util.caches import intern_dict
 from synapse.util.frozenutils import freeze
 
@@ -209,7 +212,6 @@ class EventBase(metaclass=abc.ABCMeta):
     content: DictProperty[JsonDict] = DictProperty("content")
     hashes: DictProperty[Dict[str, str]] = DictProperty("hashes")
     origin_server_ts: DictProperty[int] = DictProperty("origin_server_ts")
-    room_id: DictProperty[str] = DictProperty("room_id")
     sender: DictProperty[str] = DictProperty("sender")
     # TODO state_key should be Optional[str]. This is generally asserted in Synapse
     # by calling is_state() first (which ensures it is not None), but it is hard (not possible?)
@@ -224,6 +226,10 @@ class EventBase(metaclass=abc.ABCMeta):
     def event_id(self) -> str:
         raise NotImplementedError()
 
+    @property
+    def room_id(self) -> str:
+        raise NotImplementedError()
+
     @property
     def membership(self) -> str:
         return self.content["membership"]
@@ -386,6 +392,10 @@ class FrozenEvent(EventBase):
     def event_id(self) -> str:
         return self._event_id
 
+    @property
+    def room_id(self) -> str:
+        return self._dict["room_id"]
+
 
 class FrozenEventV2(EventBase):
     format_version = EventFormatVersions.ROOM_V3  # All events of this type are V2
@@ -443,6 +453,10 @@ class FrozenEventV2(EventBase):
         self._event_id = "$" + encode_base64(compute_event_reference_hash(self)[1])
         return self._event_id
 
+    @property
+    def room_id(self) -> str:
+        return self._dict["room_id"]
+
     def prev_event_ids(self) -> List[str]:
         """Returns the list of prev event IDs. The order matches the order
         specified in the event, though there is no meaning to it.
@@ -481,6 +495,67 @@ class FrozenEventV3(FrozenEventV2):
         return self._event_id
 
 
+class FrozenEventV4(FrozenEventV3):
+    """FrozenEventV4 for MSC4291 room IDs are hashes"""
+
+    format_version = EventFormatVersions.ROOM_V11_HYDRA_PLUS
+
+    """Override the room_id for m.room.create events"""
+
+    def __init__(
+        self,
+        event_dict: JsonDict,
+        room_version: RoomVersion,
+        internal_metadata_dict: Optional[JsonDict] = None,
+        rejected_reason: Optional[str] = None,
+    ):
+        super().__init__(
+            event_dict=event_dict,
+            room_version=room_version,
+            internal_metadata_dict=internal_metadata_dict,
+            rejected_reason=rejected_reason,
+        )
+        self._room_id: Optional[str] = None
+
+    @property
+    def room_id(self) -> str:
+        # if we have calculated the room ID already, don't do it again.
+        if self._room_id:
+            return self._room_id
+
+        is_create_event = self.type == EventTypes.Create and self.get_state_key() == ""
+
+        # for non-create events: use the supplied value from the JSON, as per FrozenEventV3
+        if not is_create_event:
+            self._room_id = self._dict["room_id"]
+            assert self._room_id is not None
+            return self._room_id
+
+        # for create events: calculate the room ID
+        from synapse.crypto.event_signing import compute_event_reference_hash
+
+        self._room_id = "!" + encode_base64(
+            compute_event_reference_hash(self)[1], urlsafe=True
+        )
+        return self._room_id
+
+    def auth_event_ids(self) -> StrCollection:
+        """Returns the list of auth event IDs. The order matches the order
+        specified in the event, though there is no meaning to it.
+        Returns:
+            The list of event IDs of this event's auth_events
+            Includes the creation event ID for convenience of all the codepaths
+            which expects the auth chain to include the creator ID, even though
+            it's explicitly not included on the wire. Excludes the create event
+            for the create event itself.
+        """
+        create_event_id = "$" + self.room_id[1:]
+        assert create_event_id not in self._dict["auth_events"]
+        if self.type == EventTypes.Create and self.get_state_key() == "":
+            return self._dict["auth_events"]  # should be []
+        return self._dict["auth_events"] + [create_event_id]
+
+
 def _event_type_from_format_version(
     format_version: int,
 ) -> Type[Union[FrozenEvent, FrozenEventV2, FrozenEventV3]]:
@@ -500,6 +575,8 @@ def _event_type_from_format_version(
         return FrozenEventV2
     elif format_version == EventFormatVersions.ROOM_V4_PLUS:
         return FrozenEventV3
+    elif format_version == EventFormatVersions.ROOM_V11_HYDRA_PLUS:
+        return FrozenEventV4
     else:
         raise Exception("No event format %r" % (format_version,))
 
@@ -559,6 +636,23 @@ def relation_from_event(event: EventBase) -> Optional[_EventRelation]:
     return _EventRelation(parent_id, rel_type, aggregation_key)
 
 
+def is_creator(create: EventBase, user_id: str) -> bool:
+    """
+    Return true if the provided user ID is the room creator.
+
+    This includes additional creators in MSC4289.
+    """
+    assert create.type == EventTypes.Create
+    if create.sender == user_id:
+        return True
+    if create.room_version.msc4289_creator_power_enabled:
+        additional_creators = set(
+            create.content.get(EventContentFields.ADDITIONAL_CREATORS, [])
+        )
+        return user_id in additional_creators
+    return False
+
+
 @attr.s(slots=True, frozen=True, auto_attribs=True)
 class StrippedStateEvent:
     """

synapse/events/builder.py

@@ -82,7 +82,8 @@ class EventBuilder:
 
     room_version: RoomVersion
 
-    room_id: str
+    # MSC4291 makes the room ID == the create event ID. This means the create event has no room_id.
+    room_id: Optional[str]
     type: str
     sender: str
 
@@ -142,7 +143,14 @@ class EventBuilder:
         Returns:
             The signed and hashed event.
         """
+        # Create events always have empty auth_events.
+        if self.type == EventTypes.Create and self.is_state() and self.state_key == "":
+            auth_event_ids = []
+
+        # Calculate auth_events for non-create events
         if auth_event_ids is None:
+            # Every non-create event must have a room ID
+            assert self.room_id is not None
             state_ids = await self._state.compute_state_after_events(
                 self.room_id,
                 prev_event_ids,
@@ -224,12 +232,31 @@ class EventBuilder:
             "auth_events": auth_events,
             "prev_events": prev_events,
             "type": self.type,
-            "room_id": self.room_id,
             "sender": self.sender,
             "content": self.content,
             "unsigned": self.unsigned,
             "depth": depth,
         }
+        if self.room_id is not None:
+            event_dict["room_id"] = self.room_id
+
+        if self.room_version.msc4291_room_ids_as_hashes:
+            # In MSC4291: the create event has no room ID as the create event ID /is/ the room ID.
+            if (
+                self.type == EventTypes.Create
+                and self.is_state()
+                and self._state_key == ""
+            ):
+                assert self.room_id is None
+            else:
+                # All other events do not reference the create event in auth_events, as the room ID
+                # /is/ the create event. However, the rest of the code (for consistency between room
+                # versions) assume that the create event remains part of the auth events. c.f. event
+                # class which automatically adds the create event when `.auth_event_ids()` is called
+                assert self.room_id is not None
+                create_event_id = "$" + self.room_id[1:]
+                auth_event_ids.remove(create_event_id)
+                event_dict["auth_events"] = auth_event_ids
 
         if self.is_state():
             event_dict["state_key"] = self._state_key
@@ -285,7 +312,7 @@ class EventBuilderFactory:
             room_version=room_version,
             type=key_values["type"],
             state_key=key_values.get("state_key"),
-            room_id=key_values["room_id"],
+            room_id=key_values.get("room_id"),
             sender=key_values["sender"],
             content=key_values.get("content", {}),
             unsigned=key_values.get("unsigned", {}),

synapse/events/utils.py

@@ -176,9 +176,12 @@ def prune_event_dict(room_version: RoomVersion, event_dict: JsonDict) -> JsonDic
         if room_version.updated_redaction_rules:
             # MSC2176 rules state that create events cannot have their `content` redacted.
             new_content = event_dict["content"]
-        elif not room_version.implicit_room_creator:
+        if not room_version.implicit_room_creator:
             # Some room versions give meaning to `creator`
             add_fields("creator")
+        if room_version.msc4291_room_ids_as_hashes:
+            # room_id is not allowed on the create event as it's derived from the event ID
+            allowed_keys.remove("room_id")
 
     elif event_type == EventTypes.JoinRules:
         add_fields("join_rule")
@@ -527,6 +530,10 @@ def serialize_event(
     if config.as_client_event:
         d = config.event_format(d)
 
+    # Ensure the room_id field is set for create events in MSC4291 rooms
+    if e.type == EventTypes.Create and e.room_version.msc4291_room_ids_as_hashes:
+        d["room_id"] = e.room_id
+
     # If the event is a redaction, the field with the redacted event ID appears
     # in a different location depending on the room version. e.redacts handles
     # fetching from the proper location; copy it to the other location for forwards-
@@ -872,6 +879,14 @@ def strip_event(event: EventBase) -> JsonDict:
     Stripped state events can only have the `sender`, `type`, `state_key` and `content`
     properties present.
     """
+    # MSC4311: Ensure the create event is available on invites and knocks.
+    # TODO: Implement the rest of MSC4311
+    if (
+        event.room_version.msc4291_room_ids_as_hashes
+        and event.type == EventTypes.Create
+        and event.get_state_key() == ""
+    ):
+        return event.get_pdu_json()
 
     return {
         "type": event.type,

synapse/events/validator.py

@@ -183,8 +183,18 @@ class EventValidator:
         fields an event would have
         """
 
+        create_event_as_room_id = (
+            event.room_version.msc4291_room_ids_as_hashes
+            and event.type == EventTypes.Create
+            and hasattr(event, "state_key")
+            and event.state_key == ""
+        )
+
         strings = ["room_id", "sender", "type"]
 
+        if create_event_as_room_id:
+            strings.remove("room_id")
+
         if hasattr(event, "state_key"):
             strings.append("state_key")
 
@@ -192,7 +202,14 @@ class EventValidator:
             if not isinstance(getattr(event, s), str):
                 raise SynapseError(400, "Not '%s' a string type" % (s,))
 
-        RoomID.from_string(event.room_id)
+        if not create_event_as_room_id:
+            assert event.room_id is not None
+            RoomID.from_string(event.room_id)
+            if event.room_version.msc4291_room_ids_as_hashes and not RoomID.is_valid(
+                event.room_id
+            ):
+                raise SynapseError(400, f"Invalid room ID '{event.room_id}'")
+
         UserID.from_string(event.sender)
 
         if event.type == EventTypes.Message:

synapse/federation/federation_base.py

@@ -343,6 +343,21 @@ def event_from_pdu_json(pdu_json: JsonDict, room_version: RoomVersion) -> EventB
     if room_version.strict_canonicaljson:
         validate_canonicaljson(pdu_json)
 
+    # enforce that MSC4291 auth events don't include the create event.
+    # N.B. if they DO include a spurious create event, it'll fail auth checks elsewhere, so we don't
+    # need to do expensive DB lookups to find which event ID is the create event here.
+    if room_version.msc4291_room_ids_as_hashes:
+        room_id = pdu_json.get("room_id")
+        if room_id:
+            create_event_id = "$" + room_id[1:]
+            auth_events = pdu_json.get("auth_events")
+            if auth_events:
+                if create_event_id in auth_events:
+                    raise SynapseError(
+                        400,
+                        "auth_events must not contain the create event",
+                        Codes.BAD_JSON,
+                    )
     event = make_event_from_dict(pdu_json, room_version)
     return event
 

synapse/handlers/auth.py

@@ -282,7 +282,9 @@ class AuthHandler:
         # response.
         self._extra_attributes: Dict[str, SsoLoginExtraAttributes] = {}
 
-        self.msc3861_oauth_delegation_enabled = hs.config.experimental.msc3861.enabled
+        self._auth_delegation_enabled = (
+            hs.config.mas.enabled or hs.config.experimental.msc3861.enabled
+        )
 
     async def validate_user_via_ui_auth(
         self,
@@ -333,7 +335,7 @@ class AuthHandler:
             LimitExceededError if the ratelimiter's failed request count for this
                 user is too high to proceed
         """
-        if self.msc3861_oauth_delegation_enabled:
+        if self._auth_delegation_enabled:
             raise SynapseError(
                 HTTPStatus.INTERNAL_SERVER_ERROR, "UIA shouldn't be used with MSC3861"
             )

synapse/handlers/event_auth.py

@@ -23,6 +23,8 @@ from typing import TYPE_CHECKING, List, Mapping, Optional, Union
 
 from synapse import event_auth
 from synapse.api.constants import (
+    CREATOR_POWER_LEVEL,
+    EventContentFields,
     EventTypes,
     JoinRules,
     Membership,
@@ -141,6 +143,8 @@ class EventAuthHandler:
         Raises:
             SynapseError if no appropriate user is found.
         """
+        create_event_id = current_state_ids[(EventTypes.Create, "")]
+        create_event = await self._store.get_event(create_event_id)
         power_level_event_id = current_state_ids.get((EventTypes.PowerLevels, ""))
         invite_level = 0
         users_default_level = 0
@@ -156,15 +160,28 @@ class EventAuthHandler:
 
         # Find the user with the highest power level (only interested in local
         # users).
+        user_power_level = 0
+        chosen_user = None
         local_users_in_room = await self._store.get_local_users_in_room(room_id)
-        chosen_user = max(
-            local_users_in_room,
-            key=lambda user: users.get(user, users_default_level),
-            default=None,
-        )
+        if create_event.room_version.msc4289_creator_power_enabled:
+            creators = set(
+                create_event.content.get(EventContentFields.ADDITIONAL_CREATORS, [])
+            )
+            creators.add(create_event.sender)
+            local_creators = creators.intersection(set(local_users_in_room))
+            if len(local_creators) > 0:
+                chosen_user = local_creators.pop()  # random creator
+                user_power_level = CREATOR_POWER_LEVEL
+        else:
+            chosen_user = max(
+                local_users_in_room,
+                key=lambda user: users.get(user, users_default_level),
+                default=None,
+            )
+            # Return the chosen if they can issue invites.
+            if chosen_user:
+                user_power_level = users.get(chosen_user, users_default_level)
 
-        # Return the chosen if they can issue invites.
-        user_power_level = users.get(chosen_user, users_default_level)
         if chosen_user and user_power_level >= invite_level:
             logger.debug(
                 "Found a user who can issue invites  %s with power level %d >= invite level %d",

synapse/handlers/message.py

@@ -22,7 +22,7 @@
 import logging
 import random
 from http import HTTPStatus
-from typing import TYPE_CHECKING, Any, Dict, List, Mapping, Optional, Tuple
+from typing import TYPE_CHECKING, Any, Dict, List, Mapping, Optional, Sequence, Tuple
 
 from canonicaljson import encode_canonical_json
 
@@ -55,7 +55,11 @@ from synapse.api.urls import ConsentURIBuilder
 from synapse.event_auth import validate_event_for_room_version
 from synapse.events import EventBase, relation_from_event
 from synapse.events.builder import EventBuilder
-from synapse.events.snapshot import EventContext, UnpersistedEventContextBase
+from synapse.events.snapshot import (
+    EventContext,
+    UnpersistedEventContext,
+    UnpersistedEventContextBase,
+)
 from synapse.events.utils import SerializeEventConfig, maybe_upsert_event_field
 from synapse.events.validator import EventValidator
 from synapse.handlers.directory import DirectoryHandler
@@ -66,6 +70,7 @@ from synapse.metrics.background_process_metrics import run_as_background_process
 from synapse.replication.http.send_events import ReplicationSendEventsRestServlet
 from synapse.storage.databases.main.events_worker import EventRedactBehaviour
 from synapse.types import (
+    JsonDict,
     PersistedEventPosition,
     Requester,
     RoomAlias,
@@ -683,7 +688,10 @@ class EventCreationHandler:
                         Codes.USER_ACCOUNT_SUSPENDED,
                     )
 
-        if event_dict["type"] == EventTypes.Create and event_dict["state_key"] == "":
+        is_create_event = (
+            event_dict["type"] == EventTypes.Create and event_dict["state_key"] == ""
+        )
+        if is_create_event:
             room_version_id = event_dict["content"]["room_version"]
             maybe_room_version_obj = KNOWN_ROOM_VERSIONS.get(room_version_id)
             if not maybe_room_version_obj:
@@ -789,6 +797,7 @@ class EventCreationHandler:
         """
         # the only thing the user can do is join the server notices room.
         if builder.type == EventTypes.Member:
+            assert builder.room_id is not None
             membership = builder.content.get("membership", None)
             if membership == Membership.JOIN:
                 return await self.store.is_server_notice_room(builder.room_id)
@@ -1254,13 +1263,40 @@ class EventCreationHandler:
                 for_verification=False,
             )
 
+        if (
+            builder.room_version.msc4291_room_ids_as_hashes
+            and builder.type == EventTypes.Create
+            and builder.is_state()
+        ):
+            if builder.room_id is not None:
+                raise SynapseError(
+                    400,
+                    "Cannot resend m.room.create event",
+                    Codes.INVALID_PARAM,
+                )
+        else:
+            assert builder.room_id is not None
+
         if prev_event_ids is not None:
             assert len(prev_event_ids) <= 10, (
                 "Attempting to create an event with %i prev_events"
                 % (len(prev_event_ids),)
             )
         else:
-            prev_event_ids = await self.store.get_prev_events_for_room(builder.room_id)
+            if builder.room_id:
+                prev_event_ids = await self.store.get_prev_events_for_room(
+                    builder.room_id
+                )
+            else:
+                prev_event_ids = []  # can only happen for the create event in MSC4291 rooms
+
+        if builder.type == EventTypes.Create and builder.is_state():
+            if len(prev_event_ids) != 0:
+                raise SynapseError(
+                    400,
+                    "Cannot resend m.room.create event",
+                    Codes.INVALID_PARAM,
+                )
 
         # We now ought to have some `prev_events` (unless it's a create event).
         #
@@ -1520,6 +1556,98 @@ class EventCreationHandler:
 
         return result
 
+    async def create_and_send_new_client_events(
+        self,
+        requester: Requester,
+        room_id: str,
+        prev_event_id: Optional[str],
+        event_dicts: Sequence[JsonDict],
+        ratelimit: bool = True,
+        ignore_shadow_ban: bool = False,
+    ) -> None:
+        """Helper to create and send a batch of new client events.
+
+        This supports sending membership events in very limited circumstances
+        (namely that the event is valid as is and doesn't need federation
+        requests or anything). Callers should prefer to use `update_membership`,
+        which correctly handles membership events in all cases. We allow
+        sending membership events here as its useful when copying e.g. bans
+        between rooms.
+
+        All other events and state events are supported.
+
+        Args:
+            requester: The requester sending the events.
+            room_id: The room ID to send the events in.
+            prev_event_id: The event ID to use as the previous event for the first
+                of the events, must have already been persisted.
+            event_dicts: A sequence of event dictionaries to create and send.
+            ratelimit: Whether to rate limit this send.
+            ignore_shadow_ban: True if shadow-banned users should be allowed to
+                send these events.
+        """
+
+        if not event_dicts:
+            # Nothing to do.
+            return
+
+        if prev_event_id is None:
+            # Pick the latest forward extremity as the previous event ID.
+            prev_event_ids = await self.store.get_forward_extremities_for_room(room_id)
+            prev_event_ids.sort(key=lambda x: x[2])  # Sort by depth.
+            prev_event_id = prev_event_ids[-1][0]
+
+        state_groups = await self._storage_controllers.state.get_state_group_for_events(
+            [prev_event_id]
+        )
+        if prev_event_id not in state_groups:
+            # This should only happen if we got passed a prev event ID that
+            # hasn't been persisted yet.
+            raise Exception("Previous event ID not found ")
+
+        current_state_group = state_groups[prev_event_id]
+        state_map = await self._storage_controllers.state.get_state_ids_for_group(
+            current_state_group
+        )
+
+        events_and_contexts_to_send = []
+        state_map = dict(state_map)
+        depth = None
+
+        for event_dict in event_dicts:
+            event, context = await self.create_event(
+                requester=requester,
+                event_dict=event_dict,
+                prev_event_ids=[prev_event_id],
+                depth=depth,
+                # Take a copy to ensure each event gets a unique copy of
+                # state_map since it is modified below.
+                state_map=dict(state_map),
+                for_batch=True,
+            )
+            events_and_contexts_to_send.append((event, context))
+
+            prev_event_id = event.event_id
+            depth = event.depth + 1
+            if event.is_state():
+                # If this is a state event, we need to update the state map
+                # so that it can be used for the next event.
+                state_map[(event.type, event.state_key)] = event.event_id
+
+        datastore = self.hs.get_datastores().state
+        events_and_context = (
+            await UnpersistedEventContext.batch_persist_unpersisted_contexts(
+                events_and_contexts_to_send, room_id, current_state_group, datastore
+            )
+        )
+
+        await self.handle_new_client_event(
+            requester,
+            events_and_context,
+            ignore_shadow_ban=ignore_shadow_ban,
+            ratelimit=ratelimit,
+        )
+
     async def _persist_events(
         self,
         requester: Requester,
@@ -2137,6 +2265,7 @@ class EventCreationHandler:
                 original_event.room_version, third_party_result
             )
             self.validator.validate_builder(builder)
+            assert builder.room_id is not None
         except SynapseError as e:
             raise Exception(
                 "Third party rules module created an invalid event: " + e.msg,

synapse/handlers/room.py

@@ -82,6 +82,7 @@ from synapse.types import (
     Requester,
     RoomAlias,
     RoomID,
+    RoomIdWithDomain,
     RoomStreamToken,
     StateMap,
     StrCollection,
@@ -93,7 +94,9 @@ from synapse.types import (
 from synapse.types.handlers import ShutdownRoomParams, ShutdownRoomResponse
 from synapse.types.state import StateFilter
 from synapse.util import stringutils
+from synapse.util.async_helpers import concurrently_execute
 from synapse.util.caches.response_cache import ResponseCache
+from synapse.util.iterutils import batch_iter
 from synapse.util.stringutils import parse_and_validate_server_name
 from synapse.visibility import filter_events_for_client
 
@@ -194,7 +197,13 @@ class RoomCreationHandler:
         )
 
     async def upgrade_room(
-        self, requester: Requester, old_room_id: str, new_version: RoomVersion
+        self,
+        requester: Requester,
+        old_room_id: str,
+        new_version: RoomVersion,
+        additional_creators: Optional[List[str]],
+        auto_member: bool = False,
+        ratelimit: bool = True,
     ) -> str:
         """Replace a room with a new room with a different version
 
@@ -202,6 +211,9 @@ class RoomCreationHandler:
             requester: the user requesting the upgrade
             old_room_id: the id of the room to be replaced
             new_version: the new room version to use
+            additional_creators: additional room creators, for MSC4289.
+            auto_member: Whether to automatically join local users to the new
+                room and send out invites to remote users.
 
         Returns:
             the new room id
@@ -209,11 +221,12 @@ class RoomCreationHandler:
         Raises:
             ShadowBanError if the requester is shadow-banned.
         """
-        await self.creation_ratelimiter.ratelimit(requester, update=False)
+        if ratelimit:
+            await self.creation_ratelimiter.ratelimit(requester, update=False)
 
-        # then apply the ratelimits
-        await self.common_request_ratelimiter.ratelimit(requester)
-        await self.creation_ratelimiter.ratelimit(requester)
+            # then apply the ratelimits
+            await self.common_request_ratelimiter.ratelimit(requester)
+            await self.creation_ratelimiter.ratelimit(requester)
 
         user_id = requester.user.to_string()
 
@@ -234,8 +247,29 @@ class RoomCreationHandler:
         old_room = await self.store.get_room(old_room_id)
         if old_room is None:
             raise NotFoundError("Unknown room id %s" % (old_room_id,))
+        old_room_is_public, _ = old_room
 
-        new_room_id = self._generate_room_id()
+        creation_event_with_context = None
+        if new_version.msc4291_room_ids_as_hashes:
+            old_room_create_event = await self.store.get_create_event_for_room(
+                old_room_id
+            )
+            creation_content = self._calculate_upgraded_room_creation_content(
+                old_room_create_event,
+                tombstone_event_id=None,
+                new_room_version=new_version,
+                additional_creators=additional_creators,
+            )
+            creation_event_with_context = await self._generate_create_event_for_room_id(
+                requester,
+                creation_content,
+                old_room_is_public,
+                new_version,
+            )
+            (create_event, _) = creation_event_with_context
+            new_room_id = create_event.room_id
+        else:
+            new_room_id = self._generate_room_id()
 
         # Try several times, it could fail with PartialStateConflictError
         # in _upgrade_room, cf comment in except block.
@@ -284,6 +318,9 @@ class RoomCreationHandler:
                     new_version,
                     tombstone_event,
                     tombstone_context,
+                    additional_creators,
+                    creation_event_with_context,
+                    auto_member=auto_member,
                 )
 
                 return ret
@@ -307,6 +344,11 @@ class RoomCreationHandler:
         new_version: RoomVersion,
         tombstone_event: EventBase,
         tombstone_context: synapse.events.snapshot.EventContext,
+        additional_creators: Optional[List[str]],
+        creation_event_with_context: Optional[
+            Tuple[EventBase, synapse.events.snapshot.EventContext]
+        ] = None,
+        auto_member: bool = False,
     ) -> str:
         """
         Args:
@@ -318,6 +360,10 @@ class RoomCreationHandler:
             new_version: the version to upgrade the room to
             tombstone_event: the tombstone event to send to the old room
             tombstone_context: the context for the tombstone event
+            additional_creators: additional room creators, for MSC4289.
+            creation_event_with_context: The new room's create event, for room IDs as create event IDs.
+            auto_member: Whether to automatically join local users to the new
+                room and send out invites to remote users.
 
         Raises:
             ShadowBanError if the requester is shadow-banned.
@@ -327,14 +373,16 @@ class RoomCreationHandler:
 
         logger.info("Creating new room %s to replace %s", new_room_id, old_room_id)
 
-        # create the new room. may raise a `StoreError` in the exceedingly unlikely
-        # event of a room ID collision.
-        await self.store.store_room(
-            room_id=new_room_id,
-            room_creator_user_id=user_id,
-            is_public=old_room[0],
-            room_version=new_version,
-        )
+        # We've already stored the room if we have the create event
+        if not creation_event_with_context:
+            # create the new room. may raise a `StoreError` in the exceedingly unlikely
+            # event of a room ID collision.
+            await self.store.store_room(
+                room_id=new_room_id,
+                room_creator_user_id=user_id,
+                is_public=old_room[0],
+                room_version=new_version,
+            )
 
         await self.clone_existing_room(
             requester,
@@ -342,6 +390,9 @@ class RoomCreationHandler:
             new_room_id=new_room_id,
             new_room_version=new_version,
             tombstone_event_id=tombstone_event.event_id,
+            additional_creators=additional_creators,
+            creation_event_with_context=creation_event_with_context,
+            auto_member=auto_member,
         )
 
         # now send the tombstone
@@ -375,6 +426,7 @@ class RoomCreationHandler:
             old_room_id,
             new_room_id,
             old_room_state,
+            additional_creators,
         )
 
         return new_room_id
@@ -385,6 +437,7 @@ class RoomCreationHandler:
         old_room_id: str,
         new_room_id: str,
         old_room_state: StateMap[str],
+        additional_creators: Optional[List[str]],
     ) -> None:
         """Send updated power levels in both rooms after an upgrade
 
@@ -393,7 +446,7 @@ class RoomCreationHandler:
             old_room_id: the id of the room to be replaced
             new_room_id: the id of the replacement room
             old_room_state: the state map for the old room
-
+            additional_creators: Additional creators in the new room.
         Raises:
             ShadowBanError if the requester is shadow-banned.
         """
@@ -449,6 +502,14 @@ class RoomCreationHandler:
             except AuthError as e:
                 logger.warning("Unable to update PLs in old room: %s", e)
 
+        new_room_version = await self.store.get_room_version(new_room_id)
+        if new_room_version.msc4289_creator_power_enabled:
+            self._remove_creators_from_pl_users_map(
+                old_room_pl_state.content.get("users", {}),
+                requester.user.to_string(),
+                additional_creators,
+            )
+
         await self.event_creation_handler.create_and_send_nonmember_event(
             requester,
             {
@@ -463,6 +524,36 @@ class RoomCreationHandler:
             ratelimit=False,
         )
 
+    def _calculate_upgraded_room_creation_content(
+        self,
+        old_room_create_event: EventBase,
+        tombstone_event_id: Optional[str],
+        new_room_version: RoomVersion,
+        additional_creators: Optional[List[str]],
+    ) -> JsonDict:
+        creation_content: JsonDict = {
+            "room_version": new_room_version.identifier,
+            "predecessor": {
+                "room_id": old_room_create_event.room_id,
+            },
+        }
+        if tombstone_event_id is not None:
+            creation_content["predecessor"]["event_id"] = tombstone_event_id
+        if (
+            additional_creators is not None
+            and new_room_version.msc4289_creator_power_enabled
+        ):
+            creation_content["additional_creators"] = additional_creators
+        # Check if old room was non-federatable
+        if not old_room_create_event.content.get(EventContentFields.FEDERATE, True):
+            # If so, mark the new room as non-federatable as well
+            creation_content[EventContentFields.FEDERATE] = False
+        # Copy the room type as per MSC3818.
+        room_type = old_room_create_event.content.get(EventContentFields.ROOM_TYPE)
+        if room_type is not None:
+            creation_content[EventContentFields.ROOM_TYPE] = room_type
+        return creation_content
+
     async def clone_existing_room(
         self,
         requester: Requester,
@@ -470,6 +561,11 @@ class RoomCreationHandler:
         new_room_id: str,
         new_room_version: RoomVersion,
         tombstone_event_id: str,
+        additional_creators: Optional[List[str]],
+        creation_event_with_context: Optional[
+            Tuple[EventBase, synapse.events.snapshot.EventContext]
+        ] = None,
+        auto_member: bool = False,
     ) -> None:
         """Populate a new room based on an old room
 
@@ -480,24 +576,27 @@ class RoomCreationHandler:
                 created with _generate_room_id())
             new_room_version: the new room version to use
             tombstone_event_id: the ID of the tombstone event in the old room.
+            additional_creators: additional room creators, for MSC4289.
+            creation_event_with_context: The create event of the new room, if the new room supports
+            room ID as create event ID hash.
+            auto_member: Whether to automatically join local users to the new
+                room and send out invites to remote users.
         """
         user_id = requester.user.to_string()
 
-        creation_content: JsonDict = {
-            "room_version": new_room_version.identifier,
-            "predecessor": {"room_id": old_room_id, "event_id": tombstone_event_id},
-        }
-
-        # Check if old room was non-federatable
-
         # Get old room's create event
         old_room_create_event = await self.store.get_create_event_for_room(old_room_id)
 
-        # Check if the create event specified a non-federatable room
-        if not old_room_create_event.content.get(EventContentFields.FEDERATE, True):
-            # If so, mark the new room as non-federatable as well
-            creation_content[EventContentFields.FEDERATE] = False
-
+        if creation_event_with_context:
+            create_event, _ = creation_event_with_context
+            creation_content = create_event.content
+        else:
+            creation_content = self._calculate_upgraded_room_creation_content(
+                old_room_create_event,
+                tombstone_event_id,
+                new_room_version,
+                additional_creators=additional_creators,
+            )
         initial_state = {}
 
         # Replicate relevant room events
@@ -513,11 +612,8 @@ class RoomCreationHandler:
             (EventTypes.PowerLevels, ""),
         ]
 
-        # Copy the room type as per MSC3818.
         room_type = old_room_create_event.content.get(EventContentFields.ROOM_TYPE)
         if room_type is not None:
-            creation_content[EventContentFields.ROOM_TYPE] = room_type
-
             # If the old room was a space, copy over the rooms in the space.
             if room_type == RoomTypes.SPACE:
                 types_to_copy.append((EventTypes.SpaceChild, None))
@@ -589,6 +685,14 @@ class RoomCreationHandler:
         if current_power_level_int < needed_power_level:
             user_power_levels[user_id] = needed_power_level
 
+        if new_room_version.msc4289_creator_power_enabled:
+            # the creator(s) cannot be in the users map
+            self._remove_creators_from_pl_users_map(
+                user_power_levels,
+                user_id,
+                additional_creators,
+            )
+
         # We construct what the body of a call to /createRoom would look like for passing
         # to the spam checker. We don't include a preset here, as we expect the
         # initial state to contain everything we need.
@@ -607,7 +711,7 @@ class RoomCreationHandler:
                 additional_fields=spam_check[1],
             )
 
-        await self._send_events_for_new_room(
+        _, last_event_id, _ = await self._send_events_for_new_room(
             requester,
             new_room_id,
             new_room_version,
@@ -617,36 +721,228 @@ class RoomCreationHandler:
             invite_list=[],
             initial_state=initial_state,
             creation_content=creation_content,
+            creation_event_with_context=creation_event_with_context,
         )
 
         # Transfer membership events
-        old_room_member_state_ids = (
-            await self._storage_controllers.state.get_current_state_ids(
-                old_room_id, StateFilter.from_types([(EventTypes.Member, None)])
-            )
-        )
+        ban_event_ids = await self.store.get_ban_event_ids_in_room(old_room_id)
+        if ban_event_ids:
+            ban_events = await self.store.get_events_as_list(ban_event_ids)
 
-        # map from event_id to BaseEvent
-        old_room_member_state_events = await self.store.get_events(
-            old_room_member_state_ids.values()
-        )
-        for old_event in old_room_member_state_events.values():
-            # Only transfer ban events
-            if (
-                "membership" in old_event.content
-                and old_event.content["membership"] == "ban"
-            ):
-                await self.room_member_handler.update_membership(
-                    requester,
-                    UserID.from_string(old_event.state_key),
-                    new_room_id,
-                    "ban",
-                    ratelimit=False,
-                    content=old_event.content,
+            # Add any banned users to the new room.
+            #
+            # Note generally we should send membership events via
+            # `update_membership`, however in this case its fine to bypass as
+            # these bans don't need any special treatment, i.e. the sender is in
+            # the room and they don't need any extra signatures, etc.
+            for batched_ban_events in batch_iter(ban_events, 1000):
+                await self.event_creation_handler.create_and_send_new_client_events(
+                    requester=requester,
+                    room_id=new_room_id,
+                    prev_event_id=last_event_id,
+                    event_dicts=[
+                        {
+                            "type": EventTypes.Member,
+                            "state_key": ban_event.state_key,
+                            "room_id": new_room_id,
+                            "sender": requester.user.to_string(),
+                            "content": ban_event.content,
+                        }
+                        for ban_event in batched_ban_events
+                    ],
+                    ratelimit=False,  # We ratelimit the entire upgrade, not individual events.
                 )
 
-        # XXX invites/joins
-        # XXX 3pid invites
+        if auto_member:
+            logger.info("Joining local users to %s", new_room_id)
+
+            # 1. Copy over all joins for local
+            joined_profiles = await self.store.get_users_in_room_with_profiles(
+                old_room_id
+            )
+
+            local_user_ids = [
+                user_id for user_id in joined_profiles if self.hs.is_mine_id(user_id)
+            ]
+
+            logger.info("Local user IDs %s", local_user_ids)
+
+            for batched_local_user_ids in batch_iter(local_user_ids, 1000):
+                invites_to_send = []
+
+                # For each local user we create an invite event (from the
+                # upgrading user) plus a join event.
+                for local_user_id in batched_local_user_ids:
+                    if local_user_id == user_id:
+                        # Ignore the upgrading user, as they are already in the
+                        # new room.
+                        continue
+
+                    invites_to_send.append(
+                        {
+                            "type": EventTypes.Member,
+                            "state_key": local_user_id,
+                            "room_id": new_room_id,
+                            "sender": requester.user.to_string(),
+                            "content": {
+                                "membership": Membership.INVITE,
+                            },
+                        }
+                    )
+
+                    # If the user has profile information in the previous join,
+                    # add it to the content.
+                    #
+                    # We could instead copy over the contents from the old join
+                    # event, however a) that would require us to fetch all the
+                    # old join events (which is slow), and b) generally the join
+                    # events have no extra information in them. (We also believe
+                    # that most clients don't copy this information over either,
+                    # but we could be wrong.)
+                    content_profile = {}
+                    user_profile = joined_profiles[local_user_id]
+                    if user_profile.display_name:
+                        content_profile["displayname"] = user_profile.display_name
+                    if user_profile.avatar_url:
+                        content_profile["avatar_url"] = user_profile.avatar_url
+
+                    invites_to_send.append(
+                        {
+                            "type": EventTypes.Member,
+                            "state_key": local_user_id,
+                            "room_id": new_room_id,
+                            "sender": local_user_id,
+                            "content": {
+                                "membership": Membership.JOIN,
+                                **content_profile,
+                            },
+                        }
+                    )
+
+                await self.event_creation_handler.create_and_send_new_client_events(
+                    requester=requester,
+                    room_id=new_room_id,
+                    prev_event_id=None,
+                    event_dicts=invites_to_send,
+                    ratelimit=False,  # We ratelimit the entire upgrade, not individual events.
+                )
+
+            # Invite other users if the room is not public. If the room *is*
+            # public then users can simply directly join, and inviting them as
+            # well may lead to confusion.
+
+            join_rule_content = initial_state.get((EventTypes.JoinRules, ""), None)
+            is_public = False
+            if join_rule_content:
+                is_public = join_rule_content["join_rule"] == JoinRules.PUBLIC
+
+            if not is_public:
+                # Copy invites
+                # TODO: Copy over 3pid invites as well.
+                invited_users = await self.store.get_invited_users_in_room(
+                    room_id=old_room_id
+                )
+
+                # For local users we can just batch send the invites.
+                local_invited_users = [
+                    user_id for user_id in invited_users if self.hs.is_mine_id(user_id)
+                ]
+
+                logger.info(
+                    "Joining local user IDs %s to new room %s",
+                    local_invited_users,
+                    new_room_id,
+                )
+
+                for batched_local_invited_users in batch_iter(
+                    local_invited_users, 1000
+                ):
+                    invites_to_send = []
+                    leaves_to_send = []
+
+                    # For each local user we create an invite event (from the
+                    # upgrading user), and reject the invite event in the old
+                    # room.
+                    #
+                    # This ensures that the user ends up with a single invite to
+                    # the new room (rather than multiple invites which may be
+                    # noisy and confusing).
+                    for local_user_id in batched_local_invited_users:
+                        leaves_to_send.append(
+                            {
+                                "type": EventTypes.Member,
+                                "state_key": local_user_id,
+                                "room_id": old_room_id,
+                                "sender": local_user_id,
+                                "content": {
+                                    "membership": Membership.LEAVE,
+                                },
+                            }
+                        )
+                        invites_to_send.append(
+                            {
+                                "type": EventTypes.Member,
+                                "state_key": local_user_id,
+                                "room_id": new_room_id,
+                                "sender": requester.user.to_string(),
+                                "content": {
+                                    "membership": Membership.INVITE,
+                                },
+                            }
+                        )
+
+                    await self.event_creation_handler.create_and_send_new_client_events(
+                        requester=requester,
+                        room_id=old_room_id,
+                        prev_event_id=None,
+                        event_dicts=leaves_to_send,
+                        ratelimit=False,  # We ratelimit the entire upgrade, not individual events.
+                    )
+                    await self.event_creation_handler.create_and_send_new_client_events(
+                        requester=requester,
+                        room_id=new_room_id,
+                        prev_event_id=None,
+                        event_dicts=invites_to_send,
+                        ratelimit=False,
+                    )
+
+                # For remote users we send invites one by one, as we need to
+                # send each one to the remote server.
+                #
+                # We also invite joined remote users who were in the old room.
+                remote_user_ids = [
+                    user_id
+                    for user_id in itertools.chain(invited_users, joined_profiles)
+                    if not self.hs.is_mine_id(user_id)
+                ]
+
+                logger.debug("Inviting remote user IDs %s", remote_user_ids)
+
+                async def remote_invite(remote_user: str) -> None:
+                    try:
+                        await self.room_member_handler.update_membership(
+                            requester,
+                            UserID.from_string(remote_user),
+                            new_room_id,
+                            Membership.INVITE,
+                            ratelimit=False,  # We ratelimit the entire upgrade, not individual events.
+                        )
+                    except SynapseError as e:
+                        # If we fail to invite a remote user, we log it but continue
+                        # on with the upgrade.
+                        logger.warning(
+                            "Failed to invite remote user %s to new room %s: %s",
+                            remote_user,
+                            new_room_id,
+                            e,
+                        )
+
+                # We do this concurrently, as it can take a while to invite
+                await concurrently_execute(
+                    remote_invite,
+                    remote_user_ids,
+                    10,
+                )
 
     async def _move_aliases_to_new_room(
         self,
@@ -775,6 +1071,25 @@ class RoomCreationHandler:
 
         await self.auth_blocking.check_auth_blocking(requester=requester)
 
+        if ratelimit:
+            # Limit the rate of room creations,
+            # using both the limiter specific to room creations as well
+            # as the general request ratelimiter.
+            #
+            # Note that we don't rate limit the individual
+            # events in the room — room creation isn't atomic and
+            # historically it was very janky if half the events in the
+            # initial state don't make it because of rate limiting.
+
+            # First check the room creation ratelimiter without updating it
+            # (this is so we don't consume a token if the other ratelimiter doesn't
+            # allow us to proceed)
+            await self.creation_ratelimiter.ratelimit(requester, update=False)
+
+            # then apply the ratelimits
+            await self.common_request_ratelimiter.ratelimit(requester)
+            await self.creation_ratelimiter.ratelimit(requester)
+
         if (
             self._server_notices_mxid is not None
             and user_id == self._server_notices_mxid
@@ -806,37 +1121,6 @@ class RoomCreationHandler:
                     Codes.MISSING_PARAM,
                 )
 
-        if not is_requester_admin:
-            spam_check = await self._spam_checker_module_callbacks.user_may_create_room(
-                user_id, config
-            )
-            if spam_check != self._spam_checker_module_callbacks.NOT_SPAM:
-                raise SynapseError(
-                    403,
-                    "You are not permitted to create rooms",
-                    errcode=spam_check[0],
-                    additional_fields=spam_check[1],
-                )
-
-        if ratelimit:
-            # Limit the rate of room creations,
-            # using both the limiter specific to room creations as well
-            # as the general request ratelimiter.
-            #
-            # Note that we don't rate limit the individual
-            # events in the room — room creation isn't atomic and
-            # historically it was very janky if half the events in the
-            # initial state don't make it because of rate limiting.
-
-            # First check the room creation ratelimiter without updating it
-            # (this is so we don't consume a token if the other ratelimiter doesn't
-            # allow us to proceed)
-            await self.creation_ratelimiter.ratelimit(requester, update=False)
-
-            # then apply the ratelimits
-            await self.common_request_ratelimiter.ratelimit(requester)
-            await self.creation_ratelimiter.ratelimit(requester)
-
         room_version_id = config.get(
             "room_version", self.config.server.default_room_version.identifier
         )
@@ -912,6 +1196,7 @@ class RoomCreationHandler:
         power_level_content_override = config.get("power_level_content_override")
         if (
             power_level_content_override
+            and not room_version.msc4289_creator_power_enabled  # this validation doesn't apply in MSC4289 rooms
             and "users" in power_level_content_override
             and user_id not in power_level_content_override["users"]
         ):
@@ -928,11 +1213,54 @@ class RoomCreationHandler:
 
         self._validate_room_config(config, visibility)
 
-        room_id = await self._generate_and_create_room_id(
-            creator_id=user_id,
-            is_public=is_public,
-            room_version=room_version,
-        )
+        # Run the spam checker after other validation
+        if not is_requester_admin:
+            spam_check = await self._spam_checker_module_callbacks.user_may_create_room(
+                user_id, config
+            )
+            if spam_check != self._spam_checker_module_callbacks.NOT_SPAM:
+                raise SynapseError(
+                    403,
+                    "You are not permitted to create rooms",
+                    errcode=spam_check[0],
+                    additional_fields=spam_check[1],
+                )
+
+        creation_content = config.get("creation_content", {})
+        # override any attempt to set room versions via the creation_content
+        creation_content["room_version"] = room_version.identifier
+
+        # trusted private chats have the invited users marked as additional creators
+        if (
+            room_version.msc4289_creator_power_enabled
+            and config.get("preset", None) == RoomCreationPreset.TRUSTED_PRIVATE_CHAT
+            and len(config.get("invite", [])) > 0
+        ):
+            # the other user(s) are additional creators
+            invitees = config.get("invite", [])
+            # we don't want to replace any additional_creators additionally specified, and we want
+            # to remove duplicates.
+            creation_content[EventContentFields.ADDITIONAL_CREATORS] = list(
+                set(creation_content.get(EventContentFields.ADDITIONAL_CREATORS, []))
+                | set(invitees)
+            )
+
+        creation_event_with_context = None
+        if room_version.msc4291_room_ids_as_hashes:
+            creation_event_with_context = await self._generate_create_event_for_room_id(
+                requester,
+                creation_content,
+                is_public,
+                room_version,
+            )
+            (create_event, _) = creation_event_with_context
+            room_id = create_event.room_id
+        else:
+            room_id = await self._generate_and_create_room_id(
+                creator_id=user_id,
+                is_public=is_public,
+                room_version=room_version,
+            )
 
         # Check whether this visibility value is blocked by a third party module
         allowed_by_third_party_rules = await (
@@ -969,11 +1297,6 @@ class RoomCreationHandler:
         for val in raw_initial_state:
             initial_state[(val["type"], val.get("state_key", ""))] = val["content"]
 
-        creation_content = config.get("creation_content", {})
-
-        # override any attempt to set room versions via the creation_content
-        creation_content["room_version"] = room_version.identifier
-
         (
             last_stream_id,
             last_sent_event_id,
@@ -990,6 +1313,7 @@ class RoomCreationHandler:
             power_level_content_override=power_level_content_override,
             creator_join_profile=creator_join_profile,
             ignore_forced_encryption=ignore_forced_encryption,
+            creation_event_with_context=creation_event_with_context,
         )
 
         # we avoid dropping the lock between invites, as otherwise joins can
@@ -1055,6 +1379,38 @@ class RoomCreationHandler:
 
         return room_id, room_alias, last_stream_id
 
+    async def _generate_create_event_for_room_id(
+        self,
+        creator: Requester,
+        creation_content: JsonDict,
+        is_public: bool,
+        room_version: RoomVersion,
+    ) -> Tuple[EventBase, synapse.events.snapshot.EventContext]:
+        (
+            creation_event,
+            new_unpersisted_context,
+        ) = await self.event_creation_handler.create_event(
+            creator,
+            {
+                "content": creation_content,
+                "sender": creator.user.to_string(),
+                "type": EventTypes.Create,
+                "state_key": "",
+            },
+            prev_event_ids=[],
+            depth=1,
+            state_map={},
+            for_batch=False,
+        )
+        await self.store.store_room(
+            room_id=creation_event.room_id,
+            room_creator_user_id=creator.user.to_string(),
+            is_public=is_public,
+            room_version=room_version,
+        )
+        creation_context = await new_unpersisted_context.persist(creation_event)
+        return (creation_event, creation_context)
+
     async def _send_events_for_new_room(
         self,
         creator: Requester,
@@ -1068,6 +1424,9 @@ class RoomCreationHandler:
         power_level_content_override: Optional[JsonDict] = None,
         creator_join_profile: Optional[JsonDict] = None,
         ignore_forced_encryption: bool = False,
+        creation_event_with_context: Optional[
+            Tuple[EventBase, synapse.events.snapshot.EventContext]
+        ] = None,
     ) -> Tuple[int, str, int]:
         """Sends the initial events into a new room. Sends the room creation, membership,
         and power level events into the room sequentially, then creates and batches up the
@@ -1104,7 +1463,10 @@ class RoomCreationHandler:
                 user in this room.
             ignore_forced_encryption:
                 Ignore encryption forced by `encryption_enabled_by_default_for_room_type` setting.
-
+            creation_event_with_context:
+                Set in MSC4291 rooms where the create event determines the room ID. If provided,
+                does not create an additional create event but instead appends the remaining new
+                events onto the provided create event.
         Returns:
             A tuple containing the stream ID, event ID and depth of the last
             event sent to the room.
@@ -1169,13 +1531,26 @@ class RoomCreationHandler:
 
         preset_config, config = self._room_preset_config(room_config)
 
-        # MSC2175 removes the creator field from the create event.
-        if not room_version.implicit_room_creator:
-            creation_content["creator"] = creator_id
-        creation_event, unpersisted_creation_context = await create_event(
-            EventTypes.Create, creation_content, False
-        )
-        creation_context = await unpersisted_creation_context.persist(creation_event)
+        if creation_event_with_context is None:
+            # MSC2175 removes the creator field from the create event.
+            if not room_version.implicit_room_creator:
+                creation_content["creator"] = creator_id
+            creation_event, unpersisted_creation_context = await create_event(
+                EventTypes.Create, creation_content, False
+            )
+            creation_context = await unpersisted_creation_context.persist(
+                creation_event
+            )
+        else:
+            (creation_event, creation_context) = creation_event_with_context
+            # we had to do the above already in order to have a room ID, so just updates local vars
+            # and continue.
+            depth = 2
+            prev_event = [creation_event.event_id]
+            state_map[(creation_event.type, creation_event.state_key)] = (
+                creation_event.event_id
+            )
+
         logger.debug("Sending %s in new room", EventTypes.Member)
         ev = await self.event_creation_handler.handle_new_client_event(
             requester=creator,
@@ -1224,7 +1599,9 @@ class RoomCreationHandler:
             # Please update the docs for `default_power_level_content_override` when
             # updating the `events` dict below
             power_level_content: JsonDict = {
-                "users": {creator_id: 100},
+                "users": {creator_id: 100}
+                if not room_version.msc4289_creator_power_enabled
+                else {},
                 "users_default": 0,
                 "events": {
                     EventTypes.Name: 50,
@@ -1232,7 +1609,9 @@ class RoomCreationHandler:
                     EventTypes.RoomHistoryVisibility: 100,
                     EventTypes.CanonicalAlias: 50,
                     EventTypes.RoomAvatar: 50,
-                    EventTypes.Tombstone: 100,
+                    EventTypes.Tombstone: 150
+                    if room_version.msc4289_creator_power_enabled
+                    else 100,
                     EventTypes.ServerACL: 100,
                     EventTypes.RoomEncryption: 100,
                 },
@@ -1245,7 +1624,13 @@ class RoomCreationHandler:
                 "historical": 100,
             }
 
-            if config["original_invitees_have_ops"]:
+            # original_invitees_have_ops is set on preset:trusted_private_chat which will already
+            # have set these users as additional_creators, hence don't set the PL for creators as
+            # that is invalid.
+            if (
+                config["original_invitees_have_ops"]
+                and not room_version.msc4289_creator_power_enabled
+            ):
                 for invitee in invite_list:
                     power_level_content["users"][invitee] = 100
 
@@ -1418,6 +1803,19 @@ class RoomCreationHandler:
             )
         return preset_name, preset_config
 
+    def _remove_creators_from_pl_users_map(
+        self,
+        users_map: Dict[str, int],
+        creator: str,
+        additional_creators: Optional[List[str]],
+    ) -> None:
+        creators = [creator]
+        if additional_creators:
+            creators.extend(additional_creators)
+        for creator in creators:
+            # the creator(s) cannot be in the users map
+            users_map.pop(creator, None)
+
     def _generate_room_id(self) -> str:
         """Generates a random room ID.
 
@@ -1435,7 +1833,7 @@ class RoomCreationHandler:
             A random room ID of the form "!opaque_id:domain".
         """
         random_string = stringutils.random_string(18)
-        return RoomID(random_string, self.hs.hostname).to_string()
+        return RoomIdWithDomain(random_string, self.hs.hostname).to_string()
 
     async def _generate_and_create_room_id(
         self,

synapse/handlers/room_member.py

@@ -42,7 +42,7 @@ from synapse.api.errors import (
 )
 from synapse.api.ratelimiting import Ratelimiter
 from synapse.event_auth import get_named_level, get_power_level_event
-from synapse.events import EventBase
+from synapse.events import EventBase, is_creator
 from synapse.events.snapshot import EventContext
 from synapse.handlers.pagination import PURGE_ROOM_ACTION_NAME
 from synapse.handlers.profile import MAX_AVATAR_URL_LEN, MAX_DISPLAYNAME_LEN
@@ -1160,9 +1160,8 @@ class RoomMemberHandler(metaclass=abc.ABCMeta):
 
         elif effective_membership_state == Membership.KNOCK:
             if not is_host_in_room:
-                # The knock needs to be sent over federation instead
-                remote_room_hosts.append(get_domain_from_id(room_id))
-
+                # we used to add the domain of the room ID to remote_room_hosts.
+                # This is not safe in MSC4291 rooms which do not have a domain.
                 content["membership"] = Membership.KNOCK
 
                 try:
@@ -1921,7 +1920,7 @@ class RoomMemberMasterHandler(RoomMemberHandler):
             check_complexity
             and self.hs.config.server.limit_remote_rooms.admins_can_join
         ):
-            check_complexity = not await self.store.is_server_admin(user)
+            check_complexity = not await self.store.is_server_admin(user.to_string())
 
         if check_complexity:
             # Fetch the room complexity
@@ -2324,6 +2323,7 @@ def get_users_which_can_issue_invite(auth_events: StateMap[EventBase]) -> List[s
 
     # Check which members are able to invite by ensuring they're joined and have
     # the necessary power level.
+    create_event = auth_events[(EventTypes.Create, "")]
     for (event_type, state_key), event in auth_events.items():
         if event_type != EventTypes.Member:
             continue
@@ -2331,8 +2331,12 @@ def get_users_which_can_issue_invite(auth_events: StateMap[EventBase]) -> List[s
         if event.membership != Membership.JOIN:
             continue
 
+        if create_event.room_version.msc4289_creator_power_enabled and is_creator(
+            create_event, state_key
+        ):
+            result.append(state_key)
         # Check if the user has a custom power level.
-        if users.get(state_key, users_default_level) >= invite_level:
+        elif users.get(state_key, users_default_level) >= invite_level:
             result.append(state_key)
 
     return result

synapse/handlers/thread_subscriptions.py

@@ -1,9 +1,15 @@
 import logging
+from http import HTTPStatus
 from typing import TYPE_CHECKING, Optional
 
-from synapse.api.errors import AuthError, NotFoundError
-from synapse.storage.databases.main.thread_subscriptions import ThreadSubscription
-from synapse.types import UserID
+from synapse.api.constants import RelationTypes
+from synapse.api.errors import AuthError, Codes, NotFoundError, SynapseError
+from synapse.events import relation_from_event
+from synapse.storage.databases.main.thread_subscriptions import (
+    AutomaticSubscriptionConflicted,
+    ThreadSubscription,
+)
+from synapse.types import EventOrderings, UserID
 
 if TYPE_CHECKING:
     from synapse.server import HomeServer
@@ -55,42 +61,79 @@ class ThreadSubscriptionsHandler:
         room_id: str,
         thread_root_event_id: str,
         *,
-        automatic: bool,
+        automatic_event_id: Optional[str],
     ) -> Optional[int]:
         """Sets or updates a user's subscription settings for a specific thread root.
 
         Args:
             requester_user_id: The ID of the user whose settings are being updated.
             thread_root_event_id: The event ID of the thread root.
-            automatic: whether the user was subscribed by an automatic decision by
-                their client.
+            automatic_event_id: if the user was subscribed by an automatic decision by
+                their client, the event ID that caused this.
 
         Returns:
             The stream ID for this update, if the update isn't no-opped.
 
         Raises:
             NotFoundError if the user cannot access the thread root event, or it isn't
-            known to this homeserver.
+            known to this homeserver. Ditto for the automatic cause event if supplied.
+
+            SynapseError(400, M_NOT_IN_THREAD): if client supplied an automatic cause event
+            but user cannot access the event.
+
+            SynapseError(409, M_SKIPPED): if client requested an automatic subscription
+            but it was skipped because the cause event is logically later than an unsubscription.
         """
         # First check that the user can access the thread root event
         # and that it exists
         try:
-            event = await self.event_handler.get_event(
+            thread_root_event = await self.event_handler.get_event(
                 user_id, room_id, thread_root_event_id
             )
-            if event is None:
+            if thread_root_event is None:
                 raise NotFoundError("No such thread root")
         except AuthError:
             logger.info("rejecting thread subscriptions change (thread not accessible)")
             raise NotFoundError("No such thread root")
 
-        return await self.store.subscribe_user_to_thread(
+        if automatic_event_id:
+            autosub_cause_event = await self.event_handler.get_event(
+                user_id, room_id, automatic_event_id
+            )
+            if autosub_cause_event is None:
+                raise NotFoundError("Automatic subscription event not found")
+            relation = relation_from_event(autosub_cause_event)
+            if (
+                relation is None
+                or relation.rel_type != RelationTypes.THREAD
+                or relation.parent_id != thread_root_event_id
+            ):
+                raise SynapseError(
+                    HTTPStatus.BAD_REQUEST,
+                    "Automatic subscription must use an event in the thread",
+                    errcode=Codes.MSC4306_NOT_IN_THREAD,
+                )
+
+            automatic_event_orderings = EventOrderings.from_event(autosub_cause_event)
+        else:
+            automatic_event_orderings = None
+
+        outcome = await self.store.subscribe_user_to_thread(
             user_id.to_string(),
-            event.room_id,
+            room_id,
             thread_root_event_id,
-            automatic=automatic,
+            automatic_event_orderings=automatic_event_orderings,
         )
 
+        if isinstance(outcome, AutomaticSubscriptionConflicted):
+            raise SynapseError(
+                HTTPStatus.CONFLICT,
+                "Automatic subscription obsoleted by an unsubscription request.",
+                errcode=Codes.MSC4306_CONFLICTING_UNSUBSCRIPTION,
+            )
+
+        return outcome
+
     async def unsubscribe_user_from_thread(
         self, user_id: UserID, room_id: str, thread_root_event_id: str
     ) -> Optional[int]:

synapse/http/server.py

@@ -337,7 +337,7 @@ class _AsyncResource(resource.Resource, metaclass=abc.ABCMeta):
                     callback_return = await self._async_render(request)
                 except LimitExceededError as e:
                     if e.pause:
-                        self._clock.sleep(e.pause)
+                        await self._clock.sleep(e.pause)
                     raise
 
                 if callback_return is not None:

synapse/metrics/__init__.py

@@ -170,7 +170,9 @@ class LaterGauge(Collector):
     ]
 
     def collect(self) -> Iterable[Metric]:
-        g = GaugeMetricFamily(self.name, self.desc, labels=self.labelnames)
+        # The decision to add `SERVER_NAME_LABEL` is from the `LaterGauge` usage itself
+        # (we don't enforce it here, one level up).
+        g = GaugeMetricFamily(self.name, self.desc, labels=self.labelnames)  # type: ignore[missing-server-name-label]
 
         try:
             calls = self.caller()
@@ -304,7 +306,9 @@ class InFlightGauge(Generic[MetricsEntry], Collector):
 
         Note: may be called by a separate thread.
         """
-        in_flight = GaugeMetricFamily(
+        # The decision to add `SERVER_NAME_LABEL` is from the `GaugeBucketCollector`
+        # usage itself (we don't enforce it here, one level up).
+        in_flight = GaugeMetricFamily(  # type: ignore[missing-server-name-label]
             self.name + "_total", self.desc, labels=self.labels
         )
 
@@ -328,7 +332,9 @@ class InFlightGauge(Generic[MetricsEntry], Collector):
         yield in_flight
 
         for name in self.sub_metrics:
-            gauge = GaugeMetricFamily(
+            # The decision to add `SERVER_NAME_LABEL` is from the `InFlightGauge` usage
+            # itself (we don't enforce it here, one level up).
+            gauge = GaugeMetricFamily(  # type: ignore[missing-server-name-label]
                 "_".join([self.name, name]), "", labels=self.labels
             )
             for key, metrics in metrics_by_key.items():
@@ -483,7 +489,9 @@ class GaugeBucketCollector(Collector):
         # that bucket or below.
         accumulated_values = itertools.accumulate(bucket_values)
 
-        return GaugeHistogramMetricFamilyWithLabels(
+        # The decision to add `SERVER_NAME_LABEL` is from the `GaugeBucketCollector`
+        # usage itself (we don't enforce it here, one level up).
+        return GaugeHistogramMetricFamilyWithLabels(  # type: ignore[missing-server-name-label]
             name=self._name,
             documentation=self._documentation,
             labelnames=self._labelnames,
@@ -519,16 +527,19 @@ class CPUMetrics(Collector):
             line = s.read()
             raw_stats = line.split(") ", 1)[1].split(" ")
 
-            user = GaugeMetricFamily("process_cpu_user_seconds_total", "")
+            # This is a process-level metric, so it does not have the `SERVER_NAME_LABEL`.
+            user = GaugeMetricFamily("process_cpu_user_seconds_total", "")  # type: ignore[missing-server-name-label]
             user.add_metric([], float(raw_stats[11]) / self.ticks_per_sec)
             yield user
 
-            sys = GaugeMetricFamily("process_cpu_system_seconds_total", "")
+            # This is a process-level metric, so it does not have the `SERVER_NAME_LABEL`.
+            sys = GaugeMetricFamily("process_cpu_system_seconds_total", "")  # type: ignore[missing-server-name-label]
             sys.add_metric([], float(raw_stats[12]) / self.ticks_per_sec)
             yield sys
 
 
-REGISTRY.register(CPUMetrics())
+# This is a process-level metric, so it does not have the `SERVER_NAME_LABEL`.
+REGISTRY.register(CPUMetrics())  # type: ignore[missing-server-name-label]
 
 
 #

synapse/metrics/_gc.py

@@ -83,7 +83,8 @@ gc_time = Histogram(  # type: ignore[missing-server-name-label]
 
 class GCCounts(Collector):
     def collect(self) -> Iterable[Metric]:
-        cm = GaugeMetricFamily("python_gc_counts", "GC object counts", labels=["gen"])
+        # This is a process-level metric, so it does not have the `SERVER_NAME_LABEL`.
+        cm = GaugeMetricFamily("python_gc_counts", "GC object counts", labels=["gen"])  # type: ignore[missing-server-name-label]
         for n, m in enumerate(gc.get_count()):
             cm.add_metric([str(n)], m)
 
@@ -102,7 +103,8 @@ def install_gc_manager() -> None:
     if running_on_pypy:
         return
 
-    REGISTRY.register(GCCounts())
+    # This is a process-level metric, so it does not have the `SERVER_NAME_LABEL`.
+    REGISTRY.register(GCCounts())  # type: ignore[missing-server-name-label]
 
     gc.disable()
 
@@ -177,7 +179,8 @@ class PyPyGCStats(Collector):
         #
         #     Total time spent in GC:  0.073                  # s.total_gc_time
 
-        pypy_gc_time = CounterMetricFamily(
+        # This is a process-level metric, so it does not have the `SERVER_NAME_LABEL`.
+        pypy_gc_time = CounterMetricFamily(  # type: ignore[missing-server-name-label]
             "pypy_gc_time_seconds_total",
             "Total time spent in PyPy GC",
             labels=[],
@@ -185,7 +188,8 @@ class PyPyGCStats(Collector):
         pypy_gc_time.add_metric([], s.total_gc_time / 1000)
         yield pypy_gc_time
 
-        pypy_mem = GaugeMetricFamily(
+        # This is a process-level metric, so it does not have the `SERVER_NAME_LABEL`.
+        pypy_mem = GaugeMetricFamily(  # type: ignore[missing-server-name-label]
             "pypy_memory_bytes",
             "Memory tracked by PyPy allocator",
             labels=["state", "class", "kind"],
@@ -209,4 +213,5 @@ class PyPyGCStats(Collector):
 
 
 if running_on_pypy:
-    REGISTRY.register(PyPyGCStats())
+    # This is a process-level metric, so it does not have the `SERVER_NAME_LABEL`.
+    REGISTRY.register(PyPyGCStats())  # type: ignore[missing-server-name-label]

synapse/metrics/_reactor_metrics.py

@@ -115,7 +115,8 @@ class ReactorLastSeenMetric(Collector):
         self._call_wrapper = call_wrapper
 
     def collect(self) -> Iterable[Metric]:
-        cm = GaugeMetricFamily(
+        # This is a process-level metric, so it does not have the `SERVER_NAME_LABEL`.
+        cm = GaugeMetricFamily(  # type: ignore[missing-server-name-label]
             "python_twisted_reactor_last_seen",
             "Seconds since the Twisted reactor was last seen",
         )
@@ -166,4 +167,5 @@ except Exception as e:
 
 
 if wrapper:
-    REGISTRY.register(ReactorLastSeenMetric(wrapper))
+    # This is a process-level metric, so it does not have the `SERVER_NAME_LABEL`.
+    REGISTRY.register(ReactorLastSeenMetric(wrapper))  # type: ignore[missing-server-name-label]

synapse/metrics/background_process_metrics.py

@@ -167,7 +167,9 @@ class _Collector(Collector):
             yield from m.collect()
 
 
-REGISTRY.register(_Collector())
+# The `SERVER_NAME_LABEL` is included in the individual metrics added to this registry,
+# so we don't need to worry about it on the collector itself.
+REGISTRY.register(_Collector())  # type: ignore[missing-server-name-label]
 
 
 class _BackgroundProcess:

synapse/metrics/jemalloc.py

@@ -188,7 +188,8 @@ def _setup_jemalloc_stats() -> None:
         def collect(self) -> Iterable[Metric]:
             stats.refresh_stats()
 
-            g = GaugeMetricFamily(
+            # This is a process-level metric, so it does not have the `SERVER_NAME_LABEL`.
+            g = GaugeMetricFamily(  # type: ignore[missing-server-name-label]
                 "jemalloc_stats_app_memory_bytes",
                 "The stats reported by jemalloc",
                 labels=["type"],
@@ -230,7 +231,8 @@ def _setup_jemalloc_stats() -> None:
 
             yield g
 
-    REGISTRY.register(JemallocCollector())
+    # This is a process-level metric, so it does not have the `SERVER_NAME_LABEL`.
+    REGISTRY.register(JemallocCollector())  # type: ignore[missing-server-name-label]
 
     logger.debug("Added jemalloc stats")
 

synapse/module_api/__init__.py

@@ -342,7 +342,9 @@ class ModuleApi:
         self._device_handler = hs.get_device_handler()
         self.custom_template_dir = hs.config.server.custom_template_directory
         self._callbacks = hs.get_module_api_callbacks()
-        self.msc3861_oauth_delegation_enabled = hs.config.experimental.msc3861.enabled
+        self._auth_delegation_enabled = (
+            hs.config.mas.enabled or hs.config.experimental.msc3861.enabled
+        )
         self._event_serializer = hs.get_event_client_serializer()
 
         try:
@@ -549,7 +551,7 @@ class ModuleApi:
 
         Added in Synapse v1.46.0.
         """
-        if self.msc3861_oauth_delegation_enabled:
+        if self._auth_delegation_enabled:
             raise ConfigError(
                 "Cannot use password auth provider callbacks when OAuth delegation is enabled"
             )
@@ -759,7 +761,7 @@ class ModuleApi:
         Returns:
             True if the user is a server admin, False otherwise.
         """
-        return await self._store.is_server_admin(UserID.from_string(user_id))
+        return await self._store.is_server_admin(user_id)
 
     async def set_user_admin(self, user_id: str, admin: bool) -> None:
         """Sets if a user is a server admin.

synapse/push/bulk_push_rule_evaluator.py

@@ -25,6 +25,7 @@ from typing import (
     Any,
     Collection,
     Dict,
+    FrozenSet,
     List,
     Mapping,
     Optional,
@@ -477,8 +478,18 @@ class BulkPushRuleEvaluator:
             event.room_version.msc3931_push_features,
             self.hs.config.experimental.msc1767_enabled,  # MSC3931 flag
             self.hs.config.experimental.msc4210_enabled,
+            self.hs.config.experimental.msc4306_enabled,
         )
 
+        msc4306_thread_subscribers: Optional[FrozenSet[str]] = None
+        if self.hs.config.experimental.msc4306_enabled and thread_id != MAIN_TIMELINE:
+            # pull out, in batch, all local subscribers to this thread
+            # (in the common case, they will all be getting processed for push
+            # rules right now)
+            msc4306_thread_subscribers = await self.store.get_subscribers_to_thread(
+                event.room_id, thread_id
+            )
+
         for uid, rules in rules_by_user.items():
             if event.sender == uid:
                 continue
@@ -503,7 +514,13 @@ class BulkPushRuleEvaluator:
                 # current user, it'll be added to the dict later.
                 actions_by_user[uid] = []
 
-            actions = evaluator.run(rules, uid, display_name)
+            msc4306_thread_subscription_state: Optional[bool] = None
+            if msc4306_thread_subscribers is not None:
+                msc4306_thread_subscription_state = uid in msc4306_thread_subscribers
+
+            actions = evaluator.run(
+                rules, uid, display_name, msc4306_thread_subscription_state
+            )
             if "notify" in actions:
                 # Push rules say we should notify the user of this event
                 actions_by_user[uid] = actions

synapse/replication/tcp/streams/_base.py

@@ -739,7 +739,7 @@ class ThreadSubscriptionsStream(_StreamFromIdGen):
     NAME = "thread_subscriptions"
     ROW_TYPE = ThreadSubscriptionsStreamRow
 
-    def __init__(self, hs: Any):
+    def __init__(self, hs: "HomeServer"):
         self.store = hs.get_datastores().main
         super().__init__(
             hs.get_instance_name(),
@@ -751,7 +751,7 @@ class ThreadSubscriptionsStream(_StreamFromIdGen):
         self, instance_name: str, from_token: int, to_token: int, limit: int
     ) -> StreamUpdateResult:
         updates = await self.store.get_updated_thread_subscriptions(
-            from_token, to_token, limit
+            from_id=from_token, to_id=to_token, limit=limit
         )
         rows = [
             (

synapse/rest/admin/__init__.py

@@ -272,11 +272,15 @@ def register_servlets(hs: "HomeServer", http_server: HttpServer) -> None:
     # Admin servlets below may not work on workers.
     if hs.config.worker.worker_app is not None:
         # Some admin servlets can be mounted on workers when MSC3861 is enabled.
+        # Note that this is only for MSC3861 mode, as modern MAS using the
+        # matrix_authentication_service integration uses the dedicated MAS API.
         if hs.config.experimental.msc3861.enabled:
             register_servlets_for_msc3861_delegation(hs, http_server)
 
         return
 
+    auth_delegated = hs.config.mas.enabled or hs.config.experimental.msc3861.enabled
+
     register_servlets_for_client_rest_resource(hs, http_server)
     BlockRoomRestServlet(hs).register(http_server)
     ListRoomRestServlet(hs).register(http_server)
@@ -287,10 +291,10 @@ def register_servlets(hs: "HomeServer", http_server: HttpServer) -> None:
     DeleteRoomStatusByRoomIdRestServlet(hs).register(http_server)
     JoinRoomAliasServlet(hs).register(http_server)
     VersionServlet(hs).register(http_server)
-    if not hs.config.experimental.msc3861.enabled:
+    if not auth_delegated:
         UserAdminServlet(hs).register(http_server)
     UserMembershipRestServlet(hs).register(http_server)
-    if not hs.config.experimental.msc3861.enabled:
+    if not auth_delegated:
         UserTokenRestServlet(hs).register(http_server)
     UserRestServletV2(hs).register(http_server)
     UsersRestServletV2(hs).register(http_server)
@@ -307,7 +311,7 @@ def register_servlets(hs: "HomeServer", http_server: HttpServer) -> None:
     RoomEventContextServlet(hs).register(http_server)
     RateLimitRestServlet(hs).register(http_server)
     UsernameAvailableRestServlet(hs).register(http_server)
-    if not hs.config.experimental.msc3861.enabled:
+    if not auth_delegated:
         ListRegistrationTokensRestServlet(hs).register(http_server)
         NewRegistrationTokenRestServlet(hs).register(http_server)
         RegistrationTokenRestServlet(hs).register(http_server)
@@ -341,16 +345,18 @@ def register_servlets_for_client_rest_resource(
     hs: "HomeServer", http_server: HttpServer
 ) -> None:
     """Register only the servlets which need to be exposed on /_matrix/client/xxx"""
+    auth_delegated = hs.config.mas.enabled or hs.config.experimental.msc3861.enabled
+
     WhoisRestServlet(hs).register(http_server)
     PurgeHistoryStatusRestServlet(hs).register(http_server)
     PurgeHistoryRestServlet(hs).register(http_server)
     # The following resources can only be run on the main process.
     if hs.config.worker.worker_app is None:
         DeactivateAccountRestServlet(hs).register(http_server)
-        if not hs.config.experimental.msc3861.enabled:
+        if not auth_delegated:
             ResetPasswordRestServlet(hs).register(http_server)
     SearchUsersRestServlet(hs).register(http_server)
-    if not hs.config.experimental.msc3861.enabled:
+    if not auth_delegated:
         UserRegisterServlet(hs).register(http_server)
         AccountValidityRenewServlet(hs).register(http_server)
 

synapse/rest/admin/rooms.py

@@ -627,6 +627,15 @@ class MakeRoomAdminRestServlet(ResolveRoomIdMixin, RestServlet):
             ]
             admin_users.sort(key=lambda user: user_power[user])
 
+            if create_event.room_version.msc4289_creator_power_enabled:
+                creators = create_event.content.get("additional_creators", []) + [
+                    create_event.sender
+                ]
+                for creator in creators:
+                    if self.is_mine_id(creator):
+                        # include the creator as they won't be in the PL users map.
+                        admin_users.append(creator)
+
             if not admin_users:
                 raise SynapseError(
                     HTTPStatus.BAD_REQUEST, "No local admin user in room"
@@ -666,7 +675,11 @@ class MakeRoomAdminRestServlet(ResolveRoomIdMixin, RestServlet):
         # updated power level event.
         new_pl_content = dict(pl_content)
         new_pl_content["users"] = dict(pl_content.get("users", {}))
-        new_pl_content["users"][user_to_add] = new_pl_content["users"][admin_user_id]
+        # give the new user the same PL as the admin, default to 100 in case there is no PL event.
+        # This means in v12+ rooms we get PL100 if the creator promotes us.
+        new_pl_content["users"][user_to_add] = new_pl_content["users"].get(
+            admin_user_id, 100
+        )
 
         fake_requester = create_requester(
             admin_user_id,

synapse/rest/admin/users.py

@@ -109,7 +109,9 @@ class UsersRestServletV2(RestServlet):
         self.auth = hs.get_auth()
         self.admin_handler = hs.get_admin_handler()
         self._msc3866_enabled = hs.config.experimental.msc3866.enabled
-        self._msc3861_enabled = hs.config.experimental.msc3861.enabled
+        self._auth_delegation_enabled = (
+            hs.config.mas.enabled or hs.config.experimental.msc3861.enabled
+        )
 
     async def on_GET(self, request: SynapseRequest) -> Tuple[int, JsonDict]:
         await assert_requester_is_admin(self.auth, request)
@@ -121,10 +123,10 @@ class UsersRestServletV2(RestServlet):
         name = parse_string(request, "name", encoding="utf-8")
 
         guests = parse_boolean(request, "guests", default=True)
-        if self._msc3861_enabled and guests:
+        if self._auth_delegation_enabled and guests:
             raise SynapseError(
                 HTTPStatus.BAD_REQUEST,
-                "The guests parameter is not supported when MSC3861 is enabled.",
+                "The guests parameter is not supported when delegating to MAS.",
                 errcode=Codes.INVALID_PARAM,
             )
 
@@ -998,7 +1000,7 @@ class UserAdminServlet(RestServlet):
                 "Only local users can be admins of this homeserver",
             )
 
-        is_admin = await self.store.is_server_admin(target_user)
+        is_admin = await self.store.is_server_admin(target_user.to_string())
 
         return HTTPStatus.OK, {"admin": is_admin}
 

synapse/rest/client/account.py

@@ -613,7 +613,7 @@ class ThreepidRestServlet(RestServlet):
     # ThreePidBindRestServelet.PostBody with an `alias_generator` to handle
     # `threePidCreds` versus `three_pid_creds`.
     async def on_POST(self, request: SynapseRequest) -> Tuple[int, JsonDict]:
-        if self.hs.config.experimental.msc3861.enabled:
+        if self.hs.config.mas.enabled or self.hs.config.experimental.msc3861.enabled:
             raise NotFoundError(errcode=Codes.UNRECOGNIZED)
 
         if not self.hs.config.registration.enable_3pid_changes:
@@ -905,18 +905,19 @@ class AccountStatusRestServlet(RestServlet):
 
 
 def register_servlets(hs: "HomeServer", http_server: HttpServer) -> None:
+    auth_delegated = hs.config.mas.enabled or hs.config.experimental.msc3861.enabled
+
     ThreepidRestServlet(hs).register(http_server)
     WhoamiRestServlet(hs).register(http_server)
 
-    if not hs.config.experimental.msc3861.enabled:
+    if not auth_delegated:
         DeactivateAccountRestServlet(hs).register(http_server)
 
-    # These servlets are only registered on the main process
     if hs.config.worker.worker_app is None:
         ThreepidBindRestServlet(hs).register(http_server)
         ThreepidUnbindRestServlet(hs).register(http_server)
 
-        if not hs.config.experimental.msc3861.enabled:
+        if not auth_delegated:
             EmailPasswordRequestTokenRestServlet(hs).register(http_server)
             PasswordRestServlet(hs).register(http_server)
             EmailThreepidRequestTokenRestServlet(hs).register(http_server)
@@ -926,5 +927,5 @@ def register_servlets(hs: "HomeServer", http_server: HttpServer) -> None:
             ThreepidAddRestServlet(hs).register(http_server)
             ThreepidDeleteRestServlet(hs).register(http_server)
 
-        if hs.config.experimental.msc3720_enabled:
-            AccountStatusRestServlet(hs).register(http_server)
+    if hs.config.experimental.msc3720_enabled:
+        AccountStatusRestServlet(hs).register(http_server)

synapse/rest/client/auth.py

@@ -20,10 +20,11 @@
 #
 
 import logging
-from typing import TYPE_CHECKING, cast
+from typing import TYPE_CHECKING
 
 from twisted.web.server import Request
 
+from synapse.api.auth.mas import MasDelegatedAuth
 from synapse.api.constants import LoginType
 from synapse.api.errors import LoginError, SynapseError
 from synapse.api.urls import CLIENT_API_PREFIX
@@ -66,22 +67,30 @@ class AuthRestServlet(RestServlet):
         if not session:
             raise SynapseError(400, "No session supplied")
 
-        if (
-            self.hs.config.experimental.msc3861.enabled
-            and stagetype == "org.matrix.cross_signing_reset"
-        ):
-            # If MSC3861 is enabled, we can assume self._auth is an instance of MSC3861DelegatedAuth
-            # We import lazily here because of the authlib requirement
-            from synapse.api.auth.msc3861_delegated import MSC3861DelegatedAuth
+        if stagetype == "org.matrix.cross_signing_reset":
+            if self.hs.config.mas.enabled:
+                assert isinstance(self.auth, MasDelegatedAuth)
 
-            auth = cast(MSC3861DelegatedAuth, self.auth)
-
-            url = await auth.account_management_url()
-            if url is not None:
+                url = await self.auth.account_management_url()
                 url = f"{url}?action=org.matrix.cross_signing_reset"
-            else:
-                url = await auth.issuer()
-            respond_with_redirect(request, str.encode(url))
+                return respond_with_redirect(
+                    request,
+                    url.encode(),
+                )
+
+            elif self.hs.config.experimental.msc3861.enabled:
+                # If MSC3861 is enabled, we can assume self._auth is an instance of MSC3861DelegatedAuth
+                # We import lazily here because of the authlib requirement
+                from synapse.api.auth.msc3861_delegated import MSC3861DelegatedAuth
+
+                assert isinstance(self.auth, MSC3861DelegatedAuth)
+
+                base = await self.auth.account_management_url()
+                if base is not None:
+                    url = f"{base}?action=org.matrix.cross_signing_reset"
+                else:
+                    url = await self.auth.issuer()
+                return respond_with_redirect(request, url.encode())
 
         if stagetype == LoginType.RECAPTCHA:
             html = self.recaptcha_template.render(

synapse/rest/client/auth_metadata.py

@@ -15,6 +15,7 @@ import logging
 import typing
 from typing import Tuple, cast
 
+from synapse.api.auth.mas import MasDelegatedAuth
 from synapse.api.errors import Codes, SynapseError
 from synapse.http.server import HttpServer
 from synapse.http.servlet import RestServlet
@@ -48,13 +49,18 @@ class AuthIssuerServlet(RestServlet):
         self._auth = hs.get_auth()
 
     async def on_GET(self, request: SynapseRequest) -> Tuple[int, JsonDict]:
-        if self._config.experimental.msc3861.enabled:
+        if self._config.mas.enabled:
+            assert isinstance(self._auth, MasDelegatedAuth)
+            return 200, {"issuer": await self._auth.issuer()}
+
+        elif self._config.experimental.msc3861.enabled:
             # If MSC3861 is enabled, we can assume self._auth is an instance of MSC3861DelegatedAuth
             # We import lazily here because of the authlib requirement
             from synapse.api.auth.msc3861_delegated import MSC3861DelegatedAuth
 
-            auth = cast(MSC3861DelegatedAuth, self._auth)
-            return 200, {"issuer": await auth.issuer()}
+            assert isinstance(self._auth, MSC3861DelegatedAuth)
+            return 200, {"issuer": await self._auth.issuer()}
+
         else:
             # Wouldn't expect this to be reached: the servelet shouldn't have been
             # registered. Still, fail gracefully if we are registered for some reason.
@@ -82,13 +88,18 @@ class AuthMetadataServlet(RestServlet):
         self._auth = hs.get_auth()
 
     async def on_GET(self, request: SynapseRequest) -> Tuple[int, JsonDict]:
-        if self._config.experimental.msc3861.enabled:
+        if self._config.mas.enabled:
+            assert isinstance(self._auth, MasDelegatedAuth)
+            return 200, await self._auth.auth_metadata()
+
+        elif self._config.experimental.msc3861.enabled:
             # If MSC3861 is enabled, we can assume self._auth is an instance of MSC3861DelegatedAuth
             # We import lazily here because of the authlib requirement
             from synapse.api.auth.msc3861_delegated import MSC3861DelegatedAuth
 
             auth = cast(MSC3861DelegatedAuth, self._auth)
             return 200, await auth.auth_metadata()
+
         else:
             # Wouldn't expect this to be reached: the servlet shouldn't have been
             # registered. Still, fail gracefully if we are registered for some reason.
@@ -100,7 +111,6 @@ class AuthMetadataServlet(RestServlet):
 
 
 def register_servlets(hs: "HomeServer", http_server: HttpServer) -> None:
-    # We use the MSC3861 values as they are used by multiple MSCs
-    if hs.config.experimental.msc3861.enabled:
+    if hs.config.mas.enabled or hs.config.experimental.msc3861.enabled:
         AuthIssuerServlet(hs).register(http_server)
         AuthMetadataServlet(hs).register(http_server)

synapse/rest/client/devices.py

@@ -144,7 +144,9 @@ class DeviceRestServlet(RestServlet):
         self.device_handler = handler
         self.auth_handler = hs.get_auth_handler()
         self._msc3852_enabled = hs.config.experimental.msc3852_enabled
-        self._msc3861_oauth_delegation_enabled = hs.config.experimental.msc3861.enabled
+        self._auth_delegation_enabled = (
+            hs.config.mas.enabled or hs.config.experimental.msc3861.enabled
+        )
 
     async def on_GET(
         self, request: SynapseRequest, device_id: str
@@ -196,7 +198,7 @@ class DeviceRestServlet(RestServlet):
             pass
 
         else:
-            if self._msc3861_oauth_delegation_enabled:
+            if self._auth_delegation_enabled:
                 raise UnrecognizedRequestError(code=404)
 
             await self.auth_handler.validate_user_via_ui_auth(
@@ -573,7 +575,8 @@ class DehydratedDeviceV2Servlet(RestServlet):
 
 
 def register_servlets(hs: "HomeServer", http_server: HttpServer) -> None:
-    if not hs.config.experimental.msc3861.enabled:
+    auth_delegated = hs.config.mas.enabled or hs.config.experimental.msc3861.enabled
+    if not auth_delegated:
         DeleteDevicesRestServlet(hs).register(http_server)
     DevicesRestServlet(hs).register(http_server)
     DeviceRestServlet(hs).register(http_server)

synapse/rest/client/keys.py

@@ -23,8 +23,9 @@
 import logging
 import re
 from collections import Counter
-from typing import TYPE_CHECKING, Any, Dict, Optional, Tuple, cast
+from typing import TYPE_CHECKING, Any, Dict, Optional, Tuple
 
+from synapse.api.auth.mas import MasDelegatedAuth
 from synapse.api.errors import (
     InteractiveAuthIncompleteError,
     InvalidAPICallError,
@@ -404,19 +405,11 @@ class SigningKeyUploadServlet(RestServlet):
         if is_cross_signing_setup:
             # With MSC3861, UIA is not possible. Instead, the auth service has to
             # explicitly mark the master key as replaceable.
-            if self.hs.config.experimental.msc3861.enabled:
+            if self.hs.config.mas.enabled:
                 if not master_key_updatable_without_uia:
-                    # If MSC3861 is enabled, we can assume self.auth is an instance of MSC3861DelegatedAuth
-                    # We import lazily here because of the authlib requirement
-                    from synapse.api.auth.msc3861_delegated import MSC3861DelegatedAuth
-
-                    auth = cast(MSC3861DelegatedAuth, self.auth)
-
-                    uri = await auth.account_management_url()
-                    if uri is not None:
-                        url = f"{uri}?action=org.matrix.cross_signing_reset"
-                    else:
-                        url = await auth.issuer()
+                    assert isinstance(self.auth, MasDelegatedAuth)
+                    url = await self.auth.account_management_url()
+                    url = f"{url}?action=org.matrix.cross_signing_reset"
 
                     # We use a dummy session ID as this isn't really a UIA flow, but we
                     # reuse the same API shape for better client compatibility.
@@ -437,6 +430,41 @@ class SigningKeyUploadServlet(RestServlet):
                             "then try again.",
                         },
                     )
+
+            elif self.hs.config.experimental.msc3861.enabled:
+                if not master_key_updatable_without_uia:
+                    # If MSC3861 is enabled, we can assume self.auth is an instance of MSC3861DelegatedAuth
+                    # We import lazily here because of the authlib requirement
+                    from synapse.api.auth.msc3861_delegated import MSC3861DelegatedAuth
+
+                    assert isinstance(self.auth, MSC3861DelegatedAuth)
+
+                    uri = await self.auth.account_management_url()
+                    if uri is not None:
+                        url = f"{uri}?action=org.matrix.cross_signing_reset"
+                    else:
+                        url = await self.auth.issuer()
+
+                    # We use a dummy session ID as this isn't really a UIA flow, but we
+                    # reuse the same API shape for better client compatibility.
+                    raise InteractiveAuthIncompleteError(
+                        "dummy",
+                        {
+                            "session": "dummy",
+                            "flows": [
+                                {"stages": ["org.matrix.cross_signing_reset"]},
+                            ],
+                            "params": {
+                                "org.matrix.cross_signing_reset": {
+                                    "url": url,
+                                },
+                            },
+                            "msg": "To reset your end-to-end encryption cross-signing "
+                            f"identity, you first need to approve it at {url} and "
+                            "then try again.",
+                        },
+                    )
+
             else:
                 # Without MSC3861, we require UIA.
                 await self.auth_handler.validate_user_via_ui_auth(

synapse/rest/client/login.py

@@ -715,7 +715,7 @@ class CasTicketServlet(RestServlet):
 
 
 def register_servlets(hs: "HomeServer", http_server: HttpServer) -> None:
-    if hs.config.experimental.msc3861.enabled:
+    if hs.config.mas.enabled or hs.config.experimental.msc3861.enabled:
         return
 
     LoginRestServlet(hs).register(http_server)

synapse/rest/client/logout.py

@@ -86,7 +86,7 @@ class LogoutAllRestServlet(RestServlet):
 
 
 def register_servlets(hs: "HomeServer", http_server: HttpServer) -> None:
-    if hs.config.experimental.msc3861.enabled:
+    if hs.config.mas.enabled or hs.config.experimental.msc3861.enabled:
         return
 
     LogoutRestServlet(hs).register(http_server)

synapse/rest/client/register.py

@@ -1044,7 +1044,7 @@ def _calculate_registration_flows(
 
 
 def register_servlets(hs: "HomeServer", http_server: HttpServer) -> None:
-    if hs.config.experimental.msc3861.enabled:
+    if hs.config.mas.enabled or hs.config.experimental.msc3861.enabled:
         RegisterAppServiceOnlyRestServlet(hs).register(http_server)
         return
 

synapse/rest/client/room_upgrade_rest_servlet.py

@@ -24,6 +24,7 @@ from typing import TYPE_CHECKING, Tuple
 
 from synapse.api.errors import Codes, ShadowBanError, SynapseError
 from synapse.api.room_versions import KNOWN_ROOM_VERSIONS
+from synapse.event_auth import check_valid_additional_creators
 from synapse.handlers.worker_lock import NEW_EVENT_DURING_PURGE_LOCK_NAME
 from synapse.http.server import HttpServer
 from synapse.http.servlet import (
@@ -85,13 +86,18 @@ class RoomUpgradeRestServlet(RestServlet):
                 "Your homeserver does not support this room version",
                 Codes.UNSUPPORTED_ROOM_VERSION,
             )
+        additional_creators = None
+        if new_version.msc4289_creator_power_enabled:
+            additional_creators = content.get("additional_creators")
+            if additional_creators is not None:
+                check_valid_additional_creators(additional_creators)
 
         try:
             async with self._worker_lock_handler.acquire_read_write_lock(
                 NEW_EVENT_DURING_PURGE_LOCK_NAME, room_id, write=False
             ):
                 new_room_id = await self._room_creation_handler.upgrade_room(
-                    requester, room_id, new_version
+                    requester, room_id, new_version, additional_creators
                 )
         except ShadowBanError:
             # Generate a random room ID.

synapse/rest/client/thread_subscriptions.py

@@ -1,7 +1,6 @@
 from http import HTTPStatus
-from typing import TYPE_CHECKING, Tuple
+from typing import TYPE_CHECKING, Optional, Tuple
 
-from synapse._pydantic_compat import StrictBool
 from synapse.api.errors import Codes, NotFoundError, SynapseError
 from synapse.http.server import HttpServer
 from synapse.http.servlet import (
@@ -12,6 +11,7 @@ from synapse.http.site import SynapseRequest
 from synapse.rest.client._base import client_patterns
 from synapse.types import JsonDict, RoomID
 from synapse.types.rest import RequestBodyModel
+from synapse.util.pydantic_models import AnyEventId
 
 if TYPE_CHECKING:
     from synapse.server import HomeServer
@@ -32,7 +32,12 @@ class ThreadSubscriptionsRestServlet(RestServlet):
         self.handler = hs.get_thread_subscriptions_handler()
 
     class PutBody(RequestBodyModel):
-        automatic: StrictBool
+        automatic: Optional[AnyEventId]
+        """
+        If supplied, the event ID of an event giving rise to this automatic subscription.
+
+        If omitted, this subscription is a manual subscription.
+        """
 
     async def on_GET(
         self, request: SynapseRequest, room_id: str, thread_root_id: str
@@ -63,15 +68,15 @@ class ThreadSubscriptionsRestServlet(RestServlet):
             raise SynapseError(
                 HTTPStatus.BAD_REQUEST, "Invalid event ID", errcode=Codes.INVALID_PARAM
             )
-        requester = await self.auth.get_user_by_req(request)
-
         body = parse_and_validate_json_object_from_request(request, self.PutBody)
 
+        requester = await self.auth.get_user_by_req(request)
+
         await self.handler.subscribe_user_to_thread(
             requester.user,
             room_id,
             thread_root_id,
-            automatic=body.automatic,
+            automatic_event_id=body.automatic,
         )
 
         return HTTPStatus.OK, {}

synapse/rest/synapse/client/__init__.py

@@ -56,8 +56,9 @@ def build_synapse_client_resource_tree(hs: "HomeServer") -> Mapping[str, Resourc
         "/_synapse/client/unsubscribe": UnsubscribeResource(hs),
     }
 
-    # Expose the JWKS endpoint if OAuth2 delegation is enabled
-    if hs.config.experimental.msc3861.enabled:
+    if hs.config.mas.enabled:
+        resources["/_synapse/mas"] = MasResource(hs)
+    elif hs.config.experimental.msc3861.enabled:
         from synapse.rest.synapse.client.jwks import JwksResource
 
         resources["/_synapse/jwks"] = JwksResource(hs)

synapse/rest/synapse/mas/_base.py

@@ -16,6 +16,7 @@
 
 from typing import TYPE_CHECKING, cast
 
+from synapse.api.auth.mas import MasDelegatedAuth
 from synapse.api.errors import SynapseError
 from synapse.http.server import DirectServeJsonResource
 
@@ -27,14 +28,21 @@ if TYPE_CHECKING:
 
 class MasBaseResource(DirectServeJsonResource):
     def __init__(self, hs: "HomeServer"):
-        # Importing this module requires authlib, which is an optional
-        # dependency but required if msc3861 is enabled
-        from synapse.api.auth.msc3861_delegated import MSC3861DelegatedAuth
+        auth = hs.get_auth()
+        if hs.config.mas.enabled:
+            assert isinstance(auth, MasDelegatedAuth)
+
+            self._is_request_from_mas = auth.is_request_using_the_shared_secret
+        else:
+            # Importing this module requires authlib, which is an optional
+            # dependency but required if msc3861 is enabled
+            from synapse.api.auth.msc3861_delegated import MSC3861DelegatedAuth
+
+            assert isinstance(auth, MSC3861DelegatedAuth)
+
+            self._is_request_from_mas = auth.is_request_using_the_admin_token
 
         DirectServeJsonResource.__init__(self, extract_context=True)
-        auth = hs.get_auth()
-        assert isinstance(auth, MSC3861DelegatedAuth)
-        self.msc3861_auth = auth
         self.store = cast("GenericWorkerStore", hs.get_datastores().main)
         self.hostname = hs.hostname
 
@@ -43,5 +51,5 @@ class MasBaseResource(DirectServeJsonResource):
 
         Throws a 403 if the request is not coming from MAS.
         """
-        if not self.msc3861_auth.is_request_using_the_admin_token(request):
+        if not self._is_request_from_mas(request):
             raise SynapseError(403, "This endpoint must only be called by MAS")

synapse/rest/well_known.py

@@ -18,11 +18,12 @@
 #
 #
 import logging
-from typing import TYPE_CHECKING, Optional, Tuple, cast
+from typing import TYPE_CHECKING, Optional, Tuple
 
 from twisted.web.resource import Resource
 from twisted.web.server import Request
 
+from synapse.api.auth.mas import MasDelegatedAuth
 from synapse.api.errors import NotFoundError
 from synapse.http.server import DirectServeJsonResource
 from synapse.http.site import SynapseRequest
@@ -52,18 +53,25 @@ class WellKnownBuilder:
                 "base_url": self._config.registration.default_identity_server
             }
 
-        # We use the MSC3861 values as they are used by multiple MSCs
-        if self._config.experimental.msc3861.enabled:
+        if self._config.mas.enabled:
+            assert isinstance(self._auth, MasDelegatedAuth)
+
+            result["org.matrix.msc2965.authentication"] = {
+                "issuer": await self._auth.issuer(),
+                "account": await self._auth.account_management_url(),
+            }
+
+        elif self._config.experimental.msc3861.enabled:
             # If MSC3861 is enabled, we can assume self._auth is an instance of MSC3861DelegatedAuth
             # We import lazily here because of the authlib requirement
             from synapse.api.auth.msc3861_delegated import MSC3861DelegatedAuth
 
-            auth = cast(MSC3861DelegatedAuth, self._auth)
+            assert isinstance(self._auth, MSC3861DelegatedAuth)
 
             result["org.matrix.msc2965.authentication"] = {
-                "issuer": await auth.issuer(),
+                "issuer": await self._auth.issuer(),
             }
-            account_management_url = await auth.account_management_url()
+            account_management_url = await self._auth.account_management_url()
             if account_management_url is not None:
                 result["org.matrix.msc2965.authentication"]["account"] = (
                     account_management_url

synapse/server.py

@@ -40,6 +40,7 @@ from twisted.web.resource import Resource
 
 from synapse.api.auth import Auth
 from synapse.api.auth.internal import InternalAuth
+from synapse.api.auth.mas import MasDelegatedAuth
 from synapse.api.auth_blocking import AuthBlocking
 from synapse.api.filtering import Filtering
 from synapse.api.ratelimiting import Ratelimiter, RequestRatelimiter
@@ -451,6 +452,8 @@ class HomeServer(metaclass=abc.ABCMeta):
 
     @cache_in_self
     def get_auth(self) -> Auth:
+        if self.config.mas.enabled:
+            return MasDelegatedAuth(self)
         if self.config.experimental.msc3861.enabled:
             from synapse.api.auth.msc3861_delegated import MSC3861DelegatedAuth
 

synapse/state/__init__.py

@@ -54,6 +54,7 @@ from synapse.logging.opentracing import tag_args, trace
 from synapse.metrics import SERVER_NAME_LABEL
 from synapse.replication.http.state import ReplicationUpdateCurrentStateRestServlet
 from synapse.state import v1, v2
+from synapse.storage.databases.main.event_federation import StateDifference
 from synapse.storage.databases.main.events_worker import EventRedactBehaviour
 from synapse.types import StateMap, StrCollection
 from synapse.types.state import StateFilter
@@ -990,17 +991,35 @@ class StateResolutionStore:
         )
 
     def get_auth_chain_difference(
-        self, room_id: str, state_sets: List[Set[str]]
-    ) -> Awaitable[Set[str]]:
-        """Given sets of state events figure out the auth chain difference (as
+        self,
+        room_id: str,
+        state_sets: List[Set[str]],
+        conflicted_state: Optional[Set[str]],
+        additional_backwards_reachable_conflicted_events: Optional[Set[str]],
+    ) -> Awaitable[StateDifference]:
+        """ "Given sets of state events figure out the auth chain difference (as
         per state res v2 algorithm).
 
-        This equivalent to fetching the full auth chain for each set of state
+        This is equivalent to fetching the full auth chain for each set of state
         and returning the events that don't appear in each and every auth
         chain.
 
+        If conflicted_state is not None, calculate and return the conflicted sub-graph as per
+        state res v2.1. The event IDs in the conflicted state MUST be a subset of the event IDs in
+        state_sets.
+
+        If additional_backwards_reachable_conflicted_events is set, the provided events are included
+        when calculating the conflicted subgraph. This is primarily useful for calculating the
+        subgraph across a combination of persisted and unpersisted events.
+
         Returns:
-            An awaitable that resolves to a set of event IDs.
+            information on the auth chain difference, and also the conflicted subgraph if
+            conflicted_state is not None
         """
 
-        return self.main_store.get_auth_chain_difference(room_id, state_sets)
+        return self.main_store.get_auth_chain_difference_extended(
+            room_id,
+            state_sets,
+            conflicted_state,
+            additional_backwards_reachable_conflicted_events,
+        )

synapse/state/v2.py

@@ -39,10 +39,11 @@ from typing import (
 )
 
 from synapse import event_auth
-from synapse.api.constants import EventTypes
+from synapse.api.constants import CREATOR_POWER_LEVEL, EventTypes
 from synapse.api.errors import AuthError
-from synapse.api.room_versions import RoomVersion
-from synapse.events import EventBase
+from synapse.api.room_versions import RoomVersion, StateResolutionVersions
+from synapse.events import EventBase, is_creator
+from synapse.storage.databases.main.event_federation import StateDifference
 from synapse.types import MutableStateMap, StateMap, StrCollection
 
 logger = logging.getLogger(__name__)
@@ -52,7 +53,7 @@ class Clock(Protocol):
     # This is usually synapse.util.Clock, but it's replaced with a FakeClock in tests.
     # We only ever sleep(0) though, so that other async functions can make forward
     # progress without waiting for stateres to complete.
-    def sleep(self, duration_ms: float) -> Awaitable[None]: ...
+    async def sleep(self, duration_ms: float) -> None: ...
 
 
 class StateResolutionStore(Protocol):
@@ -63,8 +64,12 @@ class StateResolutionStore(Protocol):
     ) -> Awaitable[Dict[str, EventBase]]: ...
 
     def get_auth_chain_difference(
-        self, room_id: str, state_sets: List[Set[str]]
-    ) -> Awaitable[Set[str]]: ...
+        self,
+        room_id: str,
+        state_sets: List[Set[str]],
+        conflicted_state: Optional[Set[str]],
+        additional_backwards_reachable_conflicted_events: Optional[set[str]],
+    ) -> Awaitable[StateDifference]: ...
 
 
 # We want to await to the reactor occasionally during state res when dealing
@@ -123,12 +128,17 @@ async def resolve_events_with_store(
     logger.debug("%d conflicted state entries", len(conflicted_state))
     logger.debug("Calculating auth chain difference")
 
-    # Also fetch all auth events that appear in only some of the state sets'
-    # auth chains.
+    conflicted_set: Optional[Set[str]] = None
+    if room_version.state_res == StateResolutionVersions.V2_1:
+        # calculate the conflicted subgraph
+        conflicted_set = set(itertools.chain.from_iterable(conflicted_state.values()))
     auth_diff = await _get_auth_chain_difference(
-        room_id, state_sets, event_map, state_res_store
+        room_id,
+        state_sets,
+        event_map,
+        state_res_store,
+        conflicted_set,
     )
-
     full_conflicted_set = set(
         itertools.chain(
             itertools.chain.from_iterable(conflicted_state.values()), auth_diff
@@ -168,15 +178,26 @@ async def resolve_events_with_store(
 
     logger.debug("sorted %d power events", len(sorted_power_events))
 
+    # v2.1 starts iterative auth checks from the empty set and not the unconflicted state.
+    # It relies on IAC behaviour which populates the base state with the events from auth_events
+    # if the state tuple is missing from the base state. This ensures the base state is only
+    # populated from auth_events rather than whatever the unconflicted state is (which could be
+    # completely bogus).
+    base_state = (
+        {}
+        if room_version.state_res == StateResolutionVersions.V2_1
+        else unconflicted_state
+    )
+
     # Now sequentially auth each one
     resolved_state = await _iterative_auth_checks(
         clock,
         room_id,
         room_version,
-        sorted_power_events,
-        unconflicted_state,
-        event_map,
-        state_res_store,
+        event_ids=sorted_power_events,
+        base_state=base_state,
+        event_map=event_map,
+        state_res_store=state_res_store,
     )
 
     logger.debug("resolved power events")
@@ -239,13 +260,23 @@ async def _get_power_level_for_sender(
     event = await _get_event(room_id, event_id, event_map, state_res_store)
 
     pl = None
+    create = None
     for aid in event.auth_event_ids():
         aev = await _get_event(
             room_id, aid, event_map, state_res_store, allow_none=True
         )
         if aev and (aev.type, aev.state_key) == (EventTypes.PowerLevels, ""):
             pl = aev
-            break
+        if aev and (aev.type, aev.state_key) == (EventTypes.Create, ""):
+            create = aev
+
+    if event.type != EventTypes.Create:
+        # we should always have a create event
+        assert create is not None
+
+    if create and create.room_version.msc4289_creator_power_enabled:
+        if is_creator(create, event.sender):
+            return CREATOR_POWER_LEVEL
 
     if pl is None:
         # Couldn't find power level. Check if they're the creator of the room
@@ -286,6 +317,7 @@ async def _get_auth_chain_difference(
     state_sets: Sequence[StateMap[str]],
     unpersisted_events: Dict[str, EventBase],
     state_res_store: StateResolutionStore,
+    conflicted_state: Optional[Set[str]],
 ) -> Set[str]:
     """Compare the auth chains of each state set and return the set of events
     that only appear in some, but not all of the auth chains.
@@ -294,11 +326,18 @@ async def _get_auth_chain_difference(
         state_sets: The input state sets we are trying to resolve across.
         unpersisted_events: A map from event ID to EventBase containing all unpersisted
             events involved in this resolution.
-        state_res_store:
+        state_res_store: A way to retrieve events and extract graph information on the auth chains.
+        conflicted_state: which event IDs are conflicted. Used in v2.1 for calculating the conflicted
+            subgraph.
 
     Returns:
-        The auth difference of the given state sets, as a set of event IDs.
+        The auth difference of the given state sets, as a set of event IDs. Also includes the
+        conflicted subgraph if `conflicted_state` is set.
     """
+    is_state_res_v21 = conflicted_state is not None
+    num_conflicted_state = (
+        len(conflicted_state) if conflicted_state is not None else None
+    )
 
     # The `StateResolutionStore.get_auth_chain_difference` function assumes that
     # all events passed to it (and their auth chains) have been persisted
@@ -318,14 +357,19 @@ async def _get_auth_chain_difference(
     # the event's auth chain with the events in `unpersisted_events` *plus* their
     # auth event IDs.
     events_to_auth_chain: Dict[str, Set[str]] = {}
+    # remember the forward links when doing the graph traversal, we'll need it for v2.1 checks
+    # This is a map from an event to the set of events that contain it as an auth event.
+    event_to_next_event: Dict[str, Set[str]] = {}
     for event in unpersisted_events.values():
         chain = {event.event_id}
         events_to_auth_chain[event.event_id] = chain
 
         to_search = [event]
         while to_search:
-            for auth_id in to_search.pop().auth_event_ids():
+            next_event = to_search.pop()
+            for auth_id in next_event.auth_event_ids():
                 chain.add(auth_id)
+                event_to_next_event.setdefault(auth_id, set()).add(next_event.event_id)
                 auth_event = unpersisted_events.get(auth_id)
                 if auth_event:
                     to_search.append(auth_event)
@@ -335,6 +379,8 @@ async def _get_auth_chain_difference(
     #
     # Note: If there are no `unpersisted_events` (which is the common case), we can do a
     # much simpler calculation.
+    additional_backwards_reachable_conflicted_events: Set[str] = set()
+    unpersisted_conflicted_events: Set[str] = set()
     if unpersisted_events:
         # The list of state sets to pass to the store, where each state set is a set
         # of the event ids making up the state. This is similar to `state_sets`,
@@ -372,7 +418,16 @@ async def _get_auth_chain_difference(
                     )
                 else:
                     set_ids.add(event_id)
-
+        if conflicted_state:
+            for conflicted_event_id in conflicted_state:
+                # presence in this map means it is unpersisted.
+                event_chain = events_to_auth_chain.get(conflicted_event_id)
+                if event_chain is not None:
+                    unpersisted_conflicted_events.add(conflicted_event_id)
+                    # tell the DB layer that we have some unpersisted conflicted events
+                    additional_backwards_reachable_conflicted_events.update(
+                        e for e in event_chain if e not in unpersisted_events
+                    )
         # The auth chain difference of the unpersisted events of the state sets
         # is calculated by taking the difference between the union and
         # intersections.
@@ -384,12 +439,89 @@ async def _get_auth_chain_difference(
         auth_difference_unpersisted_part = ()
         state_sets_ids = [set(state_set.values()) for state_set in state_sets]
 
-    difference = await state_res_store.get_auth_chain_difference(
-        room_id, state_sets_ids
-    )
-    difference.update(auth_difference_unpersisted_part)
+    if conflicted_state:
+        # to ensure that conflicted state is a subset of state set IDs, we need to remove UNPERSISTED
+        # conflicted state set ids as we removed them above.
+        conflicted_state = conflicted_state - unpersisted_conflicted_events
 
-    return difference
+    difference = await state_res_store.get_auth_chain_difference(
+        room_id,
+        state_sets_ids,
+        conflicted_state,
+        additional_backwards_reachable_conflicted_events,
+    )
+    difference.auth_difference.update(auth_difference_unpersisted_part)
+
+    # if we're doing v2.1 we may need to add or expand the conflicted subgraph
+    if (
+        is_state_res_v21
+        and difference.conflicted_subgraph is not None
+        and unpersisted_events
+    ):
+        # we always include the conflicted events themselves in the subgraph.
+        if conflicted_state:
+            difference.conflicted_subgraph.update(conflicted_state)
+        # we may need to expand the subgraph in the case where the subgraph starts in the DB and
+        # ends in unpersisted events. To do this, we first need to see where the subgraph got up to,
+        # which we can do by finding the intersection between the additional backwards reachable
+        # conflicted events and the conflicted subgraph. Events in both sets mean A) some unpersisted
+        # conflicted event could backwards reach it and B) some persisted conflicted event could forward
+        # reach it.
+        subgraph_frontier = difference.conflicted_subgraph.intersection(
+            additional_backwards_reachable_conflicted_events
+        )
+        # we can now combine the 2 scenarios:
+        #  - subgraph starts in DB and ends in unpersisted
+        #  - subgraph starts in unpersisted and ends in unpersisted
+        # by expanding the frontier into unpersisted events.
+        # The frontier is currently all persisted events. We want to expand this into unpersisted
+        # events. Mark every forwards reachable event from the frontier in the forwards_conflicted_set
+        # but NOT the backwards conflicted set. This mirrors what the DB layer does but in reverse:
+        # we supplied events which are backwards reachable to the DB and now the DB is providing
+        # forwards reachable events from the DB.
+        forwards_conflicted_set: Set[str] = set()
+        # we include unpersisted conflicted events here to process exclusive unpersisted subgraphs
+        search_queue = subgraph_frontier.union(unpersisted_conflicted_events)
+        while search_queue:
+            frontier_event = search_queue.pop()
+            next_event_ids = event_to_next_event.get(frontier_event, set())
+            search_queue.update(next_event_ids)
+            forwards_conflicted_set.add(frontier_event)
+
+        # we've already calculated the backwards form as this is the auth chain for each
+        # unpersisted conflicted event.
+        backwards_conflicted_set: Set[str] = set()
+        for uce in unpersisted_conflicted_events:
+            backwards_conflicted_set.update(events_to_auth_chain.get(uce, []))
+
+        # the unpersisted conflicted subgraph is the intersection of the backwards/forwards sets
+        conflicted_subgraph_unpersisted_part = backwards_conflicted_set.intersection(
+            forwards_conflicted_set
+        )
+        # print(f"event_to_next_event={event_to_next_event}")
+        # print(f"unpersisted_conflicted_events={unpersisted_conflicted_events}")
+        # print(f"unperssited backwards_conflicted_set={backwards_conflicted_set}")
+        # print(f"unperssited forwards_conflicted_set={forwards_conflicted_set}")
+        difference.conflicted_subgraph.update(conflicted_subgraph_unpersisted_part)
+
+    if difference.conflicted_subgraph:
+        old_events = difference.auth_difference.union(
+            conflicted_state if conflicted_state else set()
+        )
+        additional_events = difference.conflicted_subgraph.difference(old_events)
+
+        logger.debug(
+            "v2.1 %s additional events replayed=%d num_conflicts=%d conflicted_subgraph=%d auth_difference=%d",
+            room_id,
+            len(additional_events),
+            num_conflicted_state,
+            len(difference.conflicted_subgraph),
+            len(difference.auth_difference),
+        )
+        # State res v2.1 includes the conflicted subgraph in the difference
+        return difference.auth_difference.union(difference.conflicted_subgraph)
+
+    return difference.auth_difference
 
 
 def _seperate(

synapse/storage/databases/main/deviceinbox.py

@@ -1031,7 +1031,7 @@ class DeviceInboxWorkerStore(SQLBaseStore):
 
             # We sleep a bit so that we don't hammer the database in a tight
             # loop first time we run this.
-            self._clock.sleep(1)
+            await self._clock.sleep(1)
 
     async def get_devices_with_messages(
         self, user_id: str, device_ids: StrCollection

synapse/storage/databases/main/event_federation.py

@@ -114,6 +114,12 @@ _LONGEST_BACKOFF_PERIOD_MILLISECONDS = (
 assert 0 < _LONGEST_BACKOFF_PERIOD_MILLISECONDS <= ((2**31) - 1)
 
 
+# We use 2^53-1 as a "very large number", it has no particular
+# importance other than knowing synapse can support it (given canonical json
+# requires it).
+MAX_CHAIN_LENGTH = (2**53) - 1
+
+
 # All the info we need while iterating the DAG while backfilling
 @attr.s(frozen=True, slots=True, auto_attribs=True)
 class BackfillQueueNavigationItem:
@@ -123,6 +129,14 @@ class BackfillQueueNavigationItem:
     type: str
 
 
+@attr.s(frozen=True, slots=True, auto_attribs=True)
+class StateDifference:
+    # The event IDs in the auth difference.
+    auth_difference: Set[str]
+    # The event IDs in the conflicted state subgraph. Used in v2.1 only.
+    conflicted_subgraph: Optional[Set[str]]
+
+
 class _NoChainCoverIndex(Exception):
     def __init__(self, room_id: str):
         super().__init__("Unexpectedly no chain cover for events in %s" % (room_id,))
@@ -471,17 +485,41 @@ class EventFederationWorkerStore(
         return results
 
     async def get_auth_chain_difference(
-        self, room_id: str, state_sets: List[Set[str]]
+        self,
+        room_id: str,
+        state_sets: List[Set[str]],
     ) -> Set[str]:
-        """Given sets of state events figure out the auth chain difference (as
+        state_diff = await self.get_auth_chain_difference_extended(
+            room_id, state_sets, None, None
+        )
+        return state_diff.auth_difference
+
+    async def get_auth_chain_difference_extended(
+        self,
+        room_id: str,
+        state_sets: List[Set[str]],
+        conflicted_set: Optional[Set[str]],
+        additional_backwards_reachable_conflicted_events: Optional[Set[str]],
+    ) -> StateDifference:
+        """ "Given sets of state events figure out the auth chain difference (as
         per state res v2 algorithm).
 
-        This equivalent to fetching the full auth chain for each set of state
+        This is equivalent to fetching the full auth chain for each set of state
         and returning the events that don't appear in each and every auth
         chain.
 
+        If conflicted_set is not None, calculate and return the conflicted sub-graph as per
+        state res v2.1. The event IDs in the conflicted set MUST be a subset of the event IDs in
+        state_sets.
+
+        If additional_backwards_reachable_conflicted_events is set, the provided events are included
+        when calculating the conflicted subgraph. This is primarily useful for calculating the
+        subgraph across a combination of persisted and unpersisted events. The event IDs in this set
+        MUST be a subset of the event IDs in state_sets.
+
         Returns:
-            The set of the difference in auth chains.
+            information on the auth chain difference, and also the conflicted subgraph if
+            conflicted_set is not None
         """
 
         # Check if we have indexed the room so we can use the chain cover
@@ -495,6 +533,8 @@ class EventFederationWorkerStore(
                     self._get_auth_chain_difference_using_cover_index_txn,
                     room_id,
                     state_sets,
+                    conflicted_set,
+                    additional_backwards_reachable_conflicted_events,
                 )
             except _NoChainCoverIndex:
                 # For whatever reason we don't actually have a chain cover index
@@ -503,25 +543,48 @@ class EventFederationWorkerStore(
                 if not self.tests_allow_no_chain_cover_index:
                     raise
 
-        return await self.db_pool.runInteraction(
+        # It's been 4 years since we added chain cover, so we expect all rooms to have it.
+        # If they don't, we will error out when trying to do state res v2.1
+        if conflicted_set is not None:
+            raise _NoChainCoverIndex(room_id)
+
+        auth_diff = await self.db_pool.runInteraction(
             "get_auth_chain_difference",
             self._get_auth_chain_difference_txn,
             state_sets,
         )
+        return StateDifference(auth_difference=auth_diff, conflicted_subgraph=None)
 
     def _get_auth_chain_difference_using_cover_index_txn(
-        self, txn: LoggingTransaction, room_id: str, state_sets: List[Set[str]]
-    ) -> Set[str]:
+        self,
+        txn: LoggingTransaction,
+        room_id: str,
+        state_sets: List[Set[str]],
+        conflicted_set: Optional[Set[str]] = None,
+        additional_backwards_reachable_conflicted_events: Optional[Set[str]] = None,
+    ) -> StateDifference:
         """Calculates the auth chain difference using the chain index.
 
         See docs/auth_chain_difference_algorithm.md for details
         """
+        is_state_res_v21 = conflicted_set is not None
 
         # First we look up the chain ID/sequence numbers for all the events, and
         # work out the chain/sequence numbers reachable from each state set.
 
         initial_events = set(state_sets[0]).union(*state_sets[1:])
 
+        if is_state_res_v21:
+            # Sanity check v2.1 fields
+            assert conflicted_set is not None
+            assert conflicted_set.issubset(initial_events)
+            # It's possible for the conflicted_set to be empty if all the conflicts are in
+            # unpersisted events, so we don't assert that conflicted_set has len > 0
+            if additional_backwards_reachable_conflicted_events:
+                assert additional_backwards_reachable_conflicted_events.issubset(
+                    initial_events
+                )
+
         # Map from event_id -> (chain ID, seq no)
         chain_info: Dict[str, Tuple[int, int]] = {}
 
@@ -557,14 +620,14 @@ class EventFederationWorkerStore(
         events_missing_chain_info = initial_events.difference(chain_info)
 
         # The result set to return, i.e. the auth chain difference.
-        result: Set[str] = set()
+        auth_difference_result: Set[str] = set()
 
         if events_missing_chain_info:
             # For some reason we have events we haven't calculated the chain
             # index for, so we need to handle those separately. This should only
             # happen for older rooms where the server doesn't have all the auth
             # events.
-            result = self._fixup_auth_chain_difference_sets(
+            auth_difference_result = self._fixup_auth_chain_difference_sets(
                 txn,
                 room_id,
                 state_sets=state_sets,
@@ -583,6 +646,45 @@ class EventFederationWorkerStore(
 
             fetch_chain_info(new_events_to_fetch)
 
+        # State Res v2.1 needs extra data structures to calculate the conflicted subgraph which
+        # are outlined below.
+
+        # A subset of chain_info for conflicted events only, as we need to
+        # loop all conflicted chain positions. Map from event_id -> (chain ID, seq no)
+        conflicted_chain_positions: Dict[str, Tuple[int, int]] = {}
+        # For each chain, remember the positions where conflicted events are.
+        # We need this for calculating the forward reachable events.
+        conflicted_chain_to_seq: Dict[int, Set[int]] = {}  # chain_id => {seq_num}
+        #  A subset of chain_info for additional backwards reachable events only, as we need to
+        # loop all additional backwards reachable events for calculating backwards reachable events.
+        additional_backwards_reachable_positions: Dict[
+            str, Tuple[int, int]
+        ] = {}  # event_id => (chain_id, seq_num)
+        # These next two fields are critical as the intersection of them is the conflicted subgraph.
+        # We'll populate them when we walk the chain links.
+        # chain_id => max(seq_num) backwards reachable (e.g 4 means 1,2,3,4 are backwards reachable)
+        conflicted_backwards_reachable: Dict[int, int] = {}
+        # chain_id => min(seq_num) forwards reachable (e.g 4 means 4,5,6..n are forwards reachable)
+        conflicted_forwards_reachable: Dict[int, int] = {}
+
+        # populate the v2.1 data structures
+        if is_state_res_v21:
+            assert conflicted_set is not None
+            # provide chain positions for each conflicted event
+            for conflicted_event_id in conflicted_set:
+                (chain_id, seq_num) = chain_info[conflicted_event_id]
+                conflicted_chain_positions[conflicted_event_id] = (chain_id, seq_num)
+                conflicted_chain_to_seq.setdefault(chain_id, set()).add(seq_num)
+            if additional_backwards_reachable_conflicted_events:
+                for (
+                    additional_event_id
+                ) in additional_backwards_reachable_conflicted_events:
+                    (chain_id, seq_num) = chain_info[additional_event_id]
+                    additional_backwards_reachable_positions[additional_event_id] = (
+                        chain_id,
+                        seq_num,
+                    )
+
         # Corresponds to `state_sets`, except as a map from chain ID to max
         # sequence number reachable from the state set.
         set_to_chain: List[Dict[int, int]] = []
@@ -600,6 +702,8 @@ class EventFederationWorkerStore(
 
         # (We need to take a copy of `seen_chains` as the function mutates it)
         for links in self._get_chain_links(txn, set(seen_chains)):
+            # `links` encodes the backwards reachable events _from a single chain_ all the way to
+            # the root of the graph.
             for chains in set_to_chain:
                 for chain_id in links:
                     if chain_id not in chains:
@@ -608,6 +712,87 @@ class EventFederationWorkerStore(
                     _materialize(chain_id, chains[chain_id], links, chains)
 
                 seen_chains.update(chains)
+            if is_state_res_v21:
+                # Apply v2.1 conflicted event reachability checks.
+                #
+                #  A <-- B <-- C <-- D <-- E
+                #
+                # Backwards reachable from C = {A,B}
+                # Forwards reachable from C = {D,E}
+
+                # this handles calculating forwards reachable information and updates
+                # conflicted_forwards_reachable.
+                accumulate_forwards_reachable_events(
+                    conflicted_forwards_reachable,
+                    links,
+                    conflicted_chain_positions,
+                )
+
+                # handle backwards reachable information
+                for (
+                    conflicted_chain_id,
+                    conflicted_chain_seq,
+                ) in conflicted_chain_positions.values():
+                    if conflicted_chain_id not in links:
+                        # This conflicted event does not lie on the path to the root.
+                        continue
+
+                    # The conflicted chain position itself encodes reachability information
+                    # _within_ the chain. Set it now before walking to other links.
+                    conflicted_backwards_reachable[conflicted_chain_id] = max(
+                        conflicted_chain_seq,
+                        conflicted_backwards_reachable.get(conflicted_chain_id, 0),
+                    )
+
+                    # Build backwards reachability paths. This is the same as what the auth difference
+                    # code does. We find which chain the conflicted event
+                    # belongs to then walk it backwards to the root. We store reachability info
+                    # for all conflicted events in the same map 'conflicted_backwards_reachable'
+                    # as we don't care about the paths themselves.
+                    _materialize(
+                        conflicted_chain_id,
+                        conflicted_chain_seq,
+                        links,
+                        conflicted_backwards_reachable,
+                    )
+                # Mark some extra events as backwards reachable. This is used when we have some
+                # unpersisted events and want to know the subgraph across the persisted/unpersisted
+                # boundary:
+                #                  |
+                #  A <-- B <-- C <-|- D <-- E <-- F
+                #     persisted    |   unpersisted
+                #
+                # Assume {B,E} are conflicted, we want to return {B,C,D,E}
+                #
+                # The unpersisted code ensures it passes C as an additional backwards reachable
+                # event. C is NOT a conflicted event, but we do need to consider it as part of
+                # the backwards reachable set. When we then calculate the forwards reachable set
+                # from B, C will be in both the backwards and forwards reachable sets and hence
+                # will be included in the conflicted subgraph.
+                for (
+                    additional_chain_id,
+                    additional_chain_seq,
+                ) in additional_backwards_reachable_positions.values():
+                    if additional_chain_id not in links:
+                        # The additional backwards reachable event does not lie on the path to the root.
+                        continue
+
+                    # the additional event chain position itself encodes reachability information.
+                    # It means that position and all positions earlier in that chain are backwards reachable
+                    # by some unpersisted conflicted event.
+                    conflicted_backwards_reachable[additional_chain_id] = max(
+                        additional_chain_seq,
+                        conflicted_backwards_reachable.get(additional_chain_id, 0),
+                    )
+
+                    # Now walk the chains back, marking backwards reachable events.
+                    # This is the same thing we do for auth difference / conflicted events.
+                    _materialize(
+                        additional_chain_id,  # walk all links back, marking them as backwards reachable
+                        additional_chain_seq,
+                        links,
+                        conflicted_backwards_reachable,
+                    )
 
         # Now for each chain we figure out the maximum sequence number reachable
         # from *any* state set and the minimum sequence number reachable from
@@ -616,7 +801,7 @@ class EventFederationWorkerStore(
 
         # Mapping from chain ID to the range of sequence numbers that should be
         # pulled from the database.
-        chain_to_gap: Dict[int, Tuple[int, int]] = {}
+        auth_diff_chain_to_gap: Dict[int, Tuple[int, int]] = {}
 
         for chain_id in seen_chains:
             min_seq_no = min(chains.get(chain_id, 0) for chains in set_to_chain)
@@ -629,15 +814,76 @@ class EventFederationWorkerStore(
                 for seq_no in range(min_seq_no + 1, max_seq_no + 1):
                     event_id = chain_to_event.get(chain_id, {}).get(seq_no)
                     if event_id:
-                        result.add(event_id)
+                        auth_difference_result.add(event_id)
                     else:
-                        chain_to_gap[chain_id] = (min_seq_no, max_seq_no)
+                        auth_diff_chain_to_gap[chain_id] = (min_seq_no, max_seq_no)
                         break
 
-        if not chain_to_gap:
-            # If there are no gaps to fetch, we're done!
-            return result
+        conflicted_subgraph_result: Set[str] = set()
+        # Mapping from chain ID to the range of sequence numbers that should be
+        # pulled from the database.
+        conflicted_subgraph_chain_to_gap: Dict[int, Tuple[int, int]] = {}
+        if is_state_res_v21:
+            # also include the conflicted subgraph using backward/forward reachability info from all
+            # the conflicted events. To calculate this, we want to extract the intersection between
+            # the backwards and forwards reachability sets, e.g:
+            #     A <- B <- C <- D <- E
+            #    Assume B and D are conflicted so we want {C} as the conflicted subgraph.
+            #    B_backwards={A}, B_forwards={C,D,E}
+            #    D_backwards={A,B,C} D_forwards={E}
+            #    ALL_backwards={A,B,C} ALL_forwards={C,D,E}
+            #    Intersection(ALL_backwards, ALL_forwards) = {C}
+            #
+            # It's worth noting that once we have the ALL_ sets, we no longer care about the paths.
+            # We're dealing with chains and not singular events, but we've already got the ALL_ sets.
+            # As such, we can inspect each chain in isolation and check for overlapping sequence
+            # numbers:
+            #             1,2,3,4,5 Seq Num
+            #    Chain N [A,B,C,D,E]
+            #
+            # if (N,4) is in the backwards set and (N,2) is in the forwards set, then the
+            # intersection is events between 2 < 4. We will include the conflicted events themselves
+            # in the subgraph, but they will already be, hence the full set of events is {B,C,D}.
+            for chain_id, backwards_seq_num in conflicted_backwards_reachable.items():
+                forwards_seq_num = conflicted_forwards_reachable.get(chain_id)
+                if forwards_seq_num is None:
+                    continue  # this chain isn't in both sets so can't intersect
+                if forwards_seq_num > backwards_seq_num:
+                    continue  # this chain is in both sets but they don't overap
+                for seq_no in range(
+                    forwards_seq_num, backwards_seq_num + 1
+                ):  # inclusive of both
+                    event_id = chain_to_event.get(chain_id, {}).get(seq_no)
+                    if event_id:
+                        conflicted_subgraph_result.add(event_id)
+                    else:
+                        conflicted_subgraph_chain_to_gap[chain_id] = (
+                            # _fetch_event_ids_from_chains_txn is exclusive of the min value
+                            forwards_seq_num - 1,
+                            backwards_seq_num,
+                        )
+                        break
 
+        if auth_diff_chain_to_gap:
+            auth_difference_result.update(
+                self._fetch_event_ids_from_chains_txn(txn, auth_diff_chain_to_gap)
+            )
+        if conflicted_subgraph_chain_to_gap:
+            conflicted_subgraph_result.update(
+                self._fetch_event_ids_from_chains_txn(
+                    txn, conflicted_subgraph_chain_to_gap
+                )
+            )
+
+        return StateDifference(
+            auth_difference=auth_difference_result,
+            conflicted_subgraph=conflicted_subgraph_result,
+        )
+
+    def _fetch_event_ids_from_chains_txn(
+        self, txn: LoggingTransaction, chains: Dict[int, Tuple[int, int]]
+    ) -> Set[str]:
+        result: Set[str] = set()
         if isinstance(self.database_engine, PostgresEngine):
             # We can use `execute_values` to efficiently fetch the gaps when
             # using postgres.
@@ -651,7 +897,7 @@ class EventFederationWorkerStore(
 
             args = [
                 (chain_id, min_no, max_no)
-                for chain_id, (min_no, max_no) in chain_to_gap.items()
+                for chain_id, (min_no, max_no) in chains.items()
             ]
 
             rows = txn.execute_values(sql, args)
@@ -662,10 +908,9 @@ class EventFederationWorkerStore(
                 SELECT event_id FROM event_auth_chains
                 WHERE chain_id = ? AND ? < sequence_number AND sequence_number <= ?
             """
-            for chain_id, (min_no, max_no) in chain_to_gap.items():
+            for chain_id, (min_no, max_no) in chains.items():
                 txn.execute(sql, (chain_id, min_no, max_no))
                 result.update(r for (r,) in txn)
-
         return result
 
     def _fixup_auth_chain_difference_sets(
@@ -2165,6 +2410,7 @@ def _materialize(
     origin_sequence_number: int,
     links: Dict[int, List[Tuple[int, int, int]]],
     materialized: Dict[int, int],
+    backwards: bool = True,
 ) -> None:
     """Helper function for fetching auth chain links. For a given origin chain
     ID / sequence number and a dictionary of links, updates the materialized
@@ -2181,6 +2427,7 @@ def _materialize(
             target sequence number.
         materialized: dict to update with new reachability information, as a
             map from chain ID to max sequence number reachable.
+        backwards: If True, walks backwards down the chains. If False, walks forwards from the chains.
     """
 
     # Do a standard graph traversal.
@@ -2195,12 +2442,104 @@ def _materialize(
             target_chain_id,
             target_sequence_number,
         ) in chain_links:
-            # Ignore any links that are higher up the chain
-            if sequence_number > s:
-                continue
+            if backwards:
+                # Ignore any links that are higher up the chain
+                if sequence_number > s:
+                    continue
 
-            # Check if we have already visited the target chain before, if so we
-            # can skip it.
-            if materialized.get(target_chain_id, 0) < target_sequence_number:
-                stack.append((target_chain_id, target_sequence_number))
-                materialized[target_chain_id] = target_sequence_number
+                # Check if we have already visited the target chain before, if so we
+                # can skip it.
+                if materialized.get(target_chain_id, 0) < target_sequence_number:
+                    stack.append((target_chain_id, target_sequence_number))
+                    materialized[target_chain_id] = target_sequence_number
+            else:
+                # Ignore any links that are lower down the chain.
+                if sequence_number < s:
+                    continue
+                # Check if we have already visited the target chain before, if so we
+                # can skip it.
+                if (
+                    materialized.get(target_chain_id, MAX_CHAIN_LENGTH)
+                    > target_sequence_number
+                ):
+                    stack.append((target_chain_id, target_sequence_number))
+                    materialized[target_chain_id] = target_sequence_number
+
+
+def _generate_forward_links(
+    links: Dict[int, List[Tuple[int, int, int]]],
+) -> Dict[int, List[Tuple[int, int, int]]]:
+    """Reverse the input links from the given backwards links"""
+    new_links: Dict[int, List[Tuple[int, int, int]]] = {}
+    for origin_chain_id, chain_links in links.items():
+        for origin_seq_num, target_chain_id, target_seq_num in chain_links:
+            new_links.setdefault(target_chain_id, []).append(
+                (target_seq_num, origin_chain_id, origin_seq_num)
+            )
+    return new_links
+
+
+def accumulate_forwards_reachable_events(
+    conflicted_forwards_reachable: Dict[int, int],
+    back_links: Dict[int, List[Tuple[int, int, int]]],
+    conflicted_chain_positions: Dict[str, Tuple[int, int]],
+) -> None:
+    """Accumulate new forwards reachable events using the back_links provided.
+
+    Accumulating forwards reachable information is quite different from backwards reachable information
+    because _get_chain_links returns the entire linkage information for backwards reachable events,
+    but not _forwards_ reachable events. We are only interested in the forwards reachable information
+    that is encoded in the backwards reachable links, so we can just invert all the operations we do
+    for backwards reachable events to calculate a subset of forwards reachable information. The
+    caveat with this approach is that it is a _subset_. This means new back_links may encode new
+    forwards reachable information which we also need. Consider this scenario:
+
+    A <-- B <-- C <--- D <-- E <-- F         Chain 1
+                |
+                `----- G <-- H <-- I         Chain 2
+                             |
+                             `---- J <-- K   Chain 3
+
+    Now consider what happens when B is a conflicted event. _get_chain_links returns the conflicted
+    chain and ALL links heading towards the root of the graph. This means we will know the
+    Chain 1 to Chain 2 link via C (as all links for the chain are returned, not strictly ones with
+    a lower sequence number), but we will NOT know the Chain 2 to Chain 3 link via H. We can be
+    blissfully unaware of Chain 3 entirely, if and only if there isn't some other conflicted event
+    on that chain. Consider what happens when K is /also/ conflicted. _get_chain_links will generate
+    two iterations: one for B and one for K. It's important that we re-evaluate the forwards reachable
+    information for B to include Chain 3 when we process the K iteration, hence we are "accumulating"
+    forwards reachability information.
+
+    NB: We don't consider 'additional backwards reachable events' here because they have no effect
+    on forwards reachability calculations, only backwards.
+
+    Args:
+      conflicted_forwards_reachable: The materialised dict of forwards reachable information.
+      The output to this function are stored here.
+      back_links: One iteration of _get_chain_links which encodes backwards reachable information.
+      conflicted_chain_positions: The conflicted events.
+    """
+    # links go backwards but we want them to go forwards as well for v2.1
+    fwd_links = _generate_forward_links(back_links)
+
+    # for each conflicted event, accumulate forwards reachability information
+    for (
+        conflicted_chain_id,
+        conflicted_chain_seq,
+    ) in conflicted_chain_positions.values():
+        # the conflicted event itself encodes reachability information
+        # e.g if D was conflicted, it encodes E,F as forwards reachable.
+        conflicted_forwards_reachable[conflicted_chain_id] = min(
+            conflicted_chain_seq,
+            conflicted_forwards_reachable.get(conflicted_chain_id, MAX_CHAIN_LENGTH),
+        )
+        # Walk from the conflicted event forwards to explore the links.
+        # This function checks if we've visited the chain before and skips reprocessing, so this
+        # does not repeatedly traverse the graph.
+        _materialize(
+            conflicted_chain_id,
+            conflicted_chain_seq,
+            fwd_links,
+            conflicted_forwards_reachable,
+            backwards=False,
+        )

synapse/storage/databases/main/events.py

@@ -51,7 +51,12 @@ from synapse.api.constants import (
 )
 from synapse.api.errors import PartialStateConflictError
 from synapse.api.room_versions import RoomVersions
-from synapse.events import EventBase, StrippedStateEvent, relation_from_event
+from synapse.events import (
+    EventBase,
+    StrippedStateEvent,
+    is_creator,
+    relation_from_event,
+)
 from synapse.events.snapshot import EventContext
 from synapse.events.utils import parse_stripped_state_event
 from synapse.logging.opentracing import trace
@@ -481,17 +486,27 @@ class PersistEventsStore:
         pl_id = state[(EventTypes.PowerLevels, "")]
         pl_event = await self.store.get_event(pl_id, allow_none=True)
 
-        if pl_event is None:
-            # per the spec, if a power level event isn't in the room, grant the creator
-            # level 100 and all other users 0
-            create_id = state[(EventTypes.Create, "")]
-            create_event = await self.store.get_event(create_id, allow_none=True)
-            if create_event is None:
-                # not sure how this would happen but if it does then just deny the redaction
-                logger.warning("No create event found for room %s", event.room_id)
-                return False
-            if create_event.sender == event.sender:
+        create_id = state[(EventTypes.Create, "")]
+        create_event = await self.store.get_event(create_id, allow_none=True)
+
+        if create_event is None:
+            # not sure how this would happen but if it does then just deny the redaction
+            logger.warning("No create event found for room %s", event.room_id)
+            return False
+
+        if create_event.room_version.msc4289_creator_power_enabled:
+            # per the spec, grant the creator infinite power level and all other users 0
+            if is_creator(create_event, event.sender):
                 return True
+            if pl_event is None:
+                # per the spec, users other than the room creator have power level
+                # 0, which is less than the default to redact events (50).
+                return False
+        else:
+            # per the spec, if a power level event isn't in the room, grant the creator
+            # level 100 (the default redaction level is 50) and all other users 0
+            if pl_event is None:
+                return create_event.sender == event.sender
 
         assert pl_event is not None
         sender_level = pl_event.content.get("users", {}).get(event.sender)

synapse/storage/databases/main/push_rule.py

@@ -110,6 +110,7 @@ def _load_rules(
         msc3381_polls_enabled=experimental_config.msc3381_polls_enabled,
         msc4028_push_encrypted_events=experimental_config.msc4028_push_encrypted_events,
         msc4210_enabled=experimental_config.msc4210_enabled,
+        msc4306_enabled=experimental_config.msc4306_enabled,
     )
 
     return filtered_rules

synapse/storage/databases/main/registration.py

@@ -674,7 +674,7 @@ class RegistrationWorkerStore(StatsStore, CacheInvalidationWorkerStore):
         )
 
     @cached(max_entries=100000)
-    async def is_server_admin(self, user: UserID) -> bool:
+    async def is_server_admin(self, user: str) -> bool:
         """Determines if a user is an admin of this homeserver.
 
         Args:
@@ -685,7 +685,7 @@ class RegistrationWorkerStore(StatsStore, CacheInvalidationWorkerStore):
         """
         res = await self.db_pool.simple_select_one_onecol(
             table="users",
-            keyvalues={"name": user.to_string()},
+            keyvalues={"name": user},
             retcol="admin",
             allow_none=True,
             desc="is_server_admin",