add suppport for persisting batched events over replication

add fucntions to persist events as a batch, encapsulate some logic in a helper function
add some comments and sned events as a batch to be persisted
2025-12-09 01:30:18 +00:00 · 2022-09-19 11:49:33 -07:00 · 2022-09-19 11:48:59 -07:00 · 2022-09-19 11:48:35 -07:00 · 2022-09-19 10:43:03 -07:00 · 2022-09-19 10:11:15 -07:00
172 changed files with 2973 additions and 721 deletions
--- a/.ci/scripts/postgres_exec.py
+++ b/.ci/scripts/postgres_exec.py
@@ -1,31 +0,0 @@
-#!/usr/bin/env python
-# Copyright 2019 The Matrix.org Foundation C.I.C.
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-import sys
-
-import psycopg2
-
-# a very simple replacment for `psql`, to make up for the lack of the postgres client
-# libraries in the synapse docker image.
-
-# We use "postgres" as a database because it's bound to exist and the "synapse" one
-# doesn't exist yet.
-db_conn = psycopg2.connect(
-    user="postgres", host="localhost", password="postgres", dbname="postgres"
-)
-db_conn.autocommit = True
-cur = db_conn.cursor()
-for c in sys.argv[1:]:
-    cur.execute(c)
--- a/.ci/scripts/test_export_data_command.sh
+++ b/.ci/scripts/test_export_data_command.sh
@@ -32,7 +32,7 @@ else
 fi

 # Create the PostgreSQL database.
-poetry run .ci/scripts/postgres_exec.py "CREATE DATABASE synapse"
+psql -c "CREATE DATABASE synapse"

 # Port the SQLite databse to postgres so we can check command works against postgres
 echo "+++ Port SQLite3 databse to postgres"
--- a/.ci/scripts/test_synapse_port_db.sh
+++ b/.ci/scripts/test_synapse_port_db.sh
@@ -2,27 +2,27 @@
 #
 # Test script for 'synapse_port_db'.
 #   - configures synapse and a postgres server.
-#   - runs the port script on a prepopulated test sqlite db
-#   - also runs it against an new sqlite db
+#   - runs the port script on a prepopulated test sqlite db. Checks that the
+#     return code is zero.
+#   - reruns the port script on the same sqlite db, targetting the same postgres db.
+#     Checks that the return code is zero.
+#   - runs the port script against a new sqlite db. Checks the return code is zero.
 #
 # Expects Synapse to have been already installed with `poetry install --extras postgres`.
 # Expects `poetry` to be available on the `PATH`.

-set -xe
+set -xe -o pipefail
 cd "$(dirname "$0")/../.."

 echo "--- Generate the signing key"
-
-# Generate the server's signing key.
 poetry run synapse_homeserver --generate-keys -c .ci/sqlite-config.yaml

 echo "--- Prepare test database"
-
-# Make sure the SQLite3 database is using the latest schema and has no pending background update.
+# Make sure the SQLite3 database is using the latest schema and has no pending background updates.
 poetry run update_synapse_database --database-config .ci/sqlite-config.yaml --run-background-updates

 # Create the PostgreSQL database.
-poetry run .ci/scripts/postgres_exec.py "CREATE DATABASE synapse"
+psql -c "CREATE DATABASE synapse"

 echo "+++ Run synapse_port_db against test database"
 # TODO: this invocation of synapse_port_db (and others below) used to be prepended with `coverage run`,
@@ -45,9 +45,23 @@ rm .ci/test_db.db
 poetry run update_synapse_database --database-config .ci/sqlite-config.yaml --run-background-updates

 # re-create the PostgreSQL database.
-poetry run .ci/scripts/postgres_exec.py \
-  "DROP DATABASE synapse" \
-  "CREATE DATABASE synapse"
+psql \
+  -c "DROP DATABASE synapse" \
+  -c "CREATE DATABASE synapse"

 echo "+++ Run synapse_port_db against empty database"
 poetry run synapse_port_db --sqlite-database .ci/test_db.db --postgres-config .ci/postgres-config.yaml
+
+echo "--- Create a brand new postgres database from schema"
+cp .ci/postgres-config.yaml .ci/postgres-config-unported.yaml
+sed -i -e 's/database: synapse/database: synapse_unported/' .ci/postgres-config-unported.yaml
+psql -c "CREATE DATABASE synapse_unported"
+poetry run update_synapse_database --database-config .ci/postgres-config-unported.yaml --run-background-updates
+
+echo "+++ Comparing ported schema with unported schema"
+# Ignore the tables that portdb creates. (Should it tidy them up when the porting is completed?)
+psql synapse -c "DROP TABLE port_from_sqlite3;"
+pg_dump --format=plain --schema-only --no-tablespaces --no-acl --no-owner synapse_unported > unported.sql
+pg_dump --format=plain --schema-only --no-tablespaces --no-acl --no-owner synapse          >   ported.sql
+# By default, `diff` returns zero if there are no changes and nonzero otherwise
+diff -u unported.sql ported.sql | tee schema_diff
--- a/.dockerignore
+++ b/.dockerignore
@@ -11,5 +11,6 @@
 !build_rust.py

 rust/target
+synapse/*.so

 **/__pycache__
--- a/.github/workflows/tests.yml
+++ b/.github/workflows/tests.yml
@@ -32,9 +32,11 @@ jobs:
    steps:
      - uses: actions/checkout@v2
      - uses: actions/setup-python@v2
-      - run: pip install .
-      - run: scripts-dev/generate_sample_config.sh --check
-      - run: scripts-dev/config-lint.sh
+      - uses: matrix-org/setup-python-poetry@v1
+        with:
+          extras: "all"
+      - run: poetry run scripts-dev/generate_sample_config.sh --check
+      - run: poetry run scripts-dev/config-lint.sh

  check-schema-delta:
    runs-on: ubuntu-latest
@@ -76,7 +78,6 @@ jobs:
      - uses: actions/checkout@v2
        with:
          ref: ${{ github.event.pull_request.head.sha }}
-          fetch-depth: 0
      - uses: matrix-org/setup-python-poetry@v1
        with:
          extras: "all"
@@ -361,18 +362,22 @@ jobs:

    steps:
      - uses: actions/checkout@v2
-      - run: sudo apt-get -qq install xmlsec1
+      - run: sudo apt-get -qq install xmlsec1 postgresql-client
      - uses: matrix-org/setup-python-poetry@v1
        with:
          extras: "postgres"
      - run: .ci/scripts/test_export_data_command.sh
+        env:
+          PGHOST: localhost
+          PGUSER: postgres
+          PGPASSWORD: postgres
+          PGDATABASE: postgres
+

  portdb:
    if: ${{ !failure() && !cancelled() }} # Allow previous steps to be skipped, but not fail
    needs: linting-done
    runs-on: ubuntu-latest
-    env:
-      TOP: ${{ github.workspace }}
    strategy:
      matrix:
        include:
@@ -398,12 +403,27 @@ jobs:

    steps:
      - uses: actions/checkout@v2
-      - run: sudo apt-get -qq install xmlsec1
+      - run: sudo apt-get -qq install xmlsec1 postgresql-client
      - uses: matrix-org/setup-python-poetry@v1
        with:
          python-version: ${{ matrix.python-version }}
          extras: "postgres"
      - run: .ci/scripts/test_synapse_port_db.sh
+        id: run_tester_script
+        env:
+          PGHOST: localhost
+          PGUSER: postgres
+          PGPASSWORD: postgres
+          PGDATABASE: postgres
+      - name: "Upload schema differences"
+        uses: actions/upload-artifact@v3
+        if: ${{ failure() && !cancelled() && steps.run_tester_script.outcome == 'failure' }}
+        with:
+          name: Schema dumps
+          path: |
+            unported.sql
+            ported.sql
+            schema_diff

  complement:
    if: "${{ !failure() && !cancelled() }}"
@@ -468,10 +488,6 @@ jobs:
  tests-done:
    if: ${{ always() }}
    needs:
-      - check-sampleconfig
-      - lint
-      - lint-crlf
-      - lint-newsfile
      - trial
      - trial-olddeps
      - sytest
@@ -486,5 +502,7 @@ jobs:
          needs: ${{ toJSON(needs) }}

          # The newsfile lint may be skipped on non PR builds
-          skippable:
+          # Cargo test is skipped if there is no changes on Rust code
+          skippable: |
            lint-newsfile
+            cargo-test
--- a/CHANGES.md
+++ b/CHANGES.md
@@ -1,3 +1,111 @@
+Synapse 1.67.0 (2022-09-13)
+===========================
+
+This release removes using the deprecated direct TCP replication configuration
+for workers. Server admins should use Redis instead. See the [upgrade
+notes](https://matrix-org.github.io/synapse/v1.67/upgrade.html#upgrading-to-v1670).
+
+The minimum version of `poetry` supported for managing source checkouts is now
+1.2.0.
+
+**Notice:** from the next major release (1.68.0) installing Synapse from a source
+checkout will require a recent Rust compiler. Those using packages or
+`pip install matrix-synapse` will not be affected. See the [upgrade
+notes](https://matrix-org.github.io/synapse/v1.67/upgrade.html#upgrading-to-v1670).
+
+**Notice:** from the next major release (1.68.0), running Synapse with a SQLite
+database will require SQLite version 3.27.0 or higher. (The [current minimum
+ version is SQLite 3.22.0](https://github.com/matrix-org/synapse/blob/release-v1.67/synapse/storage/engines/sqlite.py#L69-L78).)
+See [#12983](https://github.com/matrix-org/synapse/issues/12983) and the [upgrade notes](https://matrix-org.github.io/synapse/v1.67/upgrade.html#upgrading-to-v1670) for more details.
+
+
+No significant changes since 1.67.0rc1.
+
+
+Synapse 1.67.0rc1 (2022-09-06)
+==============================
+
+Features
+--------
+
+- Support setting the registration shared secret in a file, via a new `registration_shared_secret_path` configuration option. ([\#13614](https://github.com/matrix-org/synapse/issues/13614))
+- Change the default startup behaviour so that any missing "additional" configuration files (signing key, etc) are generated automatically. ([\#13615](https://github.com/matrix-org/synapse/issues/13615))
+- Improve performance of sending messages in rooms with thousands of local users. ([\#13634](https://github.com/matrix-org/synapse/issues/13634))
+
+
+Bugfixes
+--------
+
+- Fix a bug introduced in Synapse 1.13 where the [List Rooms admin API](https://matrix-org.github.io/synapse/develop/admin_api/rooms.html#list-room-api) would return integers instead of booleans for the `federatable` and `public` fields when using a Sqlite database. ([\#13509](https://github.com/matrix-org/synapse/issues/13509))
+- Fix bug that user cannot `/forget` rooms after the last member has left the room. ([\#13546](https://github.com/matrix-org/synapse/issues/13546))
+- Faster Room Joins: fix `/make_knock` blocking indefinitely when the room in question is a partial-stated room. ([\#13583](https://github.com/matrix-org/synapse/issues/13583))
+- Fix loading the current stream position behind the actual position. ([\#13585](https://github.com/matrix-org/synapse/issues/13585))
+- Fix a longstanding bug in `register_new_matrix_user` which meant it was always necessary to explicitly give a server URL. ([\#13616](https://github.com/matrix-org/synapse/issues/13616))
+- Fix the running of [MSC1763](https://github.com/matrix-org/matrix-spec-proposals/pull/1763) retention purge_jobs in deployments with background jobs running on a worker by forcing them back onto the main worker. Contributed by Brad @ Beeper. ([\#13632](https://github.com/matrix-org/synapse/issues/13632))
+- Fix a long-standing bug that downloaded media for URL previews was not deleted while database background updates were running. ([\#13657](https://github.com/matrix-org/synapse/issues/13657))
+- Fix [MSC3030](https://github.com/matrix-org/matrix-spec-proposals/pull/3030) `/timestamp_to_event` endpoint to return the correct next event when the events have the same timestamp. ([\#13658](https://github.com/matrix-org/synapse/issues/13658))
+- Fix bug where we wedge media plugins if clients disconnect early. Introduced in v1.22.0. ([\#13660](https://github.com/matrix-org/synapse/issues/13660))
+- Fix a long-standing bug which meant that keys for unwhitelisted servers were not returned by `/_matrix/key/v2/query`. ([\#13683](https://github.com/matrix-org/synapse/issues/13683))
+- Fix a bug introduced in Synapse v1.20.0 that would cause the unstable unread counts from [MSC2654](https://github.com/matrix-org/matrix-spec-proposals/pull/2654) to be calculated even if the feature is disabled. ([\#13694](https://github.com/matrix-org/synapse/issues/13694))
+
+
+Updates to the Docker image
+---------------------------
+
+- Update docker image to use a stable version of poetry. ([\#13688](https://github.com/matrix-org/synapse/issues/13688))
+
+
+Improved Documentation
+----------------------
+
+- Improve the description of the ["chain cover index"](https://matrix-org.github.io/synapse/latest/auth_chain_difference_algorithm.html) used internally by Synapse. ([\#13602](https://github.com/matrix-org/synapse/issues/13602))
+- Document how ["monthly active users"](https://matrix-org.github.io/synapse/latest/usage/administration/monthly_active_users.html) is calculated and used. ([\#13617](https://github.com/matrix-org/synapse/issues/13617))
+- Improve documentation around user registration. ([\#13640](https://github.com/matrix-org/synapse/issues/13640))
+- Remove documentation of legacy `frontend_proxy` worker app. ([\#13645](https://github.com/matrix-org/synapse/issues/13645))
+- Clarify documentation that HTTP replication traffic can be protected with a shared secret. ([\#13656](https://github.com/matrix-org/synapse/issues/13656))
+- Remove unintentional colons from [config manual](https://matrix-org.github.io/synapse/latest/usage/configuration/config_documentation.html) headers. ([\#13665](https://github.com/matrix-org/synapse/issues/13665))
+- Update docs to make enabling metrics more clear. ([\#13678](https://github.com/matrix-org/synapse/issues/13678))
+- Clarify `(room_id, event_id)` global uniqueness and how we should scope our database schemas. ([\#13701](https://github.com/matrix-org/synapse/issues/13701))
+
+
+Deprecations and Removals
+-------------------------
+
+- Drop support for calling `/_matrix/client/v3/rooms/{roomId}/invite` without an `id_access_token`, which was not permitted by the spec. Contributed by @Vetchu. ([\#13241](https://github.com/matrix-org/synapse/issues/13241))
+- Remove redundant `_get_joined_users_from_context` cache. Contributed by Nick @ Beeper (@fizzadar). ([\#13569](https://github.com/matrix-org/synapse/issues/13569))
+- Remove the ability to use direct TCP replication with workers. Direct TCP replication was deprecated in Synapse v1.18.0. Workers now require using Redis. ([\#13647](https://github.com/matrix-org/synapse/issues/13647))
+- Remove support for unstable [private read receipts](https://github.com/matrix-org/matrix-spec-proposals/pull/2285). ([\#13653](https://github.com/matrix-org/synapse/issues/13653), [\#13692](https://github.com/matrix-org/synapse/issues/13692))
+
+
+Internal Changes
+----------------
+
+- Extend the release script to wait for GitHub Actions to finish and to be usable as a guide for the whole process. ([\#13483](https://github.com/matrix-org/synapse/issues/13483))
+- Add experimental configuration option to allow disabling legacy Prometheus metric names. ([\#13540](https://github.com/matrix-org/synapse/issues/13540))
+- Cache user IDs instead of profiles to reduce cache memory usage. Contributed by Nick @ Beeper (@fizzadar). ([\#13573](https://github.com/matrix-org/synapse/issues/13573), [\#13600](https://github.com/matrix-org/synapse/issues/13600))
+- Optimize how Synapse calculates domains to fetch from during backfill. ([\#13575](https://github.com/matrix-org/synapse/issues/13575))
+- Comment about a better future where we can get the state diff between two events. ([\#13586](https://github.com/matrix-org/synapse/issues/13586))
+- Instrument `_check_sigs_and_hash_and_fetch` to trace time spent in child concurrent calls for understandable traces in Jaeger. ([\#13588](https://github.com/matrix-org/synapse/issues/13588))
+- Improve performance of `@cachedList`. ([\#13591](https://github.com/matrix-org/synapse/issues/13591))
+- Minor speed up of fetching large numbers of push rules. ([\#13592](https://github.com/matrix-org/synapse/issues/13592))
+- Optimise push action fetching queries. Contributed by Nick @ Beeper (@fizzadar). ([\#13597](https://github.com/matrix-org/synapse/issues/13597))
+- Rename `event_map` to `unpersisted_events` when computing the auth differences. ([\#13603](https://github.com/matrix-org/synapse/issues/13603))
+- Refactor `get_users_in_room(room_id)` mis-use with dedicated `get_current_hosts_in_room(room_id)` function. ([\#13605](https://github.com/matrix-org/synapse/issues/13605))
+- Use dedicated `get_local_users_in_room(room_id)` function to find local users when calculating `join_authorised_via_users_server` of a `/make_join` request. ([\#13606](https://github.com/matrix-org/synapse/issues/13606))
+- Refactor `get_users_in_room(room_id)` mis-use to lookup single local user with dedicated `check_local_user_in_room(...)` function. ([\#13608](https://github.com/matrix-org/synapse/issues/13608))
+- Drop unused column `application_services_state.last_txn`. ([\#13627](https://github.com/matrix-org/synapse/issues/13627))
+- Improve readability of Complement CI logs by printing failure results last. ([\#13639](https://github.com/matrix-org/synapse/issues/13639))
+- Generalise the `@cancellable` annotation so it can be used on functions other than just servlet methods. ([\#13662](https://github.com/matrix-org/synapse/issues/13662))
+- Introduce a `CommonUsageMetrics` class to share some usage metrics between the Prometheus exporter and the phone home stats. ([\#13671](https://github.com/matrix-org/synapse/issues/13671))
+- Add some logging to help track down #13444. ([\#13679](https://github.com/matrix-org/synapse/issues/13679))
+- Update poetry lock file for v1.2.0. ([\#13689](https://github.com/matrix-org/synapse/issues/13689))
+- Add cache to `is_partial_state_room`. ([\#13693](https://github.com/matrix-org/synapse/issues/13693))
+- Update the Grafana dashboard that is included with Synapse in the `contrib` directory. ([\#13697](https://github.com/matrix-org/synapse/issues/13697))
+- Only run trial CI on all python versions on non-PRs. ([\#13698](https://github.com/matrix-org/synapse/issues/13698))
+- Fix typechecking with latest types-jsonschema. ([\#13712](https://github.com/matrix-org/synapse/issues/13712))
+- Reduce number of CI checks we run for PRs. ([\#13713](https://github.com/matrix-org/synapse/issues/13713))
+
+
 Synapse 1.66.0 (2022-08-31)
 ===========================

--- a/changelog.d/13162.misc
+++ b/changelog.d/13162.misc
@@ -0,0 +1 @@
+Bump the minimum dependency of `matrix_common` to 1.3.0 to make use of the `MXCUri` class. Use `MXCUri` to simplify media retention test code.
--- a/changelog.d/13241.removal
+++ b/changelog.d/13241.removal
@@ -1 +0,0 @@
-Drop support for calling `/_matrix/client/v3/rooms/{roomId}/invite` without an `id_access_token`, which was not permitted by the spec. Contributed by @Vetchu.
--- a/changelog.d/13480.doc
+++ b/changelog.d/13480.doc
@@ -0,0 +1 @@
+Note that `libpq` is required on ARM-based Macs.
--- a/changelog.d/13483.misc
+++ b/changelog.d/13483.misc
@@ -1 +0,0 @@
-Extend the release script to wait for GitHub Actions to finish and to be usable as a guide for the whole process.
--- a/changelog.d/13487.misc
+++ b/changelog.d/13487.misc
@@ -0,0 +1 @@
+Refactor ` _send_events_for_new_room` to separate creating and sending events.
--- a/changelog.d/13509.bugfix
+++ b/changelog.d/13509.bugfix
@@ -1 +0,0 @@
-Fix a bug introduced in Synapse 1.13 where the [List Rooms admin API](https://matrix-org.github.io/synapse/develop/admin_api/rooms.html#list-room-api) would return integers instead of booleans for the `federatable` and `public` fields when using a Sqlite database.
--- a/changelog.d/13540.misc
+++ b/changelog.d/13540.misc
@@ -1 +0,0 @@
-Add experimental configuration option to allow disabling legacy Prometheus metric names.
--- a/changelog.d/13546.bugfix
+++ b/changelog.d/13546.bugfix
@@ -1 +0,0 @@
-Fix bug that user cannot `/forget` rooms after the last member has left the room.
--- a/changelog.d/13569.removal
+++ b/changelog.d/13569.removal
@@ -1 +0,0 @@
-Remove redundant `_get_joined_users_from_context` cache. Contributed by Nick @ Beeper (@fizzadar).
--- a/changelog.d/13573.misc
+++ b/changelog.d/13573.misc
@@ -1 +0,0 @@
-Cache user IDs instead of profiles to reduce cache memory usage. Contributed by Nick @ Beeper (@fizzadar).
--- a/changelog.d/13575.misc
+++ b/changelog.d/13575.misc
@@ -1 +0,0 @@
-Optimize how Synapse calculates domains to fetch from during backfill.
--- a/changelog.d/13583.bugfix
+++ b/changelog.d/13583.bugfix
@@ -1 +0,0 @@
-Faster Room Joins: fix `/make_knock` blocking indefinitely when the room in question is a partial-stated room.
--- a/changelog.d/13585.bugfix
+++ b/changelog.d/13585.bugfix
@@ -1 +0,0 @@
-Fix loading the current stream position behind the actual position.
--- a/changelog.d/13586.misc
+++ b/changelog.d/13586.misc
@@ -1 +0,0 @@
-Comment about a better future where we can get the state diff between two events.
--- a/changelog.d/13588.misc
+++ b/changelog.d/13588.misc
@@ -1 +0,0 @@
-Instrument `_check_sigs_and_hash_and_fetch` to trace time spent in child concurrent calls for understandable traces in Jaeger.
--- a/changelog.d/13589.feature
+++ b/changelog.d/13589.feature
@@ -0,0 +1 @@
+Keep track when we fail to process a pulled event over federation so we can intelligently back-off in the future.
--- a/changelog.d/13591.misc
+++ b/changelog.d/13591.misc
@@ -1 +0,0 @@
-Improve performance of `@cachedList`.
--- a/changelog.d/13592.misc
+++ b/changelog.d/13592.misc
@@ -1 +0,0 @@
-Minor speed up of fetching large numbers of push rules.
--- a/changelog.d/13597.misc
+++ b/changelog.d/13597.misc
@@ -1 +0,0 @@
- Optimise push action fetching queries. Contributed by Nick @ Beeper (@fizzadar).
--- a/changelog.d/13600.misc
+++ b/changelog.d/13600.misc
@@ -1 +0,0 @@
-Cache user IDs instead of profiles to reduce cache memory usage. Contributed by Nick @ Beeper (@fizzadar).
--- a/changelog.d/13602.doc
+++ b/changelog.d/13602.doc
@@ -1 +0,0 @@
-Improve the description of the ["chain cover index"](https://matrix-org.github.io/synapse/latest/auth_chain_difference_algorithm.html) used internally by Synapse.
--- a/changelog.d/13603.misc
+++ b/changelog.d/13603.misc
@@ -1 +0,0 @@
-Rename `event_map` to `unpersisted_events` when computing the auth differences.
--- a/changelog.d/13605.misc
+++ b/changelog.d/13605.misc
@@ -1 +0,0 @@
-Refactor `get_users_in_room(room_id)` mis-use with dedicated `get_current_hosts_in_room(room_id)` function.
--- a/changelog.d/13606.misc
+++ b/changelog.d/13606.misc
@@ -1 +0,0 @@
-Use dedicated `get_local_users_in_room(room_id)` function to find local users when calculating `join_authorised_via_users_server` of a `/make_join` request.
--- a/changelog.d/13608.misc
+++ b/changelog.d/13608.misc
@@ -1 +0,0 @@
-Refactor `get_users_in_room(room_id)` mis-use to lookup single local user with dedicated `check_local_user_in_room(...)` function.
--- a/changelog.d/13614.feature
+++ b/changelog.d/13614.feature
@@ -1 +0,0 @@
-Support setting the registration shared secret in a file, via a new `registration_shared_secret_path` configuration option.
--- a/changelog.d/13615.feature
+++ b/changelog.d/13615.feature
@@ -1 +0,0 @@
-Change the default startup behaviour so that any missing "additional" configuration files (signing key, etc) are generated automatically.
--- a/changelog.d/13616.bugfix
+++ b/changelog.d/13616.bugfix
@@ -1 +0,0 @@
-Fix a longstanding bug in `register_new_matrix_user` which meant it was always necessary to explicitly give a server URL.
--- a/changelog.d/13617.doc
+++ b/changelog.d/13617.doc
@@ -1 +0,0 @@
-Document how ["monthly active users"](https://matrix-org.github.io/synapse/latest/usage/administration/monthly_active_users.html) is calculated and used.
--- a/changelog.d/13627.misc
+++ b/changelog.d/13627.misc
@@ -1 +0,0 @@
-Drop unused column `application_services_state.last_txn`.
--- a/changelog.d/13632.bugfix
+++ b/changelog.d/13632.bugfix
@@ -1 +0,0 @@
-Fix the running of MSC1763 retention purge_jobs in deployments with background jobs running on a worker by forcing them back onto the main worker. Contributed by Brad @ Beeper.
--- a/changelog.d/13634.feature
+++ b/changelog.d/13634.feature
@@ -1 +0,0 @@
-Improve performance of sending messages in rooms with thousands of local users.
--- a/changelog.d/13639.misc
+++ b/changelog.d/13639.misc
@@ -1 +0,0 @@
-Improve readability of Complement CI logs by printing failure results last.
--- a/changelog.d/13640.doc
+++ b/changelog.d/13640.doc
@@ -1 +0,0 @@
-Improve documentation around user registration.
--- a/changelog.d/13645.doc
+++ b/changelog.d/13645.doc
@@ -1 +0,0 @@
-Remove documentation of legacy `frontend_proxy` worker app.
--- a/changelog.d/13647.removal
+++ b/changelog.d/13647.removal
@@ -1 +0,0 @@
-Remove the ability to use direct TCP replication with workers. Direct TCP replication was deprecated in Synapse v1.18.0. Workers now require using Redis.
--- a/changelog.d/13653.removal
+++ b/changelog.d/13653.removal
@@ -1 +0,0 @@
-Remove support for unstable [private read receipts](https://github.com/matrix-org/matrix-spec-proposals/pull/2285).
--- a/changelog.d/13656.doc
+++ b/changelog.d/13656.doc
@@ -1 +0,0 @@
-Clarify documentation that HTTP replication traffic can be protected with a shared secret.
--- a/changelog.d/13657.bugfix
+++ b/changelog.d/13657.bugfix
@@ -1 +0,0 @@
-Fix a long-standing bug that downloaded media for URL previews was not deleted while database background updates were running.
--- a/changelog.d/13658.bugfix
+++ b/changelog.d/13658.bugfix
@@ -1 +0,0 @@
-Fix MSC3030 `/timestamp_to_event` endpoint to return the correct next event when the events have the same timestamp.
--- a/changelog.d/13660.bugfix
+++ b/changelog.d/13660.bugfix
@@ -1 +0,0 @@
-Fix bug where we wedge media plugins if clients disconnect early. Introduced in v1.22.0.
--- a/changelog.d/13662.misc
+++ b/changelog.d/13662.misc
@@ -1 +0,0 @@
-Generalise the `@cancellable` annotation so it can be used on functions other than just servlet methods.
--- a/changelog.d/13665.doc
+++ b/changelog.d/13665.doc
@@ -1 +0,0 @@
-Remove unintentional colons from [config manual](https://matrix-org.github.io/synapse/latest/usage/configuration/config_documentation.html) headers.
--- a/changelog.d/13671.misc
+++ b/changelog.d/13671.misc
@@ -1 +0,0 @@
-Introduce a `CommonUsageMetrics` class to share some usage metrics between the Prometheus exporter and the phone home stats.
--- a/changelog.d/13678.doc
+++ b/changelog.d/13678.doc
@@ -1 +0,0 @@
-Update docs to make enabling metrics more clear.
--- a/changelog.d/13679.misc
+++ b/changelog.d/13679.misc
@@ -1 +0,0 @@
-Add some logging to help track down #13444.
--- a/changelog.d/13683.bugfix
+++ b/changelog.d/13683.bugfix
@@ -1 +0,0 @@
-Fix a long-standing bug which meant that keys for unwhitelisted servers were not returned by `/_matrix/key/v2/query`.
--- a/changelog.d/13688.docker
+++ b/changelog.d/13688.docker
@@ -1 +0,0 @@
-Update docker image to use a stable version of poetry.
--- a/changelog.d/13689.misc
+++ b/changelog.d/13689.misc
@@ -1 +0,0 @@
-Update poetry lock file for v1.2.0.
--- a/changelog.d/13692.removal
+++ b/changelog.d/13692.removal
@@ -1 +0,0 @@
-Remove support for unstable [private read receipts](https://github.com/matrix-org/matrix-spec-proposals/pull/2285).
--- a/changelog.d/13693.misc
+++ b/changelog.d/13693.misc
@@ -1 +0,0 @@
-Add cache to `is_partial_state_room`.
--- a/changelog.d/13694.bugfix
+++ b/changelog.d/13694.bugfix
@@ -1 +0,0 @@
-Fix a bug introduced in Synapse v1.20.0 that would cause the unstable unread counts from [MSC2654](https://github.com/matrix-org/matrix-spec-proposals/pull/2654) to be calculated even if the feature is disabled.
--- a/changelog.d/13697.misc
+++ b/changelog.d/13697.misc
@@ -1 +0,0 @@
-Update the Grafana dashboard that is included with Synapse in the `contrib` directory.
--- a/changelog.d/13698.misc
+++ b/changelog.d/13698.misc
@@ -1 +0,0 @@
-Only run trial CI on all python versions on non-PRs.
--- a/changelog.d/13701.doc
+++ b/changelog.d/13701.doc
@@ -1 +0,0 @@
-Clarify `(room_id, event_id)` global uniqueness and how we should scope our database schemas.
--- a/changelog.d/13703.misc
+++ b/changelog.d/13703.misc
@@ -0,0 +1 @@
+Add & populate `event_stream_ordering` column on receipts table for future optimisation of push action processing. Contributed by Nick @ Beeper (@fizzadar).
--- a/changelog.d/13712.misc
+++ b/changelog.d/13712.misc
@@ -1 +0,0 @@
-Fix typechecking with latest types-jsonschema.
--- a/changelog.d/13713.misc
+++ b/changelog.d/13713.misc
@@ -1 +0,0 @@
-Reduce number of CI checks we run for PRs.
--- a/changelog.d/13723.bugfix
+++ b/changelog.d/13723.bugfix
@@ -0,0 +1 @@
+Fix a long-standing bug where previously rejected events could end up in room state because they pass auth checks given the current state of the room.
--- a/changelog.d/13727.doc
+++ b/changelog.d/13727.doc
@@ -0,0 +1 @@
+Fix a typo in the documentation for the login ratelimiting configuration.
--- a/changelog.d/13729.misc
+++ b/changelog.d/13729.misc
@@ -0,0 +1 @@
+Strip number suffix from instance name to consolidate services that traces are spread over.
--- a/changelog.d/13736.feature
+++ b/changelog.d/13736.feature
@@ -0,0 +1 @@
+Improve validation of request bodies for the following client-server API endpoints: [`/account/3pid/add`](https://spec.matrix.org/v1.3/client-server-api/#post_matrixclientv3account3pidadd), [`/account/3pid/bind`](https://spec.matrix.org/v1.3/client-server-api/#post_matrixclientv3account3pidbind), [`/account/3pid/delete`](https://spec.matrix.org/v1.3/client-server-api/#post_matrixclientv3account3piddelete) and [`/account/3pid/unbind`](https://spec.matrix.org/v1.3/client-server-api/#post_matrixclientv3account3pidunbind).
--- a/changelog.d/13745.misc
+++ b/changelog.d/13745.misc
@@ -0,0 +1 @@
+Remove old queries to join room memberships to current state events. Contributed by Nick @ Beeper (@fizzadar).
--- a/changelog.d/13749.bugfix
+++ b/changelog.d/13749.bugfix
@@ -0,0 +1 @@
+Fix a long standing bug where device lists would remain cached when remote users left and rejoined the last room shared with the local homeserver.
--- a/changelog.d/13753.misc
+++ b/changelog.d/13753.misc
@@ -0,0 +1 @@
+Prepatory work for storing thread IDs for notifications and receipts.
--- a/changelog.d/13759.misc
+++ b/changelog.d/13759.misc
@@ -0,0 +1 @@
+Add a check for editable installs if the Rust library needs rebuilding.
--- a/changelog.d/13761.misc
+++ b/changelog.d/13761.misc
@@ -0,0 +1 @@
+Tag traces with the instance name to be able to easily jump into the right logs and filter traces by instance.
--- a/changelog.d/13765.misc
+++ b/changelog.d/13765.misc
@@ -0,0 +1 @@
+Concurrently fetch room push actions when calculating badge counts. Contributed by Nick @ Beeper (@fizzadar).
--- a/changelog.d/13766.bugfix
+++ b/changelog.d/13766.bugfix
@@ -0,0 +1 @@
+Fix a long-standing bug where the `cache_invalidation_stream_seq` sequence would begin at 1 instead of 2.
--- a/changelog.d/13769.misc
+++ b/changelog.d/13769.misc
@@ -0,0 +1 @@
+Add a stub Rust crate.
--- a/changelog.d/13770.misc
+++ b/changelog.d/13770.misc
@@ -0,0 +1 @@
+Update the script which makes full schema dumps.
--- a/changelog.d/13778.misc
+++ b/changelog.d/13778.misc
@@ -0,0 +1 @@
+Add a stub Rust crate.
--- a/changelog.d/13780.misc
+++ b/changelog.d/13780.misc
@@ -0,0 +1 @@
+Deduplicate `is_server_notices_room`.
--- a/changelog.d/13784.misc
+++ b/changelog.d/13784.misc
@@ -0,0 +1 @@
+Simplify the dependency DAG in the tests workflow.
--- a/changelog.d/13785.doc
+++ b/changelog.d/13785.doc
@@ -0,0 +1 @@
+Add docs for common fix of deleting the `matrix_synapse.egg-info/` directory for fixing Python dependency problems.
--- a/changelog.d/13788.misc
+++ b/changelog.d/13788.misc
@@ -0,0 +1 @@
+Remove an old, incorrect migration file.
--- a/changelog.d/13789.bugfix
+++ b/changelog.d/13789.bugfix
@@ -0,0 +1 @@
+Fix a long-standing spec compliance bug where Synapse would accept a trailing slash on the end of `/get_missing_events` federation requests.
--- a/changelog.d/13794.doc
+++ b/changelog.d/13794.doc
@@ -0,0 +1 @@
+Update request log format documentation to mention the format used when the authenticated user is controlling another user.
--- a/changelog.d/13795.misc
+++ b/changelog.d/13795.misc
@@ -0,0 +1 @@
+Remove unused method in `synapse.api.auth.Auth`.
--- a/changelog.d/13798.misc
+++ b/changelog.d/13798.misc
@@ -0,0 +1 @@
+Fix a memory leak when running the unit tests.
--- a/changelog.d/13801.feature
+++ b/changelog.d/13801.feature
@@ -0,0 +1 @@
+Add `listeners[x].request_id_header` config to specify which request header to extract and use as the request ID in order to correlate requests from a reverse-proxy.
--- a/changelog.d/13802.misc
+++ b/changelog.d/13802.misc
@@ -0,0 +1 @@
+Use partial indices on SQLite.
--- a/changelog.d/13808.misc
+++ b/changelog.d/13808.misc
@@ -0,0 +1 @@
+Check that portdb generates the same postgres schema as that in the source tree.
--- a/changelog.d/13810.feature
+++ b/changelog.d/13810.feature
@@ -0,0 +1 @@
+Add an admin API endpoint to find a user based on its external ID in an auth provider.
--- a/changelog.d/13811.misc
+++ b/changelog.d/13811.misc
@@ -0,0 +1 @@
+Fix Docker build when Rust .so has been build locally first.
--- a/changelog.d/13814.feature
+++ b/changelog.d/13814.feature
@@ -0,0 +1 @@
+Keep track when we fail to process a pulled event over federation so we can intelligently back-off in the future.
--- a/changelog.d/13819.misc
+++ b/changelog.d/13819.misc
@@ -0,0 +1 @@
+complement: init postgres DB directly inside the target image instead of the base postgres image to fix building using Buildah.
--- a/changelog.d/13822.misc
+++ b/changelog.d/13822.misc
@@ -0,0 +1 @@
+Support providing an index predicate clause when doing upserts.
--- a/changelog.d/13825.bugfix
+++ b/changelog.d/13825.bugfix
@@ -0,0 +1 @@
+Delete associated data from `event_failed_pull_attempts`, `insertion_events`, `insertion_event_extremities`, `insertion_event_extremities`, `insertion_event_extremities` when purging the room.
--- a/changelog.d/13826.bugfix
+++ b/changelog.d/13826.bugfix
@@ -0,0 +1 @@
+Fix a long standing bug where device lists would remain cached when remote users left and rejoined the last room shared with the local homeserver.
--- a/changelog.d/13827.misc
+++ b/changelog.d/13827.misc
@@ -0,0 +1 @@
+Minor speedups to linting in CI.
--- a/debian/changelog
+++ b/debian/changelog
@@ -1,8 +1,18 @@
-matrix-synapse-py3 (1.66.0ubuntu1) UNRELEASED; urgency=medium
+matrix-synapse-py3 (1.67.0) stable; urgency=medium

+  * New Synapse release 1.67.0.
+
+ -- Synapse Packaging team <packages@matrix.org>  Tue, 13 Sep 2022 09:19:56 +0100
+
+matrix-synapse-py3 (1.67.0~rc1) stable; urgency=medium
+
+  [ Erik Johnston ]
  * Use stable poetry 1.2.0 version, rather than a prerelease.

- -- Erik Johnston <erik@matrix.org>  Thu, 01 Sep 2022 13:48:31 +0100
+  [ Synapse Packaging team ]
+  * New Synapse release 1.67.0rc1.
+
+ -- Synapse Packaging team <packages@matrix.org>  Tue, 06 Sep 2022 09:01:06 +0100

 matrix-synapse-py3 (1.66.0) stable; urgency=medium

--- a/docker/Dockerfile
+++ b/docker/Dockerfile
@@ -31,7 +31,9 @@ ARG PYTHON_VERSION=3.9
 ###
 ### Stage 0: generate requirements.txt
 ###
-FROM docker.io/python:${PYTHON_VERSION}-slim as requirements
+# We hardcode the use of Debian bullseye here because this could change upstream
+# and other Dockerfiles used for testing are expecting bullseye.
+FROM docker.io/python:${PYTHON_VERSION}-slim-bullseye as requirements

 # RUN --mount is specific to buildkit and is documented at
 # https://github.com/moby/buildkit/blob/master/frontend/dockerfile/docs/syntax.md#build-mounts-run---mount.
@@ -76,7 +78,7 @@ RUN if [ -z "$TEST_ONLY_IGNORE_POETRY_LOCKFILE" ]; then \
 ###
 ### Stage 1: builder
 ###
-FROM docker.io/python:${PYTHON_VERSION}-slim as builder
+FROM docker.io/python:${PYTHON_VERSION}-slim-bullseye as builder

 # install the OS build deps
 RUN \
@@ -137,7 +139,7 @@ RUN if [ -z "$TEST_ONLY_IGNORE_POETRY_LOCKFILE" ]; then \
 ### Stage 2: runtime
 ###

-FROM docker.io/python:${PYTHON_VERSION}-slim
+FROM docker.io/python:${PYTHON_VERSION}-slim-bullseye

 LABEL org.opencontainers.image.url='https://matrix.org/docs/projects/server/synapse'
 LABEL org.opencontainers.image.documentation='https://github.com/matrix-org/synapse/blob/master/docker/README.md'
--- a/docker/complement/Dockerfile
+++ b/docker/complement/Dockerfile
@@ -17,7 +17,17 @@ ARG SYNAPSE_VERSION=latest
 # the same debian version as Synapse's docker image (so the versions of the
 # shared libraries match).

-FROM postgres:13-bullseye AS postgres_base
+# now build the final image, based on the Synapse image.
+
+FROM matrixdotorg/synapse-workers:$SYNAPSE_VERSION
+    # copy the postgres installation over from the image we built above
+    RUN adduser --system --uid 999 postgres --home /var/lib/postgresql
+    COPY --from=postgres:13-bullseye /usr/lib/postgresql /usr/lib/postgresql
+    COPY --from=postgres:13-bullseye /usr/share/postgresql /usr/share/postgresql
+    RUN mkdir /var/run/postgresql && chown postgres /var/run/postgresql
+    ENV PATH="${PATH}:/usr/lib/postgresql/13/bin"
+    ENV PGDATA=/var/lib/postgresql/data
+
    # initialise the database cluster in /var/lib/postgresql
    RUN gosu postgres initdb --locale=C --encoding=UTF-8 --auth-host password

@@ -25,18 +35,6 @@ FROM postgres:13-bullseye AS postgres_base
    RUN echo "ALTER USER postgres PASSWORD 'somesecret'" | gosu postgres postgres --single
    RUN echo "CREATE DATABASE synapse" | gosu postgres postgres --single

-# now build the final image, based on the Synapse image.
-
-FROM matrixdotorg/synapse-workers:$SYNAPSE_VERSION
-    # copy the postgres installation over from the image we built above
-    RUN adduser --system --uid 999 postgres --home /var/lib/postgresql
-    COPY --from=postgres_base /var/lib/postgresql /var/lib/postgresql
-    COPY --from=postgres_base /usr/lib/postgresql /usr/lib/postgresql
-    COPY --from=postgres_base /usr/share/postgresql /usr/share/postgresql
-    RUN mkdir /var/run/postgresql && chown postgres /var/run/postgresql
-    ENV PATH="${PATH}:/usr/lib/postgresql/13/bin"
-    ENV PGDATA=/var/lib/postgresql/data
-
    # Extend the shared homeserver config to disable rate-limiting,
    # set Complement's static shared secret, enable registration, amongst other
    # tweaks to get Synapse ready for testing.
--- a/Show More
+++ b/Show More
Author	SHA1	Message	Date
H. Shay	404a6e3d79	add suppport for persisting batched events over replication	2022-09-19 11:49:33 -07:00
H. Shay	99aa2136c2	add fucntions to persist events as a batch, encapsulate some logic in a helper function	2022-09-19 11:48:59 -07:00
H. Shay	d950d99ab5	add some comments and sned events as a batch to be persisted	2022-09-19 11:48:35 -07:00
H. Shay	abc497bae6	Merge branch 'shay/batch_events' of https://github.com/matrix-org/synapse into shay/batch_events	2022-09-19 10:43:03 -07:00
H. Shay	2bb3af5310	fix tests to reflect new reality	2022-09-19 10:11:15 -07:00
H. Shay	fb5ac9208a	ditch auth events and pass state map instead	2022-09-19 10:11:14 -07:00
H. Shay	5e72a85f33	split out creating events for batches and add helper methods for duplicated code	2022-09-19 10:11:14 -07:00
H. Shay	b80daa3b2b	reduce duplicated code	2022-09-19 10:11:14 -07:00
H. Shay	86d135ac63	fix test to align with new behaviour	2022-09-19 10:11:14 -07:00
H. Shay	f0b65d057b	add function to calculate event context without pulling from db	2022-09-19 10:11:14 -07:00
H. Shay	466d2c5dfb	create events and contexts seperately	2022-09-19 10:11:14 -07:00
H. Shay	0a91b06ede	add function to create events without computing event context	2022-09-19 10:11:14 -07:00
H. Shay	ab297fa0a1	fix newsfragment	2022-09-19 10:11:14 -07:00
H. Shay	b419e55d1e	newsfragment	2022-09-19 10:11:14 -07:00
H. Shay	8f8a264fb2	fix tests to accomodate new structure of DAG after room creation	2022-09-19 10:11:14 -07:00
H. Shay	7020aeac52	batch some events to send	2022-09-19 10:11:14 -07:00
H. Shay	d06fcba002	split out creating and sending event	2022-09-19 10:11:14 -07:00
Eric Eastwood	44be42338e	Add support to purge rows from MSC2716 and other tables when purging a room (#13825 ) `event_failed_pull_attempts` added in https://github.com/matrix-org/synapse/pull/13589 MSC2716 related tables added in: - https://github.com/matrix-org/synapse/pull/10245/files#diff-3d42dfb44d02f7de3aada105e0bdc1cc9dd7f953cbf0f36c5d0f50827bf0320aR1 - Renamed in https://github.com/matrix-org/synapse/pull/10838/files#diff-2730bfbe9e688b55e46f9371aefe67dac2bd2b2b7d9d6b92774eea1fcfae156dR1 - https://github.com/matrix-org/synapse/pull/10498/files#diff-c52bbfbb5921a3f6f023b24343668479d966fac164f13b7c39d2197ce3afa7a5R1	2022-09-16 10:56:56 -05:00
Mathieu Velten	d5292b8017	Fix Docker build when Rust .so has been build locally first (#13811 ) Signed-off-by: Mathieu Velten <mathieuv@matrix.org>	2022-09-16 15:38:54 +00:00
David Robertson	642c4b253d	Compare ported to unported PG schemas in portdb test job (#13808 )	2022-09-16 16:25:54 +01:00
David Robertson	5e84461653	Minor speedups to CI linting (#13827 )	2022-09-16 16:18:32 +01:00
Sean Quah	d64e85197a	Remove error spam when users query the keys of departed remote users (#13826 ) The error message introduced in #13749 has turned out to be very spammy. Remove it for now.	2022-09-16 16:16:05 +01:00
Mathieu Velten	384dca53d6	complement: init postgres DB directly inside the target image (#13819 ) Doing so in the base postgres image doesn't work with buildah because changes in a declared VOLUME in the Dockerfile is supposed to be discarded, cf https://docs.docker.com/engine/reference/builder/#volume Signed-off-by: Mathieu Velten <mathieuv@matrix.org>	2022-09-16 17:12:45 +02:00
Quentin Gliech	74f60cec92	Add an admin API endpoint to find a user based on its external ID in an auth provider. (#13810 )	2022-09-16 12:29:03 +00:00
reivilibre	f7a77ad717	Update request log format documentation to mention the format used when the authenticated user is controlling another user. (#13794 )	2022-09-16 11:48:41 +00:00
Sean Quah	b73cbb8215	Avoid putting rejected events in room state (#13723 ) Signed-off-by: Sean Quah <seanq@matrix.org>	2022-09-16 12:45:04 +01:00
Eric Eastwood	6986bcbf39	Document common fix of Poetry problems by removing egg-info (#13785 ) `matrix_synapse.egg-info/` Mentioned at https://matrix.to/#/!vcyiEtMVHIhWXcJAfl:sw1v.org/$aKy_IjrKwb70aTVZWeW_6zt0k7OIZ1YkyZpkP9uiRaM?via=matrix.org&via=element.io&via=beeper.com and many other places.	2022-09-15 16:28:03 -05:00
Eric Eastwood	5093cbf88d	Be able to correlate timeouts in reverse-proxy layer in front of Synapse (pull request ID from header) (#13801 ) Fix https://github.com/matrix-org/synapse/issues/13685 New config: ```diff listeners: - port: 8008 tls: false type: http x_forwarded: true + request_id_header: "cf-ray" bind_addresses: ['::1', '127.0.0.1', '0.0.0.0'] ```	2022-09-15 15:32:25 -05:00
Eric Eastwood	140af0cdb6	Record any exception when processing a pulled event (#13814 ) Part of https://github.com/matrix-org/synapse/issues/13700 and https://github.com/matrix-org/synapse/issues/13356 Follow-up to https://github.com/matrix-org/synapse/pull/13589	2022-09-15 14:40:49 -05:00
Patrick Cloke	b2b0c85279	Support providing an index predicate for upserts. (#13822 ) This is useful to upsert against a table which has a unique partial index while avoiding conflicts.	2022-09-15 18:28:48 +00:00
David Robertson	742f9f9d78	A third batch of Pydantic validation for rest/client/account.py (#13736 )	2022-09-15 18:36:02 +01:00
Andrew Morgan	918c74bfb5	Add a `MXCUri` class to make working with mxc uri's easier. (#13162 )	2022-09-15 12:57:16 +00:00
Eric Eastwood	957e3d74fc	Keep track when we try and fail to process a pulled event (#13589 ) We can follow-up this PR with: 1. Only try to backfill from an event if we haven't tried recently -> https://github.com/matrix-org/synapse/issues/13622 1. When we decide to backfill that event again, process it in the background so it doesn't block and make `/messages` slow when we know it will probably fail again -> https://github.com/matrix-org/synapse/issues/13623 1. Generally track failures everywhere we try and fail to pull an event over federation -> https://github.com/matrix-org/synapse/issues/13700 Fix https://github.com/matrix-org/synapse/issues/13621 Part of https://github.com/matrix-org/synapse/issues/13356 Mentioned in [internal doc](https://docs.google.com/document/d/1lvUoVfYUiy6UaHB6Rb4HicjaJAU40-APue9Q4vzuW3c/edit#bookmark=id.qv7cj51sv9i5)	2022-09-14 13:57:50 -05:00
Patrick Cloke	666ae87729	Update event push action and receipt tables to support threads. (#13753 ) Adds a `thread_id` column to the `event_push_actions`, `event_push_actions_staging`, and `event_push_summary` tables. This will notifications to be segmented by the thread in a future pull request. The `thread_id` column stores the root event ID or the special value `"main"`. The `thread_id` column for `event_push_actions` and `event_push_summary` is backfilled with `"main"` for all existing rows. New entries into `event_push_actions` and `event_push_actions_staging` will get the proper thread ID. `receipts_linearized` and `receipts_graph` also gain a `thread_id` column, which is similar, except `NULL` is a special value meaning the receipt is "unthreaded". See MSC3771 and MSC3773 for where this data will be useful.	2022-09-14 17:11:16 +00:00
Patrick Cloke	f2d12ccabe	Use partial indices on SQLIte. (#13802 ) Partial indices have been supported since SQLite 3.8, but Synapse now requires >= 3.27, so we can enable support for them. This requires rebuilding previous indices which were partial on PostgreSQL, but not on SQLite.	2022-09-14 12:01:42 -04:00
reivilibre	6302753012	Deduplicate `is_server_notices_room`. (#13780 )	2022-09-14 15:53:18 +00:00
reivilibre	cf65433de2	Fix a memory leak when running the unit tests. (#13798 )	2022-09-14 15:29:05 +00:00
Quentin Gliech	eaed4e6113	Remove unused method in `synapse.api.auth.Auth`. (#13795 ) Clean-up from `b19060a29b` (#13094) and `73af10f419` (#13093) which removed all callers.	2022-09-14 10:33:54 -04:00
David Robertson	51a77e990b	Remove incorrect migration file from `state` logical DB (#13788 ) * Remove incorrect migration file from `state` logical DB The table `ex_outlier_stream` is part of the `main` logical DB; it should not have been created in the `state` logical DB. We remove this migration now as a tidy-up. Note: we cannot `DROP TABLE IF EXISTS ex_outlier_stream` in a new migration, because some (most) instances of Synapse host both of these logical DBs on the same DB cluster. * Changelog	2022-09-14 14:16:12 +01:00
Erik Johnston	69beef22c2	Merge remote-tracking branch 'origin/develop' into shay/batch_events	2022-09-14 11:13:37 +01:00
Sean Quah	c73774467e	Fix bug in device list caching when remote users leave rooms (#13749 ) When a remote user leaves the last room shared with the homeserver, we have to mark their device list as unsubscribed, otherwise we would hold on to a stale device list in our cache. Crucially, the device list would remain cached even after the remote user rejoined the room, which could lead to E2EE failures until the next change to the remote user's device list. Fixes #13651. Signed-off-by: Sean Quah <seanq@matrix.org>	2022-09-14 10:42:57 +01:00
reivilibre	21687ec189	Fix a long-standing spec compliance bug where Synapse would accept a trailing slash on the end of `/get_missing_events` federation requests. (#13789 ) * Don't accept a trailing slash on the end of /get_missing_events * Newsfile Signed-off-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org> Signed-off-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org>	2022-09-14 09:28:12 +01:00
Mathieu Velten	12dacecabd	Make sequence `cache_invalidation_stream_seq` begin at `2` (#13766 ) Signed-off-by: Mathieu Velten <mathieuv@matrix.org> Co-authored-by: Sean Quah <8349537+squahtx@users.noreply.github.com>	2022-09-13 16:14:28 +02:00
Erik Johnston	9772e362aa	Merge branch 'master' into develop	2022-09-13 12:11:53 +01:00
David Robertson	b60d47ab2c	Updates to the schema dump script (#13770 )	2022-09-13 10:53:11 +01:00
David Robertson	540afb0bfc	Simplify CI tests DAG (#13784 ) * Simplify CI tests DAG * Changelog	2022-09-13 10:17:23 +01:00
Richard van der Hoff	41df25bbbd	installation.md: require libpq on M1 macs (#13480 )	2022-09-13 09:01:21 +00:00
Erik Johnston	80bb098d87	Fixup changelog	2022-09-13 09:55:10 +01:00
Erik Johnston	4b678b20a2	1.67.0	2022-09-13 09:20:28 +01:00
Nick Mills-Barrett	cdbb641232	Add receipts event stream ordering (#13703 )	2022-09-13 08:16:37 +01:00
Mathieu Velten	fa2f3d8d0c	Fix GHA skippable syntax (#13778 ) Signed-off-by: Mathieu Velten <mathieuv@matrix.org>	2022-09-12 17:31:23 +00:00
Brendan Abolivier	7571337445	Fix typo in ratelimiting documentation (#13727 )	2022-09-12 14:11:18 +01:00
Erik Johnston	dd7484b562	Fix CI on non-PR builds (#13769 ) Mark cargo-test as skippable since it only runs on Rust code change.	2022-09-12 13:26:33 +01:00
Nick Mills-Barrett	da41a7cd61	Remove check current state membership up to date (#13745 ) * Remove checks for membership column in current_state_events * Add schema script to force through the `current_state_events_membership` background job Contributed by Nick @ Beeper (@fizzadar).	2022-09-12 12:58:33 +01:00
Erik Johnston	ebfeac7c5d	Check if Rust lib needs rebuilding. (#13759 ) This protects against the common mistake of failing to remember to rebuild Rust code after making changes.	2022-09-12 10:03:42 +00:00
Nick Mills-Barrett	4c4889cac0	Concurrently collect room unread counts for push badges (#13765 ) Most of the time this function is heavily cached, but when that isn't the case fetching the counts room by room slows down push delivery on users with many (thousands) of rooms. Signed off by Nick @ Beeper.	2022-09-09 19:00:21 +01:00
Eric Eastwood	a911ffb42c	Tag trace with instance name (#13761 ) We tag the Synapse instance name so that it's an easy jumping off point into the logs. Can also be used to filter for an instance that is under load. As suggested by @clokep and @reivilibre in, - https://github.com/matrix-org/synapse/pull/13729#discussion_r964719258 - https://github.com/matrix-org/synapse/pull/13729#discussion_r964733578	2022-09-09 11:31:37 -05:00
Eric Eastwood	f694bb71b7	Strip number suffix from instance name to consolidate services that traces are spread over (#13729 ) The problem with many services is that it makes it hard to find which service has the trace you want, see https://github.com/jaegertracing/jaeger-ui/issues/985 Previously, we split traces out into services based on their instance name like `matrix.org client_reader-1`, etc but there are many worker instances of the same `client_reader` so there is a lot to click through. With this PR, all of the traces are just collected under the worker type like `client_reader`, `event_persister` 😇 Note: A Synapse worker instance name is an opaque string with the number convention only being our own thing for the `matrix.org` deployment. But seems pretty sensible to group things this way.	2022-09-09 11:30:06 -05:00
David Robertson	9d11842562	Also cite upgrade notes	2022-09-08 12:04:29 +01:00
David Robertson	a7c71686ca	Add minimum version bump for sqlite to the release notes (#13742 ) * Notify that SQLite min version will be bumped * Mention in upgrade notes Co-authored-by: reivilibre <oliverw@matrix.org>	2022-09-08 12:00:03 +01:00
Erik Johnston	ec2fe7bb53	Fixup changelog	2022-09-06 13:04:22 +01:00
Erik Johnston	571f565c1f	Update changelog	2022-09-06 09:25:32 +01:00
Erik Johnston	cdf7fb737b	1.67.0rc1	2022-09-06 09:01:26 +01:00
H. Shay	6630daba47	fix tests to reflect new reality	2022-09-01 14:47:10 -07:00
H. Shay	070f1bb720	ditch auth events and pass state map instead	2022-09-01 14:46:57 -07:00
H. Shay	059746dec4	split out creating events for batches and add helper methods for duplicated code	2022-09-01 14:46:25 -07:00
H. Shay	2d06a39dd0	reduce duplicated code	2022-08-31 15:30:05 -07:00
Shay	172b651832	Merge branch 'develop' into shay/batch_events	2022-08-24 14:18:31 -07:00
H. Shay	b38f3e9217	fix test to align with new behaviour	2022-08-24 14:08:47 -07:00
H. Shay	28eca036bb	add function to calculate event context without pulling from db	2022-08-24 14:08:31 -07:00
H. Shay	49686ede02	create events and contexts seperately	2022-08-24 14:08:04 -07:00
H. Shay	af2a5f3893	add function to create events without computing event context	2022-08-24 14:07:36 -07:00
H. Shay	066045f03e	Merge branch 'shay/batch_events' of https://github.com/matrix-org/synapse into shay/batch_events	2022-08-11 21:29:58 -07:00
H. Shay	47bd7cff4e	fix newsfragment	2022-08-11 21:29:35 -07:00
Shay	0da05c5a2c	Merge branch 'develop' into shay/batch_events	2022-08-11 21:21:30 -07:00
H. Shay	740a48de76	newsfragment	2022-08-11 21:20:10 -07:00
H. Shay	a7ed07944f	Merge branch 'shay/batch_events' of https://github.com/matrix-org/synapse into shay/batch_events	2022-08-11 21:15:57 -07:00
H. Shay	b8ed35841d	fix tests to accomodate new structure of DAG after room creation	2022-08-11 21:15:16 -07:00
H. Shay	e215109b74	batch some events to send	2022-08-11 21:13:59 -07:00
Shay	9adffc2c95	Merge branch 'develop' into shay/batch_events	2022-08-09 09:04:46 -07:00
H. Shay	a08f32f8ed	split out creating and sending event	2022-08-04 12:16:26 -07:00
				`@@ -0,0 +1 @@`
				Bump the minimum dependency of `matrix_common` to 1.3.0 to make use of the `MXCUri` class. Use `MXCUri` to simplify media retention test code.
				`@@ -1 +0,0 @@`
				Drop support for calling `/_matrix/client/v3/rooms/{roomId}/invite` without an `id_access_token`, which was not permitted by the spec. Contributed by @Vetchu.
				`@@ -0,0 +1 @@`
				Note that `libpq` is required on ARM-based Macs.
				`@@ -1 +0,0 @@`
				`Extend the release script to wait for GitHub Actions to finish and to be usable as a guide for the whole process.`
				`@@ -0,0 +1 @@`
				Refactor ` _send_events_for_new_room` to separate creating and sending events.
				`@@ -1 +0,0 @@`
				Fix a bug introduced in Synapse 1.13 where the [List Rooms admin API](https://matrix-org.github.io/synapse/develop/admin_api/rooms.html#list-room-api) would return integers instead of booleans for the `federatable` and `public` fields when using a Sqlite database.
				`@@ -1 +0,0 @@`
				`Add experimental configuration option to allow disabling legacy Prometheus metric names.`
				`@@ -1 +0,0 @@`
				Fix bug that user cannot `/forget` rooms after the last member has left the room.
				`@@ -1 +0,0 @@`
				Remove redundant `_get_joined_users_from_context` cache. Contributed by Nick @ Beeper (@fizzadar).