SYN-187: Set a more sensible default for the content_addr

Added the needed libssl-dev package

.gitignore

@@ -1,6 +1,7 @@
 *.pyc
 .*.swp
 
+.DS_Store
 _trial_temp/
 logs/
 dbs/
@@ -11,6 +12,14 @@ docs/build/
 
 cmdclient_config.json
 homeserver*.db
+homeserver*.log
+homeserver*.pid
+homeserver*.yaml
+
+*.signing.key
+*.tls.crt
+*.tls.dh
+*.tls.key
 
 .coverage
 htmlcov
@@ -24,7 +33,8 @@ graph/*.svg
 graph/*.png
 graph/*.dot
 
-webclient/config.js
-webclient/test/environment-protractor.js
+**/webclient/config.js
+**/webclient/test/coverage/
+**/webclient/test/environment-protractor.js
 
 uploads

CHANGES.rst

@@ -1,3 +1,64 @@
+Changes in synapse 0.5.3 (2014-11-27)
+=====================================
+
+ * Fix bug that caused joining a remote room to fail if a single event was not
+   signed correctly.
+ * Fix bug which caused servers to continuously try and fetch events from other
+   servers.
+
+Changes in synapse 0.5.2 (2014-11-26)
+=====================================
+
+Fix major bug that caused rooms to disappear from peoples initial sync.
+
+Changes in synapse 0.5.1 (2014-11-26)
+=====================================
+See UPGRADES.rst for specific instructions on how to upgrade.
+
+ * Fix bug where we served up an Event that did not match its signatures.
+ * Fix regression where we no longer correctly handled the case where a
+   homeserver receives an event for a room it doesn't recognise (but is in.)
+
+Changes in synapse 0.5.0 (2014-11-19)
+=====================================
+This release includes changes to the federation protocol and client-server API
+that is not backwards compatible.
+
+This release also changes the internal database schemas and so requires servers to
+drop their current history. See UPGRADES.rst for details.
+
+Homeserver:
+ * Add authentication and authorization to the federation protocol. Events are
+   now signed by their originating homeservers.
+ * Implement the new authorization model for rooms.
+ * Split out web client into a seperate repository: matrix-angular-sdk.
+ * Change the structure of PDUs.
+ * Fix bug where user could not join rooms via an alias containing 4-byte
+   UTF-8 characters.
+ * Merge concept of PDUs and Events internally.
+ * Improve logging by adding request ids to log lines.
+ * Implement a very basic room initial sync API.
+ * Implement the new invite/join federation APIs.
+
+Webclient:
+ * The webclient has been moved to a seperate repository.
+
+Changes in synapse 0.4.2 (2014-10-31)
+=====================================
+
+Homeserver:
+ * Fix bugs where we did not notify users of correct presence updates.
+ * Fix bug where we did not handle sub second event stream timeouts.
+
+Webclient:
+ * Add ability to click on messages to see JSON.
+ * Add ability to redact messages.
+ * Add ability to view and edit all room state JSON.
+ * Handle incoming redactions.
+ * Improve feedback on errors.
+ * Fix bugs in mobile CSS.
+ * Fix bugs with desktop notifications.
+
 Changes in synapse 0.4.1 (2014-10-17)
 =====================================
 Webclient:
@@ -5,16 +66,18 @@ Webclient:
 
 Changes in synpase 0.4.0 (2014-10-17)
 =====================================
-This server includes changes to the federation protocol that is not backwards
-compatible.
+This release includes changes to the federation protocol and client-server API
+that is not backwards compatible.
 
-The Matrix specification has been moved to a separate git repository.
+The Matrix specification has been moved to a separate git repository:
+http://github.com/matrix-org/matrix-doc
 
 You will also need an updated syutil and config. See UPGRADES.rst.
 
 Homeserver:
- * Sign federation transactions.
- * Rename timestamp keys in PDUs.
+ * Sign federation transactions to assert strong identity over federation.
+ * Rename timestamp keys in PDUs and events from 'ts' and 'hsob_ts' to 'origin_server_ts'.
+
 
 Changes in synapse 0.3.4 (2014-09-25)
 =====================================

MANIFEST.in

@@ -1,3 +1,4 @@
 recursive-include docs *
 recursive-include tests *.py
-recursive-include synapse/persistence/schema *.sql
+recursive-include synapse/storage/schema *.sql
+recursive-include syweb/webclient *

README.rst

@@ -4,9 +4,9 @@ Introduction
 Matrix is an ambitious new ecosystem for open federated Instant Messaging and
 VoIP.  The basics you need to know to get up and running are:
 
-- Chatrooms are distributed and do not exist on any single server.  Rooms
-  can be found using aliases like ``#matrix:matrix.org`` or
-  ``#test:localhost:8008`` or they can be ephemeral.
+- Everything in Matrix happens in a room.  Rooms are distributed and do not
+  exist on any single server.  Rooms can be located using convenience aliases 
+  like ``#matrix:matrix.org`` or ``#test:localhost:8008``.
 
 - Matrix user IDs look like ``@matthew:matrix.org`` (although in the future
   you will normally refer to yourself and others using a 3PID: email
@@ -17,56 +17,12 @@ The overall architecture is::
       client <----> homeserver <=====================> homeserver <----> client
              https://somewhere.org/_matrix      https://elsewhere.net/_matrix
 
-WARNING
-=======
-
-**Synapse is currently in a state of rapid development, and not all features
-are yet functional. Critically, some security features are still in
-development, which means Synapse can *not* be considered secure or reliable at
-this point.**  For instance:
-
-- **SSL Certificates used by server-server federation are not yet validated.**
-- **Room permissions are not yet enforced on traffic received via federation.**
-- **Homeservers do not yet cryptographically sign their events to avoid
-  tampering**
-- Default configuration provides open signup to the service from the internet
-
-Despite this, we believe Synapse is more than useful as a way for experimenting
-and exploring Synapse, and the missing features will land shortly. **Until
-then, please do *NOT* use Synapse for any remotely important or secure
-communication.**
-
-
-Quick Start
-===========
-
-System requirements:
-- POSIX-compliant system (tested on Linux & OSX)
-- Python 2.7
-
-To get up and running:
-
-- To simply play with an **existing** homeserver you can
-  just go straight to http://matrix.org/alpha.
-
-- To run your own **private** homeserver on localhost:8008, generate a basic
-  config file: ``./synctl start`` will give you instructions on how to do this.
-  For this purpose, you can use 'localhost' or your hostname as a server name.
-  Once you've done so, running ``./synctl start`` again will start your private
-  home sserver. You will find a webclient running at http://localhost:8008.
-  Please use a recent Chrome or Firefox for now (or Safari if you don't need
-  VoIP support).
-
-- To run a **public** homeserver and let it exchange messages with other
-  homeservers and participate in the global Matrix federation, you must expose
-  port 8448 to the internet and edit homeserver.yaml to specify server_name
-  (the public DNS entry for this server) and then run ``synctl start``. If you
-  changed the server_name, you may need to move the old database
-  (homeserver.db) out of the way first. Then come join ``#matrix:matrix.org``
-  and say hi! :)
-
-For more detailed setup instructions, please see further down this document.
+``#matrix:matrix.org`` is the official support room for Matrix, and can be
+accessed by the web client at http://matrix.org/alpha or via an IRC bridge at
+irc://irc.freenode.net/matrix.
 
+Synapse is currently in rapid development, but as of version 0.5 we believe it 
+is sufficiently stable to be run as an internet-facing service for real usage!
 
 About Matrix
 ============
@@ -76,10 +32,10 @@ which handle:
 
 - Creating and managing fully distributed chat rooms with no
   single points of control or failure
-- Eventually-consistent cryptographically secure[1] synchronisation of room
+- Eventually-consistent cryptographically secure synchronisation of room
   state across a global open network of federated servers and services
 - Sending and receiving extensible messages in a room with (optional)
-  end-to-end encryption[2]
+  end-to-end encryption[1]
 - Inviting, joining, leaving, kicking, banning room members
 - Managing user accounts (registration, login, logout)
 - Using 3rd Party IDs (3PIDs) such as email addresses, phone numbers,
@@ -111,56 +67,122 @@ Synapse ships with two basic demo Matrix clients: webclient (a basic group chat
 web client demo implemented in AngularJS) and cmdclient (a basic Python
 command line utility which lets you easily see what the JSON APIs are up to).
 
-We'd like to invite you to take a look at the Matrix spec, try to run a
-homeserver, and join the existing Matrix chatrooms already out there,
-experiment with the APIs and the demo clients, and let us know your thoughts at
-https://github.com/matrix-org/synapse/issues or at matrix@matrix.org.
+Meanwhile, iOS and Android SDKs and clients are currently in development and available from:
 
-Thanks for trying Matrix!
+- https://github.com/matrix-org/matrix-ios-sdk
+- https://github.com/matrix-org/matrix-android-sdk
 
-[1] Cryptographic signing of messages isn't turned on yet
+We'd like to invite you to join #matrix:matrix.org (via http://matrix.org/alpha), run a homeserver, take a look at the Matrix spec at
+http://matrix.org/docs/spec, experiment with the APIs and the demo
+clients, and report any bugs via http://matrix.org/jira.
 
-[2] End-to-end encryption is currently in development
+Thanks for using Matrix!
 
+[1] End-to-end encryption is currently in development
 
 Homeserver Installation
 =======================
 
-First, the dependencies need to be installed.  Start by installing
-'python2.7-dev' and the various tools of the compiler toolchain.
+System requirements:
+- POSIX-compliant system (tested on Linux & OSX)
+- Python 2.7
 
-Installing prerequisites on Ubuntu::
+Synapse is written in python but some of the libraries is uses are written in
+C. So before we can install synapse itself we need a working C compiler and the
+header files for python C extensions.
 
-    $ sudo apt-get install build-essential python2.7-dev libffi-dev
+Installing prerequisites on Ubuntu or Debian::
+
+    $ sudo apt-get install build-essential python2.7-dev libffi-dev \
+                           python-pip python-setuptools sqlite3 \
+                           libssl-dev
 
 Installing prerequisites on Mac OS X::
 
     $ xcode-select --install
+    
+To install the synapse homeserver run::
+
+    $ pip install --user --process-dependency-links https://github.com/matrix-org/synapse/tarball/master
+
+This installs synapse, along with the libraries it uses, into
+``$HOME/.local/lib/`` on Linux or ``$HOME/Library/Python/2.7/lib/`` on OSX.
+
+Troubleshooting Installation
+----------------------------
+
+Synapse requires pip 1.7 or later, so if your OS provides too old a version and 
+you get errors about ``error: no such option: --process-dependency-links`` you 
+may need to manually upgrade it::
+
+    $ sudo pip install --upgrade pip
+    
+If pip crashes mid-installation for reason (e.g. lost terminal), pip may
+refuse to run until you remove the temporary installation directory it
+created. To reset the installation::
+
+    $ rm -rf /tmp/pip_install_matrix
+    
+pip seems to leak *lots* of memory during installation.  For instance, a Linux 
+host with 512MB of RAM may run out of memory whilst installing Twisted.  If this 
+happens, you will have to individually install the dependencies which are 
+failing, e.g.::
+
+    $ pip install --user twisted
+
+Running Your Homeserver
+=======================
+
+To actually run your new homeserver, pick a working directory for Synapse to run 
+(e.g. ``~/.synapse``), and::
+
+    $ mkdir ~/.synapse
+    $ cd ~/.synapse
+    
+    $ # on Linux
+    $ ~/.local/bin/synctl start
+    
+    $ # on OSX
+    $ ~/Library/Python/2.7/bin/synctl start
+
+Troubleshooting Running
+-----------------------
+
+If ``synctl`` fails with ``pkg_resources.DistributionNotFound`` errors you may 
+need a newer version of setuptools than that provided by your OS.::
+
+    $ sudo pip install setuptools --upgrade
+
+If synapse fails with ``missing "sodium.h"`` crypto errors, you may need 
+to manually upgrade PyNaCL, as synapse uses NaCl (http://nacl.cr.yp.to/) for 
+encryption and digital signatures.
+Unfortunately PyNACL currently has a few issues
+(https://github.com/pyca/pynacl/issues/53) and
+(https://github.com/pyca/pynacl/issues/79) that mean it may not install
+correctly, causing all tests to fail with errors about missing "sodium.h". To
+fix try re-installing from PyPI or directly from
+(https://github.com/pyca/pynacl)::
+
+    $ # Install from PyPI
+    $ pip install --user --upgrade --force pynacl
+    $ # Install from github
+    $ pip install --user https://github.com/pyca/pynacl/tarball/master
+
+
+Homeserver Development
+======================
+
+To check out a homeserver for development, clone the git repo into a working
+directory of your choice::
+
+    $ git clone https://github.com/matrix-org/synapse.git
+    $ cd synapse
 
 The homeserver has a number of external dependencies, that are easiest
 to install by making setup.py do so, in --user mode::
 
     $ python setup.py develop --user
 
-You'll need a version of setuptools new enough to know about git, so you
-may need to also run::
-
-    $ sudo apt-get install python-pip
-    $ sudo pip install --upgrade setuptools
-
-If you don't have access to github, then you may need to install ``syutil``
-manually by checking it out and running ``python setup.py develop --user`` on
-it too.
-
-If you get errors about ``sodium.h`` being missing, you may also need to
-manually install a newer PyNaCl via pip as setuptools installs an old one. Or
-you can check PyNaCl out of git directly (https://github.com/pyca/pynacl) and
-installing it. Installing PyNaCl using pip may also work (remember to remove
-any other versions installed by setuputils in, for example, ~/.local/lib).
-
-On OSX, if you encounter ``clang: error: unknown argument: '-mno-fused-madd'``
-you will need to ``export CFLAGS=-Qunused-arguments``.
-
 This will run a process of downloading and installing into your
 user's .local/lib directory all of the required dependencies that are
 missing.
@@ -180,8 +202,8 @@ This should end with a 'PASSED' result::
 Upgrading an existing homeserver
 ================================
 
-Before upgrading an existing homeserver to a new version, please refer to
-UPGRADE.rst for any additional instructions.
+IMPORTANT: Before upgrading an existing homeserver to a new version, please
+refer to UPGRADE.rst for any additional instructions.
 
 
 Setting up Federation
@@ -202,18 +224,15 @@ IDs:
    domain name.
 
 For the first form, simply pass the required hostname (of the machine) as the
---host parameter::
+--server-name parameter::
 
-    $ python synapse/app/homeserver.py \
+    $ python -m synapse.app.homeserver \
         --server-name machine.my.domain.name \
         --config-path homeserver.config \
         --generate-config
-    $ python synapse/app/homeserver.py --config-path homeserver.config
+    $ python -m synapse.app.homeserver --config-path homeserver.config
 
-Alternatively, you can run synapse via synctl - running ``synctl start`` to
-generate a homeserver.yaml config file, where you can then edit server-name to
-specify machine.my.domain.name, and then set the actual server running again
-with synctl start.
+Alternatively, you can run ``synctl start`` to guide you through the process.
 
 For the second form, first create your SRV record and publish it in DNS. This
 needs to be named _matrix._tcp.YOURDOMAIN, and point at at least one hostname
@@ -221,17 +240,19 @@ and port where the server is running.  (At the current time synapse does not
 support clustering multiple servers into a single logical homeserver).  The DNS
 record would then look something like::
 
+    $ dig -t srv _matrix._tcp.machine.my.domaine.name
     _matrix._tcp    IN      SRV     10 0 8448 machine.my.domain.name.
 
+
 At this point, you should then run the homeserver with the hostname of this
 SRV record, as that is the name other machines will expect it to have::
 
-    $ python synapse/app/homeserver.py \
+    $ python -m synapse.app.homeserver \
         --server-name YOURDOMAIN \
         --bind-port 8448 \
         --config-path homeserver.config \
         --generate-config
-    $ python synapse/app/homeserver.py --config-path homeserver.config
+    $ python -m synapse.app.homeserver --config-path homeserver.config
 
 
 You may additionally want to pass one or more "-v" options, in order to
@@ -250,6 +271,8 @@ private federation (``localhost:8080``, ``localhost:8081`` and
 http://localhost:8080. Simply run::
 
     $ demo/start.sh
+    
+This is mainly useful just for development purposes.
 
 Running The Demo Web Client
 ===========================
@@ -308,13 +331,14 @@ time.
 Where's the spec?!
 ==================
 
-For now, please go spelunking in the ``docs/`` directory to find out.
+The source of the matrix spec lives at https://github.com/matrix-org/matrix-doc.  
+A recent HTML snapshot of this lives at http://matrix.org/docs/spec
 
 
 Building Internal API Documentation
 ===================================
 
-Before building internal API documentation install spinx and
+Before building internal API documentation install sphinx and
 sphinxcontrib-napoleon::
 
     $ pip install sphinx

UPGRADE.rst

@@ -1,3 +1,48 @@
+Upgrading to v0.5.1
+===================
+
+Depending on precisely when you installed v0.5.0 you may have ended up with
+a stale release of the reference matrix webclient installed as a python module.
+To uninstall it and ensure you are depending on the latest module, please run::
+
+    $ pip uninstall syweb
+
+Upgrading to v0.5.0
+===================
+
+The webclient has been split out into a seperate repository/pacakage in this
+release. Before you restart your homeserver you will need to pull in the
+webclient package by running::
+
+  python setup.py develop --user
+
+This release completely changes the database schema and so requires upgrading
+it before starting the new version of the homeserver.
+
+The script "database-prepare-for-0.5.0.sh" should be used to upgrade the
+database. This will save all user information, such as logins and profiles, 
+but will otherwise purge the database. This includes messages, which
+rooms the home server was a member of and room alias mappings.
+
+If you would like to keep your history, please take a copy of your database
+file and ask for help in #matrix:matrix.org. The upgrade process is,
+unfortunately, non trivial and requires human intervention to resolve any
+resulting conflicts during the upgrade process.
+
+Before running the command the homeserver should be first completely 
+shutdown. To run it, simply specify the location of the database, e.g.:
+
+  ./database-prepare-for-0.5.0.sh "homeserver.db"
+
+Once this has successfully completed it will be safe to restart the 
+homeserver. You may notice that the homeserver takes a few seconds longer to 
+restart than usual as it reinitializes the database.
+
+On startup of the new version, users can either rejoin remote rooms using room
+aliases or by being reinvited. Alternatively, if any other homeserver sends a
+message to a room that the homeserver was previously in the local HS will 
+automatically rejoin the room.
+
 Upgrading to v0.4.0
 ===================
 

VERSION

@@ -1 +1 @@
-0.4.1
+0.5.3a

database-prepare-for-0.5.0.sh

@@ -0,0 +1,21 @@
+#!/bin/bash
+
+# This is will prepare a synapse database for running with v0.5.0 of synapse. 
+# It will store all the user information, but will *delete* all messages and
+# room data.
+
+set -e
+
+cp "$1" "$1.bak"
+
+DUMP=$(sqlite3 "$1" << 'EOF'
+.dump users
+.dump access_tokens
+.dump presence
+.dump profiles
+EOF
+)
+
+rm "$1"
+
+sqlite3 "$1" <<< "$DUMP" 

demo/clean.sh

@@ -14,3 +14,4 @@ fi
 find "$DIR" -name "*.log" -delete
 find "$DIR" -name "*.db" -delete
 
+rm -rf $DIR/etc

demo/start.sh

@@ -8,6 +8,14 @@ cd "$DIR/.."
 
 mkdir -p demo/etc
 
+# Check the --no-rate-limit param
+PARAMS=""
+if [ $# -eq 1 ]; then
+    if [ $1 = "--no-rate-limit" ]; then
+	    PARAMS="--rc-messages-per-second 1000 --rc-message-burst-count 1000"
+    fi
+fi
+
 for port in 8080 8081 8082; do
     echo "Starting server on port $port... "
 
@@ -23,7 +31,8 @@ for port in 8080 8081 8082; do
         -d "$DIR/$port.db" \
         -D --pid-file "$DIR/$port.pid" \
         --manhole $((port + 1000)) \
-        --tls-dh-params-path "demo/demo.tls.dh"
+        --tls-dh-params-path "demo/demo.tls.dh" \
+		$PARAMS $SYNAPSE_PARAMS
 
     python -m synapse.app.homeserver \
         --config-path "demo/etc/$port.config" \
@@ -31,7 +40,4 @@ for port in 8080 8081 8082; do
 
 done
 
-echo "Starting webclient on port 8000..."
-python "demo/webserver.py" -p 8000 -P "$DIR/webserver.pid" "webclient"
-
 cd "$CWD"

docs/ancient_architecture_notes.rst

@@ -1,3 +1,9 @@
+.. WARNING::
+  These architecture notes are spectacularly old, and date back to when Synapse 
+  was just federation code in isolation.  This should be merged into the main
+  spec.
+  
+
 = Server to Server =
 
 == Server to Server Stack ==

docs/architecture.rst

@@ -0,0 +1,68 @@
+Synapse Architecture
+====================
+
+As of the end of Oct 2014, Synapse's overall architecture looks like::
+
+        synapse
+        .-----------------------------------------------------.
+        |                          Notifier                   |
+        |                            ^  |                     |
+        |                            |  |                     |
+        |                  .------------|------.              |
+        |                  | handlers/  |      |              |
+        |                  |            v      |              |
+        |                  | Event*Handler <--------> rest/* <=> Client
+        |                  | Rooms*Handler     |              |
+  HSes <=> federation/* <==> FederationHandler |              |
+        |      |           | PresenceHandler   |              |
+        |      |           | TypingHandler     |              |
+        |      |           '-------------------'              |
+        |      |                 |     |                      |
+        |      |              state/*  |                      |
+        |      |                 |     |                      |
+        |      |                 v     v                      |
+        |      `--------------> storage/*                     |
+        |                          |                          |
+        '--------------------------|--------------------------'
+                                   v
+                                .----.
+                                | DB |
+                                '----'
+
+* Handlers: business logic of synapse itself.  Follows a set contract of BaseHandler:
+
+  - BaseHandler gives us onNewRoomEvent which: (TODO: flesh this out and make it less cryptic):
+ 
+    + handle_state(event)
+    + auth(event)
+    + persist_event(event)
+    + notify notifier or federation(event)
+   
+  - PresenceHandler: use distributor to get EDUs out of Federation.  Very
+    lightweight logic built on the distributor
+  - TypingHandler: use distributor to get EDUs out of Federation.  Very
+    lightweight logic built on the distributor
+  - EventsHandler: handles the events stream...
+  - FederationHandler: - gets PDU from Federation Layer; turns into an event;
+    follows basehandler functionality.
+  - RoomsHandler: does all the room logic, including members - lots of classes in
+    RoomsHandler.
+  - ProfileHandler: talks to the storage to store/retrieve profile info.
+
+* EventFactory: generates events of particular event types.
+* Notifier: Backs the events handler
+* REST: Interfaces handlers and events to the outside world via HTTP/JSON.
+  Converts events back and forth from JSON.
+* Federation: holds the HTTP client & server to talk to other servers.  Does
+  replication to make sure there's nothing missing in the graph.  Handles
+  reliability.  Handles txns.
+* Distributor: generic event bus. used for presence & typing only currently. 
+  Notifier could be implemented using Distributor - so far we are only using for
+  things which actually /require/ dynamic pluggability however as it can
+  obfuscate the actual flow of control.
+* Auth: helper singleton to say whether a given event is allowed to do a given
+  thing  (TODO: put this on the diagram)
+* State: helper singleton: does state conflict resolution. You give it an event
+  and it tells you if it actually updates the state or not, and annotates the
+  event up properly and handles merge conflict resolution.
+* Storage: abstracts the storage engine.

docs/client-server/model/presence.rst

@@ -1,149 +0,0 @@
-API Efficiency
-==============
-
-A simple implementation of presence messaging has the ability to cause a large
-amount of Internet traffic relating to presence updates. In order to minimise
-the impact of such a feature, the following observations can be made:
-
- * There is no point in a Home Server polling status for peers in a user's
-   presence list if the user has no clients connected that care about it.
-
- * It is highly likely that most presence subscriptions will be symmetric - a
-   given user watching another is likely to in turn be watched by that user.
-
- * It is likely that most subscription pairings will be between users who share
-   at least one Room in common, and so their Home Servers are actively
-   exchanging message PDUs or transactions relating to that Room.
-
- * Presence update messages do not need realtime guarantees. It is acceptable to
-   delay delivery of updates for some small amount of time (10 seconds to a
-   minute).
-
-The general model of presence information is that of a HS registering its
-interest in receiving presence status updates from other HSes, which then
-promise to send them when required. Rather than actively polling for the
-currentt state all the time, HSes can rely on their relative stability to only
-push updates when required.
-
-A Home Server should not rely on the longterm validity of this presence
-information, however, as this would not cover such cases as a user's server
-crashing and thus failing to inform their peers that users it used to host are
-no longer available online. Therefore, each promise of future updates should
-carry with a timeout value (whether explicit in the message, or implicit as some
-defined default in the protocol), after which the receiving HS should consider
-the information potentially stale and request it again.
-
-However, because of the likelyhood that two home servers are exchanging messages
-relating to chat traffic in a room common to both of them, the ongoing receipt
-of these messages can be taken by each server as an implicit notification that
-the sending server is still up and running, and therefore that no status changes
-have happened; because if they had the server would have sent them. A second,
-larger timeout should be applied to this implicit inference however, to protect
-against implementation bugs or other reasons that the presence state cache may
-become invalid; eventually the HS should re-enquire the current state of users
-and update them with its own.
-
-The following workflows can therefore be used to handle presence updates:
-
- 1 When a user first appears online their HS sends a message to each other HS
-   containing at least one user to be watched; each message carrying both a
-   notification of the sender's new online status, and a request to obtain and
-   watch the target users' presence information. This message implicitly
-   promises the sending HS will now push updates to the target HSes.
-
- 2 The target HSes then respond a single message each, containing the current
-   status of the requested user(s). These messages too implicitly promise the
-   target HSes will themselves push updates to the sending HS.
-
-   As these messages arrive at the sending user's HS they can be pushed to the
-   user's client(s), possibly batched again to ensure not too many small
-   messages which add extra protocol overheads.
-
-At this point, all the user's clients now have the current presence status
-information for this moment in time, and have promised to send each other
-updates in future.
-
- 3 The HS maintains two watchdog timers per peer HS it is exchanging presence
-   information with. The first timer should have a relatively small expiry
-   (perhaps 1 minute), and the second timer should have a much longer time
-   (perhaps 1 hour).
-
- 4 Any time any kind of message is received from a peer HS, the short-term
-   presence timer associated with it is reset.
-
- 5 Whenever either of these timers expires, an HS should push a status reminder
-   to the target HS whose timer has now expired, and request again from that
-   server the status of the subscribed users.
-
- 6 On receipt of one of these presence status reminders, an HS can reset both
-   of its presence watchdog timers.
-
-To avoid bursts of traffic, implementations should attempt to stagger the expiry
-of the longer-term watchdog timers for different peer HSes.
-
-When individual users actively change their status (either by explicit requests
-from clients, or inferred changes due to idle timers or client timeouts), the HS
-should batch up any status changes for some reasonable amount of time (10
-seconds to a minute). This allows for reduced protocol overheads in the case of
-multiple messages needing to be sent to the same peer HS; as is the likely
-scenario in many cases, such as a given human user having multiple user
-accounts.
-
-
-API Requirements
-================
-
-The data model presented here puts the following requirements on the APIs:
-
-Client-Server
--------------
-
-Requests that a client can make to its Home Server
-
- * get/set current presence state
-   Basic enumeration + ability to set a custom piece of text
-
- * report per-device idle time
-   After some (configurable?) idle time the device should send a single message
-   to set the idle duration. The HS can then infer a "start of idle" instant and
-   use that to keep the device idleness up to date. At some later point the
-   device can cancel this idleness.
-
- * report per-device type
-   Inform the server that this device is a "mobile" device, or perhaps some
-   other to-be-defined category of reduced capability that could be presented to
-   other users.
-
- * start/stop presence polling for my presence list
-   It is likely that these messages could be implicitly inferred by other
-   messages, though having explicit control is always useful.
-
- * get my presence list
-   [implicit poll start?]
-   It is possible that the HS doesn't yet have current presence information when
-   the client requests this. There should be a "don't know" type too.
-
- * add/remove a user to my presence list
-
-Server-Server
--------------
-
-Requests that Home Servers make to others
-
- * request permission to add a user to presence list
-
- * allow/deny a request to add to a presence list
-
- * perform a combined presence state push and subscription request
-   For each sending user ID, the message contains their new status.
-   For each receiving user ID, the message should contain an indication on
-   whether the sending server is also interested in receiving status from that
-   user; either as an immediate update response now, or as a promise to send
-   future updates.
-
-Server to Client
-----------------
-
-[[TODO(paul): There also needs to be some way for a user's HS to push status
-updates of the presence list to clients, but the general server-client event
-model currently lacks a space to do that.]]

docs/client-server/model/profiles.rst

@@ -1,232 +0,0 @@
-========
-Profiles
-========
-
-A description of Synapse user profile metadata support.
-
-
-Overview
-========
-
-Internally within Synapse users are referred to by an opaque ID, which consists
-of some opaque localpart combined with the domain name of their home server.
-Obviously this does not yield a very nice user experience; users would like to
-see readable names for other users that are in some way meaningful to them.
-Additionally, users like to be able to publish "profile" details to inform other
-users of other information about them.
-
-It is also conceivable that since we are attempting to provide a
-worldwide-applicable messaging system, that users may wish to present different
-subsets of information in their profile to different other people, from a
-privacy and permissions perspective.
-
-A Profile consists of a display name, an (optional?) avatar picture, and a set
-of other metadata fields that the user may wish to publish (email address, phone
-numbers, website URLs, etc...). We put no requirements on the display name other
-than it being a valid Unicode string. Since it is likely that users will end up
-having multiple accounts (perhaps by necessity of being hosted in multiple
-places, perhaps by choice of wanting multiple distinct identifies), it would be
-useful that a metadata field type exists that can refer to another Synapse User
-ID, so that clients and HSes can make use of this information.
-
-Metadata Fields
----------------
-
-[[TODO(paul): Likely this list is incomplete; more fields can be defined as we
-think of them. At the very least, any sort of supported ID for the 3rd Party ID
-servers should be accounted for here.]]
-
- * Synapse Directory Server username(s)
-
- * Email address
-
- * Phone number - classify "home"/"work"/"mobile"/custom?
- 
- * Twitter/Facebook/Google+/... social networks
-
- * Location - keep this deliberately vague to allow people to choose how
-     granular it is
- 
- * "Bio" information - date of birth, etc...
-
- * Synapse User ID of another account
-
- * Web URL
-
- * Freeform description text
-
-
-Visibility Permissions
-======================
-
-A home server implementation could offer the ability to set permissions on
-limited visibility of those fields. When another user requests access to the
-target user's profile, their own identity should form part of that request. The
-HS implementation can then decide which fields to make available to the
-requestor.
-
-A particular detail of implementation could allow the user to create one or more
-ACLs; where each list is granted permission to see a given set of non-public
-fields (compare to Google+ Circles) and contains a set of other people allowed
-to use it. By giving these ACLs strong identities within the HS, they can be
-referenced in communications with it, granting other users who encounter these
-the "ACL Token" to use the details in that ACL.
-
-If we further allow an ACL Token to be present on Room join requests or stored
-by 3PID servers, then users of these ACLs gain the extra convenience of not
-having to manually curate people in the access list; anyone in the room or with
-knowledge of the 3rd Party ID is automatically granted access. Every HS and
-client implementation would have to be aware of the existence of these ACL
-Token, and include them in requests if present, but not every HS implementation
-needs to actually provide the full permissions model. This can be used as a
-distinguishing feature among competing implementations. However, servers MUST
-NOT serve profile information from a cache if there is a chance that its limited
-understanding could lead to information leakage.
-
-
-Client Concerns of Multiple Accounts
-====================================
-
-Because a given person may want to have multiple Synapse User accounts, client
-implementations should allow the use of multiple accounts simultaneously
-(especially in the field of mobile phone clients, which generally don't support
-running distinct instances of the same application). Where features like address
-books, presence lists or rooms are presented, the client UI should remember to
-make distinct with user account is in use for each.
-
-
-Directory Servers
-=================
-
-Directory Servers can provide a forward mapping from human-readable names to
-User IDs. These can provide a service similar to giving domain-namespaced names
-for Rooms; in this case they can provide a way for a user to reference their
-User ID in some external form (e.g. that can be printed on a business card).
-
-The format for Synapse user name will consist of a localpart specific to the
-directory server, and the domain name of that directory server:
-
-  @localname:some.domain.name
-
-The localname is separated from the domain name using a colon, so as to ensure
-the localname can still contain periods, as users may want this for similarity
-to email addresses or the like, which typically can contain them. The format is
-also visually quite distinct from email addresses, phone numbers, etc... so
-hopefully reasonably "self-describing" when written on e.g. a business card
-without surrounding context.
-
-[[TODO(paul): we might have to think about this one - too close to email?
-  Twitter? Also it suggests a format scheme for room names of
-  #localname:domain.name, which I quite like]]
-
-Directory server administrators should be able to make some kind of policy
-decision on how these are allocated. Servers within some "closed" domain (such
-as company-specific ones) may wish to verify the validity of a mapping using
-their own internal mechanisms; "public" naming servers can operate on a FCFS
-basis. There are overlapping concerns here with the idea of the 3rd party
-identity servers as well, though in this specific case we are creating a new
-namespace to allocate names into.
-
-It would also be nice from a user experience perspective if the profile that a
-given name links to can also declare that name as part of its metadata.
-Furthermore as a security and consistency perspective it would be nice if each
-end (the directory server and the user's home server) check the validity of the
-mapping in some way. This needs investigation from a security perspective to
-ensure against spoofing.
-
-One such model may be that the user starts by declaring their intent to use a
-given user name link to their home server, which then contacts the directory
-service. At some point later (maybe immediately for "public open FCFS servers",
-maybe after some kind of human intervention for verification) the DS decides to
-honour this link, and includes it in its served output. It should also tell the
-HS of this fact, so that the HS can present this as fact when requested for the
-profile information. For efficiency, it may further wish to provide the HS with
-a cryptographically-signed certificate as proof, so the HS serving the profile
-can provide that too when asked, avoiding requesting HSes from constantly having
-to contact the DS to verify this mapping. (Note: This is similar to the security
-model often applied in DNS to verify PTR <-> A bidirectional mappings).
-
-
-Identity Servers
-================
-
-The identity servers should support the concept of pointing a 3PID being able to
-store an ACL Token as well as the main User ID. It is however, beyond scope to
-do any kind of verification that any third-party IDs that the profile is
-claiming match up to the 3PID mappings.
-
-
-User Interface and Expectations Concerns
-========================================
-
-Given the weak "security" of some parts of this model as compared to what users
-might expect, some care should be taken on how it is presented to users,
-specifically in the naming or other wording of user interface components.
-
-Most notably mere knowledge of an ACL Pointer is enough to read the information
-stored in it. It is possible that Home or Identity Servers could leak this
-information, allowing others to see it. This is a security-vs-convenience
-balancing choice on behalf of the user who would choose, or not, to make use of
-such a feature to publish their information.
-
-Additionally, unless some form of strong end-to-end user-based encryption is
-used, a user of ACLs for information privacy has to trust other home servers not
-to lie about the identify of the user requesting access to the Profile.
-
-
-API Requirements
-================
-
-The data model presented here puts the following requirements on the APIs:
-
-Client-Server
--------------
-
-Requests that a client can make to its Home Server
-
- * get/set my Display Name
-   This should return/take a simple "text/plain" field
-
- * get/set my Avatar URL
-   The avatar image data itself is not stored by this API; we'll just store a
-   URL to let the clients fetch it. Optionally HSes could integrate this with
-   their generic content attacmhent storage service, allowing a user to set
-   upload their profile Avatar and update the URL to point to it.
-
- * get/add/remove my metadata fields
-   Also we need to actually define types of metadata
-
- * get another user's Display Name / Avatar / metadata fields
-
-[[TODO(paul): At some later stage we should consider the API for:
-
- * get/set ACL permissions on my metadata fields
-
- * manage my ACL tokens
-]]
-
-Server-Server
--------------
-
-Requests that Home Servers make to others
-
- * get a user's Display Name / Avatar
-
- * get a user's full profile - name/avatar + MD fields
-   This request must allow for specifying the User ID of the requesting user,
-   for permissions purposes. It also needs to take into account any ACL Tokens
-   the requestor has.
-
- * push a change of Display Name to observers (overlaps with the presence API)
-
-Room Event PDU Types
---------------------
-
-Events that are pushed from Home Servers to other Home Servers or clients.
-
- * user Display Name change
- 
- * user Avatar change
-   [[TODO(paul): should the avatar image itself be stored in all the room
-   histories? maybe this event should just be a hint to clients that they should
-   re-fetch the avatar image]]

docs/client-server/model/protocol_examples.rst

@@ -1,64 +0,0 @@
-PUT /send/abc/ HTTP/1.1
-Host: ...
-Content-Length: ...
-Content-Type: application/json
-
-{
-    "origin": "localhost:5000",
-    "pdus": [
-        {
-            "content": {},
-            "context": "tng",
-            "depth": 12,
-            "is_state": false,
-            "origin": "localhost:5000",
-            "pdu_id": 1404381396854,
-            "pdu_type": "feedback",
-            "prev_pdus": [
-                [
-                    "1404381395883",
-                    "localhost:6000"
-                ]
-            ],
-            "ts": 1404381427581
-        }
-    ],
-    "prev_ids": [
-        "1404381396852"
-    ],
-    "ts": 1404381427823
-}
-
-HTTP/1.1 200 OK
-...
-
-======================================
-
-GET /pull/-1/ HTTP/1.1
-Host: ...
-Content-Length: 0
-
-HTTP/1.1 200 OK
-Content-Length: ...
-Content-Type: application/json
-
-{
-    origin: ...,
-    prev_ids: ...,
-    data: [
-        {
-            data_id: ...,
-            prev_pdus: [...],
-            depth: ...,
-            ts: ...,
-            context: ...,
-            origin: ...,
-            content: {
-                ...
-            }
-        },
-        ...,
-    ]
-}
-
-

docs/client-server/model/room-join-workflow.rst

@@ -1,113 +0,0 @@
-==================
-Room Join Workflow
-==================
-
-An outline of the workflows required when a user joins a room.
-
-Discovery
-=========
-
-To join a room, a user has to discover the room by some mechanism in order to
-obtain the (opaque) Room ID and a candidate list of likely home servers that
-contain it.
-
-Sending an Invitation
----------------------
-
-The most direct way a user discovers the existence of a room is from a
-invitation from some other user who is a member of that room.
-
-The inviter's HS sets the membership status of the invitee to "invited" in the
-"m.members" state key by sending a state update PDU. The HS then broadcasts this
-PDU among the existing members in the usual way. An invitation message is also
-sent to the invited user, containing the Room ID and the PDU ID of this
-invitation state change and potentially a list of some other home servers to use
-to accept the invite. The user's client can then choose to display it in some
-way to alert the user.
-
-[[TODO(paul): At present, no API has been designed or described to actually send
-that invite to the invited user. Likely it will be some facet of the larger
-user-user API required for presence, profile management, etc...]]
-
-Directory Service
------------------
-
-Alternatively, the user may discover the channel via a directory service; either
-by performing a name lookup, or some kind of browse or search acitivty. However
-this is performed, the end result is that the user's home server requests the
-Room ID and candidate list from the directory service.
-
-[[TODO(paul): At present, no API has been designed or described for this
-directory service]]
-
-
-Joining
-=======
-
-Once the ID and home servers are obtained, the user can then actually join the
-room.
-
-Accepting an Invite
--------------------
-
-If a user has received and accepted an invitation to join a room, the invitee's
-home server can now send an invite acceptance message to a chosen candidate
-server from the list given in the invitation, citing also the PDU ID of the
-invitation as "proof" of their invite. (This is required as due to late message
-propagation it could be the case that the acceptance is received before the
-invite by some servers). If this message is allowed by the candidate server, it
-generates a new PDU that updates the invitee's membership status to "joined",
-referring back to the acceptance PDU, and broadcasts that as a state change in
-the usual way. The newly-invited user is now a full member of the room, and
-state propagation proceeds as usual.
-
-Joining a Public Room
----------------------
-
-If a user has discovered the existence of a room they wish to join but does not
-have an active invitation, they can request to join it directly by sending a
-join message to a candidate server on the list provided by the directory
-service. As this list may be out of date, the HS should be prepared to retry
-other candidates if the chosen one is no longer aware of the room, because it
-has no users as members in it.
-
-Once a candidate server that is aware of the room has been found, it can
-broadcast an update PDU to add the member into the "m.members" key setting their
-state directly to "joined" (i.e. bypassing the two-phase invite semantics),
-remembering to include the new user's HS in that list.
-
-Knocking on a Semi-Public Room
-------------------------------
-
-If a user requests to join a room but the join mode of the room is "knock", the
-join is not immediately allowed. Instead, if the user wishes to proceed, they
-can instead post a "knock" message, which informs other members of the room that
-the would-be joiner wishes to become a member and sets their membership value to
-"knocked". If any of them wish to accept this, they can then send an invitation
-in the usual way described above. Knowing that the user has already knocked and
-expressed an interest in joining, the invited user's home server should
-immediately accept that invitation on the user's behalf, and go on to join the
-room in the usual way.
-
-[[NOTE(Erik): Though this may confuse users who expect 'X has joined' to
-actually be a user initiated action, i.e. they may expect that 'X' is actually
-looking at synapse right now?]]
-
-[[NOTE(paul): Yes, a fair point maybe we should suggest HSes don't do that, and
-just offer an invite to the user as normal]]
-
-Private and Non-Existent Rooms
-------------------------------
-
-If a user requests to join a room but the room is either unknown by the home
-server receiving the request, or is known by the join mode is "invite" and the
-user has not been invited, the server must respond that the room does not exist.
-This is to prevent leaking information about the existence and identity of
-private rooms.
-
-
-Outstanding Questions
-=====================
-
- * Do invitations or knocks time out and expire at some point? If so when? Time
-   is hard in distributed systems.

docs/client-server/model/rooms.rst

@@ -1,274 +0,0 @@
-===========
-Rooms Model
-===========
-
-A description of the general data model used to implement Rooms, and the
-user-level visible effects and implications.
-
-
-Overview
-========
-
-"Rooms" in Synapse are shared messaging channels over which all the participant
-users can exchange messages. Rooms have an opaque persistent identify, a
-globally-replicated set of state (consisting principly of a membership set of
-users, and other management and miscellaneous metadata), and a message history.
-
-
-Room Identity and Naming
-========================
-
-Rooms can be arbitrarily created by any user on any home server; at which point
-the home server will sign the message that creates the channel, and the
-fingerprint of this signature becomes the strong persistent identify of the
-room. This now identifies the room to any home server in the network regardless
-of its original origin. This allows the identify of the room to outlive any
-particular server. Subject to appropriate permissions [to be discussed later],
-any current member of a room can invite others to join it, can post messages
-that become part of its history, and can change the persistent state of the room
-(including its current set of permissions).
-
-Home servers can provide a directory service, allowing a lookup from a
-convenient human-readable form of room label to a room ID. This mapping is
-scoped to the particular home server domain and so simply represents that server
-administrator's opinion of what room should take that label; it does not have to
-be globally replicated and does not form part of the stored state of that room.
-
-This room name takes the form
-
-  #localname:some.domain.name
-
-for similarity and consistency with user names on directories.
-
-To join a room (and therefore to be allowed to inspect past history, post new
-messages to it, and read its state), a user must become aware of the room's
-fingerprint ID. There are two mechanisms to allow this:
-
- * An invite message from someone else in the room
-
- * A referral from a room directory service
-
-As room IDs are opaque and ephemeral, they can serve as a mechanism to create
-"ad-hoc" rooms deliberately unnamed, for small group-chats or even private
-one-to-one message exchange.
-
-
-Stored State and Permissions
-============================
-
-Every room has a globally-replicated set of stored state. This state is a set of
-key/value or key/subkey/value pairs. The value of every (sub)key is a
-JSON-representable object. The main key of a piece of stored state establishes
-its meaning; some keys store sub-keys to allow a sub-structure within them [more
-detail below]. Some keys have special meaning to Synapse, as they relate to
-management details of the room itself, storing such details as user membership,
-and permissions of users to alter the state of the room itself. Other keys may
-store information to present to users, which the system does not directly rely
-on. The key space itself is namespaced, allowing 3rd party extensions, subject
-to suitable permission.
-
-Permission management is based on the concept of "power-levels". Every user
-within a room has an integer assigned, being their "power-level" within that
-room. Along with its actual data value, each key (or subkey) also stores the
-minimum power-level a user must have in order to write to that key, the
-power-level of the last user who actually did write to it, and the PDU ID of
-that state change.
-
-To be accepted as valid, a change must NOT:
-
- * Be made by a user having a power-level lower than required to write to the
-   state key
-
- * Alter the required power-level for that state key to a value higher than the
-   user has
-
- * Increase that user's own power-level
-
- * Grant any other user a power-level higher than the level of the user making
-   the change
-
-[[TODO(paul): consider if relaxations should be allowed; e.g. is the current
-outright-winner allowed to raise their own level, to allow for "inflation"?]]
-
-
-Room State Keys
-===============
-
-[[TODO(paul): if this list gets too big it might become necessary to move it
-into its own doc]]
-
-The following keys have special semantics or meaning to Synapse itself:
-
-m.member (has subkeys)
-  Stores a sub-key for every Synapse User ID which is currently a member of
-  this room. Its value gives the membership type ("knocked", "invited",
-  "joined").
-
-m.power_levels
-  Stores a mapping from Synapse User IDs to their power-level in the room. If
-  they are not present in this mapping, the default applies.
-
-  The reason to store this as a single value rather than a value with subkeys
-  is that updates to it are atomic; allowing a number of colliding-edit
-  problems to be avoided.
-
-m.default_level
-  Gives the default power-level for members of the room that do not have one
-  specified in their membership key.
-
-m.invite_level
-  If set, gives the minimum power-level required for members to invite others
-  to join, or to accept knock requests from non-members requesting access. If
-  absent, then invites are not allowed. An invitation involves setting their
-  membership type to "invited", in addition to sending the invite message.
-
-m.join_rules
-  Encodes the rules on how non-members can join the room. Has the following
-  possibilities:
-    "public" - a non-member can join the room directly
-    "knock" - a non-member cannot join the room, but can post a single "knock"
-        message requesting access, which existing members may approve or deny
-    "invite" - non-members cannot join the room without an invite from an
-        existing member
-    "private" - nobody who is not in the 'may_join' list or already a member
-        may join by any mechanism
-
-  In any of the first three modes, existing members with sufficient permission
-  can send invites to non-members if allowed by the "m.invite_level" key. A
-  "private" room is not allowed to have the "m.invite_level" set.
-
-  A client may use the value of this key to hint at the user interface
-  expectations to provide; in particular, a private chat with one other use
-  might warrant specific handling in the client.
-
-m.may_join
-  A list of User IDs that are always allowed to join the room, regardless of any
-  of the prevailing join rules and invite levels. These apply even to private
-  rooms. These are stored in a single list with normal update-powerlevel
-  permissions applied; users cannot arbitrarily remove themselves from the list.
-
-m.add_state_level
-  The power-level required for a user to be able to add new state keys.
-
-m.public_history
-  If set and true, anyone can request the history of the room, without needing
-  to be a member of the room.
-
-m.archive_servers
-  For "public" rooms with public history, gives a list of home servers that
-  should be included in message distribution to the room, even if no users on
-  that server are present. These ensure that a public room can still persist
-  even if no users are currently members of it. This list should be consulted by
-  the dirctory servers as the candidate list they respond with.
-
-The following keys are provided by Synapse for user benefit, but their value is
-not otherwise used by Synapse.
-
-m.name
-  Stores a short human-readable name for the room, such that clients can display
-  to a user to assist in identifying which room is which.
-  
-  This name specifically is not the strong ID used by the message transport
-  system to refer to the room, because it may be changed from time to time.
-
-m.topic
-  Stores the current human-readable topic
-
-
-Room Creation Templates
-=======================
-
-A client (or maybe home server?) could offer a few templates for the creation of
-new rooms. For example, for a simple private one-to-one chat the channel could
-assign the creator a power-level of 1, requiring a level of 1 to invite, and
-needing an invite before members can join. An invite is then sent to the other
-party, and if accepted and the other user joins, the creator's power-level can
-now be reduced to 0. This now leaves a room with two participants in it being
-unable to add more.
-
-
-Rooms that Continue History
-===========================
-
-An option that could be considered for room creation, is that when a new room is
-created the creator could specify a PDU ID into an existing room, as the history
-continuation point. This would be stored as an extra piece of meta-data on the
-initial PDU of the room's creation. (It does not appear in the normal previous
-PDU linkage).
-
-This would allow users in rooms to "fork" a room, if it is considered that the
-conversations in the room no longer fit its original purpose, and wish to
-diverge. Existing permissions on the original room would continue to apply of
-course, for viewing that history. If both rooms are considered "public" we might
-also want to define a message to post into the original room to represent this
-fork point, and give a reference to the new room.
-
-
-User Direct Message Rooms
-=========================
-
-There is no need to build a mechanism for directly sending messages between
-users, because a room can handle this ability. To allow direct user-to-user chat
-messaging we simply need to be able to create rooms with specific set of
-permissions to allow this direct messaging.
-
-Between any given pair of user IDs that wish to exchange private messages, there
-will exist a single shared Room, created lazily by either side. These rooms will
-need a certain amount of special handling in both home servers and display on
-clients, but as much as possible should be treated by the lower layers of code
-the same as other rooms.
-
-Specially, a client would likely offer a special menu choice associated with
-another user (in room member lists, presence list, etc..) as "direct chat". That
-would perform all the necessary steps to create the private chat room. Receiving
-clients should display these in a special way too as the room name is not
-important; instead it should distinguish them on the Display Name of the other
-party.
-
-Home Servers will need a client-API option to request setting up a new user-user
-chat room, which will then need special handling within the server. It will
-create a new room with the following 
-
-  m.member: the proposing user
-  m.join_rules: "private"
-  m.may_join: both users
-  m.power_levels: empty
-  m.default_level: 0
-  m.add_state_level: 0
-  m.public_history: False
-
-Having created the room, it can send an invite message to the other user in the
-normal way - the room permissions state that no users can be set to the invited
-state, but because they're in the may_join list then they'd be allowed to join
-anyway.
-
-In this arrangement there is now a room with both users may join but neither has
-the power to invite any others. Both users now have the confidence that (at
-least within the messaging system itself) their messages remain private and
-cannot later be provably leaked to a third party. They can freely set the topic
-or name if they choose and add or edit any other state of the room. The update
-powerlevel of each of these fixed properties should be 1, to lock out the users
-from being able to alter them.
-
-
-Anti-Glare
-==========
-
-There exists the possibility of a race condition if two users who have no chat
-history with each other simultaneously create a room and invite the other to it.
-This is called a "glare" situation. There are two possible ideas for how to
-resolve this:
-
- * Each Home Server should persist the mapping of (user ID pair) to room ID, so
-   that duplicate requests can be suppressed. On receipt of a room creation
-   request that the HS thinks there already exists a room for, the invitation to
-   join can be rejected if:
-      a) the HS believes the sending user is already a member of the room (and
-         maybe their HS has forgotten this fact), or
-      b) the proposed room has a lexicographically-higher ID than the existing
-         room (to resolve true race condition conflicts)
-      
- * The room ID for a private 1:1 chat has a special form, determined by
-   concatenting the User IDs of both members in a deterministic order, such that
-   it doesn't matter which side creates it first; the HSes can just ignore
-   (or merge?) received PDUs that create the room twice.

docs/client-server/model/terminology.rst

@@ -1,86 +0,0 @@
-===========
-Terminology
-===========
-
-A list of definitions of specific terminology used among these documents.
-These terms were originally taken from the server-server documentation, and may
-not currently match the exact meanings used in other places; though as a
-medium-term goal we should encourage the unification of this terminology.
-
-
-Terms
-=====
-
-Backfilling:
-  The process of synchronising historic state from one home server to another,
-  to backfill the event storage so that scrollback can be presented to the
-  client(s).  (Formerly, and confusingly, called 'pagination')
-
-Context:
-  A single human-level entity of interest (currently, a chat room)
-
-EDU (Ephemeral Data Unit):
-  A message that relates directly to a given pair of home servers that are
-  exchanging it. EDUs are short-lived messages that related only to one single
-  pair of servers; they are not persisted for a long time and are not forwarded
-  on to other servers. Because of this, they have no internal ID nor previous
-  EDUs reference chain.
-
-Event:
-  A record of activity that records a single thing that happened on to a context
-  (currently, a chat room). These are the "chat messages" that Synapse makes
-  available.
-  [[NOTE(paul): The current server-server implementation calls these simply
-  "messages" but the term is too ambiguous here; I've called them Events]]
-
-PDU (Persistent Data Unit):
-  A message that relates to a single context, irrespective of the server that
-  is communicating it. PDUs either encode a single Event, or a single State
-  change. A PDU is referred to by its PDU ID; the pair of its origin server
-  and local reference from that server.
-
-PDU ID:
-  The pair of PDU Origin and PDU Reference, that together globally uniquely
-  refers to a specific PDU.
-
-PDU Origin:
-  The name of the origin server that generated a given PDU. This may not be the
-  server from which it has been received, due to the way they are copied around
-  from server to server. The origin always records the original server that
-  created it.
-
-PDU Reference:
-  A local ID used to refer to a specific PDU from a given origin server. These
-  references are opaque at the protocol level, but may optionally have some
-  structured meaning within a given origin server or implementation.
-
-Presence:
-  The concept of whether a user is currently online, how available they declare
-  they are, and so on. See also: doc/model/presence
-
-Profile:
-  A set of metadata about a user, such as a display name, provided for the
-  benefit of other users. See also: doc/model/profiles
-
-Room ID:
-  An opaque string (of as-yet undecided format) that identifies a particular
-  room and used in PDUs referring to it.
-
-Room Alias:
-  A human-readable string of the form #name:some.domain that users can use as a
-  pointer to identify a room; a Directory Server will map this to its Room ID
-
-State:
-  A set of metadata maintained about a Context, which is replicated among the
-  servers in addition to the history of Events.
-
-User ID:
-  A string of the form @localpart:domain.name that identifies a user for
-  wire-protocol purposes. The localpart is meaningless outside of a particular
-  home server. This takes a human-readable form that end-users can use directly
-  if they so wish, avoiding the 3PIDs.
-
-Transaction:
-  A message which relates to the communication between a given pair of servers.
-  A transaction contains possibly-empty lists of PDUs and EDUs.
-

docs/client-server/model/third-party-id.rst

@@ -1,108 +0,0 @@
-======================
-Third Party Identities
-======================
-
-A description of how email addresses, mobile phone numbers and other third
-party identifiers can be used to authenticate and discover users in Matrix.
-
-
-Overview
-========
-
-New users need to authenticate their account. An email or SMS text message can 
-be a convenient form of authentication. Users already have email addresses 
-and phone numbers for contacts in their address book. They want to communicate
-with those contacts in Matrix without manually exchanging a Matrix User ID with 
-them.
-
-Third Party IDs
----------------
-
-[[TODO(markjh): Describe the format of a 3PID]]
-
-
-Third Party ID Associations
----------------------------
-
-An Associaton is a binding between a Matrix User ID and a Third Party ID (3PID).
-Each 3PID can be associated with one Matrix User ID at a time.
-
-[[TODO(markjh): JSON format of the association.]]
-
-Verification 
-------------
-
-An Assocation must be verified by a trusted Verification Server. Email 
-addresses and phone numbers can be verified by sending a token to the address 
-which a client can supply to the verifier to confirm ownership.
-
-An email Verification Server may be capable of verifying all email 3PIDs or may
-be restricted to verifying addresses for a particular domain. A phone number
-Verification Server may be capable of verifying all phone numbers or may be
-restricted to verifying numbers for a given country or phone prefix.
-
-Verification Servers fulfil a similar role to Certificate Authorities in PKI so
-a similar level of vetting should be required before clients trust their
-signatures.
-
-A Verification Server may wish to check for existing Associations for a 3PID 
-before creating a new Association.
-
-Discovery
----------
-
-Users can discover Associations using a trusted Identity Server. Each 
-Association will be signed by the Identity Server. An Identity Server may store
-the entire space of Associations or may delegate to other Identity Servers when
-looking up Associations.
-
-Each Association returned from an Identity Server must be signed by a 
-Verification Server. Clients should check these signatures.
-
-Identity Servers fulfil a similar role to DNS servers.
-
-Privacy
--------
-
-A User may publish the association between their phone number and Matrix User ID
-on the Identity Server without publishing the number in their Profile hosted on
-their Home Server.
-
-Identity Servers should refrain from publishing reverse mappings and should 
-take steps, such as rate limiting, to prevent attackers enumerating the space of
-mappings.
-
-Federation
-==========
-
-Delegation
-----------
-
-Verification Servers could delegate signing to another server by issuing 
-certificate to that server allowing it to verify and sign a subset of 3PID on 
-its behalf. It would be necessary to provide a language for describing which
-subset of 3PIDs that server had authority to validate. Alternatively it could 
-delegate the verification step to another server but sign the resulting
-association itself.
-
-The 3PID space will have a heirachical structure like DNS so Identity Servers
-can delegate lookups to other servers. An Identity Server should be prepared 
-to host or delegate any valid association within the subset of the 3PIDs it is 
-resonsible for.
-
-Multiple Root Verification Servers
-----------------------------------
-
-There can be multiple root Verification Servers and an Association could be
-signed by multiple servers if different clients trust different subsets of
-the verification servers.
-
-Multiple Root Identity Servers
-------------------------------
-
-There can be be multiple root Identity Servers. Clients will add each
-Association to all root Identity Servers.
-
-[[TODO(markjh): Describe how clients find the list of root Identity Servers]]
-
-

docs/client-server/web/README

@@ -1,5 +0,0 @@
-To get this running:
-  ln -s ../swagger_matrix
-  python -m SimpleHTTPServer
-
-Go to http://localhost:8000/swagger.html

docs/client-server/web/files/backbone-min.js

@@ -1,38 +0,0 @@
-// Backbone.js 0.9.2
-
-// (c) 2010-2012 Jeremy Ashkenas, DocumentCloud Inc.
-// Backbone may be freely distributed under the MIT license.
-// For all details and documentation:
-// http://backbonejs.org
-(function(){var l=this,y=l.Backbone,z=Array.prototype.slice,A=Array.prototype.splice,g;g="undefined"!==typeof exports?exports:l.Backbone={};g.VERSION="0.9.2";var f=l._;!f&&"undefined"!==typeof require&&(f=require("underscore"));var i=l.jQuery||l.Zepto||l.ender;g.setDomLibrary=function(a){i=a};g.noConflict=function(){l.Backbone=y;return this};g.emulateHTTP=!1;g.emulateJSON=!1;var p=/\s+/,k=g.Events={on:function(a,b,c){var d,e,f,g,j;if(!b)return this;a=a.split(p);for(d=this._callbacks||(this._callbacks=
-{});e=a.shift();)f=(j=d[e])?j.tail:{},f.next=g={},f.context=c,f.callback=b,d[e]={tail:g,next:j?j.next:f};return this},off:function(a,b,c){var d,e,h,g,j,q;if(e=this._callbacks){if(!a&&!b&&!c)return delete this._callbacks,this;for(a=a?a.split(p):f.keys(e);d=a.shift();)if(h=e[d],delete e[d],h&&(b||c))for(g=h.tail;(h=h.next)!==g;)if(j=h.callback,q=h.context,b&&j!==b||c&&q!==c)this.on(d,j,q);return this}},trigger:function(a){var b,c,d,e,f,g;if(!(d=this._callbacks))return this;f=d.all;a=a.split(p);for(g=
-z.call(arguments,1);b=a.shift();){if(c=d[b])for(e=c.tail;(c=c.next)!==e;)c.callback.apply(c.context||this,g);if(c=f){e=c.tail;for(b=[b].concat(g);(c=c.next)!==e;)c.callback.apply(c.context||this,b)}}return this}};k.bind=k.on;k.unbind=k.off;var o=g.Model=function(a,b){var c;a||(a={});b&&b.parse&&(a=this.parse(a));if(c=n(this,"defaults"))a=f.extend({},c,a);b&&b.collection&&(this.collection=b.collection);this.attributes={};this._escapedAttributes={};this.cid=f.uniqueId("c");this.changed={};this._silent=
-{};this._pending={};this.set(a,{silent:!0});this.changed={};this._silent={};this._pending={};this._previousAttributes=f.clone(this.attributes);this.initialize.apply(this,arguments)};f.extend(o.prototype,k,{changed:null,_silent:null,_pending:null,idAttribute:"id",initialize:function(){},toJSON:function(){return f.clone(this.attributes)},get:function(a){return this.attributes[a]},escape:function(a){var b;if(b=this._escapedAttributes[a])return b;b=this.get(a);return this._escapedAttributes[a]=f.escape(null==
-b?"":""+b)},has:function(a){return null!=this.get(a)},set:function(a,b,c){var d,e;f.isObject(a)||null==a?(d=a,c=b):(d={},d[a]=b);c||(c={});if(!d)return this;d instanceof o&&(d=d.attributes);if(c.unset)for(e in d)d[e]=void 0;if(!this._validate(d,c))return!1;this.idAttribute in d&&(this.id=d[this.idAttribute]);var b=c.changes={},h=this.attributes,g=this._escapedAttributes,j=this._previousAttributes||{};for(e in d){a=d[e];if(!f.isEqual(h[e],a)||c.unset&&f.has(h,e))delete g[e],(c.silent?this._silent:
-b)[e]=!0;c.unset?delete h[e]:h[e]=a;!f.isEqual(j[e],a)||f.has(h,e)!=f.has(j,e)?(this.changed[e]=a,c.silent||(this._pending[e]=!0)):(delete this.changed[e],delete this._pending[e])}c.silent||this.change(c);return this},unset:function(a,b){(b||(b={})).unset=!0;return this.set(a,null,b)},clear:function(a){(a||(a={})).unset=!0;return this.set(f.clone(this.attributes),a)},fetch:function(a){var a=a?f.clone(a):{},b=this,c=a.success;a.success=function(d,e,f){if(!b.set(b.parse(d,f),a))return!1;c&&c(b,d)};
-a.error=g.wrapError(a.error,b,a);return(this.sync||g.sync).call(this,"read",this,a)},save:function(a,b,c){var d,e;f.isObject(a)||null==a?(d=a,c=b):(d={},d[a]=b);c=c?f.clone(c):{};if(c.wait){if(!this._validate(d,c))return!1;e=f.clone(this.attributes)}a=f.extend({},c,{silent:!0});if(d&&!this.set(d,c.wait?a:c))return!1;var h=this,i=c.success;c.success=function(a,b,e){b=h.parse(a,e);if(c.wait){delete c.wait;b=f.extend(d||{},b)}if(!h.set(b,c))return false;i?i(h,a):h.trigger("sync",h,a,c)};c.error=g.wrapError(c.error,
-h,c);b=this.isNew()?"create":"update";b=(this.sync||g.sync).call(this,b,this,c);c.wait&&this.set(e,a);return b},destroy:function(a){var a=a?f.clone(a):{},b=this,c=a.success,d=function(){b.trigger("destroy",b,b.collection,a)};if(this.isNew())return d(),!1;a.success=function(e){a.wait&&d();c?c(b,e):b.trigger("sync",b,e,a)};a.error=g.wrapError(a.error,b,a);var e=(this.sync||g.sync).call(this,"delete",this,a);a.wait||d();return e},url:function(){var a=n(this,"urlRoot")||n(this.collection,"url")||t();
-return this.isNew()?a:a+("/"==a.charAt(a.length-1)?"":"/")+encodeURIComponent(this.id)},parse:function(a){return a},clone:function(){return new this.constructor(this.attributes)},isNew:function(){return null==this.id},change:function(a){a||(a={});var b=this._changing;this._changing=!0;for(var c in this._silent)this._pending[c]=!0;var d=f.extend({},a.changes,this._silent);this._silent={};for(c in d)this.trigger("change:"+c,this,this.get(c),a);if(b)return this;for(;!f.isEmpty(this._pending);){this._pending=
-{};this.trigger("change",this,a);for(c in this.changed)!this._pending[c]&&!this._silent[c]&&delete this.changed[c];this._previousAttributes=f.clone(this.attributes)}this._changing=!1;return this},hasChanged:function(a){return!arguments.length?!f.isEmpty(this.changed):f.has(this.changed,a)},changedAttributes:function(a){if(!a)return this.hasChanged()?f.clone(this.changed):!1;var b,c=!1,d=this._previousAttributes,e;for(e in a)if(!f.isEqual(d[e],b=a[e]))(c||(c={}))[e]=b;return c},previous:function(a){return!arguments.length||
-!this._previousAttributes?null:this._previousAttributes[a]},previousAttributes:function(){return f.clone(this._previousAttributes)},isValid:function(){return!this.validate(this.attributes)},_validate:function(a,b){if(b.silent||!this.validate)return!0;var a=f.extend({},this.attributes,a),c=this.validate(a,b);if(!c)return!0;b&&b.error?b.error(this,c,b):this.trigger("error",this,c,b);return!1}});var r=g.Collection=function(a,b){b||(b={});b.model&&(this.model=b.model);b.comparator&&(this.comparator=b.comparator);
-this._reset();this.initialize.apply(this,arguments);a&&this.reset(a,{silent:!0,parse:b.parse})};f.extend(r.prototype,k,{model:o,initialize:function(){},toJSON:function(a){return this.map(function(b){return b.toJSON(a)})},add:function(a,b){var c,d,e,g,i,j={},k={},l=[];b||(b={});a=f.isArray(a)?a.slice():[a];c=0;for(d=a.length;c<d;c++){if(!(e=a[c]=this._prepareModel(a[c],b)))throw Error("Can't add an invalid model to a collection");g=e.cid;i=e.id;j[g]||this._byCid[g]||null!=i&&(k[i]||this._byId[i])?
-l.push(c):j[g]=k[i]=e}for(c=l.length;c--;)a.splice(l[c],1);c=0;for(d=a.length;c<d;c++)(e=a[c]).on("all",this._onModelEvent,this),this._byCid[e.cid]=e,null!=e.id&&(this._byId[e.id]=e);this.length+=d;A.apply(this.models,[null!=b.at?b.at:this.models.length,0].concat(a));this.comparator&&this.sort({silent:!0});if(b.silent)return this;c=0;for(d=this.models.length;c<d;c++)if(j[(e=this.models[c]).cid])b.index=c,e.trigger("add",e,this,b);return this},remove:function(a,b){var c,d,e,g;b||(b={});a=f.isArray(a)?
-a.slice():[a];c=0;for(d=a.length;c<d;c++)if(g=this.getByCid(a[c])||this.get(a[c]))delete this._byId[g.id],delete this._byCid[g.cid],e=this.indexOf(g),this.models.splice(e,1),this.length--,b.silent||(b.index=e,g.trigger("remove",g,this,b)),this._removeReference(g);return this},push:function(a,b){a=this._prepareModel(a,b);this.add(a,b);return a},pop:function(a){var b=this.at(this.length-1);this.remove(b,a);return b},unshift:function(a,b){a=this._prepareModel(a,b);this.add(a,f.extend({at:0},b));return a},
-shift:function(a){var b=this.at(0);this.remove(b,a);return b},get:function(a){return null==a?void 0:this._byId[null!=a.id?a.id:a]},getByCid:function(a){return a&&this._byCid[a.cid||a]},at:function(a){return this.models[a]},where:function(a){return f.isEmpty(a)?[]:this.filter(function(b){for(var c in a)if(a[c]!==b.get(c))return!1;return!0})},sort:function(a){a||(a={});if(!this.comparator)throw Error("Cannot sort a set without a comparator");var b=f.bind(this.comparator,this);1==this.comparator.length?
-this.models=this.sortBy(b):this.models.sort(b);a.silent||this.trigger("reset",this,a);return this},pluck:function(a){return f.map(this.models,function(b){return b.get(a)})},reset:function(a,b){a||(a=[]);b||(b={});for(var c=0,d=this.models.length;c<d;c++)this._removeReference(this.models[c]);this._reset();this.add(a,f.extend({silent:!0},b));b.silent||this.trigger("reset",this,b);return this},fetch:function(a){a=a?f.clone(a):{};void 0===a.parse&&(a.parse=!0);var b=this,c=a.success;a.success=function(d,
-e,f){b[a.add?"add":"reset"](b.parse(d,f),a);c&&c(b,d)};a.error=g.wrapError(a.error,b,a);return(this.sync||g.sync).call(this,"read",this,a)},create:function(a,b){var c=this,b=b?f.clone(b):{},a=this._prepareModel(a,b);if(!a)return!1;b.wait||c.add(a,b);var d=b.success;b.success=function(e,f){b.wait&&c.add(e,b);d?d(e,f):e.trigger("sync",a,f,b)};a.save(null,b);return a},parse:function(a){return a},chain:function(){return f(this.models).chain()},_reset:function(){this.length=0;this.models=[];this._byId=
-{};this._byCid={}},_prepareModel:function(a,b){b||(b={});a instanceof o?a.collection||(a.collection=this):(b.collection=this,a=new this.model(a,b),a._validate(a.attributes,b)||(a=!1));return a},_removeReference:function(a){this==a.collection&&delete a.collection;a.off("all",this._onModelEvent,this)},_onModelEvent:function(a,b,c,d){("add"==a||"remove"==a)&&c!=this||("destroy"==a&&this.remove(b,d),b&&a==="change:"+b.idAttribute&&(delete this._byId[b.previous(b.idAttribute)],this._byId[b.id]=b),this.trigger.apply(this,
-arguments))}});f.each("forEach,each,map,reduce,reduceRight,find,detect,filter,select,reject,every,all,some,any,include,contains,invoke,max,min,sortBy,sortedIndex,toArray,size,first,initial,rest,last,without,indexOf,shuffle,lastIndexOf,isEmpty,groupBy".split(","),function(a){r.prototype[a]=function(){return f[a].apply(f,[this.models].concat(f.toArray(arguments)))}});var u=g.Router=function(a){a||(a={});a.routes&&(this.routes=a.routes);this._bindRoutes();this.initialize.apply(this,arguments)},B=/:\w+/g,
-C=/\*\w+/g,D=/[-[\]{}()+?.,\\^$|#\s]/g;f.extend(u.prototype,k,{initialize:function(){},route:function(a,b,c){g.history||(g.history=new m);f.isRegExp(a)||(a=this._routeToRegExp(a));c||(c=this[b]);g.history.route(a,f.bind(function(d){d=this._extractParameters(a,d);c&&c.apply(this,d);this.trigger.apply(this,["route:"+b].concat(d));g.history.trigger("route",this,b,d)},this));return this},navigate:function(a,b){g.history.navigate(a,b)},_bindRoutes:function(){if(this.routes){var a=[],b;for(b in this.routes)a.unshift([b,
-this.routes[b]]);b=0;for(var c=a.length;b<c;b++)this.route(a[b][0],a[b][1],this[a[b][1]])}},_routeToRegExp:function(a){a=a.replace(D,"\\$&").replace(B,"([^/]+)").replace(C,"(.*?)");return RegExp("^"+a+"$")},_extractParameters:function(a,b){return a.exec(b).slice(1)}});var m=g.History=function(){this.handlers=[];f.bindAll(this,"checkUrl")},s=/^[#\/]/,E=/msie [\w.]+/;m.started=!1;f.extend(m.prototype,k,{interval:50,getHash:function(a){return(a=(a?a.location:window.location).href.match(/#(.*)$/))?a[1]:
-""},getFragment:function(a,b){if(null==a)if(this._hasPushState||b){var a=window.location.pathname,c=window.location.search;c&&(a+=c)}else a=this.getHash();a.indexOf(this.options.root)||(a=a.substr(this.options.root.length));return a.replace(s,"")},start:function(a){if(m.started)throw Error("Backbone.history has already been started");m.started=!0;this.options=f.extend({},{root:"/"},this.options,a);this._wantsHashChange=!1!==this.options.hashChange;this._wantsPushState=!!this.options.pushState;this._hasPushState=
-!(!this.options.pushState||!window.history||!window.history.pushState);var a=this.getFragment(),b=document.documentMode;if(b=E.exec(navigator.userAgent.toLowerCase())&&(!b||7>=b))this.iframe=i('<iframe src="javascript:0" tabindex="-1" />').hide().appendTo("body")[0].contentWindow,this.navigate(a);this._hasPushState?i(window).bind("popstate",this.checkUrl):this._wantsHashChange&&"onhashchange"in window&&!b?i(window).bind("hashchange",this.checkUrl):this._wantsHashChange&&(this._checkUrlInterval=setInterval(this.checkUrl,
-this.interval));this.fragment=a;a=window.location;b=a.pathname==this.options.root;if(this._wantsHashChange&&this._wantsPushState&&!this._hasPushState&&!b)return this.fragment=this.getFragment(null,!0),window.location.replace(this.options.root+"#"+this.fragment),!0;this._wantsPushState&&this._hasPushState&&b&&a.hash&&(this.fragment=this.getHash().replace(s,""),window.history.replaceState({},document.title,a.protocol+"//"+a.host+this.options.root+this.fragment));if(!this.options.silent)return this.loadUrl()},
-stop:function(){i(window).unbind("popstate",this.checkUrl).unbind("hashchange",this.checkUrl);clearInterval(this._checkUrlInterval);m.started=!1},route:function(a,b){this.handlers.unshift({route:a,callback:b})},checkUrl:function(){var a=this.getFragment();a==this.fragment&&this.iframe&&(a=this.getFragment(this.getHash(this.iframe)));if(a==this.fragment)return!1;this.iframe&&this.navigate(a);this.loadUrl()||this.loadUrl(this.getHash())},loadUrl:function(a){var b=this.fragment=this.getFragment(a);return f.any(this.handlers,
-function(a){if(a.route.test(b))return a.callback(b),!0})},navigate:function(a,b){if(!m.started)return!1;if(!b||!0===b)b={trigger:b};var c=(a||"").replace(s,"");this.fragment!=c&&(this._hasPushState?(0!=c.indexOf(this.options.root)&&(c=this.options.root+c),this.fragment=c,window.history[b.replace?"replaceState":"pushState"]({},document.title,c)):this._wantsHashChange?(this.fragment=c,this._updateHash(window.location,c,b.replace),this.iframe&&c!=this.getFragment(this.getHash(this.iframe))&&(b.replace||
-this.iframe.document.open().close(),this._updateHash(this.iframe.location,c,b.replace))):window.location.assign(this.options.root+a),b.trigger&&this.loadUrl(a))},_updateHash:function(a,b,c){c?a.replace(a.toString().replace(/(javascript:|#).*$/,"")+"#"+b):a.hash=b}});var v=g.View=function(a){this.cid=f.uniqueId("view");this._configure(a||{});this._ensureElement();this.initialize.apply(this,arguments);this.delegateEvents()},F=/^(\S+)\s*(.*)$/,w="model,collection,el,id,attributes,className,tagName".split(",");
-f.extend(v.prototype,k,{tagName:"div",$:function(a){return this.$el.find(a)},initialize:function(){},render:function(){return this},remove:function(){this.$el.remove();return this},make:function(a,b,c){a=document.createElement(a);b&&i(a).attr(b);c&&i(a).html(c);return a},setElement:function(a,b){this.$el&&this.undelegateEvents();this.$el=a instanceof i?a:i(a);this.el=this.$el[0];!1!==b&&this.delegateEvents();return this},delegateEvents:function(a){if(a||(a=n(this,"events"))){this.undelegateEvents();
-for(var b in a){var c=a[b];f.isFunction(c)||(c=this[a[b]]);if(!c)throw Error('Method "'+a[b]+'" does not exist');var d=b.match(F),e=d[1],d=d[2],c=f.bind(c,this),e=e+(".delegateEvents"+this.cid);""===d?this.$el.bind(e,c):this.$el.delegate(d,e,c)}}},undelegateEvents:function(){this.$el.unbind(".delegateEvents"+this.cid)},_configure:function(a){this.options&&(a=f.extend({},this.options,a));for(var b=0,c=w.length;b<c;b++){var d=w[b];a[d]&&(this[d]=a[d])}this.options=a},_ensureElement:function(){if(this.el)this.setElement(this.el,
-!1);else{var a=n(this,"attributes")||{};this.id&&(a.id=this.id);this.className&&(a["class"]=this.className);this.setElement(this.make(this.tagName,a),!1)}}});o.extend=r.extend=u.extend=v.extend=function(a,b){var c=G(this,a,b);c.extend=this.extend;return c};var H={create:"POST",update:"PUT","delete":"DELETE",read:"GET"};g.sync=function(a,b,c){var d=H[a];c||(c={});var e={type:d,dataType:"json"};c.url||(e.url=n(b,"url")||t());if(!c.data&&b&&("create"==a||"update"==a))e.contentType="application/json",
-e.data=JSON.stringify(b.toJSON());g.emulateJSON&&(e.contentType="application/x-www-form-urlencoded",e.data=e.data?{model:e.data}:{});if(g.emulateHTTP&&("PUT"===d||"DELETE"===d))g.emulateJSON&&(e.data._method=d),e.type="POST",e.beforeSend=function(a){a.setRequestHeader("X-HTTP-Method-Override",d)};"GET"!==e.type&&!g.emulateJSON&&(e.processData=!1);return i.ajax(f.extend(e,c))};g.wrapError=function(a,b,c){return function(d,e){e=d===b?e:d;a?a(b,e,c):b.trigger("error",b,e,c)}};var x=function(){},G=function(a,
-b,c){var d;d=b&&b.hasOwnProperty("constructor")?b.constructor:function(){a.apply(this,arguments)};f.extend(d,a);x.prototype=a.prototype;d.prototype=new x;b&&f.extend(d.prototype,b);c&&f.extend(d,c);d.prototype.constructor=d;d.__super__=a.prototype;return d},n=function(a,b){return!a||!a[b]?null:f.isFunction(a[b])?a[b]():a[b]},t=function(){throw Error('A "url" property or function must be specified');}}).call(this);

docs/client-server/web/files/css

@@ -1,16 +0,0 @@
-/* latin */
-@font-face {
-  font-family: 'Droid Sans';
-  font-style: normal;
-  font-weight: 400;
-  src: local('Droid Sans'), local('DroidSans'), url(http://fonts.gstatic.com/s/droidsans/v5/s-BiyweUPV0v-yRb-cjciPk_vArhqVIZ0nv9q090hN8.woff2) format('woff2');
-  unicode-range: U+0000-00FF, U+0131, U+0152-0153, U+02C6, U+02DA, U+02DC, U+2000-206F, U+2074, U+20AC, U+2212, U+2215, U+E0FF, U+EFFD, U+F000;
-}
-/* latin */
-@font-face {
-  font-family: 'Droid Sans';
-  font-style: normal;
-  font-weight: 700;
-  src: local('Droid Sans Bold'), local('DroidSans-Bold'), url(http://fonts.gstatic.com/s/droidsans/v5/EFpQQyG9GqCrobXxL-KRMYWiMMZ7xLd792ULpGE4W_Y.woff2) format('woff2');
-  unicode-range: U+0000-00FF, U+0131, U+0152-0153, U+02C6, U+02DA, U+02DC, U+2000-206F, U+2074, U+20AC, U+2212, U+2215, U+E0FF, U+EFFD, U+F000;
-}

docs/client-server/web/files/jquery.ba-bbq.min.js

@@ -1,18 +0,0 @@
-/*
- * jQuery BBQ: Back Button & Query Library - v1.2.1 - 2/17/2010
- * http://benalman.com/projects/jquery-bbq-plugin/
- * 
- * Copyright (c) 2010 "Cowboy" Ben Alman
- * Dual licensed under the MIT and GPL licenses.
- * http://benalman.com/about/license/
- */
-(function($,p){var i,m=Array.prototype.slice,r=decodeURIComponent,a=$.param,c,l,v,b=$.bbq=$.bbq||{},q,u,j,e=$.event.special,d="hashchange",A="querystring",D="fragment",y="elemUrlAttr",g="location",k="href",t="src",x=/^.*\?|#.*$/g,w=/^.*\#/,h,C={};function E(F){return typeof F==="string"}function B(G){var F=m.call(arguments,1);return function(){return G.apply(this,F.concat(m.call(arguments)))}}function n(F){return F.replace(/^[^#]*#?(.*)$/,"$1")}function o(F){return F.replace(/(?:^[^?#]*\?([^#]*).*$)?.*/,"$1")}function f(H,M,F,I,G){var O,L,K,N,J;if(I!==i){K=F.match(H?/^([^#]*)\#?(.*)$/:/^([^#?]*)\??([^#]*)(#?.*)/);J=K[3]||"";if(G===2&&E(I)){L=I.replace(H?w:x,"")}else{N=l(K[2]);I=E(I)?l[H?D:A](I):I;L=G===2?I:G===1?$.extend({},I,N):$.extend({},N,I);L=a(L);if(H){L=L.replace(h,r)}}O=K[1]+(H?"#":L||!K[1]?"?":"")+L+J}else{O=M(F!==i?F:p[g][k])}return O}a[A]=B(f,0,o);a[D]=c=B(f,1,n);c.noEscape=function(G){G=G||"";var F=$.map(G.split(""),encodeURIComponent);h=new RegExp(F.join("|"),"g")};c.noEscape(",/");$.deparam=l=function(I,F){var H={},G={"true":!0,"false":!1,"null":null};$.each(I.replace(/\+/g," ").split("&"),function(L,Q){var K=Q.split("="),P=r(K[0]),J,O=H,M=0,R=P.split("]["),N=R.length-1;if(/\[/.test(R[0])&&/\]$/.test(R[N])){R[N]=R[N].replace(/\]$/,"");R=R.shift().split("[").concat(R);N=R.length-1}else{N=0}if(K.length===2){J=r(K[1]);if(F){J=J&&!isNaN(J)?+J:J==="undefined"?i:G[J]!==i?G[J]:J}if(N){for(;M<=N;M++){P=R[M]===""?O.length:R[M];O=O[P]=M<N?O[P]||(R[M+1]&&isNaN(R[M+1])?{}:[]):J}}else{if($.isArray(H[P])){H[P].push(J)}else{if(H[P]!==i){H[P]=[H[P],J]}else{H[P]=J}}}}else{if(P){H[P]=F?i:""}}});return H};function z(H,F,G){if(F===i||typeof F==="boolean"){G=F;F=a[H?D:A]()}else{F=E(F)?F.replace(H?w:x,""):F}return l(F,G)}l[A]=B(z,0);l[D]=v=B(z,1);$[y]||($[y]=function(F){return $.extend(C,F)})({a:k,base:k,iframe:t,img:t,input:t,form:"action",link:k,script:t});j=$[y];function s(I,G,H,F){if(!E(H)&&typeof H!=="object"){F=H;H=G;G=i}return this.each(function(){var L=$(this),J=G||j()[(this.nodeName||"").toLowerCase()]||"",K=J&&L.attr(J)||"";L.attr(J,a[I](K,H,F))})}$.fn[A]=B(s,A);$.fn[D]=B(s,D);b.pushState=q=function(I,F){if(E(I)&&/^#/.test(I)&&F===i){F=2}var H=I!==i,G=c(p[g][k],H?I:{},H?F:2);p[g][k]=G+(/#/.test(G)?"":"#")};b.getState=u=function(F,G){return F===i||typeof F==="boolean"?v(F):v(G)[F]};b.removeState=function(F){var G={};if(F!==i){G=u();$.each($.isArray(F)?F:arguments,function(I,H){delete G[H]})}q(G,2)};e[d]=$.extend(e[d],{add:function(F){var H;function G(J){var I=J[D]=c();J.getState=function(K,L){return K===i||typeof K==="boolean"?l(I,K):l(I,L)[K]};H.apply(this,arguments)}if($.isFunction(F)){H=F;return G}else{H=F.handler;F.handler=G}}})})(jQuery,this);
-/*
- * jQuery hashchange event - v1.2 - 2/11/2010
- * http://benalman.com/projects/jquery-hashchange-plugin/
- * 
- * Copyright (c) 2010 "Cowboy" Ben Alman
- * Dual licensed under the MIT and GPL licenses.
- * http://benalman.com/about/license/
- */
-(function($,i,b){var j,k=$.event.special,c="location",d="hashchange",l="href",f=$.browser,g=document.documentMode,h=f.msie&&(g===b||g<8),e="on"+d in i&&!h;function a(m){m=m||i[c][l];return m.replace(/^[^#]*#?(.*)$/,"$1")}$[d+"Delay"]=100;k[d]=$.extend(k[d],{setup:function(){if(e){return false}$(j.start)},teardown:function(){if(e){return false}$(j.stop)}});j=(function(){var m={},r,n,o,q;function p(){o=q=function(s){return s};if(h){n=$('<iframe src="javascript:0"/>').hide().insertAfter("body")[0].contentWindow;q=function(){return a(n.document[c][l])};o=function(u,s){if(u!==s){var t=n.document;t.open().close();t[c].hash="#"+u}};o(a())}}m.start=function(){if(r){return}var t=a();o||p();(function s(){var v=a(),u=q(t);if(v!==t){o(t=v,u);$(i).trigger(d)}else{if(u!==t){i[c][l]=i[c][l].replace(/#.*/,"")+"#"+u}}r=setTimeout(s,$[d+"Delay"])})()};m.stop=function(){if(!n){r&&clearTimeout(r);r=0}};return m})()})(jQuery,this);

docs/client-server/web/files/jquery.slideto.min.js

@@ -1 +0,0 @@
-(function(b){b.fn.slideto=function(a){a=b.extend({slide_duration:"slow",highlight_duration:3E3,highlight:true,highlight_color:"#FFFF99"},a);return this.each(function(){obj=b(this);b("body").animate({scrollTop:obj.offset().top},a.slide_duration,function(){a.highlight&&b.ui.version&&obj.effect("highlight",{color:a.highlight_color},a.highlight_duration)})})}})(jQuery);

docs/client-server/web/files/jquery.wiggle.min.js

@@ -1,8 +0,0 @@
-/*
-jQuery Wiggle
-Author: WonderGroup, Jordan Thomas
-URL: http://labs.wondergroup.com/demos/mini-ui/index.html
-License: MIT (http://en.wikipedia.org/wiki/MIT_License)
-*/
-jQuery.fn.wiggle=function(o){var d={speed:50,wiggles:3,travel:5,callback:null};var o=jQuery.extend(d,o);return this.each(function(){var cache=this;var wrap=jQuery(this).wrap('<div class="wiggle-wrap"></div>').css("position","relative");var calls=0;for(i=1;i<=o.wiggles;i++){jQuery(this).animate({left:"-="+o.travel},o.speed).animate({left:"+="+o.travel*2},o.speed*2).animate({left:"-="+o.travel},o.speed,function(){calls++;if(jQuery(cache).parent().hasClass('wiggle-wrap')){jQuery(cache).parent().replaceWith(cache);}
-if(calls==o.wiggles&&jQuery.isFunction(o.callback)){o.callback();}});}});};

docs/client-server/web/files/reset.css

@@ -1,125 +0,0 @@
-/* http://meyerweb.com/eric/tools/css/reset/ v2.0 | 20110126 */
-html,
-body,
-div,
-span,
-applet,
-object,
-iframe,
-h1,
-h2,
-h3,
-h4,
-h5,
-h6,
-p,
-blockquote,
-pre,
-a,
-abbr,
-acronym,
-address,
-big,
-cite,
-code,
-del,
-dfn,
-em,
-img,
-ins,
-kbd,
-q,
-s,
-samp,
-small,
-strike,
-strong,
-sub,
-sup,
-tt,
-var,
-b,
-u,
-i,
-center,
-dl,
-dt,
-dd,
-ol,
-ul,
-li,
-fieldset,
-form,
-label,
-legend,
-table,
-caption,
-tbody,
-tfoot,
-thead,
-tr,
-th,
-td,
-article,
-aside,
-canvas,
-details,
-embed,
-figure,
-figcaption,
-footer,
-header,
-hgroup,
-menu,
-nav,
-output,
-ruby,
-section,
-summary,
-time,
-mark,
-audio,
-video {
-  margin: 0;
-  padding: 0;
-  border: 0;
-  font-size: 100%;
-  font: inherit;
-  vertical-align: baseline;
-}
-/* HTML5 display-role reset for older browsers */
-article,
-aside,
-details,
-figcaption,
-figure,
-footer,
-header,
-hgroup,
-menu,
-nav,
-section {
-  display: block;
-}
-body {
-  line-height: 1;
-}
-ol,
-ul {
-  list-style: none;
-}
-blockquote,
-q {
-  quotes: none;
-}
-blockquote:before,
-blockquote:after,
-q:before,
-q:after {
-  content: '';
-  content: none;
-}
-table {
-  border-collapse: collapse;
-  border-spacing: 0;
-}

docs/client-server/web/files/swagger-oauth.js

@@ -1,211 +0,0 @@
-var appName;
-var popupMask;
-var popupDialog;
-var clientId;
-var realm;
-
-function handleLogin() {
-  var scopes = [];
-
-  if(window.swaggerUi.api.authSchemes 
-    && window.swaggerUi.api.authSchemes.oauth2
-    && window.swaggerUi.api.authSchemes.oauth2.scopes) {
-    scopes = window.swaggerUi.api.authSchemes.oauth2.scopes;
-  }
-
-  if(window.swaggerUi.api
-    && window.swaggerUi.api.info) {
-    appName = window.swaggerUi.api.info.title;
-  }
-
-  if(popupDialog.length > 0)
-    popupDialog = popupDialog.last();
-  else {
-    popupDialog = $(
-      [
-        '<div class="api-popup-dialog">',
-        '<div class="api-popup-title">Select OAuth2.0 Scopes</div>',
-        '<div class="api-popup-content">',
-          '<p>Scopes are used to grant an application different levels of access to data on behalf of the end user. Each API may declare one or more scopes.',
-            '<a href="#">Learn how to use</a>',
-          '</p>',
-          '<p><strong>' + appName + '</strong> API requires the following scopes. Select which ones you want to grant to Swagger UI.</p>',
-          '<ul class="api-popup-scopes">',
-          '</ul>',
-          '<p class="error-msg"></p>',
-          '<div class="api-popup-actions"><button class="api-popup-authbtn api-button green" type="button">Authorize</button><button class="api-popup-cancel api-button gray" type="button">Cancel</button></div>',
-        '</div>',
-        '</div>'].join(''));
-    $(document.body).append(popupDialog);
-
-    popup = popupDialog.find('ul.api-popup-scopes').empty();
-    for (i = 0; i < scopes.length; i ++) {
-      scope = scopes[i];
-      str = '<li><input type="checkbox" id="scope_' + i + '" scope="' + scope.scope + '"/>' + '<label for="scope_' + i + '">' + scope.scope;
-      if (scope.description) {
-        str += '<br/><span class="api-scope-desc">' + scope.description + '</span>';
-      }
-      str += '</label></li>';
-      popup.append(str);
-    }
-  }
-
-  var $win = $(window),
-    dw = $win.width(),
-    dh = $win.height(),
-    st = $win.scrollTop(),
-    dlgWd = popupDialog.outerWidth(),
-    dlgHt = popupDialog.outerHeight(),
-    top = (dh -dlgHt)/2 + st,
-    left = (dw - dlgWd)/2;
-
-  popupDialog.css({
-    top: (top < 0? 0 : top) + 'px',
-    left: (left < 0? 0 : left) + 'px'
-  });
-
-  popupDialog.find('button.api-popup-cancel').click(function() {
-    popupMask.hide();
-    popupDialog.hide();
-  });
-  popupDialog.find('button.api-popup-authbtn').click(function() {
-    popupMask.hide();
-    popupDialog.hide();
-
-    var authSchemes = window.swaggerUi.api.authSchemes;
-    var host = window.location;
-    var redirectUrl = host.protocol + '//' + host.host + "/o2c.html";
-    var url = null;
-
-    var p = window.swaggerUi.api.authSchemes;
-    for (var key in p) {
-      if (p.hasOwnProperty(key)) {
-        var o = p[key].grantTypes;
-        for(var t in o) {
-          if(o.hasOwnProperty(t) && t === 'implicit') {
-            var dets = o[t];
-            url = dets.loginEndpoint.url + "?response_type=token";
-            window.swaggerUi.tokenName = dets.tokenName;
-          }
-        }
-      }
-    }
-    var scopes = []
-    var o = $('.api-popup-scopes').find('input:checked');
-
-    for(k =0; k < o.length; k++) {
-      scopes.push($(o[k]).attr("scope"));
-    }
-
-    window.enabledScopes=scopes;
-
-    url += '&redirect_uri=' + encodeURIComponent(redirectUrl);
-    url += '&realm=' + encodeURIComponent(realm);
-    url += '&client_id=' + encodeURIComponent(clientId);
-    url += '&scope=' + encodeURIComponent(scopes);
-
-    window.open(url);
-  });
-
-  popupMask.show();
-  popupDialog.show();
-  return;
-}
-
-
-function handleLogout() {
-  for(key in window.authorizations.authz){
-    window.authorizations.remove(key)
-  }
-  window.enabledScopes = null;
-  $('.api-ic.ic-on').addClass('ic-off');
-  $('.api-ic.ic-on').removeClass('ic-on');
-
-  // set the info box
-  $('.api-ic.ic-warning').addClass('ic-error');
-  $('.api-ic.ic-warning').removeClass('ic-warning');
-}
-
-function initOAuth(opts) {
-  var o = (opts||{});
-  var errors = [];
-
-  appName = (o.appName||errors.push("missing appName"));
-  popupMask = (o.popupMask||$('#api-common-mask'));
-  popupDialog = (o.popupDialog||$('.api-popup-dialog'));
-  clientId = (o.clientId||errors.push("missing client id"));
-  realm = (o.realm||errors.push("missing realm"));
-
-  if(errors.length > 0){
-    log("auth unable initialize oauth: " + errors);
-    return;
-  }
-
-  $('pre code').each(function(i, e) {hljs.highlightBlock(e)});
-  $('.api-ic').click(function(s) {
-    if($(s.target).hasClass('ic-off'))
-      handleLogin();
-    else {
-      handleLogout();
-    }
-    false;
-  });
-}
-
-function onOAuthComplete(token) {
-  if(token) {
-    if(token.error) {
-      var checkbox = $('input[type=checkbox],.secured')
-      checkbox.each(function(pos){
-        checkbox[pos].checked = false;
-      });
-      alert(token.error);
-    }
-    else {
-      var b = token[window.swaggerUi.tokenName];
-      if(b){
-        // if all roles are satisfied
-        var o = null;
-        $.each($('.auth #api_information_panel'), function(k, v) {
-          var children = v;
-          if(children && children.childNodes) {
-            var requiredScopes = [];
-            $.each((children.childNodes), function (k1, v1){
-              var inner = v1.innerHTML;
-              if(inner)
-                requiredScopes.push(inner);
-            });
-            var diff = [];
-            for(var i=0; i < requiredScopes.length; i++) {
-              var s = requiredScopes[i];
-              if(window.enabledScopes && window.enabledScopes.indexOf(s) == -1) {
-                diff.push(s);
-              }
-            }
-            if(diff.length > 0){
-              o = v.parentNode;
-              $(o.parentNode).find('.api-ic.ic-on').addClass('ic-off');
-              $(o.parentNode).find('.api-ic.ic-on').removeClass('ic-on');
-
-              // sorry, not all scopes are satisfied
-              $(o).find('.api-ic').addClass('ic-warning');
-              $(o).find('.api-ic').removeClass('ic-error');
-            }
-            else {
-              o = v.parentNode;
-              $(o.parentNode).find('.api-ic.ic-off').addClass('ic-on');
-              $(o.parentNode).find('.api-ic.ic-off').removeClass('ic-off');
-
-              // all scopes are satisfied
-              $(o).find('.api-ic').addClass('ic-info');
-              $(o).find('.api-ic').removeClass('ic-warning');
-              $(o).find('.api-ic').removeClass('ic-error');          
-            }
-          }
-        });
-
-        window.authorizations.add("oauth2", new ApiKeyAuthorization("Authorization", "Bearer " + b, "header"));
-      }
-    }
-  }
-}

docs/client-server/web/files/underscore-min.js

@@ -1,32 +0,0 @@
-// Underscore.js 1.3.3
-// (c) 2009-2012 Jeremy Ashkenas, DocumentCloud Inc.
-// Underscore is freely distributable under the MIT license.
-// Portions of Underscore are inspired or borrowed from Prototype,
-// Oliver Steele's Functional, and John Resig's Micro-Templating.
-// For all details and documentation:
-// http://documentcloud.github.com/underscore
-(function(){function r(a,c,d){if(a===c)return 0!==a||1/a==1/c;if(null==a||null==c)return a===c;a._chain&&(a=a._wrapped);c._chain&&(c=c._wrapped);if(a.isEqual&&b.isFunction(a.isEqual))return a.isEqual(c);if(c.isEqual&&b.isFunction(c.isEqual))return c.isEqual(a);var e=l.call(a);if(e!=l.call(c))return!1;switch(e){case "[object String]":return a==""+c;case "[object Number]":return a!=+a?c!=+c:0==a?1/a==1/c:a==+c;case "[object Date]":case "[object Boolean]":return+a==+c;case "[object RegExp]":return a.source==
-c.source&&a.global==c.global&&a.multiline==c.multiline&&a.ignoreCase==c.ignoreCase}if("object"!=typeof a||"object"!=typeof c)return!1;for(var f=d.length;f--;)if(d[f]==a)return!0;d.push(a);var f=0,g=!0;if("[object Array]"==e){if(f=a.length,g=f==c.length)for(;f--&&(g=f in a==f in c&&r(a[f],c[f],d)););}else{if("constructor"in a!="constructor"in c||a.constructor!=c.constructor)return!1;for(var h in a)if(b.has(a,h)&&(f++,!(g=b.has(c,h)&&r(a[h],c[h],d))))break;if(g){for(h in c)if(b.has(c,h)&&!f--)break;
-g=!f}}d.pop();return g}var s=this,I=s._,o={},k=Array.prototype,p=Object.prototype,i=k.slice,J=k.unshift,l=p.toString,K=p.hasOwnProperty,y=k.forEach,z=k.map,A=k.reduce,B=k.reduceRight,C=k.filter,D=k.every,E=k.some,q=k.indexOf,F=k.lastIndexOf,p=Array.isArray,L=Object.keys,t=Function.prototype.bind,b=function(a){return new m(a)};"undefined"!==typeof exports?("undefined"!==typeof module&&module.exports&&(exports=module.exports=b),exports._=b):s._=b;b.VERSION="1.3.3";var j=b.each=b.forEach=function(a,
-c,d){if(a!=null)if(y&&a.forEach===y)a.forEach(c,d);else if(a.length===+a.length)for(var e=0,f=a.length;e<f;e++){if(e in a&&c.call(d,a[e],e,a)===o)break}else for(e in a)if(b.has(a,e)&&c.call(d,a[e],e,a)===o)break};b.map=b.collect=function(a,c,b){var e=[];if(a==null)return e;if(z&&a.map===z)return a.map(c,b);j(a,function(a,g,h){e[e.length]=c.call(b,a,g,h)});if(a.length===+a.length)e.length=a.length;return e};b.reduce=b.foldl=b.inject=function(a,c,d,e){var f=arguments.length>2;a==null&&(a=[]);if(A&&
-a.reduce===A){e&&(c=b.bind(c,e));return f?a.reduce(c,d):a.reduce(c)}j(a,function(a,b,i){if(f)d=c.call(e,d,a,b,i);else{d=a;f=true}});if(!f)throw new TypeError("Reduce of empty array with no initial value");return d};b.reduceRight=b.foldr=function(a,c,d,e){var f=arguments.length>2;a==null&&(a=[]);if(B&&a.reduceRight===B){e&&(c=b.bind(c,e));return f?a.reduceRight(c,d):a.reduceRight(c)}var g=b.toArray(a).reverse();e&&!f&&(c=b.bind(c,e));return f?b.reduce(g,c,d,e):b.reduce(g,c)};b.find=b.detect=function(a,
-c,b){var e;G(a,function(a,g,h){if(c.call(b,a,g,h)){e=a;return true}});return e};b.filter=b.select=function(a,c,b){var e=[];if(a==null)return e;if(C&&a.filter===C)return a.filter(c,b);j(a,function(a,g,h){c.call(b,a,g,h)&&(e[e.length]=a)});return e};b.reject=function(a,c,b){var e=[];if(a==null)return e;j(a,function(a,g,h){c.call(b,a,g,h)||(e[e.length]=a)});return e};b.every=b.all=function(a,c,b){var e=true;if(a==null)return e;if(D&&a.every===D)return a.every(c,b);j(a,function(a,g,h){if(!(e=e&&c.call(b,
-a,g,h)))return o});return!!e};var G=b.some=b.any=function(a,c,d){c||(c=b.identity);var e=false;if(a==null)return e;if(E&&a.some===E)return a.some(c,d);j(a,function(a,b,h){if(e||(e=c.call(d,a,b,h)))return o});return!!e};b.include=b.contains=function(a,c){var b=false;if(a==null)return b;if(q&&a.indexOf===q)return a.indexOf(c)!=-1;return b=G(a,function(a){return a===c})};b.invoke=function(a,c){var d=i.call(arguments,2);return b.map(a,function(a){return(b.isFunction(c)?c||a:a[c]).apply(a,d)})};b.pluck=
-function(a,c){return b.map(a,function(a){return a[c]})};b.max=function(a,c,d){if(!c&&b.isArray(a)&&a[0]===+a[0])return Math.max.apply(Math,a);if(!c&&b.isEmpty(a))return-Infinity;var e={computed:-Infinity};j(a,function(a,b,h){b=c?c.call(d,a,b,h):a;b>=e.computed&&(e={value:a,computed:b})});return e.value};b.min=function(a,c,d){if(!c&&b.isArray(a)&&a[0]===+a[0])return Math.min.apply(Math,a);if(!c&&b.isEmpty(a))return Infinity;var e={computed:Infinity};j(a,function(a,b,h){b=c?c.call(d,a,b,h):a;b<e.computed&&
-(e={value:a,computed:b})});return e.value};b.shuffle=function(a){var b=[],d;j(a,function(a,f){d=Math.floor(Math.random()*(f+1));b[f]=b[d];b[d]=a});return b};b.sortBy=function(a,c,d){var e=b.isFunction(c)?c:function(a){return a[c]};return b.pluck(b.map(a,function(a,b,c){return{value:a,criteria:e.call(d,a,b,c)}}).sort(function(a,b){var c=a.criteria,d=b.criteria;return c===void 0?1:d===void 0?-1:c<d?-1:c>d?1:0}),"value")};b.groupBy=function(a,c){var d={},e=b.isFunction(c)?c:function(a){return a[c]};
-j(a,function(a,b){var c=e(a,b);(d[c]||(d[c]=[])).push(a)});return d};b.sortedIndex=function(a,c,d){d||(d=b.identity);for(var e=0,f=a.length;e<f;){var g=e+f>>1;d(a[g])<d(c)?e=g+1:f=g}return e};b.toArray=function(a){return!a?[]:b.isArray(a)||b.isArguments(a)?i.call(a):a.toArray&&b.isFunction(a.toArray)?a.toArray():b.values(a)};b.size=function(a){return b.isArray(a)?a.length:b.keys(a).length};b.first=b.head=b.take=function(a,b,d){return b!=null&&!d?i.call(a,0,b):a[0]};b.initial=function(a,b,d){return i.call(a,
-0,a.length-(b==null||d?1:b))};b.last=function(a,b,d){return b!=null&&!d?i.call(a,Math.max(a.length-b,0)):a[a.length-1]};b.rest=b.tail=function(a,b,d){return i.call(a,b==null||d?1:b)};b.compact=function(a){return b.filter(a,function(a){return!!a})};b.flatten=function(a,c){return b.reduce(a,function(a,e){if(b.isArray(e))return a.concat(c?e:b.flatten(e));a[a.length]=e;return a},[])};b.without=function(a){return b.difference(a,i.call(arguments,1))};b.uniq=b.unique=function(a,c,d){var d=d?b.map(a,d):a,
-e=[];a.length<3&&(c=true);b.reduce(d,function(d,g,h){if(c?b.last(d)!==g||!d.length:!b.include(d,g)){d.push(g);e.push(a[h])}return d},[]);return e};b.union=function(){return b.uniq(b.flatten(arguments,true))};b.intersection=b.intersect=function(a){var c=i.call(arguments,1);return b.filter(b.uniq(a),function(a){return b.every(c,function(c){return b.indexOf(c,a)>=0})})};b.difference=function(a){var c=b.flatten(i.call(arguments,1),true);return b.filter(a,function(a){return!b.include(c,a)})};b.zip=function(){for(var a=
-i.call(arguments),c=b.max(b.pluck(a,"length")),d=Array(c),e=0;e<c;e++)d[e]=b.pluck(a,""+e);return d};b.indexOf=function(a,c,d){if(a==null)return-1;var e;if(d){d=b.sortedIndex(a,c);return a[d]===c?d:-1}if(q&&a.indexOf===q)return a.indexOf(c);d=0;for(e=a.length;d<e;d++)if(d in a&&a[d]===c)return d;return-1};b.lastIndexOf=function(a,b){if(a==null)return-1;if(F&&a.lastIndexOf===F)return a.lastIndexOf(b);for(var d=a.length;d--;)if(d in a&&a[d]===b)return d;return-1};b.range=function(a,b,d){if(arguments.length<=
-1){b=a||0;a=0}for(var d=arguments[2]||1,e=Math.max(Math.ceil((b-a)/d),0),f=0,g=Array(e);f<e;){g[f++]=a;a=a+d}return g};var H=function(){};b.bind=function(a,c){var d,e;if(a.bind===t&&t)return t.apply(a,i.call(arguments,1));if(!b.isFunction(a))throw new TypeError;e=i.call(arguments,2);return d=function(){if(!(this instanceof d))return a.apply(c,e.concat(i.call(arguments)));H.prototype=a.prototype;var b=new H,g=a.apply(b,e.concat(i.call(arguments)));return Object(g)===g?g:b}};b.bindAll=function(a){var c=
-i.call(arguments,1);c.length==0&&(c=b.functions(a));j(c,function(c){a[c]=b.bind(a[c],a)});return a};b.memoize=function(a,c){var d={};c||(c=b.identity);return function(){var e=c.apply(this,arguments);return b.has(d,e)?d[e]:d[e]=a.apply(this,arguments)}};b.delay=function(a,b){var d=i.call(arguments,2);return setTimeout(function(){return a.apply(null,d)},b)};b.defer=function(a){return b.delay.apply(b,[a,1].concat(i.call(arguments,1)))};b.throttle=function(a,c){var d,e,f,g,h,i,j=b.debounce(function(){h=
-g=false},c);return function(){d=this;e=arguments;f||(f=setTimeout(function(){f=null;h&&a.apply(d,e);j()},c));g?h=true:i=a.apply(d,e);j();g=true;return i}};b.debounce=function(a,b,d){var e;return function(){var f=this,g=arguments;d&&!e&&a.apply(f,g);clearTimeout(e);e=setTimeout(function(){e=null;d||a.apply(f,g)},b)}};b.once=function(a){var b=false,d;return function(){if(b)return d;b=true;return d=a.apply(this,arguments)}};b.wrap=function(a,b){return function(){var d=[a].concat(i.call(arguments,0));
-return b.apply(this,d)}};b.compose=function(){var a=arguments;return function(){for(var b=arguments,d=a.length-1;d>=0;d--)b=[a[d].apply(this,b)];return b[0]}};b.after=function(a,b){return a<=0?b():function(){if(--a<1)return b.apply(this,arguments)}};b.keys=L||function(a){if(a!==Object(a))throw new TypeError("Invalid object");var c=[],d;for(d in a)b.has(a,d)&&(c[c.length]=d);return c};b.values=function(a){return b.map(a,b.identity)};b.functions=b.methods=function(a){var c=[],d;for(d in a)b.isFunction(a[d])&&
-c.push(d);return c.sort()};b.extend=function(a){j(i.call(arguments,1),function(b){for(var d in b)a[d]=b[d]});return a};b.pick=function(a){var c={};j(b.flatten(i.call(arguments,1)),function(b){b in a&&(c[b]=a[b])});return c};b.defaults=function(a){j(i.call(arguments,1),function(b){for(var d in b)a[d]==null&&(a[d]=b[d])});return a};b.clone=function(a){return!b.isObject(a)?a:b.isArray(a)?a.slice():b.extend({},a)};b.tap=function(a,b){b(a);return a};b.isEqual=function(a,b){return r(a,b,[])};b.isEmpty=
-function(a){if(a==null)return true;if(b.isArray(a)||b.isString(a))return a.length===0;for(var c in a)if(b.has(a,c))return false;return true};b.isElement=function(a){return!!(a&&a.nodeType==1)};b.isArray=p||function(a){return l.call(a)=="[object Array]"};b.isObject=function(a){return a===Object(a)};b.isArguments=function(a){return l.call(a)=="[object Arguments]"};b.isArguments(arguments)||(b.isArguments=function(a){return!(!a||!b.has(a,"callee"))});b.isFunction=function(a){return l.call(a)=="[object Function]"};
-b.isString=function(a){return l.call(a)=="[object String]"};b.isNumber=function(a){return l.call(a)=="[object Number]"};b.isFinite=function(a){return b.isNumber(a)&&isFinite(a)};b.isNaN=function(a){return a!==a};b.isBoolean=function(a){return a===true||a===false||l.call(a)=="[object Boolean]"};b.isDate=function(a){return l.call(a)=="[object Date]"};b.isRegExp=function(a){return l.call(a)=="[object RegExp]"};b.isNull=function(a){return a===null};b.isUndefined=function(a){return a===void 0};b.has=function(a,
-b){return K.call(a,b)};b.noConflict=function(){s._=I;return this};b.identity=function(a){return a};b.times=function(a,b,d){for(var e=0;e<a;e++)b.call(d,e)};b.escape=function(a){return(""+a).replace(/&/g,"&amp;").replace(/</g,"&lt;").replace(/>/g,"&gt;").replace(/"/g,"&quot;").replace(/'/g,"&#x27;").replace(/\//g,"&#x2F;")};b.result=function(a,c){if(a==null)return null;var d=a[c];return b.isFunction(d)?d.call(a):d};b.mixin=function(a){j(b.functions(a),function(c){M(c,b[c]=a[c])})};var N=0;b.uniqueId=
-function(a){var b=N++;return a?a+b:b};b.templateSettings={evaluate:/<%([\s\S]+?)%>/g,interpolate:/<%=([\s\S]+?)%>/g,escape:/<%-([\s\S]+?)%>/g};var u=/.^/,n={"\\":"\\","'":"'",r:"\r",n:"\n",t:"\t",u2028:"\u2028",u2029:"\u2029"},v;for(v in n)n[n[v]]=v;var O=/\\|'|\r|\n|\t|\u2028|\u2029/g,P=/\\(\\|'|r|n|t|u2028|u2029)/g,w=function(a){return a.replace(P,function(a,b){return n[b]})};b.template=function(a,c,d){d=b.defaults(d||{},b.templateSettings);a="__p+='"+a.replace(O,function(a){return"\\"+n[a]}).replace(d.escape||
-u,function(a,b){return"'+\n_.escape("+w(b)+")+\n'"}).replace(d.interpolate||u,function(a,b){return"'+\n("+w(b)+")+\n'"}).replace(d.evaluate||u,function(a,b){return"';\n"+w(b)+"\n;__p+='"})+"';\n";d.variable||(a="with(obj||{}){\n"+a+"}\n");var a="var __p='';var print=function(){__p+=Array.prototype.join.call(arguments, '')};\n"+a+"return __p;\n",e=new Function(d.variable||"obj","_",a);if(c)return e(c,b);c=function(a){return e.call(this,a,b)};c.source="function("+(d.variable||"obj")+"){\n"+a+"}";return c};
-b.chain=function(a){return b(a).chain()};var m=function(a){this._wrapped=a};b.prototype=m.prototype;var x=function(a,c){return c?b(a).chain():a},M=function(a,c){m.prototype[a]=function(){var a=i.call(arguments);J.call(a,this._wrapped);return x(c.apply(b,a),this._chain)}};b.mixin(b);j("pop,push,reverse,shift,sort,splice,unshift".split(","),function(a){var b=k[a];m.prototype[a]=function(){var d=this._wrapped;b.apply(d,arguments);var e=d.length;(a=="shift"||a=="splice")&&e===0&&delete d[0];return x(d,
-this._chain)}});j(["concat","join","slice"],function(a){var b=k[a];m.prototype[a]=function(){return x(b.apply(this._wrapped,arguments),this._chain)}});m.prototype.chain=function(){this._chain=true;return this};m.prototype.value=function(){return this._wrapped}}).call(this);

docs/client-server/web/swagger.html

@@ -1,78 +0,0 @@
-<!DOCTYPE html>
-<html><head><meta http-equiv="Content-Type" content="text/html; charset=ISO-8859-1">
-  <title>Matrix Client-Server API Documentation</title>
-  <link href="./files/css" rel="stylesheet" type="text/css">
-  <link href="./files/reset.css" media="screen" rel="stylesheet" type="text/css">
-  <link href="./files/screen.css" media="screen" rel="stylesheet" type="text/css">
-  <link href="./files/reset.css" media="print" rel="stylesheet" type="text/css">
-  <link href="./files/screen.css" media="print" rel="stylesheet" type="text/css">
-  <script type="text/javascript" src="./files/shred.bundle.js"></script>
-  <script src="./files/jquery-1.8.0.min.js" type="text/javascript"></script>
-  <script src="./files/jquery.slideto.min.js" type="text/javascript"></script>
-  <script src="./files/jquery.wiggle.min.js" type="text/javascript"></script>
-  <script src="./files/jquery.ba-bbq.min.js" type="text/javascript"></script>
-  <script src="./files/handlebars-1.0.0.js" type="text/javascript"></script>
-  <script src="./files/underscore-min.js" type="text/javascript"></script>
-  <script src="./files/backbone-min.js" type="text/javascript"></script>
-  <script src="./files/swagger.js" type="text/javascript"></script>
-  <script src="./files/swagger-ui.js" type="text/javascript"></script>
-  <script src="./files/highlight.7.3.pack.js" type="text/javascript"></script>
-
-  <!-- enabling this will enable oauth2 implicit scope support -->
-  <script src="./files/swagger-oauth.js" type="text/javascript"></script>
-
-  <script type="text/javascript">
-    $(function () {
-      window.swaggerUi = new SwaggerUi({
-      url: "http://localhost:8000/swagger_matrix/api-docs",
-      dom_id: "swagger-ui-container",
-      supportedSubmitMethods: ['get', 'post', 'put', 'delete'],
-      onComplete: function(swaggerApi, swaggerUi){
-        log("Loaded SwaggerUI");
-
-        if(typeof initOAuth == "function") {
-          initOAuth({
-            clientId: "your-client-id",
-            realm: "your-realms",
-            appName: "your-app-name"
-          });
-        }
-        $('pre code').each(function(i, e) {
-          hljs.highlightBlock(e)
-        });
-      },
-      onFailure: function(data) {
-        log("Unable to Load SwaggerUI");
-      },
-      docExpansion: "none"
-    });
-
-    $('#input_apiKey').change(function() {
-      var key = $('#input_apiKey')[0].value;
-      log("key: " + key);
-      if(key && key.trim() != "") {
-        log("added key " + key);
-        window.authorizations.add("key", new ApiKeyAuthorization("access_token", key, "query"));
-      }
-    })
-    window.swaggerUi.load();
-  });
-  </script>
-</head>
-
-<body class="swagger-section">
-<div id="header">
-  <div class="swagger-ui-wrap">
-    <a id="logo" href="http://swagger.wordnik.com/">swagger</a>
-    <form id="api_selector">
-      <div class="input"><input placeholder="http://example.com/api" id="input_baseUrl" name="baseUrl" type="text"></div>
-      <div class="input"><input placeholder="access_token" id="input_apiKey" name="apiKey" type="text"></div>
-    </form>
-  </div>
-</div>
-
-<div id="message-bar" class="swagger-ui-wrap message-fail">Can't read from server.  It may not have the appropriate access-control-origin settings.</div>
-<div id="swagger-ui-container" class="swagger-ui-wrap"></div>
-
-
-</body></html>

docs/server-server/protocol-format.rst

@@ -1,59 +0,0 @@
-
-Transaction
-===========
-
-Required keys:
-
-============ =================== ===============================================
-    Key            Type                         Description
-============ =================== ===============================================
-origin       String              DNS name of homeserver making this transaction.
-ts           Integer             Timestamp in milliseconds on originating 
-                                 homeserver when this transaction started.
-previous_ids List of Strings     List of transactions that were sent immediately
-                                 prior to this transaction.
-pdus         List of Objects     List of updates contained in this transaction.
-============ =================== ===============================================
-
-
-PDU
-===
-
-Required keys:
-
-============ ================== ================================================
-    Key            Type                         Description
-============ ================== ================================================
-context      String             Event context identifier
-origin       String             DNS name of homeserver that created this PDU.
-pdu_id       String             Unique identifier for PDU within the context for
-                                the originating homeserver.
-ts           Integer            Timestamp in milliseconds on originating 
-                                homeserver when this PDU was created.
-pdu_type     String             PDU event type.
-prev_pdus    List of Pairs      The originating homeserver and PDU ids of the
-             of Strings         most recent PDUs the homeserver was aware of for
-                                this context when it made this PDU.
-depth        Integer            The maximum depth of the previous PDUs plus one.
-============ ================== ================================================
-
-Keys for state updates:
-
-================== ============ ================================================
-    Key               Type                      Description
-================== ============ ================================================
-is_state           Boolean      True if this PDU is updating state.
-state_key          String       Optional key identifying the updated state within
-                                the context.
-power_level        Integer      The asserted power level of the user performing
-                                the update.
-min_update         Integer      The required power level needed to replace this
-                                update.
-prev_state_id      String       The homeserver of the update this replaces
-prev_state_origin  String       The PDU id of the update this replaces.
-user               String       The user updating the state.
-================== ============ ================================================
-
-
-
-

docs/server-server/signing.rst

@@ -1,151 +0,0 @@
-Signing JSON
-============
-
-JSON is signed by encoding the JSON object without ``signatures`` or ``meta``
-keys using a canonical encoding. The JSON bytes are then signed using the
-signature algorithm and the signature encoded using base64 with the padding
-stripped. The resulting base64 signature is added to an object under the
-*signing key identifier* which is added to the ``signatures`` object under the
-name of the server signing it which is added back to the original JSON object
-along with the ``meta`` object.
-
-The *signing key identifier* is the concatenation of the *signing algorithm*
-and a *key version*. The *signing algorithm* identifies the algorithm used to
-sign the JSON. The currently support value for *signing algorithm* is
-``ed25519`` as implemented by NACL (http://nacl.cr.yp.to/). The *key version*
-is used to distinguish between different signing keys used by the same entity.
-
-The ``meta`` object and the ``signatures`` object are not covered by the
-signature. Therefore intermediate servers can add metadata such as time stamps
-and additional signatures.
-
-
-::
-
-  {
-     "name": "example.org",
-     "signing_keys": {
-       "ed25519:1": "XSl0kuyvrXNj6A+7/tkrB9sxSbRi08Of5uRhxOqZtEQ"
-     },
-     "meta": {
-        "retrieved_ts_ms": 922834800000
-     },
-     "signatures": {
-        "example.org": {
-           "ed25519:1": "s76RUgajp8w172am0zQb/iPTHsRnb4SkrzGoeCOSFfcBY2V/1c8QfrmdXHpvnc2jK5BD1WiJIxiMW95fMjK7Bw"
-        }
-     }
-  }
-
-::
-
-  def sign_json(json_object, signing_key, signing_name):
-      signatures = json_object.pop("signatures", {})
-      meta = json_object.pop("meta", None)
-
-      signed = signing_key.sign(encode_canonical_json(json_object))
-      signature_base64 = encode_base64(signed.signature)
-
-      key_id = "%s:%s" % (signing_key.alg, signing_key.version)
-      signatures.setdefault(sigature_name, {})[key_id] = signature_base64
-
-      json_object["signatures"] = signatures
-      if meta is not None:
-          json_object["meta"] = meta
-
-      return json_object
-
-Checking for a Signature
-------------------------
-
-To check if an entity has signed a JSON object a server does the following
-
-1. Checks if the ``signatures`` object contains an entry with the name of the
-   entity. If the entry is missing then the check fails.
-2. Removes any *signing key identifiers* from the entry with algorithms it
-   doesn't understand. If there are no *signing key identifiers* left then the
-   check fails.
-3. Looks up *verification keys* for the remaining *signing key identifiers*
-   either from a local cache or by consulting a trusted key server. If it
-   cannot find a *verification key* then the check fails.
-4. Decodes the base64 encoded signature bytes. If base64 decoding fails then
-   the check fails.
-5. Checks the signature bytes using the *verification key*. If this fails then
-   the check fails. Otherwise the check succeeds.
-
-Canonical JSON
---------------
-
-The canonical JSON encoding for a value is the shortest UTF-8 JSON encoding
-with dictionary keys lexicographically sorted by unicode codepoint. Numbers in
-the JSON value must be integers in the range [-(2**53)+1, (2**53)-1].
-
-::
-
- import json
-
- def canonical_json(value):
-     return json.dumps(
-         value,
-         ensure_ascii=False,
-         separators=(',',':'),
-         sort_keys=True,
-     ).encode("UTF-8")
-
-Grammar
-+++++++
-
-Adapted from the grammar in http://tools.ietf.org/html/rfc7159 removing
-insignificant whitespace, fractions, exponents and redundant character escapes
-
-::
-
- value     = false / null / true / object / array / number / string
- false     = %x66.61.6c.73.65
- null      = %x6e.75.6c.6c
- true      = %x74.72.75.65
- object    = %x7B [ member *( %x2C member ) ] %7D
- member    = string %x3A value
- array     = %x5B [ value *( %x2C value ) ] %5B
- number    = [ %x2D ] int
- int       = %x30 / ( %x31-39 *digit )
- digit     = %x30-39
- string    = %x22 *char %x22
- char      = unescaped / %x5C escaped
- unescaped = %x20-21 / %x23-5B / %x5D-10FFFF
- escaped   = %x22 ; "    quotation mark  U+0022
-           / %x5C ; \    reverse solidus U+005C
-           / %x62 ; b    backspace       U+0008
-           / %x66 ; f    form feed       U+000C
-           / %x6E ; n    line feed       U+000A
-           / %x72 ; r    carriage return U+000D
-           / %x74 ; t    tab             U+0009
-           / %x75.30.30.30 (%x30-37 / %x62 / %x65-66) ; u000X
-           / %x75.30.30.31 (%x30-39 / %x61-66)        ; u001X
-
-Signing Events
-==============
-
-Signing events is a more complicated process since servers can choose to redact
-non-essential event contents. Before signing the event it is encoded as
-Canonical JSON and hashed using SHA-256. The resulting hash is then stored
-in the event JSON in a ``hash`` object under a ``sha256`` key. Then all
-non-essential keys are stripped from the event object, and the resulting object
-which included the ``hash`` key is signed using the JSON signing algorithm.
-
-Servers can then transmit the entire event or the event with the non-essential
-keys removed. Receiving servers can then check the entire event if it is
-present by computing the SHA-256 of the event excluding the ``hash`` object, or
-by using the ``hash`` object included in the event if keys have been redacted.
-
-New hash functions can be introduced by adding additional keys to the ``hash``
-object. Since the ``hash`` object cannot be redacted a server shouldn't allow
-too many hashes to be listed, otherwise a server might embed illict data within
-the ``hash`` object. For similar reasons a server shouldn't allow hash values
-that are too long.
-
-[[TODO(markjh): We might want to specify a maximum number of keys for the
-``hash`` and we might want to specify the maximum output size of a hash]]
-
-[[TODO(markjh) We might want to allow the server to omit the output of well
-known hash functions like SHA-256 when none of the keys have been redacted]]

docs/server-server/specification.rst

@@ -1,231 +0,0 @@
-===========================
-Matrix Server-to-Server API
-===========================
-
-A description of the protocol used to communicate between Matrix home servers;
-also known as Federation.
-
-
-Overview
-========
-
-The server-server API is a mechanism by which two home servers can exchange
-Matrix event messages, both as a real-time push of current events, and as a
-historic fetching mechanism to synchronise past history for clients to view. It
-uses HTTP connections between each pair of servers involved as the underlying
-transport. Messages are exchanged between servers in real-time by active pushing
-from each server's HTTP client into the server of the other. Queries to fetch
-historic data for the purpose of back-filling scrollback buffers and the like
-can also be performed.
-
-
-  { Matrix clients }                              { Matrix clients }
-     ^          |                                    ^          |
-     |  events  |                                    |  events  |
-     |          V                                    |          V
- +------------------+                            +------------------+
- |                  |---------( HTTP )---------->|                  |
- |   Home Server    |                            |   Home Server    |
- |                  |<--------( HTTP )-----------|                  |
- +------------------+                            +------------------+
-
-There are three main kinds of communication that occur between home servers:
-
- * Queries
-   These are single request/response interactions between a given pair of
-   servers, initiated by one side sending an HTTP request to obtain some
-   information, and responded by the other. They are not persisted and contain
-   no long-term significant history. They simply request a snapshot state at the
-   instant the query is made.
-
- * EDUs - Ephemeral Data Units
-   These are notifications of events that are pushed from one home server to
-   another. They are not persisted and contain no long-term significant history,
-   nor does the receiving home server have to reply to them.
-
- * PDUs - Persisted Data Units
-   These are notifications of events that are broadcast from one home server to
-   any others that are interested in the same "context" (namely, a Room ID).
-   They are persisted to long-term storage and form the record of history for
-   that context.
-
-Where Queries are presented directly across the HTTP connection as GET requests
-to specific URLs, EDUs and PDUs are further wrapped in an envelope called a
-Transaction, which is transferred from the origin to the destination home server
-using a PUT request.
-
-
-Transactions and EDUs/PDUs
-==========================
-
-The transfer of EDUs and PDUs between home servers is performed by an exchange
-of Transaction messages, which are encoded as JSON objects with a dict as the
-top-level element, passed over an HTTP PUT request. A Transaction is meaningful
-only to the pair of home servers that exchanged it; they are not globally-
-meaningful.
-
-Each transaction has an opaque ID and timestamp (UNIX epoch time in
-milliseconds) generated by its origin server, an origin and destination server
-name, a list of "previous IDs", and a list of PDUs - the actual message payload
-that the Transaction carries.
-
- {"transaction_id":"916d630ea616342b42e98a3be0b74113",
-  "ts":1404835423000,
-  "origin":"red",
-  "destination":"blue",
-  "prev_ids":["e1da392e61898be4d2009b9fecce5325"],
-  "pdus":[...],
-  "edus":[...]}
-
-The "previous IDs" field will contain a list of previous transaction IDs that
-the origin server has sent to this destination. Its purpose is to act as a
-sequence checking mechanism - the destination server can check whether it has
-successfully received that Transaction, or ask for a retransmission if not.
-
-The "pdus" field of a transaction is a list, containing zero or more PDUs.[*]
-Each PDU is itself a dict containing a number of keys, the exact details of
-which will vary depending on the type of PDU. Similarly, the "edus" field is
-another list containing the EDUs. This key may be entirely absent if there are
-no EDUs to transfer.
-
-(* Normally the PDU list will be non-empty, but the server should cope with
-receiving an "empty" transaction, as this is useful for informing peers of other
-transaction IDs they should be aware of. This effectively acts as a push
-mechanism to encourage peers to continue to replicate content.)
-
-All PDUs have an ID, a context, a declaration of their type, a list of other PDU
-IDs that have been seen recently on that context (regardless of which origin
-sent them), and a nested content field containing the actual event content.
-
-[[TODO(paul): Update this structure so that 'pdu_id' is a two-element
-[origin,ref] pair like the prev_pdus are]]
-
- {"pdu_id":"a4ecee13e2accdadf56c1025af232176",
-  "context":"#example.green",
-  "origin":"green",
-  "ts":1404838188000,
-  "pdu_type":"m.text",
-  "prev_pdus":[["blue","99d16afbc857975916f1d73e49e52b65"]],
-  "content":...
-  "is_state":false}
-
-In contrast to the transaction layer, it is important to note that the prev_pdus
-field of a PDU refers to PDUs that any origin server has sent, rather than
-previous IDs that this origin has sent. This list may refer to other PDUs sent
-by the same origin as the current one, or other origins.
-
-Because of the distributed nature of participants in a Matrix conversation, it
-is impossible to establish a globally-consistent total ordering on the events.
-However, by annotating each outbound PDU at its origin with IDs of other PDUs it
-has received, a partial ordering can be constructed allowing causallity
-relationships to be preserved. A client can then display these messages to the
-end-user in some order consistent with their content and ensure that no message
-that is semantically in reply of an earlier one is ever displayed before it.
-
-PDUs fall into two main categories: those that deliver Events, and those that
-synchronise State. For PDUs that relate to State synchronisation, additional
-keys exist to support this:
-
- {...,
-  "is_state":true,
-  "state_key":TODO
-  "power_level":TODO
-  "prev_state_id":TODO
-  "prev_state_origin":TODO}
-
-[[TODO(paul): At this point we should probably have a long description of how
-State management works, with descriptions of clobbering rules, power levels, etc
-etc... But some of that detail is rather up-in-the-air, on the whiteboard, and
-so on. This part needs refining. And writing in its own document as the details
-relate to the server/system as a whole, not specifically to server-server
-federation.]]
-
-EDUs, by comparison to PDUs, do not have an ID, a context, or a list of
-"previous" IDs. The only mandatory fields for these are the type, origin and
-destination home server names, and the actual nested content.
-
- {"edu_type":"m.presence",
-  "origin":"blue",
-  "destination":"orange",
-  "content":...}
-
-
-Protocol URLs
-=============
-
-All these URLs are namespaced within a prefix of 
-
-  /_matrix/federation/v1/...
-
-For active pushing of messages representing live activity "as it happens":
-
-  PUT .../send/:transaction_id/
-    Body: JSON encoding of a single Transaction
-
-    Response: [[TODO(paul): I don't actually understand what
-    ReplicationLayer.on_transaction() is doing here, so I'm not sure what the
-    response ought to be]]
-
-  The transaction_id path argument will override any ID given in the JSON body.
-  The destination name will be set to that of the receiving server itself. Each
-  embedded PDU in the transaction body will be processed.
-
-
-To fetch a particular PDU:
-
-  GET .../pdu/:origin/:pdu_id/
-
-    Response: JSON encoding of a single Transaction containing one PDU
-
-  Retrieves a given PDU from the server. The response will contain a single new
-  Transaction, inside which will be the requested PDU.
-  
-
-To fetch all the state of a given context:
-
-  GET .../state/:context/
-
-    Response: JSON encoding of a single Transaction containing multiple PDUs
-
-  Retrieves a snapshot of the entire current state of the given context. The
-  response will contain a single Transaction, inside which will be a list of
-  PDUs that encode the state.
-
-
-To backfill events on a given context:
-
-  GET .../backfill/:context/
-    Query args: v, limit
-
-    Response: JSON encoding of a single Transaction containing multiple PDUs
-
-  Retrieves a sliding-window history of previous PDUs that occurred on the
-  given context. Starting from the PDU ID(s) given in the "v" argument, the
-  PDUs that preceeded it are retrieved, up to a total number given by the
-  "limit" argument. These are then returned in a new Transaction containing all
-  off the PDUs.
-
-
-To stream events all the events:
-
-  GET .../pull/
-    Query args: origin, v
-
-  Response: JSON encoding of a single Transaction consisting of multiple PDUs
-
-  Retrieves all of the transactions later than any version given by the "v"
-  arguments. [[TODO(paul): I'm not sure what the "origin" argument does because
-  I think at some point in the code it's got swapped around.]]
-
-
-To make a query:
-
-  GET .../query/:query_type
-    Query args: as specified by the individual query types
-
-  Response: JSON encoding of a response object
-
-  Performs a single query request on the receiving home server. The Query Type
-  part of the path specifies the kind of query being made, and its query
-  arguments have a meaning specific to that kind of query. The response is a
-  JSON-encoded object whose meaning also depends on the kind of query.

docs/server-server/versioning.rst

@@ -1,11 +0,0 @@
-Versioning is, like, hard for backfilling backwards because of the number of Home Servers involved.
-
-The way we solve this is by doing versioning as an acyclic directed graph of PDUs. For backfilling purposes, this is done on a per context basis. 
-When we send a PDU we include all PDUs that have been received for that context that hasn't been subsequently listed in a later PDU. The trivial case is a simple list of PDUs, e.g. A <- B <- C. However, if two servers send out a PDU at the same to, both B and C would point at A - a later PDU would then list both B and C.
-
-Problems with opaque version strings:
-    - How do you do clustering without mandating that a cluster can only have one transaction in flight to a given remote home server at a time. 
-      If you have multiple transactions sent at once, then you might drop one transaction, receive another with a version that is later than the dropped transaction and which point ARGH WE LOST A TRANSACTION.
-    - How do you do backfilling? A version string defines a point in a stream w.r.t. a single home server, not a point in the context.
-
-We only need to store the ends of the directed graph, we DO NOT need to do the whole one table of nodes and one of edges.

docs/sphinx/README.rst

@@ -0,0 +1 @@
+TODO: how (if at all) is this actually maintained?

pylint.cfg

@@ -0,0 +1,280 @@
+[MASTER]
+
+# Specify a configuration file.
+#rcfile=
+
+# Python code to execute, usually for sys.path manipulation such as
+# pygtk.require().
+#init-hook=
+
+# Profiled execution.
+profile=no
+
+# Add files or directories to the blacklist. They should be base names, not
+# paths.
+ignore=CVS
+
+# Pickle collected data for later comparisons.
+persistent=yes
+
+# List of plugins (as comma separated values of python modules names) to load,
+# usually to register additional checkers.
+load-plugins=
+
+
+[MESSAGES CONTROL]
+
+# Enable the message, report, category or checker with the given id(s). You can
+# either give multiple identifier separated by comma (,) or put this option
+# multiple time. See also the "--disable" option for examples.
+#enable=
+
+# Disable the message, report, category or checker with the given id(s). You
+# can either give multiple identifiers separated by comma (,) or put this
+# option multiple times (only on the command line, not in the configuration
+# file where it should appear only once).You can also use "--disable=all" to
+# disable everything first and then reenable specific checks. For example, if
+# you want to run only the similarities checker, you can use "--disable=all
+# --enable=similarities". If you want to run only the classes checker, but have
+# no Warning level messages displayed, use"--disable=all --enable=classes
+# --disable=W"
+disable=missing-docstring
+
+
+[REPORTS]
+
+# Set the output format. Available formats are text, parseable, colorized, msvs
+# (visual studio) and html. You can also give a reporter class, eg
+# mypackage.mymodule.MyReporterClass.
+output-format=text
+
+# Put messages in a separate file for each module / package specified on the
+# command line instead of printing them on stdout. Reports (if any) will be
+# written in a file name "pylint_global.[txt|html]".
+files-output=no
+
+# Tells whether to display a full report or only the messages
+reports=yes
+
+# Python expression which should return a note less than 10 (10 is the highest
+# note). You have access to the variables errors warning, statement which
+# respectively contain the number of errors / warnings messages and the total
+# number of statements analyzed. This is used by the global evaluation report
+# (RP0004).
+evaluation=10.0 - ((float(5 * error + warning + refactor + convention) / statement) * 10)
+
+# Add a comment according to your evaluation note. This is used by the global
+# evaluation report (RP0004).
+comment=no
+
+# Template used to display messages. This is a python new-style format string
+# used to format the message information. See doc for all details
+#msg-template=
+
+
+[TYPECHECK]
+
+# Tells whether missing members accessed in mixin class should be ignored. A
+# mixin class is detected if its name ends with "mixin" (case insensitive).
+ignore-mixin-members=yes
+
+# List of classes names for which member attributes should not be checked
+# (useful for classes with attributes dynamically set).
+ignored-classes=SQLObject
+
+# When zope mode is activated, add a predefined set of Zope acquired attributes
+# to generated-members.
+zope=no
+
+# List of members which are set dynamically and missed by pylint inference
+# system, and so shouldn't trigger E0201 when accessed. Python regular
+# expressions are accepted.
+generated-members=REQUEST,acl_users,aq_parent
+
+
+[MISCELLANEOUS]
+
+# List of note tags to take in consideration, separated by a comma.
+notes=FIXME,XXX,TODO
+
+
+[SIMILARITIES]
+
+# Minimum lines number of a similarity.
+min-similarity-lines=4
+
+# Ignore comments when computing similarities.
+ignore-comments=yes
+
+# Ignore docstrings when computing similarities.
+ignore-docstrings=yes
+
+# Ignore imports when computing similarities.
+ignore-imports=no
+
+
+[VARIABLES]
+
+# Tells whether we should check for unused import in __init__ files.
+init-import=no
+
+# A regular expression matching the beginning of the name of dummy variables
+# (i.e. not used).
+dummy-variables-rgx=_$|dummy
+
+# List of additional names supposed to be defined in builtins. Remember that
+# you should avoid to define new builtins when possible.
+additional-builtins=
+
+
+[BASIC]
+
+# Required attributes for module, separated by a comma
+required-attributes=
+
+# List of builtins function names that should not be used, separated by a comma
+bad-functions=map,filter,apply,input
+
+# Regular expression which should only match correct module names
+module-rgx=(([a-z_][a-z0-9_]*)|([A-Z][a-zA-Z0-9]+))$
+
+# Regular expression which should only match correct module level names
+const-rgx=(([A-Z_][A-Z0-9_]*)|(__.*__))$
+
+# Regular expression which should only match correct class names
+class-rgx=[A-Z_][a-zA-Z0-9]+$
+
+# Regular expression which should only match correct function names
+function-rgx=[a-z_][a-z0-9_]{2,30}$
+
+# Regular expression which should only match correct method names
+method-rgx=[a-z_][a-z0-9_]{2,30}$
+
+# Regular expression which should only match correct instance attribute names
+attr-rgx=[a-z_][a-z0-9_]{2,30}$
+
+# Regular expression which should only match correct argument names
+argument-rgx=[a-z_][a-z0-9_]{2,30}$
+
+# Regular expression which should only match correct variable names
+variable-rgx=[a-z_][a-z0-9_]{2,30}$
+
+# Regular expression which should only match correct attribute names in class
+# bodies
+class-attribute-rgx=([A-Za-z_][A-Za-z0-9_]{2,30}|(__.*__))$
+
+# Regular expression which should only match correct list comprehension /
+# generator expression variable names
+inlinevar-rgx=[A-Za-z_][A-Za-z0-9_]*$
+
+# Good variable names which should always be accepted, separated by a comma
+good-names=i,j,k,ex,Run,_
+
+# Bad variable names which should always be refused, separated by a comma
+bad-names=foo,bar,baz,toto,tutu,tata
+
+# Regular expression which should only match function or class names that do
+# not require a docstring.
+no-docstring-rgx=__.*__
+
+# Minimum line length for functions/classes that require docstrings, shorter
+# ones are exempt.
+docstring-min-length=-1
+
+
+[FORMAT]
+
+# Maximum number of characters on a single line.
+max-line-length=80
+
+# Regexp for a line that is allowed to be longer than the limit.
+ignore-long-lines=^\s*(# )?<?https?://\S+>?$
+
+# Allow the body of an if to be on the same line as the test if there is no
+# else.
+single-line-if-stmt=no
+
+# List of optional constructs for which whitespace checking is disabled
+no-space-check=trailing-comma,dict-separator
+
+# Maximum number of lines in a module
+max-module-lines=1000
+
+# String used as indentation unit. This is usually " " (4 spaces) or "\t" (1
+# tab).
+indent-string='    '
+
+
+[DESIGN]
+
+# Maximum number of arguments for function / method
+max-args=5
+
+# Argument names that match this expression will be ignored. Default to name
+# with leading underscore
+ignored-argument-names=_.*
+
+# Maximum number of locals for function / method body
+max-locals=15
+
+# Maximum number of return / yield for function / method body
+max-returns=6
+
+# Maximum number of branch for function / method body
+max-branches=12
+
+# Maximum number of statements in function / method body
+max-statements=50
+
+# Maximum number of parents for a class (see R0901).
+max-parents=7
+
+# Maximum number of attributes for a class (see R0902).
+max-attributes=7
+
+# Minimum number of public methods for a class (see R0903).
+min-public-methods=2
+
+# Maximum number of public methods for a class (see R0904).
+max-public-methods=20
+
+
+[IMPORTS]
+
+# Deprecated modules which should not be used, separated by a comma
+deprecated-modules=regsub,TERMIOS,Bastion,rexec
+
+# Create a graph of every (i.e. internal and external) dependencies in the
+# given file (report RP0402 must not be disabled)
+import-graph=
+
+# Create a graph of external dependencies in the given file (report RP0402 must
+# not be disabled)
+ext-import-graph=
+
+# Create a graph of internal dependencies in the given file (report RP0402 must
+# not be disabled)
+int-import-graph=
+
+
+[CLASSES]
+
+# List of interface methods to ignore, separated by a comma. This is used for
+# instance to not check methods defines in Zope's Interface base class.
+ignore-iface-methods=isImplementedBy,deferred,extends,names,namesAndDescriptions,queryDescriptionFor,getBases,getDescriptionFor,getDoc,getName,getTaggedValue,getTaggedValueTags,isEqualOrExtendedBy,setTaggedValue,isImplementedByInstancesOf,adaptWith,is_implemented_by
+
+# List of method names used to declare (i.e. assign) instance attributes.
+defining-attr-methods=__init__,__new__,setUp
+
+# List of valid names for the first argument in a class method.
+valid-classmethod-first-arg=cls
+
+# List of valid names for the first argument in a metaclass class method.
+valid-metaclass-classmethod-first-arg=mcs
+
+
+[EXCEPTIONS]
+
+# Exceptions that will emit a warning when being caught. Defaults to
+# "Exception"
+overgeneral-exceptions=Exception

scripts/check_event_hash.py

@@ -0,0 +1,47 @@
+from synapse.crypto.event_signing import *
+from syutil.base64util import encode_base64
+
+import argparse
+import hashlib
+import sys
+import json
+
+
+class dictobj(dict):
+    def __init__(self, *args, **kargs):
+        dict.__init__(self, *args, **kargs)
+        self.__dict__ = self
+
+    def get_dict(self):
+        return dict(self)
+
+    def get_full_dict(self):
+        return dict(self)
+
+
+def main():
+    parser = argparse.ArgumentParser()
+    parser.add_argument("input_json", nargs="?", type=argparse.FileType('r'),
+                        default=sys.stdin)
+    args = parser.parse_args()
+    logging.basicConfig()
+
+    event_json = dictobj(json.load(args.input_json))
+
+    algorithms = {
+        "sha256": hashlib.sha256,
+    }
+
+    for alg_name in event_json.hashes:
+        if check_event_content_hash(event_json, algorithms[alg_name]):
+            print "PASS content hash %s" % (alg_name,)
+        else:
+            print "FAIL content hash %s" % (alg_name,)
+
+    for algorithm in algorithms.values():
+        name, h_bytes = compute_event_reference_hash(event_json, algorithm)
+        print "Reference hash %s: %s" % (name, encode_base64(h_bytes))
+
+if __name__=="__main__":
+    main()
+

scripts/check_signature.py

@@ -0,0 +1,73 @@
+
+from syutil.crypto.jsonsign import verify_signed_json
+from syutil.crypto.signing_key import (
+    decode_verify_key_bytes, write_signing_keys
+)
+from syutil.base64util import decode_base64
+
+import urllib2
+import json
+import sys
+import dns.resolver
+import pprint
+import argparse
+import logging
+
+def get_targets(server_name):
+    if ":" in server_name:
+        target, port = server_name.split(":")
+        yield (target, int(port))
+        return
+    try:
+        answers = dns.resolver.query("_matrix._tcp." + server_name, "SRV")
+        for srv in answers:
+            yield (srv.target, srv.port)
+    except dns.resolver.NXDOMAIN:
+        yield (server_name, 8448)
+
+def get_server_keys(server_name, target, port):
+    url = "https://%s:%i/_matrix/key/v1" % (target, port)
+    keys = json.load(urllib2.urlopen(url))
+    verify_keys = {}
+    for key_id, key_base64 in keys["verify_keys"].items():
+        verify_key = decode_verify_key_bytes(key_id, decode_base64(key_base64))
+        verify_signed_json(keys, server_name, verify_key)
+        verify_keys[key_id] = verify_key
+    return verify_keys
+
+def main():
+
+    parser = argparse.ArgumentParser()
+    parser.add_argument("signature_name")
+    parser.add_argument("input_json", nargs="?", type=argparse.FileType('r'),
+                        default=sys.stdin)
+
+    args = parser.parse_args()
+    logging.basicConfig()
+
+    server_name = args.signature_name
+    keys = {}
+    for target, port in get_targets(server_name):
+        try:
+            keys = get_server_keys(server_name, target, port)
+            print "Using keys from https://%s:%s/_matrix/key/v1" % (target, port)
+            write_signing_keys(sys.stdout, keys.values())
+            break
+        except:
+            logging.exception("Error talking to %s:%s", target, port)
+
+    json_to_check = json.load(args.input_json)
+    print "Checking JSON:"
+    for key_id in json_to_check["signatures"][args.signature_name]:
+        try:
+            key = keys[key_id]
+            verify_signed_json(json_to_check, args.signature_name, key)
+            print "PASS %s" % (key_id,)
+        except:
+            logging.exception("Check for key %s failed" % (key_id,))
+            print "FAIL %s" % (key_id,)
+
+
+if __name__ == '__main__':
+    main()
+

scripts/hash_history.py

@@ -0,0 +1,69 @@
+from synapse.storage.pdu import PduStore
+from synapse.storage.signatures import SignatureStore
+from synapse.storage._base import SQLBaseStore
+from synapse.federation.units import Pdu
+from synapse.crypto.event_signing import (
+    add_event_pdu_content_hash, compute_pdu_event_reference_hash
+)
+from synapse.api.events.utils import prune_pdu
+from syutil.base64util import encode_base64, decode_base64
+from syutil.jsonutil import encode_canonical_json
+import sqlite3
+import sys
+
+class Store(object):
+    _get_pdu_tuples = PduStore.__dict__["_get_pdu_tuples"]
+    _get_pdu_content_hashes_txn = SignatureStore.__dict__["_get_pdu_content_hashes_txn"]
+    _get_prev_pdu_hashes_txn = SignatureStore.__dict__["_get_prev_pdu_hashes_txn"]
+    _get_pdu_origin_signatures_txn = SignatureStore.__dict__["_get_pdu_origin_signatures_txn"]
+    _store_pdu_content_hash_txn = SignatureStore.__dict__["_store_pdu_content_hash_txn"]
+    _store_pdu_reference_hash_txn = SignatureStore.__dict__["_store_pdu_reference_hash_txn"]
+    _store_prev_pdu_hash_txn = SignatureStore.__dict__["_store_prev_pdu_hash_txn"]
+    _simple_insert_txn = SQLBaseStore.__dict__["_simple_insert_txn"]
+
+
+store = Store()
+
+
+def select_pdus(cursor):
+    cursor.execute(
+        "SELECT pdu_id, origin FROM pdus ORDER BY depth ASC"
+    )
+
+    ids = cursor.fetchall()
+
+    pdu_tuples = store._get_pdu_tuples(cursor, ids)
+
+    pdus = [Pdu.from_pdu_tuple(p) for p in pdu_tuples]
+
+    reference_hashes = {}
+
+    for pdu in pdus:
+        try:
+            if pdu.prev_pdus:
+                print "PROCESS", pdu.pdu_id, pdu.origin, pdu.prev_pdus
+                for pdu_id, origin, hashes in pdu.prev_pdus:
+                    ref_alg, ref_hsh = reference_hashes[(pdu_id, origin)]
+                    hashes[ref_alg] = encode_base64(ref_hsh)
+                    store._store_prev_pdu_hash_txn(cursor,  pdu.pdu_id, pdu.origin, pdu_id, origin, ref_alg, ref_hsh)
+                print "SUCCESS", pdu.pdu_id, pdu.origin, pdu.prev_pdus
+            pdu = add_event_pdu_content_hash(pdu)
+            ref_alg, ref_hsh = compute_pdu_event_reference_hash(pdu)
+            reference_hashes[(pdu.pdu_id, pdu.origin)] = (ref_alg, ref_hsh)
+            store._store_pdu_reference_hash_txn(cursor, pdu.pdu_id, pdu.origin, ref_alg, ref_hsh)
+
+            for alg, hsh_base64 in pdu.hashes.items():
+                print alg, hsh_base64
+                store._store_pdu_content_hash_txn(cursor, pdu.pdu_id, pdu.origin, alg, decode_base64(hsh_base64))
+
+        except:
+            print "FAILED_", pdu.pdu_id, pdu.origin, pdu.prev_pdus
+
+def main():
+    conn = sqlite3.connect(sys.argv[1])
+    cursor = conn.cursor()
+    select_pdus(cursor)
+    conn.commit()
+
+if __name__=='__main__':
+    main()

setup.py

@@ -26,14 +26,16 @@ def read(fname):
     return open(os.path.join(os.path.dirname(__file__), fname)).read()
 
 setup(
-    name="SynapseHomeServer",
-    version="0.0.1",
-    packages=find_packages(exclude=["tests"]),
+    name="matrix-synapse",
+    version=read("VERSION").strip(),
+    packages=find_packages(exclude=["tests", "tests.*"]),
     description="Reference Synapse Home Server",
     install_requires=[
         "syutil==0.0.2",
+        "matrix_angular_sdk==0.5.1",
         "Twisted>=14.0.0",
         "service_identity>=1.0.0",
+        "pyopenssl>=0.14",
         "pyyaml",
         "pyasn1",
         "pynacl",
@@ -41,17 +43,22 @@ setup(
         "py-bcrypt",
     ],
     dependency_links=[
-        "git+ssh://git@github.com/matrix-org/syutil.git#egg=syutil-0.0.2",
+        "https://github.com/matrix-org/syutil/tarball/v0.0.2#egg=syutil-0.0.2",
+        "https://github.com/pyca/pynacl/tarball/d4d3175589b892f6ea7c22f466e0e223853516fa#egg=pynacl-0.3.0",
+        "https://github.com/matrix-org/matrix-angular-sdk/tarball/v0.5.1/#egg=matrix_angular_sdk-0.5.1",
     ],
     setup_requires=[
         "setuptools_trial",
-        "setuptools>=1.0.0", # Needs setuptools that supports git+ssh. It's not obvious when support for this was introduced.
+        "setuptools>=1.0.0", # Needs setuptools that supports git+ssh.
+                             # TODO: Do we need this now? we don't use git+ssh.
         "mock"
     ],
     include_package_data=True,
+    zip_safe=False,
     long_description=read("README.rst"),
     entry_points="""
     [console_scripts]
+    synctl=synapse.app.synctl:main
     synapse-homeserver=synapse.app.homeserver:run
     """
 )

synapse/__init__.py

@@ -16,4 +16,4 @@
 """ This is a reference implementation of a synapse home server.
 """
 
-__version__ = "0.4.1"
+__version__ = "0.5.3a"

synapse/api/__init__.py

@@ -12,4 +12,3 @@
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
-

synapse/api/auth.py

@@ -21,8 +21,10 @@ from synapse.api.constants import Membership, JoinRules
 from synapse.api.errors import AuthError, StoreError, Codes, SynapseError
 from synapse.api.events.room import (
     RoomMemberEvent, RoomPowerLevelsEvent, RoomRedactionEvent,
+    RoomJoinRulesEvent, RoomCreateEvent, RoomAliasesEvent,
 )
 from synapse.util.logutils import log_function
+from syutil.base64util import encode_base64
 
 import logging
 
@@ -34,69 +36,96 @@ class Auth(object):
     def __init__(self, hs):
         self.hs = hs
         self.store = hs.get_datastore()
+        self.state = hs.get_state_handler()
 
-    @defer.inlineCallbacks
-    def check(self, event, snapshot, raises=False):
+    def check(self, event, auth_events):
         """ Checks if this event is correctly authed.
 
         Returns:
             True if the auth checks pass.
-        Raises:
-            AuthError if there was a problem authorising this event. This will
-            be raised only if raises=True.
         """
         try:
-            if hasattr(event, "room_id"):
-                is_state = hasattr(event, "state_key")
+            if not hasattr(event, "room_id"):
+                raise AuthError(500, "Event has no room_id: %s" % event)
+            if auth_events is None:
+                # Oh, we don't know what the state of the room was, so we
+                # are trusting that this is allowed (at least for now)
+                logger.warn("Trusting event: %s", event.event_id)
+                return True
 
-                if event.type == RoomMemberEvent.TYPE:
-                    yield self._can_replace_state(event)
-                    allowed = yield self.is_membership_change_allowed(event)
-                    defer.returnValue(allowed)
-                    return
+            if event.type == RoomCreateEvent.TYPE:
+                # FIXME
+                return True
 
-                self._check_joined_room(
-                    member=snapshot.membership_state,
-                    user_id=snapshot.user_id,
-                    room_id=snapshot.room_id,
+            # FIXME: Temp hack
+            if event.type == RoomAliasesEvent.TYPE:
+                return True
+
+            if event.type == RoomMemberEvent.TYPE:
+                allowed = self.is_membership_change_allowed(
+                    event, auth_events
                 )
-
-                if is_state:
-                    # TODO (erikj): This really only should be called for *new*
-                    # state
-                    yield self._can_add_state(event)
-                    yield self._can_replace_state(event)
+                if allowed:
+                    logger.debug("Allowing! %s", event)
                 else:
-                    yield self._can_send_event(event)
+                    logger.debug("Denying! %s", event)
+                return allowed
 
-                if event.type == RoomPowerLevelsEvent.TYPE:
-                    yield self._check_power_levels(event)
+            self.check_event_sender_in_room(event, auth_events)
+            self._can_send_event(event, auth_events)
 
-                if event.type == RoomRedactionEvent.TYPE:
-                    yield self._check_redaction(event)
+            if event.type == RoomPowerLevelsEvent.TYPE:
+                self._check_power_levels(event, auth_events)
 
-                defer.returnValue(True)
-            else:
-                raise AuthError(500, "Unknown event: %s" % event)
+            if event.type == RoomRedactionEvent.TYPE:
+                self._check_redaction(event, auth_events)
+
+            logger.debug("Allowing! %s", event)
         except AuthError as e:
-            logger.info("Event auth check failed on event %s with msg: %s",
-                        event, e.msg)
-            if raises:
-                raise e
-        defer.returnValue(False)
+            logger.info(
+                "Event auth check failed on event %s with msg: %s",
+                event, e.msg
+            )
+            logger.info("Denying! %s", event)
+            raise
 
     @defer.inlineCallbacks
     def check_joined_room(self, room_id, user_id):
-        try:
-            member = yield self.store.get_room_member(
-                room_id=room_id,
-                user_id=user_id
-            )
-            self._check_joined_room(member, user_id, room_id)
-            defer.returnValue(member)
-        except AttributeError:
-            pass
-        defer.returnValue(None)
+        member = yield self.state.get_current_state(
+            room_id=room_id,
+            event_type=RoomMemberEvent.TYPE,
+            state_key=user_id
+        )
+        self._check_joined_room(member, user_id, room_id)
+        defer.returnValue(member)
+
+    @defer.inlineCallbacks
+    def check_host_in_room(self, room_id, host):
+        curr_state = yield self.state.get_current_state(room_id)
+
+        for event in curr_state:
+            if event.type == RoomMemberEvent.TYPE:
+                try:
+                    if self.hs.parse_userid(event.state_key).domain != host:
+                        continue
+                except:
+                    logger.warn("state_key not user_id: %s", event.state_key)
+                    continue
+
+                if event.content["membership"] == Membership.JOIN:
+                    defer.returnValue(True)
+
+        defer.returnValue(False)
+
+    def check_event_sender_in_room(self, event, auth_events):
+        key = (RoomMemberEvent.TYPE, event.user_id, )
+        member_event = auth_events.get(key)
+
+        return self._check_joined_room(
+            member_event,
+            event.user_id,
+            event.room_id
+        )
 
     def _check_joined_room(self, member, user_id, room_id):
         if not member or member.membership != Membership.JOIN:
@@ -104,46 +133,79 @@ class Auth(object):
                 user_id, room_id, repr(member)
             ))
 
-    @defer.inlineCallbacks
-    def is_membership_change_allowed(self, event):
-        target_user_id = event.state_key
-
-        # does this room even exist
-        room = yield self.store.get_room(event.room_id)
-        if not room:
-            raise AuthError(403, "Room does not exist")
-
-        # get info about the caller
-        try:
-            caller = yield self.store.get_room_member(
-                user_id=event.user_id,
-                room_id=event.room_id)
-        except:
-            caller = None
-        caller_in_room = caller and caller.membership == "join"
-
-        # get info about the target
-        try:
-            target = yield self.store.get_room_member(
-                user_id=target_user_id,
-                room_id=event.room_id)
-        except:
-            target = None
-        target_in_room = target and target.membership == "join"
-
+    @log_function
+    def is_membership_change_allowed(self, event, auth_events):
         membership = event.content["membership"]
 
-        join_rule = yield self.store.get_room_join_rule(event.room_id)
-        if not join_rule:
+        # Check if this is the room creator joining:
+        if len(event.prev_events) == 1 and Membership.JOIN == membership:
+            # Get room creation event:
+            key = (RoomCreateEvent.TYPE, "", )
+            create = auth_events.get(key)
+            if create and event.prev_events[0][0] == create.event_id:
+                if create.content["creator"] == event.state_key:
+                    return True
+
+        target_user_id = event.state_key
+
+        # get info about the caller
+        key = (RoomMemberEvent.TYPE, event.user_id, )
+        caller = auth_events.get(key)
+
+        caller_in_room = caller and caller.membership == Membership.JOIN
+        caller_invited = caller and caller.membership == Membership.INVITE
+
+        # get info about the target
+        key = (RoomMemberEvent.TYPE, target_user_id, )
+        target = auth_events.get(key)
+
+        target_in_room = target and target.membership == Membership.JOIN
+
+        key = (RoomJoinRulesEvent.TYPE, "", )
+        join_rule_event = auth_events.get(key)
+        if join_rule_event:
+            join_rule = join_rule_event.content.get(
+                "join_rule", JoinRules.INVITE
+            )
+        else:
             join_rule = JoinRules.INVITE
 
+        user_level = self._get_power_level_from_event_state(
+            event,
+            event.user_id,
+            auth_events,
+        )
+
+        ban_level, kick_level, redact_level = (
+            self._get_ops_level_from_event_state(
+                event,
+                auth_events,
+            )
+        )
+
+        logger.debug(
+            "is_membership_change_allowed: %s",
+            {
+                "caller_in_room": caller_in_room,
+                "caller_invited": caller_invited,
+                "target_in_room": target_in_room,
+                "membership": membership,
+                "join_rule": join_rule,
+                "target_user_id": target_user_id,
+                "event.user_id": event.user_id,
+            }
+        )
+
         if Membership.INVITE == membership:
             # TODO (erikj): We should probably handle this more intelligently
             # PRIVATE join rules.
 
             # Invites are valid iff caller is in the room and target isn't.
             if not caller_in_room:  # caller isn't joined
-                raise AuthError(403, "You are not in room %s." % event.room_id)
+                raise AuthError(
+                    403,
+                    "%s not in room %s." % (event.user_id, event.room_id,)
+                )
             elif target_in_room:  # the target is already in the room.
                 raise AuthError(403, "%s is already in the room." %
                                      target_user_id)
@@ -153,13 +215,10 @@ class Auth(object):
             # joined: It's a NOOP
             if event.user_id != target_user_id:
                 raise AuthError(403, "Cannot force another user to join.")
-            elif join_rule == JoinRules.PUBLIC or room.is_public:
+            elif join_rule == JoinRules.PUBLIC:
                 pass
             elif join_rule == JoinRules.INVITE:
-                if (
-                    not caller or caller.membership not in
-                    [Membership.INVITE, Membership.JOIN]
-                ):
+                if not caller_in_room and not caller_invited:
                     raise AuthError(403, "You are not invited to this room.")
             else:
                 # TODO (erikj): may_join list
@@ -169,31 +228,21 @@ class Auth(object):
             # TODO (erikj): Implement kicks.
 
             if not caller_in_room:  # trying to leave a room you aren't joined
-                raise AuthError(403, "You are not in room %s." % event.room_id)
-            elif target_user_id != event.user_id:
-                user_level = yield self.store.get_power_level(
-                    event.room_id,
-                    event.user_id,
+                raise AuthError(
+                    403,
+                    "%s not in room %s." % (target_user_id, event.room_id,)
                 )
-                _, kick_level, _ = yield self.store.get_ops_levels(event.room_id)
-
+            elif target_user_id != event.user_id:
                 if kick_level:
                     kick_level = int(kick_level)
                 else:
-                    kick_level = 50
+                    kick_level = 50  # FIXME (erikj): What should we do here?
 
                 if user_level < kick_level:
                     raise AuthError(
                         403, "You cannot kick user %s." % target_user_id
                     )
         elif Membership.BAN == membership:
-            user_level = yield self.store.get_power_level(
-                event.room_id,
-                event.user_id,
-            )
-
-            ban_level, _, _  = yield self.store.get_ops_levels(event.room_id)
-
             if ban_level:
                 ban_level = int(ban_level)
             else:
@@ -204,7 +253,36 @@ class Auth(object):
         else:
             raise AuthError(500, "Unknown membership %s" % membership)
 
-        defer.returnValue(True)
+        return True
+
+    def _get_power_level_from_event_state(self, event, user_id, auth_events):
+        key = (RoomPowerLevelsEvent.TYPE, "", )
+        power_level_event = auth_events.get(key)
+        level = None
+        if power_level_event:
+            level = power_level_event.content.get("users", {}).get(user_id)
+            if not level:
+                level = power_level_event.content.get("users_default", 0)
+        else:
+            key = (RoomCreateEvent.TYPE, "", )
+            create_event = auth_events.get(key)
+            if (create_event is not None and
+                    create_event.content["creator"] == user_id):
+                return 100
+
+        return level
+
+    def _get_ops_level_from_event_state(self, event, auth_events):
+        key = (RoomPowerLevelsEvent.TYPE, "", )
+        power_level_event = auth_events.get(key)
+
+        if power_level_event:
+            return (
+                power_level_event.content.get("ban", 50),
+                power_level_event.content.get("kick", 50),
+                power_level_event.content.get("redact", 50),
+            )
+        return None, None, None,
 
     @defer.inlineCallbacks
     def get_user_by_req(self, request):
@@ -229,7 +307,7 @@ class Auth(object):
                 default=[""]
             )[0]
             if user and access_token and ip_addr:
-                self.store.insert_client_ip(
+                yield self.store.insert_client_ip(
                     user=user,
                     access_token=access_token,
                     device_id=user_info["device_id"],
@@ -273,18 +351,88 @@ class Auth(object):
         return self.store.is_server_admin(user)
 
     @defer.inlineCallbacks
+    def add_auth_events(self, event):
+        if event.type == RoomCreateEvent.TYPE:
+            event.auth_events = []
+            return
+
+        auth_events = []
+
+        key = (RoomPowerLevelsEvent.TYPE, "", )
+        power_level_event = event.old_state_events.get(key)
+
+        if power_level_event:
+            auth_events.append(power_level_event.event_id)
+
+        key = (RoomJoinRulesEvent.TYPE, "", )
+        join_rule_event = event.old_state_events.get(key)
+
+        key = (RoomMemberEvent.TYPE, event.user_id, )
+        member_event = event.old_state_events.get(key)
+
+        key = (RoomCreateEvent.TYPE, "", )
+        create_event = event.old_state_events.get(key)
+        if create_event:
+            auth_events.append(create_event.event_id)
+
+        if join_rule_event:
+            join_rule = join_rule_event.content.get("join_rule")
+            is_public = join_rule == JoinRules.PUBLIC if join_rule else False
+        else:
+            is_public = False
+
+        if event.type == RoomMemberEvent.TYPE:
+            e_type = event.content["membership"]
+            if e_type in [Membership.JOIN, Membership.INVITE]:
+                if join_rule_event:
+                    auth_events.append(join_rule_event.event_id)
+
+                if member_event and not is_public:
+                    auth_events.append(member_event.event_id)
+        elif member_event:
+            if member_event.content["membership"] == Membership.JOIN:
+                auth_events.append(member_event.event_id)
+
+        hashes = yield self.store.get_event_reference_hashes(
+            auth_events
+        )
+        hashes = [
+            {
+                k: encode_base64(v) for k, v in h.items()
+                if k == "sha256"
+            }
+            for h in hashes
+        ]
+        event.auth_events = zip(auth_events, hashes)
+
     @log_function
-    def _can_send_event(self, event):
-        send_level = yield self.store.get_send_event_level(event.room_id)
+    def _can_send_event(self, event, auth_events):
+        key = (RoomPowerLevelsEvent.TYPE, "", )
+        send_level_event = auth_events.get(key)
+        send_level = None
+        if send_level_event:
+            send_level = send_level_event.content.get("events", {}).get(
+                event.type
+            )
+            if not send_level:
+                if hasattr(event, "state_key"):
+                    send_level = send_level_event.content.get(
+                        "state_default", 50
+                    )
+                else:
+                    send_level = send_level_event.content.get(
+                        "events_default", 0
+                    )
 
         if send_level:
             send_level = int(send_level)
         else:
             send_level = 0
 
-        user_level = yield self.store.get_power_level(
-            event.room_id,
+        user_level = self._get_power_level_from_event_state(
+            event,
             event.user_id,
+            auth_events,
         )
 
         if user_level:
@@ -294,101 +442,55 @@ class Auth(object):
 
         if user_level < send_level:
             raise AuthError(
-                403, "You don't have permission to post to the room"
+                403,
+                "You don't have permission to post that to the room. " +
+                "user_level (%d) < send_level (%d)" % (user_level, send_level)
             )
 
-        defer.returnValue(True)
+        # Check state_key
+        if hasattr(event, "state_key"):
+            if not event.state_key.startswith("_"):
+                if event.state_key.startswith("@"):
+                    if event.state_key != event.user_id:
+                        raise AuthError(
+                            403,
+                            "You are not allowed to set others state"
+                        )
+                    else:
+                        sender_domain = self.hs.parse_userid(
+                            event.user_id
+                        ).domain
 
-    @defer.inlineCallbacks
-    def _can_add_state(self, event):
-        add_level = yield self.store.get_add_state_level(event.room_id)
+                        if sender_domain != event.state_key:
+                            raise AuthError(
+                                403,
+                                "You are not allowed to set others state"
+                            )
 
-        if not add_level:
-            defer.returnValue(True)
+        return True
 
-        add_level = int(add_level)
-
-        user_level = yield self.store.get_power_level(
-            event.room_id,
+    def _check_redaction(self, event, auth_events):
+        user_level = self._get_power_level_from_event_state(
+            event,
             event.user_id,
+            auth_events,
         )
 
-        user_level = int(user_level)
-
-        if user_level < add_level:
-            raise AuthError(
-                403, "You don't have permission to add state to the room"
-            )
-
-        defer.returnValue(True)
-
-    @defer.inlineCallbacks
-    def _can_replace_state(self, event):
-        current_state = yield self.store.get_current_state(
-            event.room_id,
-            event.type,
-            event.state_key,
+        _, _, redact_level = self._get_ops_level_from_event_state(
+            event,
+            auth_events,
         )
 
-        if current_state:
-            current_state = current_state[0]
-
-        user_level = yield self.store.get_power_level(
-            event.room_id,
-            event.user_id,
-        )
-
-        if user_level:
-            user_level = int(user_level)
-        else:
-            user_level = 0
-
-        logger.debug(
-            "Checking power level for %s, %s", event.user_id, user_level
-        )
-        if current_state and hasattr(current_state, "required_power_level"):
-            req = current_state.required_power_level
-
-            logger.debug("Checked power level for %s, %s", event.user_id, req)
-            if user_level < req:
-                raise AuthError(
-                    403,
-                    "You don't have permission to change that state"
-                )
-
-    @defer.inlineCallbacks
-    def _check_redaction(self, event):
-        user_level = yield self.store.get_power_level(
-            event.room_id,
-            event.user_id,
-        )
-
-        if user_level:
-            user_level = int(user_level)
-        else:
-            user_level = 0
-
-        _, _, redact_level  = yield self.store.get_ops_levels(event.room_id)
-
-        if not redact_level:
-            redact_level = 50
-
         if user_level < redact_level:
             raise AuthError(
                 403,
                 "You don't have permission to redact events"
             )
 
-    @defer.inlineCallbacks
-    def _check_power_levels(self, event):
-        for k, v in event.content.items():
-            if k == "default":
-                continue
-
-            # FIXME (erikj): We don't want hsob_Ts in content.
-            if k == "hsob_ts":
-                continue
-
+    def _check_power_levels(self, event, auth_events):
+        user_list = event.content.get("users", {})
+        # Validate users
+        for k, v in user_list.items():
             try:
                 self.hs.parse_userid(k)
             except:
@@ -399,80 +501,69 @@ class Auth(object):
             except:
                 raise SynapseError(400, "Not a valid power level: %s" % (v,))
 
-        current_state = yield self.store.get_current_state(
-            event.room_id,
-            event.type,
-            event.state_key,
-        )
+        key = (event.type, event.state_key, )
+        current_state = auth_events.get(key)
 
         if not current_state:
             return
-        else:
-            current_state = current_state[0]
 
-        user_level = yield self.store.get_power_level(
-            event.room_id,
+        user_level = self._get_power_level_from_event_state(
+            event,
             event.user_id,
+            auth_events,
         )
 
-        if user_level:
-            user_level = int(user_level)
-        else:
-            user_level = 0
+        # Check other levels:
+        levels_to_check = [
+            ("users_default", []),
+            ("events_default", []),
+            ("ban", []),
+            ("redact", []),
+            ("kick", []),
+        ]
 
-        old_list = current_state.content
+        old_list = current_state.content.get("users")
+        for user in set(old_list.keys() + user_list.keys()):
+            levels_to_check.append(
+                (user, ["users"])
+            )
 
-        # FIXME (erikj)
-        old_people = {k: v for k, v in old_list.items() if k.startswith("@")}
-        new_people = {
-            k: v for k, v in event.content.items()
-            if k.startswith("@")
-        }
+        old_list = current_state.content.get("events")
+        new_list = event.content.get("events")
+        for ev_id in set(old_list.keys() + new_list.keys()):
+            levels_to_check.append(
+                (ev_id, ["events"])
+            )
 
-        removed = set(old_people.keys()) - set(new_people.keys())
-        added = set(new_people.keys()) - set(old_people.keys())
-        same = set(old_people.keys()) & set(new_people.keys())
+        old_state = current_state.content
+        new_state = event.content
 
-        for r in removed:
-            if int(old_list[r]) > user_level:
-                raise AuthError(
-                    403,
-                    "You don't have permission to remove user: %s" % (r, )
-                )
+        for level_to_check, dir in levels_to_check:
+            old_loc = old_state
+            for d in dir:
+                old_loc = old_loc.get(d, {})
 
-        for n in added:
-            if int(event.content[n]) > user_level:
+            new_loc = new_state
+            for d in dir:
+                new_loc = new_loc.get(d, {})
+
+            if level_to_check in old_loc:
+                old_level = int(old_loc[level_to_check])
+            else:
+                old_level = None
+
+            if level_to_check in new_loc:
+                new_level = int(new_loc[level_to_check])
+            else:
+                new_level = None
+
+            if new_level is not None and old_level is not None:
+                if new_level == old_level:
+                    continue
+
+            if old_level > user_level or new_level > user_level:
                 raise AuthError(
                     403,
                     "You don't have permission to add ops level greater "
                     "than your own"
                 )
-
-        for s in same:
-            if int(event.content[s]) != int(old_list[s]):
-                if int(event.content[s]) > user_level:
-                    raise AuthError(
-                        403,
-                        "You don't have permission to add ops level greater "
-                        "than your own"
-                    )
-
-        if "default" in old_list:
-            old_default = int(old_list["default"])
-
-            if old_default > user_level:
-                raise AuthError(
-                    403,
-                    "You don't have permission to add ops level greater than "
-                    "your own"
-                )
-
-            if "default" in event.content:
-                new_default = int(event.content["default"])
-
-                if new_default > user_level:
-                    raise AuthError(
-                        403,
-                        "You don't have permission to add ops level greater "
-                        "than your own"
-                    )

synapse/api/constants.py

@@ -58,4 +58,4 @@ class LoginType(object):
     EMAIL_CODE = u"m.login.email.code"
     EMAIL_URL = u"m.login.email.url"
     EMAIL_IDENTITY = u"m.login.email.identity"
-    RECAPTCHA = u"m.login.recaptcha"
+    RECAPTCHA = u"m.login.recaptcha"

synapse/api/errors.py

@@ -17,6 +17,8 @@
 
 import logging
 
+logger = logging.getLogger(__name__)
+
 
 class Codes(object):
     UNAUTHORIZED = "M_UNAUTHORIZED"
@@ -38,7 +40,7 @@ class CodeMessageException(Exception):
     """An exception with integer code and message string attributes."""
 
     def __init__(self, code, msg):
-        logging.error("%s: %s, %s", type(self).__name__, code, msg)
+        logger.info("%s: %s, %s", type(self).__name__, code, msg)
         super(CodeMessageException, self).__init__("%d: %s" % (code, msg))
         self.code = code
         self.msg = msg
@@ -54,7 +56,7 @@ class SynapseError(CodeMessageException):
         """Constructs a synapse error.
 
         Args:
-            code (int): The integer error code (typically an HTTP response code)
+            code (int): The integer error code (an HTTP response code)
             msg (str): The human-readable error message.
             err (str): The error code e.g 'M_FORBIDDEN'
         """
@@ -67,6 +69,7 @@ class SynapseError(CodeMessageException):
             self.errcode,
         )
 
+
 class RoomError(SynapseError):
     """An error raised when a room event fails."""
     pass
@@ -117,6 +120,7 @@ class InvalidCaptchaError(SynapseError):
             error_url=self.error_url,
         )
 
+
 class LimitExceededError(SynapseError):
     """A client has sent too many requests and is being throttled.
     """
@@ -138,7 +142,8 @@ def cs_exception(exception):
     if isinstance(exception, CodeMessageException):
         return exception.error_dict()
     else:
-        logging.error("Unknown exception type: %s", type(exception))
+        logger.error("Unknown exception type: %s", type(exception))
+        return {}
 
 
 def cs_error(msg, code=Codes.UNKNOWN, **kwargs):
@@ -156,3 +161,37 @@ def cs_error(msg, code=Codes.UNKNOWN, **kwargs):
     for key, value in kwargs.iteritems():
         err[key] = value
     return err
+
+
+class FederationError(RuntimeError):
+    """  This class is used to inform remote home servers about erroneous
+    PDUs they sent us.
+
+    FATAL: The remote server could not interpret the source event.
+        (e.g., it was missing a required field)
+    ERROR: The remote server interpreted the event, but it failed some other
+        check (e.g. auth)
+    WARN: The remote server accepted the event, but believes some part of it
+        is wrong (e.g., it referred to an invalid event)
+    """
+
+    def __init__(self, level, code, reason, affected, source=None):
+        if level not in ["FATAL", "ERROR", "WARN"]:
+            raise ValueError("Level is not valid: %s" % (level,))
+        self.level = level
+        self.code = code
+        self.reason = reason
+        self.affected = affected
+        self.source = source
+
+        msg = "%s %s: %s" % (level, code, reason,)
+        super(FederationError, self).__init__(msg)
+
+    def get_dict(self):
+        return {
+            "level": self.level,
+            "code": self.code,
+            "reason": self.reason,
+            "affected": self.affected,
+            "source": self.source if self.source else self.affected,
+        }

synapse/api/events/__init__.py

@@ -13,7 +13,6 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-from synapse.api.errors import SynapseError, Codes
 from synapse.util.jsonobject import JsonEncodedObject
 
 
@@ -56,22 +55,26 @@ class SynapseEvent(JsonEncodedObject):
         "user_id",  # sender/initiator
         "content",  # HTTP body, JSON
         "state_key",
-        "required_power_level",
         "age_ts",
         "prev_content",
-        "prev_state",
+        "replaces_state",
         "redacted_because",
+        "origin_server_ts",
     ]
 
     internal_keys = [
         "is_state",
-        "prev_events",
         "depth",
         "destinations",
         "origin",
         "outlier",
-        "power_level",
         "redacted",
+        "prev_events",
+        "hashes",
+        "signatures",
+        "prev_state",
+        "auth_events",
+        "state_hash",
     ]
 
     required_keys = [
@@ -80,10 +83,12 @@ class SynapseEvent(JsonEncodedObject):
         "content",
     ]
 
+    outlier = False
+
     def __init__(self, raises=True, **kwargs):
         super(SynapseEvent, self).__init__(**kwargs)
-        if "content" in kwargs:
-            self.check_json(self.content, raises=raises)
+        # if "content" in kwargs:
+        #     self.check_json(self.content, raises=raises)
 
     def get_content_template(self):
         """ Retrieve the JSON template for this event as a dict.
@@ -114,65 +119,25 @@ class SynapseEvent(JsonEncodedObject):
         """
         raise NotImplementedError("get_content_template not implemented.")
 
-    def check_json(self, content, raises=True):
-        """Checks the given JSON content abides by the rules of the template.
-
-        Args:
-            content : A JSON object to check.
-            raises: True to raise a SynapseError if the check fails.
-        Returns:
-            True if the content passes the template. Returns False if the check
-            fails and raises=False.
-        Raises:
-            SynapseError if the check fails and raises=True.
-        """
-        # recursively call to inspect each layer
-        err_msg = self._check_json(content, self.get_content_template())
-        if err_msg:
-            if raises:
-                raise SynapseError(400, err_msg, Codes.BAD_JSON)
-            else:
-                return False
-        else:
-            return True
-
-    def _check_json(self, content, template):
-        """Check content and template matches.
-
-        If the template is a dict, each key in the dict will be validated with
-        the content, else it will just compare the types of content and
-        template. This basic type check is required because this function will
-        be recursively called and could be called with just strs or ints.
-
-        Args:
-            content: The content to validate.
-            template: The validation template.
-        Returns:
-            str: An error message if the validation fails, else None.
-        """
-        if type(content) != type(template):
-            return "Mismatched types: %s" % template
-
-        if type(template) == dict:
-            for key in template:
-                if key not in content:
-                    return "Missing %s key" % key
-
-                if type(content[key]) != type(template[key]):
-                    return "Key %s is of the wrong type (got %s, want %s)" % (
-                        key, type(content[key]), type(template[key]))
-
-                if type(content[key]) == dict:
-                    # we must go deeper
-                    msg = self._check_json(content[key], template[key])
-                    if msg:
-                        return msg
-                elif type(content[key]) == list:
-                    # make sure each item type in content matches the template
-                    for entry in content[key]:
-                        msg = self._check_json(entry, template[key][0])
-                        if msg:
-                            return msg
+    def get_pdu_json(self, time_now=None):
+        pdu_json = self.get_full_dict()
+        pdu_json.pop("destinations", None)
+        pdu_json.pop("outlier", None)
+        pdu_json.pop("replaces_state", None)
+        pdu_json.pop("redacted", None)
+        pdu_json.pop("prev_content", None)
+        state_hash = pdu_json.pop("state_hash", None)
+        if state_hash is not None:
+            pdu_json.setdefault("unsigned", {})["state_hash"] = state_hash
+        content = pdu_json.get("content", {})
+        content.pop("prev", None)
+        if time_now is not None and "age_ts" in pdu_json:
+            age = time_now - pdu_json["age_ts"]
+            pdu_json.setdefault("unsigned", {})["age"] = int(age)
+            del pdu_json["age_ts"]
+        user_id = pdu_json.pop("user_id")
+        pdu_json["sender"] = user_id
+        return pdu_json
 
 
 class SynapseStateEvent(SynapseEvent):

synapse/api/events/factory.py

@@ -16,11 +16,13 @@
 from synapse.api.events.room import (
     RoomTopicEvent, MessageEvent, RoomMemberEvent, FeedbackEvent,
     InviteJoinEvent, RoomConfigEvent, RoomNameEvent, GenericEvent,
-    RoomPowerLevelsEvent, RoomJoinRulesEvent, RoomOpsPowerLevelsEvent,
-    RoomCreateEvent, RoomAddStateLevelEvent, RoomSendEventLevelEvent,
+    RoomPowerLevelsEvent, RoomJoinRulesEvent,
+    RoomCreateEvent,
     RoomRedactionEvent,
 )
 
+from synapse.types import EventID
+
 from synapse.util.stringutils import random_string
 
 
@@ -37,9 +39,6 @@ class EventFactory(object):
         RoomPowerLevelsEvent,
         RoomJoinRulesEvent,
         RoomCreateEvent,
-        RoomAddStateLevelEvent,
-        RoomSendEventLevelEvent,
-        RoomOpsPowerLevelsEvent,
         RoomRedactionEvent,
     ]
 
@@ -51,12 +50,26 @@ class EventFactory(object):
         self.clock = hs.get_clock()
         self.hs = hs
 
+        self.event_id_count = 0
+
+    def create_event_id(self):
+        i = str(self.event_id_count)
+        self.event_id_count += 1
+
+        local_part = str(int(self.clock.time())) + i + random_string(5)
+
+        e_id = EventID.create_local(local_part, self.hs)
+
+        return e_id.to_string()
+
     def create_event(self, etype=None, **kwargs):
         kwargs["type"] = etype
         if "event_id" not in kwargs:
-            kwargs["event_id"] = "%s@%s" % (
-                random_string(10), self.hs.hostname
-            )
+            kwargs["event_id"] = self.create_event_id()
+            kwargs["origin"] = self.hs.hostname
+        else:
+            ev_id = self.hs.parse_eventid(kwargs["event_id"])
+            kwargs["origin"] = ev_id.domain
 
         if "origin_server_ts" not in kwargs:
             kwargs["origin_server_ts"] = int(self.clock.time_msec())

synapse/api/events/room.py

@@ -154,27 +154,6 @@ class RoomPowerLevelsEvent(SynapseStateEvent):
         return {}
 
 
-class RoomAddStateLevelEvent(SynapseStateEvent):
-    TYPE = "m.room.add_state_level"
-
-    def get_content_template(self):
-        return {}
-
-
-class RoomSendEventLevelEvent(SynapseStateEvent):
-    TYPE = "m.room.send_event_level"
-
-    def get_content_template(self):
-        return {}
-
-
-class RoomOpsPowerLevelsEvent(SynapseStateEvent):
-    TYPE = "m.room.ops_levels"
-
-    def get_content_template(self):
-        return {}
-
-
 class RoomAliasesEvent(SynapseStateEvent):
     TYPE = "m.room.aliases"
 

synapse/api/events/utils.py

@@ -15,21 +15,36 @@
 
 from .room import (
     RoomMemberEvent, RoomJoinRulesEvent, RoomPowerLevelsEvent,
-    RoomAddStateLevelEvent, RoomSendEventLevelEvent, RoomOpsPowerLevelsEvent,
     RoomAliasesEvent, RoomCreateEvent,
 )
 
+
 def prune_event(event):
-    """ Prunes the given event of all keys we don't know about or think could
-    potentially be dodgy.
+    """ Returns a pruned version of the given event, which removes all keys we
+    don't know about or think could potentially be dodgy.
 
     This is used when we "redact" an event. We want to remove all fields that
     the user has specified, but we do want to keep necessary information like
     type, state_key etc.
     """
+    event_type = event.type
 
-    # Remove all extraneous fields.
-    event.unrecognized_keys = {}
+    allowed_keys = [
+        "event_id",
+        "user_id",
+        "room_id",
+        "hashes",
+        "signatures",
+        "content",
+        "type",
+        "state_key",
+        "depth",
+        "prev_events",
+        "prev_state",
+        "auth_events",
+        "origin",
+        "origin_server_ts",
+    ]
 
     new_content = {}
 
@@ -38,27 +53,33 @@ def prune_event(event):
             if field in event.content:
                 new_content[field] = event.content[field]
 
-    if event.type == RoomMemberEvent.TYPE:
+    if event_type == RoomMemberEvent.TYPE:
         add_fields("membership")
-    elif event.type == RoomCreateEvent.TYPE:
+    elif event_type == RoomCreateEvent.TYPE:
         add_fields("creator")
-    elif event.type == RoomJoinRulesEvent.TYPE:
+    elif event_type == RoomJoinRulesEvent.TYPE:
         add_fields("join_rule")
-    elif event.type == RoomPowerLevelsEvent.TYPE:
-        # TODO: Actually check these are valid user_ids etc.
-        add_fields("default")
-        for k, v in event.content.items():
-            if k.startswith("@") and isinstance(v, (int, long)):
-                new_content[k] = v
-    elif event.type == RoomAddStateLevelEvent.TYPE:
-        add_fields("level")
-    elif event.type == RoomSendEventLevelEvent.TYPE:
-        add_fields("level")
-    elif event.type == RoomOpsPowerLevelsEvent.TYPE:
-        add_fields("kick_level", "ban_level", "redact_level")
-    elif event.type == RoomAliasesEvent.TYPE:
+    elif event_type == RoomPowerLevelsEvent.TYPE:
+        add_fields(
+            "users",
+            "users_default",
+            "events",
+            "events_default",
+            "events_default",
+            "state_default",
+            "ban",
+            "kick",
+            "redact",
+        )
+    elif event_type == RoomAliasesEvent.TYPE:
         add_fields("aliases")
 
-    event.content = new_content
+    allowed_fields = {
+        k: v
+        for k, v in event.get_full_dict().items()
+        if k in allowed_keys
+    }
 
-    return event
+    allowed_fields["content"] = new_content
+
+    return type(event)(**allowed_fields)

synapse/api/events/validator.py

@@ -0,0 +1,87 @@
+# -*- coding: utf-8 -*-
+# Copyright 2014 OpenMarket Ltd
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from synapse.api.errors import SynapseError, Codes
+
+
+class EventValidator(object):
+    def __init__(self, hs):
+        pass
+
+    def validate(self, event):
+        """Checks the given JSON content abides by the rules of the template.
+
+        Args:
+            content : A JSON object to check.
+            raises: True to raise a SynapseError if the check fails.
+        Returns:
+            True if the content passes the template. Returns False if the check
+            fails and raises=False.
+        Raises:
+            SynapseError if the check fails and raises=True.
+        """
+        # recursively call to inspect each layer
+        err_msg = self._check_json_template(
+            event.content,
+            event.get_content_template()
+        )
+        if err_msg:
+            raise SynapseError(400, err_msg, Codes.BAD_JSON)
+        else:
+            return True
+
+    def _check_json_template(self, content, template):
+        """Check content and template matches.
+
+        If the template is a dict, each key in the dict will be validated with
+        the content, else it will just compare the types of content and
+        template. This basic type check is required because this function will
+        be recursively called and could be called with just strs or ints.
+
+        Args:
+            content: The content to validate.
+            template: The validation template.
+        Returns:
+            str: An error message if the validation fails, else None.
+        """
+        if type(content) != type(template):
+            return "Mismatched types: %s" % template
+
+        if type(template) == dict:
+            for key in template:
+                if key not in content:
+                    return "Missing %s key" % key
+
+                if type(content[key]) != type(template[key]):
+                    return "Key %s is of the wrong type (got %s, want %s)" % (
+                        key, type(content[key]), type(template[key]))
+
+                if type(content[key]) == dict:
+                    # we must go deeper
+                    msg = self._check_json_template(
+                        content[key],
+                        template[key]
+                    )
+                    if msg:
+                        return msg
+                elif type(content[key]) == list:
+                    # make sure each item type in content matches the template
+                    for entry in content[key]:
+                        msg = self._check_json_template(
+                            entry,
+                            template[key][0]
+                        )
+                        if msg:
+                            return msg

synapse/app/__init__.py

@@ -12,4 +12,3 @@
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
-

synapse/app/homeserver.py

@@ -26,13 +26,14 @@ from twisted.web.server import Site
 from synapse.http.server import JsonResource, RootRedirect
 from synapse.http.content_repository import ContentRepoResource
 from synapse.http.server_key_resource import LocalKey
-from synapse.http.client import MatrixHttpClient
+from synapse.http.matrixfederationclient import MatrixFederationHttpClient
 from synapse.api.urls import (
     CLIENT_PREFIX, FEDERATION_PREFIX, WEB_CLIENT_PREFIX, CONTENT_REPO_PREFIX,
     SERVER_KEY_PREFIX,
 )
 from synapse.config.homeserver import HomeServerConfig
 from synapse.crypto import context_factory
+from synapse.util.logcontext import LoggingContext
 
 from daemonize import Daemonize
 import twisted.manhole.telnet
@@ -42,6 +43,7 @@ import os
 import re
 import sys
 import sqlite3
+import syweb
 
 logger = logging.getLogger(__name__)
 
@@ -49,7 +51,7 @@ logger = logging.getLogger(__name__)
 class SynapseHomeServer(HomeServer):
 
     def build_http_client(self):
-        return MatrixHttpClient(self)
+        return MatrixFederationHttpClient(self)
 
     def build_resource_for_client(self):
         return JsonResource()
@@ -58,7 +60,9 @@ class SynapseHomeServer(HomeServer):
         return JsonResource()
 
     def build_resource_for_web_client(self):
-        return File("webclient")  # TODO configurable?
+        syweb_path = os.path.dirname(syweb.__file__)
+        webclient_path = os.path.join(syweb_path, "webclient")
+        return File(webclient_path)  # TODO configurable?
 
     def build_resource_for_content_repo(self):
         return ContentRepoResource(
@@ -112,7 +116,7 @@ class SynapseHomeServer(HomeServer):
         # extra resources to existing nodes. See self._resource_id for the key.
         resource_mappings = {}
         for (full_path, resource) in desired_tree:
-            logging.info("Attaching %s to path %s", resource, full_path)
+            logger.info("Attaching %s to path %s", resource, full_path)
             last_resource = self.root_resource
             for path_seg in full_path.split('/')[1:-1]:
                 if not path_seg in last_resource.listNames():
@@ -217,12 +221,12 @@ def setup():
 
     db_name = hs.get_db_name()
 
-    logging.info("Preparing database: %s...", db_name)
+    logger.info("Preparing database: %s...", db_name)
 
     with sqlite3.connect(db_name) as db_conn:
         prepare_database(db_conn)
 
-    logging.info("Database prepared in %s.", db_name)
+    logger.info("Database prepared in %s.", db_name)
 
     hs.get_db_pool()
 
@@ -233,14 +237,17 @@ def setup():
         f.namespace['hs'] = hs
         reactor.listenTCP(config.manhole, f, interface='127.0.0.1')
 
-    hs.start_listening(config.bind_port, config.unsecure_port)
+    bind_port = config.bind_port
+    if config.no_tls:
+        bind_port = None
+    hs.start_listening(bind_port, config.unsecure_port)
 
     if config.daemonize:
         print config.pid_file
         daemon = Daemonize(
             app="synapse-homeserver",
             pid=config.pid_file,
-            action=reactor.run,
+            action=run,
             auto_close_fds=False,
             verbose=True,
             logger=logger,
@@ -251,5 +258,15 @@ def setup():
         reactor.run()
 
 
+def run():
+    with LoggingContext("run"):
+        reactor.run()
+
+
+def main():
+    with LoggingContext("main"):
+        setup()
+
+
 if __name__ == '__main__':
-    setup()
+    main()

synapse/app/synctl.py

@@ -0,0 +1,70 @@
+#!/usr/bin/env python
+# -*- coding: utf-8 -*-
+# Copyright 2014 OpenMarket Ltd
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import sys
+import os
+import subprocess
+import signal
+
+SYNAPSE = ["python", "-m", "synapse.app.homeserver"]
+
+CONFIGFILE = "homeserver.yaml"
+PIDFILE = "homeserver.pid"
+
+GREEN = "\x1b[1;32m"
+NORMAL = "\x1b[m"
+
+
+def start():
+    if not os.path.exists(CONFIGFILE):
+        sys.stderr.write(
+            "No config file found\n"
+            "To generate a config file, run '%s -c %s --generate-config"
+            " --server-name=<server name>'\n" % (
+                " ".join(SYNAPSE), CONFIGFILE
+            )
+        )
+        sys.exit(1)
+    print "Starting ...",
+    args = SYNAPSE
+    args.extend(["--daemonize", "-c", CONFIGFILE, "--pid-file", PIDFILE])
+    subprocess.check_call(args)
+    print GREEN + "started" + NORMAL
+
+
+def stop():
+    if os.path.exists(PIDFILE):
+        pid = int(open(PIDFILE).read())
+        os.kill(pid, signal.SIGTERM)
+        print GREEN + "stopped" + NORMAL
+
+
+def main():
+    action = sys.argv[1] if sys.argv[1:] else "usage"
+    if action == "start":
+        start()
+    elif action == "stop":
+        stop()
+    elif action == "restart":
+        stop()
+        start()
+    else:
+        sys.stderr.write("Usage: %s [start|stop|restart]\n" % (sys.argv[0],))
+        sys.exit(1)
+
+
+if __name__ == "__main__":
+    main()

synapse/config/_base.py

@@ -36,7 +36,10 @@ class Config(object):
         if file_path is None:
             raise ConfigError(
                 "Missing config for %s."
-                " Try running again with --generate-config"
+                " You must specify a path for the config file. You can "
+                "do this with the -c or --config-path option. "
+                "Adding --generate-config along with --server-name "
+                "<server name> will generate a config file at the given path."
                 % (config_name,)
             )
         if not os.path.exists(file_path):
@@ -116,18 +119,25 @@ class Config(object):
             config = {}
             for key, value in vars(args).items():
                 if (key not in set(["config_path", "generate_config"])
-                    and value is not None):
+                        and value is not None):
                     config[key] = value
             with open(config_args.config_path, "w") as config_file:
                 # TODO(paul) it would be lovely if we wrote out vim- and emacs-
                 #   style mode markers into the file, to hint to people that
                 #   this is a YAML file.
                 yaml.dump(config, config_file, default_flow_style=False)
-            print "A config file has been generated in %s for server name '%s') with corresponding SSL keys and self-signed certificates. Please review this file and customise it to your needs." % (config_args.config_path, config['server_name'])
-            print "If this server name is incorrect, you will need to regenerate the SSL certificates"
+            print (
+                "A config file has been generated in %s for server name"
+                " '%s' with corresponding SSL keys and self-signed"
+                " certificates. Please review this file and customise it to"
+                " your needs."
+            ) % (
+                config_args.config_path, config['server_name']
+            )
+            print (
+                "If this server name is incorrect, you will need to regenerate"
+                " the SSL certificates"
+            )
             sys.exit(0)
 
         return cls(args)
-
-
-

synapse/config/database.py

@@ -16,6 +16,7 @@
 from ._base import Config
 import os
 
+
 class DatabaseConfig(Config):
     def __init__(self, args):
         super(DatabaseConfig, self).__init__(args)
@@ -34,4 +35,3 @@ class DatabaseConfig(Config):
     def generate_config(cls, args, config_dir_path):
         super(DatabaseConfig, cls).generate_config(args, config_dir_path)
         args.database_path = os.path.abspath(args.database_path)
-

synapse/config/email.py

@@ -35,5 +35,8 @@ class EmailConfig(Config):
         email_group.add_argument(
             "--email-smtp-server",
             default="",
-            help="The SMTP server to send emails from (e.g. for password resets)."
-        )
+            help=(
+                "The SMTP server to send emails from (e.g. for password"
+                " resets)."
+            )
+        )

synapse/config/logger.py

@@ -14,11 +14,12 @@
 # limitations under the License.
 
 from ._base import Config
-
+from synapse.util.logcontext import LoggingContextFilter
 from twisted.python.log import PythonLoggingObserver
 import logging
 import logging.config
 
+
 class LoggingConfig(Config):
     def __init__(self, args):
         super(LoggingConfig, self).__init__(args)
@@ -45,21 +46,29 @@ class LoggingConfig(Config):
 
     def setup_logging(self):
         log_format = (
-            '%(asctime)s - %(name)s - %(lineno)d - %(levelname)s - %(message)s'
+            "%(asctime)s - %(name)s - %(lineno)d - %(levelname)s - %(request)s"
+            " - %(message)s"
         )
         if self.log_config is None:
 
             level = logging.INFO
             if self.verbosity:
-               level = logging.DEBUG
+                level = logging.DEBUG
 
-               # FIXME: we need a logging.WARN for a -q quiet option
+            # FIXME: we need a logging.WARN for a -q quiet option
+            logger = logging.getLogger('')
+            logger.setLevel(level)
+            formatter = logging.Formatter(log_format)
+            if self.log_file:
+                handler = logging.FileHandler(self.log_file)
+            else:
+                handler = logging.StreamHandler()
+            handler.setFormatter(formatter)
 
-            logging.basicConfig(
-                level=level,
-                filename=self.log_file,
-                format=log_format
-            )
+            handler.addFilter(LoggingContextFilter(request=""))
+
+            logger.addHandler(handler)
+            logger.info("Test")
         else:
             logging.config.fileConfig(self.log_config)
 

synapse/config/ratelimiting.py

@@ -14,6 +14,7 @@
 
 from ._base import Config
 
+
 class RatelimitConfig(Config):
 
     def __init__(self, args):

synapse/config/repository.py

@@ -15,6 +15,7 @@
 
 from ._base import Config
 
+
 class ContentRepositoryConfig(Config):
     def __init__(self, args):
         super(ContentRepositoryConfig, self).__init__(args)

synapse/config/server.py

@@ -30,12 +30,16 @@ class ServerConfig(Config):
         self.pid_file = self.abspath(args.pid_file)
         self.webclient = True
         self.manhole = args.manhole
+        self.no_tls = args.no_tls
 
         if not args.content_addr:
             host = args.server_name
             if ':' not in host:
-                host  = "%s:%d" % (host, args.bind_port)
-            args.content_addr = "https://%s" % (host,)
+                host = "%s:%d" % (host, args.unsecure_port)
+            else:
+                host = host.split(':')[0]
+                host = "%s:%d" % (host, args.unsecure_port)
+            args.content_addr = "http://%s" % (host,)
 
         self.content_addr = args.content_addr
 
@@ -67,6 +71,8 @@ class ServerConfig(Config):
         server_group.add_argument("--content-addr", default=None,
                                   help="The host and scheme to use for the "
                                   "content repository")
+        server_group.add_argument("--no-tls", action='store_true',
+                                  help="Don't bind to the https port.")
 
     def read_signing_key(self, signing_key_path):
         signing_keys = self.read_file(signing_key_path, "signing_key")
@@ -74,7 +80,7 @@ class ServerConfig(Config):
             return syutil.crypto.signing_key.read_signing_keys(
                 signing_keys.splitlines(True)
             )
-        except Exception as e:
+        except Exception:
             raise ConfigError(
                 "Error reading signing_key."
                 " Try running again with --generate-config"
@@ -94,7 +100,7 @@ class ServerConfig(Config):
             with open(args.signing_key_path, "w") as signing_key_file:
                 syutil.crypto.signing_key.write_signing_keys(
                     signing_key_file,
-                    (syutil.crypto.SigningKey.generate("auto"),),
+                    (syutil.crypto.signing_key.generate_singing_key("auto"),),
                 )
         else:
             signing_keys = cls.read_file(args.signing_key_path, "signing_key")

synapse/config/tls.py

@@ -19,7 +19,7 @@ from OpenSSL import crypto
 import subprocess
 import os
 
-GENERATE_DH_PARAMS=False
+GENERATE_DH_PARAMS = False
 
 
 class TlsConfig(Config):

synapse/config/voip.py

@@ -33,7 +33,10 @@ class VoipConfig(Config):
         )
         group.add_argument(
             "--turn-shared-secret", type=str, default=None,
-            help="The shared secret used to compute passwords for the TURN server"
+            help=(
+                "The shared secret used to compute passwords for the TURN"
+                " server"
+            )
         )
         group.add_argument(
             "--turn-user-lifetime", type=int, default=(1000 * 60 * 60),

synapse/crypto/__init__.py

@@ -12,4 +12,3 @@
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
-

synapse/crypto/context_factory.py

@@ -16,6 +16,10 @@ from twisted.internet import ssl
 from OpenSSL import SSL
 from twisted.internet._sslverify import _OpenSSLECCurve, _defaultCurveName
 
+import logging
+
+logger = logging.getLogger(__name__)
+
 
 class ServerContextFactory(ssl.ContextFactory):
     """Factory for PyOpenSSL SSL contexts that are used to handle incoming
@@ -31,7 +35,7 @@ class ServerContextFactory(ssl.ContextFactory):
             _ecCurve = _OpenSSLECCurve(_defaultCurveName)
             _ecCurve.addECKeyToContext(context)
         except:
-            pass
+            logger.exception("Failed to enable eliptic curve for TLS")
         context.set_options(SSL.OP_NO_SSLv2 | SSL.OP_NO_SSLv3)
         context.use_certificate(config.tls_certificate)
         context.use_privatekey(config.tls_private_key)
@@ -40,4 +44,3 @@ class ServerContextFactory(ssl.ContextFactory):
 
     def getContext(self):
         return self._context
-

synapse/crypto/event_signing.py

@@ -0,0 +1,108 @@
+# -*- coding: utf-8 -*-
+
+# Copyright 2014 OpenMarket Ltd
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+from synapse.api.events.utils import prune_event
+from syutil.jsonutil import encode_canonical_json
+from syutil.base64util import encode_base64, decode_base64
+from syutil.crypto.jsonsign import sign_json
+from synapse.api.errors import SynapseError, Codes
+
+import hashlib
+import logging
+
+logger = logging.getLogger(__name__)
+
+
+def check_event_content_hash(event, hash_algorithm=hashlib.sha256):
+    """Check whether the hash for this PDU matches the contents"""
+    computed_hash = _compute_content_hash(event, hash_algorithm)
+    logger.debug("Expecting hash: %s", encode_base64(computed_hash.digest()))
+    if computed_hash.name not in event.hashes:
+        raise SynapseError(
+            400,
+            "Algorithm %s not in hashes %s" % (
+                computed_hash.name, list(event.hashes),
+            ),
+            Codes.UNAUTHORIZED,
+        )
+    message_hash_base64 = event.hashes[computed_hash.name]
+    try:
+        message_hash_bytes = decode_base64(message_hash_base64)
+    except:
+        raise SynapseError(
+            400,
+            "Invalid base64: %s" % (message_hash_base64,),
+            Codes.UNAUTHORIZED,
+        )
+    return message_hash_bytes == computed_hash.digest()
+
+
+def _compute_content_hash(event, hash_algorithm):
+    event_json = event.get_pdu_json()
+    event_json.pop("age_ts", None)
+    event_json.pop("unsigned", None)
+    event_json.pop("signatures", None)
+    event_json.pop("hashes", None)
+    event_json.pop("outlier", None)
+    event_json.pop("destinations", None)
+    event_json_bytes = encode_canonical_json(event_json)
+    return hash_algorithm(event_json_bytes)
+
+
+def compute_event_reference_hash(event, hash_algorithm=hashlib.sha256):
+    tmp_event = prune_event(event)
+    event_json = tmp_event.get_pdu_json()
+    event_json.pop("signatures", None)
+    event_json.pop("age_ts", None)
+    event_json.pop("unsigned", None)
+    event_json_bytes = encode_canonical_json(event_json)
+    hashed = hash_algorithm(event_json_bytes)
+    return (hashed.name, hashed.digest())
+
+
+def compute_event_signature(event, signature_name, signing_key):
+    tmp_event = prune_event(event)
+    redact_json = tmp_event.get_pdu_json()
+    redact_json.pop("age_ts", None)
+    redact_json.pop("unsigned", None)
+    logger.debug("Signing event: %s", redact_json)
+    redact_json = sign_json(redact_json, signature_name, signing_key)
+    return redact_json["signatures"]
+
+
+def add_hashes_and_signatures(event, signature_name, signing_key,
+                              hash_algorithm=hashlib.sha256):
+    if hasattr(event, "old_state_events"):
+        state_json_bytes = encode_canonical_json(
+            [e.event_id for e in event.old_state_events.values()]
+        )
+        hashed = hash_algorithm(state_json_bytes)
+        event.state_hash = {
+            hashed.name: encode_base64(hashed.digest())
+        }
+
+    hashed = _compute_content_hash(event, hash_algorithm=hash_algorithm)
+
+    if not hasattr(event, "hashes"):
+        event.hashes = {}
+    event.hashes[hashed.name] = encode_base64(hashed.digest())
+
+    event.signatures = compute_event_signature(
+        event,
+        signature_name=signature_name,
+        signing_key=signing_key,
+    )

synapse/crypto/keyclient.py

@@ -17,8 +17,8 @@
 from twisted.web.http import HTTPClient
 from twisted.internet.protocol import Factory
 from twisted.internet import defer, reactor
-from twisted.internet.endpoints import connectProtocol
-from synapse.http.endpoint import matrix_endpoint
+from synapse.http.endpoint import matrix_federation_endpoint
+from synapse.util.logcontext import PreserveLoggingContext
 import json
 import logging
 
@@ -31,23 +31,24 @@ def fetch_server_key(server_name, ssl_context_factory):
     """Fetch the keys for a remote server."""
 
     factory = SynapseKeyClientFactory()
-    endpoint = matrix_endpoint(
+    endpoint = matrix_federation_endpoint(
         reactor, server_name, ssl_context_factory, timeout=30
     )
 
     for i in range(5):
         try:
-            protocol = yield endpoint.connect(factory)
-            server_response, server_certificate = yield protocol.remote_key
-            defer.returnValue((server_response, server_certificate))
-            return
+            with PreserveLoggingContext():
+                protocol = yield endpoint.connect(factory)
+                server_response, server_certificate = yield protocol.remote_key
+                defer.returnValue((server_response, server_certificate))
+                return
         except Exception as e:
             logger.exception(e)
     raise IOError("Cannot get key for %s" % server_name)
 
 
 class SynapseKeyClientError(Exception):
-    """The key wasn't retireved from the remote server."""
+    """The key wasn't retrieved from the remote server."""
     pass
 
 
@@ -99,4 +100,3 @@ class SynapseKeyClientProtocol(HTTPClient):
 
 class SynapseKeyClientFactory(Factory):
     protocol = SynapseKeyClientProtocol
-

synapse/crypto/keyring.py

@@ -44,7 +44,7 @@ class Keyring(object):
             raise SynapseError(
                 400,
                 "Not signed with a supported algorithm",
-                 Codes.UNAUTHORIZED,
+                Codes.UNAUTHORIZED,
             )
         try:
             verify_key = yield self.get_server_verify_key(server_name, key_ids)
@@ -100,7 +100,7 @@ class Keyring(object):
         )
 
         if ("signatures" not in response
-            or server_name not in response["signatures"]):
+                or server_name not in response["signatures"]):
             raise ValueError("Key response not signed by remote server")
 
         if "tls_certificate" not in response:
@@ -135,7 +135,7 @@ class Keyring(object):
 
         time_now_ms = self.clock.time_msec()
 
-        self.store.store_server_certificate(
+        yield self.store.store_server_certificate(
             server_name,
             server_name,
             time_now_ms,
@@ -143,7 +143,7 @@ class Keyring(object):
         )
 
         for key_id, key in verify_keys.items():
-            self.store.store_server_verify_key(
+            yield self.store.store_server_verify_key(
                 server_name, server_name, time_now_ms, key
             )
 

synapse/federation/pdu_codec.py

@@ -1,102 +0,0 @@
-# -*- coding: utf-8 -*-
-# Copyright 2014 OpenMarket Ltd
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-from .units import Pdu
-
-import copy
-
-
-def decode_event_id(event_id, server_name):
-    parts = event_id.split("@")
-    if len(parts) < 2:
-        return (event_id, server_name)
-    else:
-        return (parts[0], "".join(parts[1:]))
-
-
-def encode_event_id(pdu_id, origin):
-    return "%s@%s" % (pdu_id, origin)
-
-
-class PduCodec(object):
-
-    def __init__(self, hs):
-        self.server_name = hs.hostname
-        self.event_factory = hs.get_event_factory()
-        self.clock = hs.get_clock()
-
-    def event_from_pdu(self, pdu):
-        kwargs = {}
-
-        kwargs["event_id"] = encode_event_id(pdu.pdu_id, pdu.origin)
-        kwargs["room_id"] = pdu.context
-        kwargs["etype"] = pdu.pdu_type
-        kwargs["prev_events"] = [
-            encode_event_id(p[0], p[1]) for p in pdu.prev_pdus
-        ]
-
-        if hasattr(pdu, "prev_state_id") and hasattr(pdu, "prev_state_origin"):
-            kwargs["prev_state"] = encode_event_id(
-                pdu.prev_state_id, pdu.prev_state_origin
-            )
-
-        kwargs.update({
-            k: v
-            for k, v in pdu.get_full_dict().items()
-            if k not in [
-                "pdu_id",
-                "context",
-                "pdu_type",
-                "prev_pdus",
-                "prev_state_id",
-                "prev_state_origin",
-            ]
-        })
-
-        return self.event_factory.create_event(**kwargs)
-
-    def pdu_from_event(self, event):
-        d = event.get_full_dict()
-
-        d["pdu_id"], d["origin"] = decode_event_id(
-            event.event_id, self.server_name
-        )
-        d["context"] = event.room_id
-        d["pdu_type"] = event.type
-
-        if hasattr(event, "prev_events"):
-            d["prev_pdus"] = [
-                decode_event_id(e, self.server_name)
-                for e in event.prev_events
-            ]
-
-        if hasattr(event, "prev_state"):
-            d["prev_state_id"], d["prev_state_origin"] = (
-                decode_event_id(event.prev_state, self.server_name)
-            )
-
-        if hasattr(event, "state_key"):
-            d["is_state"] = True
-
-        kwargs = copy.deepcopy(event.unrecognized_keys)
-        kwargs.update({
-            k: v for k, v in d.items()
-            if k not in ["event_id", "room_id", "type", "prev_events"]
-        })
-
-        if "origin_server_ts" not in kwargs:
-            kwargs["origin_server_ts"] = int(self.clock.time_msec())
-
-        return Pdu(**kwargs)

synapse/federation/persistence.py

@@ -21,8 +21,6 @@ These actions are mostly only used by the :py:mod:`.replication` module.
 
 from twisted.internet import defer
 
-from .units import Pdu
-
 from synapse.util.logutils import log_function
 
 import json
@@ -32,76 +30,6 @@ import logging
 logger = logging.getLogger(__name__)
 
 
-class PduActions(object):
-    """ Defines persistence actions that relate to handling PDUs.
-    """
-
-    def __init__(self, datastore):
-        self.store = datastore
-
-    @log_function
-    def mark_as_processed(self, pdu):
-        """ Persist the fact that we have fully processed the given `Pdu`
-
-        Returns:
-            Deferred
-        """
-        return self.store.mark_pdu_as_processed(pdu.pdu_id, pdu.origin)
-
-    @defer.inlineCallbacks
-    @log_function
-    def after_transaction(self, transaction_id, destination, origin):
-        """ Returns all `Pdu`s that we sent to the given remote home server
-        after a given transaction id.
-
-        Returns:
-            Deferred: Results in a list of `Pdu`s
-        """
-        results = yield self.store.get_pdus_after_transaction(
-            transaction_id,
-            destination
-        )
-
-        defer.returnValue([Pdu.from_pdu_tuple(p) for p in results])
-
-    @defer.inlineCallbacks
-    @log_function
-    def get_all_pdus_from_context(self, context):
-        results = yield self.store.get_all_pdus_from_context(context)
-        defer.returnValue([Pdu.from_pdu_tuple(p) for p in results])
-
-    @defer.inlineCallbacks
-    @log_function
-    def backfill(self, context, pdu_list, limit):
-        """ For a given list of PDU id and origins return the proceeding
-        `limit` `Pdu`s in the given `context`.
-
-        Returns:
-            Deferred: Results in a list of `Pdu`s.
-        """
-        results = yield self.store.get_backfill(
-            context, pdu_list, limit
-        )
-
-        defer.returnValue([Pdu.from_pdu_tuple(p) for p in results])
-
-    @log_function
-    def is_new(self, pdu):
-        """ When we receive a `Pdu` from a remote home server, we want to
-        figure out whether it is `new`, i.e. it is not some historic PDU that
-        we haven't seen simply because we haven't backfilled back that far.
-
-        Returns:
-            Deferred: Results in a `bool`
-        """
-        return self.store.is_pdu_new(
-            pdu_id=pdu.pdu_id,
-            origin=pdu.origin,
-            context=pdu.context,
-            depth=pdu.depth
-        )
-
-
 class TransactionActions(object):
     """ Defines persistence actions that relate to handling Transactions.
     """
@@ -158,7 +86,6 @@ class TransactionActions(object):
             transaction.transaction_id,
             transaction.destination,
             transaction.origin_server_ts,
-            [(p["pdu_id"], p["origin"]) for p in transaction.pdus]
         )
 
     @log_function

synapse/federation/replication.py

@@ -19,11 +19,12 @@ a given transport.
 
 from twisted.internet import defer
 
-from .units import Transaction, Pdu, Edu
+from .units import Transaction, Edu
 
-from .persistence import PduActions, TransactionActions
+from .persistence import TransactionActions
 
 from synapse.util.logutils import log_function
+from synapse.util.logcontext import PreserveLoggingContext
 
 import logging
 
@@ -57,7 +58,7 @@ class ReplicationLayer(object):
         self.transport_layer.register_request_handler(self)
 
         self.store = hs.get_datastore()
-        self.pdu_actions = PduActions(self.store)
+        # self.pdu_actions = PduActions(self.store)
         self.transaction_actions = TransactionActions(self.store)
 
         self._transaction_queue = _TransactionQueue(
@@ -72,6 +73,8 @@ class ReplicationLayer(object):
 
         self._clock = hs.get_clock()
 
+        self.event_factory = hs.get_event_factory()
+
     def set_handler(self, handler):
         """Sets the handler that the replication layer will use to communicate
         receipt of new PDUs from other home servers. The required methods are
@@ -81,7 +84,7 @@ class ReplicationLayer(object):
 
     def register_edu_handler(self, edu_type, handler):
         if edu_type in self.edu_handlers:
-            raise KeyError("Already have an EDU handler for %s" % (edu_type))
+            raise KeyError("Already have an EDU handler for %s" % (edu_type,))
 
         self.edu_handlers[edu_type] = handler
 
@@ -102,24 +105,17 @@ class ReplicationLayer(object):
           object to encode as JSON.
         """
         if query_type in self.query_handlers:
-            raise KeyError("Already have a Query handler for %s" % (query_type))
+            raise KeyError(
+                "Already have a Query handler for %s" % (query_type,)
+            )
 
         self.query_handlers[query_type] = handler
 
-    @defer.inlineCallbacks
     @log_function
     def send_pdu(self, pdu):
         """Informs the replication layer about a new PDU generated within the
         home server that should be transmitted to others.
 
-        This will fill out various attributes on the PDU object, e.g. the
-        `prev_pdus` key.
-
-        *Note:* The home server should always call `send_pdu` even if it knows
-        that it does not need to be replicated to other home servers. This is
-        in case e.g. someone else joins via a remote home server and then
-        backfills.
-
         TODO: Figure out when we should actually resolve the deferred.
 
         Args:
@@ -132,18 +128,15 @@ class ReplicationLayer(object):
         order = self._order
         self._order += 1
 
-        logger.debug("[%s] Persisting PDU", pdu.pdu_id)
-
-        # Save *before* trying to send
-        yield self.store.persist_event(pdu=pdu)
-
-        logger.debug("[%s] Persisted PDU", pdu.pdu_id)
-        logger.debug("[%s] transaction_layer.enqueue_pdu... ", pdu.pdu_id)
+        logger.debug("[%s] transaction_layer.enqueue_pdu... ", pdu.event_id)
 
         # TODO, add errback, etc.
         self._transaction_queue.enqueue_pdu(pdu, order)
 
-        logger.debug("[%s] transaction_layer.enqueue_pdu... done", pdu.pdu_id)
+        logger.debug(
+            "[%s] transaction_layer.enqueue_pdu... done",
+            pdu.event_id
+        )
 
     @log_function
     def send_edu(self, destination, edu_type, content):
@@ -158,6 +151,11 @@ class ReplicationLayer(object):
         self._transaction_queue.enqueue_edu(edu)
         return defer.succeed(None)
 
+    @log_function
+    def send_failure(self, failure, destination):
+        self._transaction_queue.enqueue_failure(failure, destination)
+        return defer.succeed(None)
+
     @log_function
     def make_query(self, destination, query_type, args,
                    retry_on_dns_fail=True):
@@ -181,7 +179,7 @@ class ReplicationLayer(object):
 
     @defer.inlineCallbacks
     @log_function
-    def backfill(self, dest, context, limit):
+    def backfill(self, dest, context, limit, extremities):
         """Requests some more historic PDUs for the given context from the
         given destination server.
 
@@ -189,12 +187,12 @@ class ReplicationLayer(object):
             dest (str): The remote home server to ask.
             context (str): The context to backfill.
             limit (int): The maximum number of PDUs to return.
+            extremities (list): List of PDU id and origins of the first pdus
+                we have seen from the context
 
         Returns:
             Deferred: Results in the received PDUs.
         """
-        extremities = yield self.store.get_oldest_pdus_in_context(context)
-
         logger.debug("backfill extrem=%s", extremities)
 
         # If there are no extremeties then we've (probably) reached the start.
@@ -208,15 +206,18 @@ class ReplicationLayer(object):
 
         transaction = Transaction(**transaction_data)
 
-        pdus = [Pdu(outlier=False, **p) for p in transaction.pdus]
+        pdus = [
+            self.event_from_pdu_json(p, outlier=False)
+            for p in transaction.pdus
+        ]
         for pdu in pdus:
-            yield self._handle_new_pdu(pdu, backfilled=True)
+            yield self._handle_new_pdu(dest, pdu, backfilled=True)
 
         defer.returnValue(pdus)
 
     @defer.inlineCallbacks
     @log_function
-    def get_pdu(self, destination, pdu_origin, pdu_id, outlier=False):
+    def get_pdu(self, destination, event_id, outlier=False):
         """Requests the PDU with given origin and ID from the remote home
         server.
 
@@ -225,7 +226,7 @@ class ReplicationLayer(object):
         Args:
             destination (str): Which home server to query
             pdu_origin (str): The home server that originally sent the pdu.
-            pdu_id (str)
+            event_id (str)
             outlier (bool): Indicates whether the PDU is an `outlier`, i.e. if
                 it's from an arbitary point in the context as opposed to part
                 of the current block of PDUs. Defaults to `False`
@@ -234,23 +235,27 @@ class ReplicationLayer(object):
             Deferred: Results in the requested PDU.
         """
 
-        transaction_data = yield self.transport_layer.get_pdu(
-            destination, pdu_origin, pdu_id)
+        transaction_data = yield self.transport_layer.get_event(
+            destination, event_id
+        )
 
         transaction = Transaction(**transaction_data)
 
-        pdu_list = [Pdu(outlier=outlier, **p) for p in transaction.pdus]
+        pdu_list = [
+            self.event_from_pdu_json(p, outlier=outlier)
+            for p in transaction.pdus
+        ]
 
         pdu = None
         if pdu_list:
             pdu = pdu_list[0]
-            yield self._handle_new_pdu(pdu)
+            yield self._handle_new_pdu(destination, pdu)
 
         defer.returnValue(pdu)
 
     @defer.inlineCallbacks
     @log_function
-    def get_state_for_context(self, destination, context):
+    def get_state_for_context(self, destination, context, event_id=None):
         """Requests all of the `current` state PDUs for a given context from
         a remote home server.
 
@@ -263,29 +268,41 @@ class ReplicationLayer(object):
         """
 
         transaction_data = yield self.transport_layer.get_context_state(
-            destination, context)
+            destination,
+            context,
+            event_id=event_id,
+        )
 
         transaction = Transaction(**transaction_data)
-
-        pdus = [Pdu(outlier=True, **p) for p in transaction.pdus]
-        for pdu in pdus:
-            yield self._handle_new_pdu(pdu)
+        pdus = [
+            self.event_from_pdu_json(p, outlier=True)
+            for p in transaction.pdus
+        ]
 
         defer.returnValue(pdus)
 
     @defer.inlineCallbacks
     @log_function
-    def on_context_pdus_request(self, context):
-        pdus = yield self.pdu_actions.get_all_pdus_from_context(
-            context
+    def get_event_auth(self, destination, context, event_id):
+        res = yield self.transport_layer.get_event_auth(
+            destination, context, event_id,
         )
-        defer.returnValue((200, self._transaction_from_pdus(pdus).get_dict()))
+
+        auth_chain = [
+            self.event_from_pdu_json(p, outlier=True)
+            for p in res["auth_chain"]
+        ]
+
+        auth_chain.sort(key=lambda e: e.depth)
+
+        defer.returnValue(auth_chain)
 
     @defer.inlineCallbacks
     @log_function
-    def on_backfill_request(self, context, versions, limit):
-
-        pdus = yield self.pdu_actions.backfill(context, versions, limit)
+    def on_backfill_request(self, origin, context, versions, limit):
+        pdus = yield self.handler.on_backfill_request(
+            origin, context, versions, limit
+        )
 
         defer.returnValue((200, self._transaction_from_pdus(pdus).get_dict()))
 
@@ -295,11 +312,17 @@ class ReplicationLayer(object):
         transaction = Transaction(**transaction_data)
 
         for p in transaction.pdus:
+            if "unsigned" in p:
+                unsigned = p["unsigned"]
+                if "age" in unsigned:
+                    p["age"] = unsigned["age"]
             if "age" in p:
                 p["age_ts"] = int(self._clock.time_msec()) - int(p["age"])
                 del p["age"]
 
-        pdu_list = [Pdu(**p) for p in transaction.pdus]
+        pdu_list = [
+            self.event_from_pdu_json(p) for p in transaction.pdus
+        ]
 
         logger.debug("[%s] Got transaction", transaction.transaction_id)
 
@@ -313,15 +336,20 @@ class ReplicationLayer(object):
 
         logger.debug("[%s] Transacition is new", transaction.transaction_id)
 
-        dl = []
-        for pdu in pdu_list:
-            dl.append(self._handle_new_pdu(pdu))
+        with PreserveLoggingContext():
+            dl = []
+            for pdu in pdu_list:
+                dl.append(self._handle_new_pdu(transaction.origin, pdu))
 
-        if hasattr(transaction, "edus"):
-            for edu in [Edu(**x) for x in transaction.edus]:
-                self.received_edu(transaction.origin, edu.edu_type, edu.content)
+            if hasattr(transaction, "edus"):
+                for edu in [Edu(**x) for x in transaction.edus]:
+                    self.received_edu(
+                        transaction.origin,
+                        edu.edu_type,
+                        edu.content
+                    )
 
-        results = yield defer.DeferredList(dl)
+            results = yield defer.DeferredList(dl)
 
         ret = []
         for r in results:
@@ -347,20 +375,22 @@ class ReplicationLayer(object):
 
     @defer.inlineCallbacks
     @log_function
-    def on_context_state_request(self, context):
-        results = yield self.store.get_current_state_for_context(
-            context
-        )
+    def on_context_state_request(self, origin, context, event_id):
+        if event_id:
+            pdus = yield self.handler.get_state_for_pdu(
+                origin,
+                context,
+                event_id,
+            )
+        else:
+            raise NotImplementedError("Specify an event")
 
-        logger.debug("Context returning %d results", len(results))
-
-        pdus = [Pdu.from_pdu_tuple(p) for p in results]
         defer.returnValue((200, self._transaction_from_pdus(pdus).get_dict()))
 
     @defer.inlineCallbacks
     @log_function
-    def on_pdu_request(self, pdu_origin, pdu_id):
-        pdu = yield self._get_persisted_pdu(pdu_id, pdu_origin)
+    def on_pdu_request(self, origin, event_id):
+        pdu = yield self._get_persisted_pdu(origin, event_id)
 
         if pdu:
             defer.returnValue(
@@ -372,20 +402,7 @@ class ReplicationLayer(object):
     @defer.inlineCallbacks
     @log_function
     def on_pull_request(self, origin, versions):
-        transaction_id = max([int(v) for v in versions])
-
-        response = yield self.pdu_actions.after_transaction(
-            transaction_id,
-            origin,
-            self.server_name
-        )
-
-        if not response:
-            response = []
-
-        defer.returnValue(
-            (200, self._transaction_from_pdus(response).get_dict())
-        )
+        raise NotImplementedError("Pull transacions not implemented")
 
     @defer.inlineCallbacks
     def on_query_request(self, query_type, args):
@@ -393,95 +410,266 @@ class ReplicationLayer(object):
             response = yield self.query_handlers[query_type](args)
             defer.returnValue((200, response))
         else:
-            defer.returnValue((404, "No handler for Query type '%s'"
-                % (query_type)
-            ))
+            defer.returnValue(
+                (404, "No handler for Query type '%s'" % (query_type, ))
+            )
 
     @defer.inlineCallbacks
+    def on_make_join_request(self, context, user_id):
+        pdu = yield self.handler.on_make_join_request(context, user_id)
+        time_now = self._clock.time_msec()
+        defer.returnValue({
+            "event": pdu.get_pdu_json(time_now),
+        })
+
+    @defer.inlineCallbacks
+    def on_invite_request(self, origin, content):
+        pdu = self.event_from_pdu_json(content)
+        ret_pdu = yield self.handler.on_invite_request(origin, pdu)
+        time_now = self._clock.time_msec()
+        defer.returnValue(
+            (
+                200,
+                {
+                    "event": ret_pdu.get_pdu_json(time_now),
+                }
+            )
+        )
+
+    @defer.inlineCallbacks
+    def on_send_join_request(self, origin, content):
+        pdu = self.event_from_pdu_json(content)
+        res_pdus = yield self.handler.on_send_join_request(origin, pdu)
+        time_now = self._clock.time_msec()
+        defer.returnValue((200, {
+            "state": [p.get_pdu_json(time_now) for p in res_pdus["state"]],
+            "auth_chain": [
+                p.get_pdu_json(time_now) for p in res_pdus["auth_chain"]
+            ],
+        }))
+
+    @defer.inlineCallbacks
+    def on_event_auth(self, origin, context, event_id):
+        time_now = self._clock.time_msec()
+        auth_pdus = yield self.handler.on_event_auth(event_id)
+        defer.returnValue(
+            (
+                200,
+                {
+                    "auth_chain": [
+                        a.get_pdu_json(time_now) for a in auth_pdus
+                    ],
+                }
+            )
+        )
+
+    @defer.inlineCallbacks
+    def make_join(self, destination, context, user_id):
+        ret = yield self.transport_layer.make_join(
+            destination=destination,
+            context=context,
+            user_id=user_id,
+        )
+
+        pdu_dict = ret["event"]
+
+        logger.debug("Got response to make_join: %s", pdu_dict)
+
+        defer.returnValue(self.event_from_pdu_json(pdu_dict))
+
+    @defer.inlineCallbacks
+    def send_join(self, destination, pdu):
+        time_now = self._clock.time_msec()
+        _, content = yield self.transport_layer.send_join(
+            destination,
+            pdu.room_id,
+            pdu.event_id,
+            pdu.get_pdu_json(time_now),
+        )
+
+        logger.debug("Got content: %s", content)
+
+        state = [
+            self.event_from_pdu_json(p, outlier=True)
+            for p in content.get("state", [])
+        ]
+
+        # FIXME: We probably want to do something with the auth_chain given
+        # to us
+
+        auth_chain = [
+            self.event_from_pdu_json(p, outlier=True)
+            for p in content.get("auth_chain", [])
+        ]
+
+        auth_chain.sort(key=lambda e: e.depth)
+
+        defer.returnValue({
+            "state": state,
+            "auth_chain": auth_chain,
+        })
+
+    @defer.inlineCallbacks
+    def send_invite(self, destination, context, event_id, pdu):
+        time_now = self._clock.time_msec()
+        code, content = yield self.transport_layer.send_invite(
+            destination=destination,
+            context=context,
+            event_id=event_id,
+            content=pdu.get_pdu_json(time_now),
+        )
+
+        pdu_dict = content["event"]
+
+        logger.debug("Got response to send_invite: %s", pdu_dict)
+
+        defer.returnValue(self.event_from_pdu_json(pdu_dict))
+
     @log_function
-    def _get_persisted_pdu(self, pdu_id, pdu_origin):
+    def _get_persisted_pdu(self, origin, event_id, do_auth=True):
         """ Get a PDU from the database with given origin and id.
 
         Returns:
             Deferred: Results in a `Pdu`.
         """
-        pdu_tuple = yield self.store.get_pdu(pdu_id, pdu_origin)
-
-        defer.returnValue(Pdu.from_pdu_tuple(pdu_tuple))
+        return self.handler.get_persisted_pdu(
+            origin, event_id, do_auth=do_auth
+        )
 
     def _transaction_from_pdus(self, pdu_list):
         """Returns a new Transaction containing the given PDUs suitable for
         transmission.
         """
-        pdus = [p.get_dict() for p in pdu_list]
-        for p in pdus:
-            if "age_ts" in pdus:
-                p["age"] = int(self.clock.time_msec()) - p["age_ts"]
-
+        time_now = self._clock.time_msec()
+        pdus = [p.get_pdu_json(time_now) for p in pdu_list]
         return Transaction(
             origin=self.server_name,
             pdus=pdus,
-            origin_server_ts=int(self._clock.time_msec()),
+            origin_server_ts=int(time_now),
             destination=None,
         )
 
     @defer.inlineCallbacks
     @log_function
-    def _handle_new_pdu(self, pdu, backfilled=False):
+    def _handle_new_pdu(self, origin, pdu, backfilled=False):
         # We reprocess pdus when we have seen them only as outliers
-        existing = yield self._get_persisted_pdu(pdu.pdu_id, pdu.origin)
+        existing = yield self._get_persisted_pdu(
+            origin, pdu.event_id, do_auth=False
+        )
 
         if existing and (not existing.outlier or pdu.outlier):
-            logger.debug("Already seen pdu %s %s", pdu.pdu_id, pdu.origin)
+            logger.debug("Already seen pdu %s", pdu.event_id)
             defer.returnValue({})
             return
 
+        state = None
+
+        # We need to make sure we have all the auth events.
+        # for e_id, _ in pdu.auth_events:
+        #     exists = yield self._get_persisted_pdu(
+        #         origin,
+        #         e_id,
+        #         do_auth=False
+        #     )
+        #
+        #     if not exists:
+        #         try:
+        #             logger.debug(
+        #                 "_handle_new_pdu fetch missing auth event %s from %s",
+        #                 e_id,
+        #                 origin,
+        #             )
+        #
+        #             yield self.get_pdu(
+        #                 origin,
+        #                 event_id=e_id,
+        #                 outlier=True,
+        #             )
+        #
+        #             logger.debug("Processed pdu %s", e_id)
+        #         except:
+        #             logger.warn(
+        #                 "Failed to get auth event %s from %s",
+        #                 e_id,
+        #                 origin
+        #             )
+
         # Get missing pdus if necessary.
-        is_new = yield self.pdu_actions.is_new(pdu)
-        if is_new and not pdu.outlier:
+        if not pdu.outlier:
             # We only backfill backwards to the min depth.
-            min_depth = yield self.store.get_min_depth_for_context(pdu.context)
+            min_depth = yield self.handler.get_min_depth_for_context(
+                pdu.room_id
+            )
+
+            logger.debug(
+                "_handle_new_pdu min_depth for %s: %d",
+                pdu.room_id, min_depth
+            )
 
             if min_depth and pdu.depth > min_depth:
-                for pdu_id, origin in pdu.prev_pdus:
-                    exists = yield self._get_persisted_pdu(pdu_id, origin)
+                for event_id, hashes in pdu.prev_events:
+                    exists = yield self._get_persisted_pdu(
+                        origin,
+                        event_id,
+                        do_auth=False
+                    )
 
                     if not exists:
-                        logger.debug("Requesting pdu %s %s", pdu_id, origin)
+                        logger.debug(
+                            "_handle_new_pdu requesting pdu %s",
+                            event_id
+                        )
 
                         try:
                             yield self.get_pdu(
-                                pdu.origin,
-                                pdu_id=pdu_id,
-                                pdu_origin=origin
+                                origin,
+                                event_id=event_id,
                             )
-                            logger.debug("Processed pdu %s %s", pdu_id, origin)
+                            logger.debug("Processed pdu %s", event_id)
                         except:
                             # TODO(erikj): Do some more intelligent retries.
                             logger.exception("Failed to get PDU")
-
-        # Persist the Pdu, but don't mark it as processed yet.
-        yield self.store.persist_event(pdu=pdu)
+            else:
+                # We need to get the state at this event, since we have reached
+                # a backward extremity edge.
+                logger.debug(
+                    "_handle_new_pdu getting state for %s",
+                    pdu.room_id
+                )
+                state = yield self.get_state_for_context(
+                    origin, pdu.room_id, pdu.event_id,
+                )
 
         if not backfilled:
-            ret = yield self.handler.on_receive_pdu(pdu, backfilled=backfilled)
+            ret = yield self.handler.on_receive_pdu(
+                origin,
+                pdu,
+                backfilled=backfilled,
+                state=state,
+            )
         else:
             ret = None
 
-        yield self.pdu_actions.mark_as_processed(pdu)
+        # yield self.pdu_actions.mark_as_processed(pdu)
 
         defer.returnValue(ret)
 
     def __str__(self):
         return "<ReplicationLayer(%s)>" % self.server_name
 
-
-class ReplicationHandler(object):
-    """This defines the methods that the :py:class:`.ReplicationLayer` will
-    use to communicate with the rest of the home server.
-    """
-    def on_receive_pdu(self, pdu):
-        raise NotImplementedError("on_receive_pdu")
+    def event_from_pdu_json(self, pdu_json, outlier=False):
+        #TODO: Check we have all the PDU keys here
+        pdu_json.setdefault("hashes", {})
+        pdu_json.setdefault("signatures", {})
+        sender = pdu_json.pop("sender", None)
+        if sender is not None:
+            pdu_json["user_id"] = sender
+        state_hash = pdu_json.get("unsigned", {}).pop("state_hash", None)
+        if state_hash is not None:
+            pdu_json["state_hash"] = state_hash
+        return self.event_factory.create_event(
+            pdu_json["type"], outlier=outlier, **pdu_json
+        )
 
 
 class _TransactionQueue(object):
@@ -509,6 +697,9 @@ class _TransactionQueue(object):
         # destination -> list of tuple(edu, deferred)
         self.pending_edus_by_dest = {}
 
+        # destination -> list of tuple(failure, deferred)
+        self.pending_failures_by_dest = {}
+
         # HACK to get unique tx id
         self._next_txn_id = int(self._clock.time_msec())
 
@@ -537,7 +728,8 @@ class _TransactionQueue(object):
                 (pdu, deferred, order)
             )
 
-            self._attempt_new_transaction(destination)
+            with PreserveLoggingContext():
+                self._attempt_new_transaction(destination)
 
             deferreds.append(deferred)
 
@@ -557,10 +749,24 @@ class _TransactionQueue(object):
                 deferred.errback(failure)
             else:
                 logger.exception("Failed to send edu", failure)
-        self._attempt_new_transaction(destination).addErrback(eb)
+
+        with PreserveLoggingContext():
+            self._attempt_new_transaction(destination).addErrback(eb)
 
         return deferred
 
+    @defer.inlineCallbacks
+    def enqueue_failure(self, failure, destination):
+        deferred = defer.Deferred()
+
+        self.pending_failures_by_dest.setdefault(
+            destination, []
+        ).append(
+            (failure, deferred)
+        )
+
+        yield deferred
+
     @defer.inlineCallbacks
     @log_function
     def _attempt_new_transaction(self, destination):
@@ -570,8 +776,9 @@ class _TransactionQueue(object):
         #  list of (pending_pdu, deferred, order)
         pending_pdus = self.pending_pdus_by_dest.pop(destination, [])
         pending_edus = self.pending_edus_by_dest.pop(destination, [])
+        pending_failures = self.pending_failures_by_dest.pop(destination, [])
 
-        if not pending_pdus and not pending_edus:
+        if not pending_pdus and not pending_edus and not pending_failures:
             return
 
         logger.debug("TX [%s] Attempting new transaction", destination)
@@ -581,7 +788,11 @@ class _TransactionQueue(object):
 
         pdus = [x[0] for x in pending_pdus]
         edus = [x[0] for x in pending_edus]
-        deferreds = [x[1] for x in pending_pdus + pending_edus]
+        failures = [x[0].get_dict() for x in pending_failures]
+        deferreds = [
+            x[1]
+            for x in pending_pdus + pending_edus + pending_failures
+        ]
 
         try:
             self.pending_transactions[destination] = 1
@@ -589,12 +800,13 @@ class _TransactionQueue(object):
             logger.debug("TX [%s] Persisting transaction...", destination)
 
             transaction = Transaction.create_new(
-                origin_server_ts=self._clock.time_msec(),
+                origin_server_ts=int(self._clock.time_msec()),
                 transaction_id=str(self._next_txn_id),
                 origin=self.server_name,
                 destination=destination,
                 pdus=pdus,
                 edus=edus,
+                pdu_failures=failures,
             )
 
             self._next_txn_id += 1
@@ -614,7 +826,9 @@ class _TransactionQueue(object):
                 if "pdus" in data:
                     for p in data["pdus"]:
                         if "age_ts" in p:
-                            p["age"] = now - int(p["age_ts"])
+                            unsigned = p.setdefault("unsigned", {})
+                            unsigned["age"] = now - int(p["age_ts"])
+                            del p["age_ts"]
                 return data
 
             code, response = yield self.transport_layer.send_transaction(

synapse/federation/transport.py

@@ -72,7 +72,7 @@ class TransportLayer(object):
         self.received_handler = None
 
     @log_function
-    def get_context_state(self, destination, context):
+    def get_context_state(self, destination, context, event_id=None):
         """ Requests all state for a given context (i.e. room) from the
         given server.
 
@@ -89,54 +89,62 @@ class TransportLayer(object):
 
         subpath = "/state/%s/" % context
 
-        return self._do_request_for_transaction(destination, subpath)
+        args = {}
+        if event_id:
+            args["event_id"] = event_id
+
+        return self._do_request_for_transaction(
+            destination, subpath, args=args
+        )
 
     @log_function
-    def get_pdu(self, destination, pdu_origin, pdu_id):
+    def get_event(self, destination, event_id):
         """ Requests the pdu with give id and origin from the given server.
 
         Args:
             destination (str): The host name of the remote home server we want
                 to get the state from.
-            pdu_origin (str): The home server which created the PDU.
-            pdu_id (str): The id of the PDU being requested.
+            event_id (str): The id of the event being requested.
 
         Returns:
             Deferred: Results in a dict received from the remote homeserver.
         """
-        logger.debug("get_pdu dest=%s, pdu_origin=%s, pdu_id=%s",
-                     destination, pdu_origin, pdu_id)
+        logger.debug("get_pdu dest=%s, event_id=%s",
+                     destination, event_id)
 
-        subpath = "/pdu/%s/%s/" % (pdu_origin, pdu_id)
+        subpath = "/event/%s/" % (event_id, )
 
         return self._do_request_for_transaction(destination, subpath)
 
     @log_function
-    def backfill(self, dest, context, pdu_tuples, limit):
+    def backfill(self, dest, context, event_tuples, limit):
         """ Requests `limit` previous PDUs in a given context before list of
         PDUs.
 
         Args:
             dest (str)
             context (str)
-            pdu_tuples (list)
+            event_tuples (list)
             limt (int)
 
         Returns:
             Deferred: Results in a dict received from the remote homeserver.
         """
         logger.debug(
-            "backfill dest=%s, context=%s, pdu_tuples=%s, limit=%s",
-            dest, context, repr(pdu_tuples), str(limit)
+            "backfill dest=%s, context=%s, event_tuples=%s, limit=%s",
+            dest, context, repr(event_tuples), str(limit)
         )
 
-        if not pdu_tuples:
+        if not event_tuples:
+            # TODO: raise?
             return
 
-        subpath = "/backfill/%s/" % context
+        subpath = "/backfill/%s/" % (context,)
 
-        args = {"v": ["%s,%s" % (i, o) for i, o in pdu_tuples]}
-        args["limit"] = limit
+        args = {
+            "v": event_tuples,
+            "limit": [str(limit)],
+        }
 
         return self._do_request_for_transaction(
             dest,
@@ -197,6 +205,72 @@ class TransportLayer(object):
 
         defer.returnValue(response)
 
+    @defer.inlineCallbacks
+    @log_function
+    def make_join(self, destination, context, user_id, retry_on_dns_fail=True):
+        path = PREFIX + "/make_join/%s/%s" % (context, user_id,)
+
+        response = yield self.client.get_json(
+            destination=destination,
+            path=path,
+            retry_on_dns_fail=retry_on_dns_fail,
+        )
+
+        defer.returnValue(response)
+
+    @defer.inlineCallbacks
+    @log_function
+    def send_join(self, destination, context, event_id, content):
+        path = PREFIX + "/send_join/%s/%s" % (
+            context,
+            event_id,
+        )
+
+        code, content = yield self.client.put_json(
+            destination=destination,
+            path=path,
+            data=content,
+        )
+
+        if not 200 <= code < 300:
+            raise RuntimeError("Got %d from send_join", code)
+
+        defer.returnValue(json.loads(content))
+
+    @defer.inlineCallbacks
+    @log_function
+    def send_invite(self, destination, context, event_id, content):
+        path = PREFIX + "/invite/%s/%s" % (
+            context,
+            event_id,
+        )
+
+        code, content = yield self.client.put_json(
+            destination=destination,
+            path=path,
+            data=content,
+        )
+
+        if not 200 <= code < 300:
+            raise RuntimeError("Got %d from send_invite", code)
+
+        defer.returnValue(json.loads(content))
+
+    @defer.inlineCallbacks
+    @log_function
+    def get_event_auth(self, destination, context, event_id):
+        path = PREFIX + "/event_auth/%s/%s" % (
+            context,
+            event_id,
+        )
+
+        response = yield self.client.get_json(
+            destination=destination,
+            path=path,
+        )
+
+        defer.returnValue(response)
+
     @defer.inlineCallbacks
     def _authenticate_request(self, request):
         json_request = {
@@ -210,7 +284,7 @@ class TransportLayer(object):
         origin = None
 
         if request.method == "PUT":
-            #TODO: Handle other method types? other content types?
+            # TODO: Handle other method types? other content types?
             try:
                 content_bytes = request.content.read()
                 content = json.loads(content_bytes)
@@ -222,11 +296,13 @@ class TransportLayer(object):
             try:
                 params = auth.split(" ")[1].split(",")
                 param_dict = dict(kv.split("=") for kv in params)
+
                 def strip_quotes(value):
                     if value.startswith("\""):
                         return value[1:-1]
                     else:
                         return value
+
                 origin = strip_quotes(param_dict["origin"])
                 key = strip_quotes(param_dict["key"])
                 sig = strip_quotes(param_dict["sig"])
@@ -247,7 +323,7 @@ class TransportLayer(object):
             if auth.startswith("X-Matrix"):
                 (origin, key, sig) = parse_auth_header(auth)
                 json_request["origin"] = origin
-                json_request["signatures"].setdefault(origin,{})[key] = sig
+                json_request["signatures"].setdefault(origin, {})[key] = sig
 
         if not json_request["signatures"]:
             raise SynapseError(
@@ -313,10 +389,10 @@ class TransportLayer(object):
         # data_id pair.
         self.server.register_path(
             "GET",
-            re.compile("^" + PREFIX + "/pdu/([^/]*)/([^/]*)/$"),
+            re.compile("^" + PREFIX + "/event/([^/]*)/$"),
             self._with_authentication(
-                lambda origin, content, query, pdu_origin, pdu_id:
-                handler.on_pdu_request(pdu_origin, pdu_id)
+                lambda origin, content, query, event_id:
+                handler.on_pdu_request(origin, event_id)
             )
         )
 
@@ -326,7 +402,11 @@ class TransportLayer(object):
             re.compile("^" + PREFIX + "/state/([^/]*)/$"),
             self._with_authentication(
                 lambda origin, content, query, context:
-                handler.on_context_state_request(context)
+                handler.on_context_state_request(
+                    origin,
+                    context,
+                    query.get("event_id", [None])[0],
+                )
             )
         )
 
@@ -336,20 +416,11 @@ class TransportLayer(object):
             self._with_authentication(
                 lambda origin, content, query, context:
                 self._on_backfill_request(
-                    context, query["v"], query["limit"]
+                    origin, context, query["v"], query["limit"]
                 )
             )
         )
 
-        self.server.register_path(
-            "GET",
-            re.compile("^" + PREFIX + "/context/([^/]*)/$"),
-            self._with_authentication(
-                lambda origin, content, query, context:
-                handler.on_context_pdus_request(context)
-            )
-        )
-
         # This is when we receive a server-server Query
         self.server.register_path(
             "GET",
@@ -357,7 +428,52 @@ class TransportLayer(object):
             self._with_authentication(
                 lambda origin, content, query, query_type:
                 handler.on_query_request(
-                    query_type, {k: v[0] for k, v in query.items()}
+                    query_type,
+                    {k: v[0].decode("utf-8") for k, v in query.items()}
+                )
+            )
+        )
+
+        self.server.register_path(
+            "GET",
+            re.compile("^" + PREFIX + "/make_join/([^/]*)/([^/]*)$"),
+            self._with_authentication(
+                lambda origin, content, query, context, user_id:
+                self._on_make_join_request(
+                    origin, content, query, context, user_id
+                )
+            )
+        )
+
+        self.server.register_path(
+            "GET",
+            re.compile("^" + PREFIX + "/event_auth/([^/]*)/([^/]*)$"),
+            self._with_authentication(
+                lambda origin, content, query, context, event_id:
+                handler.on_event_auth(
+                    origin, context, event_id,
+                )
+            )
+        )
+
+        self.server.register_path(
+            "PUT",
+            re.compile("^" + PREFIX + "/send_join/([^/]*)/([^/]*)$"),
+            self._with_authentication(
+                lambda origin, content, query, context, event_id:
+                self._on_send_join_request(
+                    origin, content, query,
+                )
+            )
+        )
+
+        self.server.register_path(
+            "PUT",
+            re.compile("^" + PREFIX + "/invite/([^/]*)/([^/]*)$"),
+            self._with_authentication(
+                lambda origin, content, query, context, event_id:
+                self._on_invite_request(
+                    origin, content, query,
                 )
             )
         )
@@ -402,7 +518,8 @@ class TransportLayer(object):
             return
 
         try:
-            code, response = yield self.received_handler.on_incoming_transaction(
+            handler = self.received_handler
+            code, response = yield handler.on_incoming_transaction(
                 transaction_data
             )
         except:
@@ -440,7 +557,7 @@ class TransportLayer(object):
         defer.returnValue(data)
 
     @log_function
-    def _on_backfill_request(self, context, v_list, limits):
+    def _on_backfill_request(self, origin, context, v_list, limits):
         if not limits:
             return defer.succeed(
                 (400, {"error": "Did not include limit param"})
@@ -448,124 +565,34 @@ class TransportLayer(object):
 
         limit = int(limits[-1])
 
-        versions = [v.split(",", 1) for v in v_list]
+        versions = v_list
 
         return self.request_handler.on_backfill_request(
-            context, versions, limit)
+            origin, context, versions, limit
+        )
 
+    @defer.inlineCallbacks
+    @log_function
+    def _on_make_join_request(self, origin, content, query, context, user_id):
+        content = yield self.request_handler.on_make_join_request(
+            context, user_id,
+        )
+        defer.returnValue((200, content))
 
-class TransportReceivedHandler(object):
-    """ Callbacks used when we receive a transaction
-    """
-    def on_incoming_transaction(self, transaction):
-        """ Called on PUT /send/<transaction_id>, or on response to a request
-        that we sent (e.g. a backfill request)
+    @defer.inlineCallbacks
+    @log_function
+    def _on_send_join_request(self, origin, content, query):
+        content = yield self.request_handler.on_send_join_request(
+            origin, content,
+        )
 
-        Args:
-            transaction (synapse.transaction.Transaction): The transaction that
-                was sent to us.
+        defer.returnValue((200, content))
 
-        Returns:
-            twisted.internet.defer.Deferred: A deferred that gets fired when
-            the transaction has finished being processed.
+    @defer.inlineCallbacks
+    @log_function
+    def _on_invite_request(self, origin, content, query):
+        content = yield self.request_handler.on_invite_request(
+            origin, content,
+        )
 
-            The result should be a tuple in the form of
-            `(response_code, respond_body)`, where `response_body` is a python
-            dict that will get serialized to JSON.
-
-            On errors, the dict should have an `error` key with a brief message
-            of what went wrong.
-        """
-        pass
-
-
-class TransportRequestHandler(object):
-    """ Handlers used when someone want's data from us
-    """
-    def on_pull_request(self, versions):
-        """ Called on GET /pull/?v=...
-
-        This is hit when a remote home server wants to get all data
-        after a given transaction. Mainly used when a home server comes back
-        online and wants to get everything it has missed.
-
-        Args:
-            versions (list): A list of transaction_ids that should be used to
-                determine what PDUs the remote side have not yet seen.
-
-        Returns:
-            Deferred: Resultsin a tuple in the form of
-            `(response_code, respond_body)`, where `response_body` is a python
-            dict that will get serialized to JSON.
-
-            On errors, the dict should have an `error` key with a brief message
-            of what went wrong.
-        """
-        pass
-
-    def on_pdu_request(self, pdu_origin, pdu_id):
-        """ Called on GET /pdu/<pdu_origin>/<pdu_id>/
-
-        Someone wants a particular PDU. This PDU may or may not have originated
-        from us.
-
-        Args:
-            pdu_origin (str)
-            pdu_id (str)
-
-        Returns:
-            Deferred: Resultsin a tuple in the form of
-            `(response_code, respond_body)`, where `response_body` is a python
-            dict that will get serialized to JSON.
-
-            On errors, the dict should have an `error` key with a brief message
-            of what went wrong.
-        """
-        pass
-
-    def on_context_state_request(self, context):
-        """ Called on GET /state/<context>/
-
-        Gets hit when someone wants all the *current* state for a given
-        contexts.
-
-        Args:
-            context (str): The name of the context that we're interested in.
-
-        Returns:
-            twisted.internet.defer.Deferred: A deferred that gets fired when
-            the transaction has finished being processed.
-
-            The result should be a tuple in the form of
-            `(response_code, respond_body)`, where `response_body` is a python
-            dict that will get serialized to JSON.
-
-            On errors, the dict should have an `error` key with a brief message
-            of what went wrong.
-        """
-        pass
-
-    def on_backfill_request(self, context, versions, limit):
-        """ Called on GET /backfill/<context>/?v=...&limit=...
-
-        Gets hit when we want to backfill backwards on a given context from
-        the given point.
-
-        Args:
-            context (str): The context to backfill
-            versions (list): A list of 2-tuples representing where to backfill
-                from, in the form `(pdu_id, origin)`
-            limit (int): How many pdus to return.
-
-        Returns:
-            Deferred: Results in a tuple in the form of
-            `(response_code, respond_body)`, where `response_body` is a python
-            dict that will get serialized to JSON.
-
-            On errors, the dict should have an `error` key with a brief message
-            of what went wrong.
-        """
-        pass
-
-    def on_query_request(self):
-        """ Called on a GET /query/<query_type> request. """
+        defer.returnValue((200, content))

synapse/federation/units.py

@@ -20,127 +20,11 @@ server protocol.
 from synapse.util.jsonobject import JsonEncodedObject
 
 import logging
-import json
-import copy
 
 
 logger = logging.getLogger(__name__)
 
 
-class Pdu(JsonEncodedObject):
-    """ A Pdu represents a piece of data sent from a server and is associated
-    with a context.
-
-    A Pdu can be classified as "state". For a given context, we can efficiently
-    retrieve all state pdu's that haven't been clobbered. Clobbering is done
-    via a unique constraint on the tuple (context, pdu_type, state_key). A pdu
-    is a state pdu if `is_state` is True.
-
-    Example pdu::
-
-        {
-            "pdu_id": "78c",
-            "origin_server_ts": 1404835423000,
-            "origin": "bar",
-            "prev_ids": [
-                ["23b", "foo"],
-                ["56a", "bar"],
-            ],
-            "content": { ... },
-        }
-
-    """
-
-    valid_keys = [
-        "pdu_id",
-        "context",
-        "origin",
-        "origin_server_ts",
-        "pdu_type",
-        "destinations",
-        "transaction_id",
-        "prev_pdus",
-        "depth",
-        "content",
-        "outlier",
-        "is_state",  # Below this are keys valid only for State Pdus.
-        "state_key",
-        "power_level",
-        "prev_state_id",
-        "prev_state_origin",
-        "required_power_level",
-        "user_id",
-    ]
-
-    internal_keys = [
-        "destinations",
-        "transaction_id",
-        "outlier",
-    ]
-
-    required_keys = [
-        "pdu_id",
-        "context",
-        "origin",
-        "origin_server_ts",
-        "pdu_type",
-        "content",
-    ]
-
-    # TODO: We need to make this properly load content rather than
-    # just leaving it as a dict. (OR DO WE?!)
-
-    def __init__(self, destinations=[], is_state=False, prev_pdus=[],
-                 outlier=False, **kwargs):
-        if is_state:
-            for required_key in ["state_key"]:
-                if required_key not in kwargs:
-                    raise RuntimeError("Key %s is required" % required_key)
-
-        super(Pdu, self).__init__(
-            destinations=destinations,
-            is_state=is_state,
-            prev_pdus=prev_pdus,
-            outlier=outlier,
-            **kwargs
-        )
-
-    @classmethod
-    def from_pdu_tuple(cls, pdu_tuple):
-        """ Converts a PduTuple to a Pdu
-
-        Args:
-            pdu_tuple (synapse.persistence.transactions.PduTuple): The tuple to
-                convert
-
-        Returns:
-            Pdu
-        """
-        if pdu_tuple:
-            d = copy.copy(pdu_tuple.pdu_entry._asdict())
-            d["origin_server_ts"] = d.pop("ts")
-
-            d["content"] = json.loads(d["content_json"])
-            del d["content_json"]
-
-            args = {f: d[f] for f in cls.valid_keys if f in d}
-            if "unrecognized_keys" in d and d["unrecognized_keys"]:
-                args.update(json.loads(d["unrecognized_keys"]))
-
-            return Pdu(
-                prev_pdus=pdu_tuple.prev_pdu_list,
-                **args
-            )
-        else:
-            return None
-
-    def __str__(self):
-        return "(%s, %s)" % (self.__class__.__name__, repr(self.__dict__))
-
-    def __repr__(self):
-        return "<%s, %s>" % (self.__class__.__name__, repr(self.__dict__))
-
-
 class Edu(JsonEncodedObject):
     """ An Edu represents a piece of data sent from one homeserver to another.
 
@@ -160,11 +44,10 @@ class Edu(JsonEncodedObject):
         "edu_type",
     ]
 
-#    TODO: SYN-103: Remove "origin" and "destination" keys.
-#    internal_keys = [
-#        "origin",
-#        "destination",
-#    ]
+    internal_keys = [
+        "origin",
+        "destination",
+    ]
 
 
 class Transaction(JsonEncodedObject):
@@ -193,6 +76,7 @@ class Transaction(JsonEncodedObject):
         "edus",
         "transaction_id",
         "destination",
+        "pdu_failures",
     ]
 
     internal_keys = [
@@ -229,7 +113,9 @@ class Transaction(JsonEncodedObject):
         transaction_id and origin_server_ts keys.
         """
         if "origin_server_ts" not in kwargs:
-            raise KeyError("Require 'origin_server_ts' to construct a Transaction")
+            raise KeyError(
+                "Require 'origin_server_ts' to construct a Transaction"
+            )
         if "transaction_id" not in kwargs:
             raise KeyError(
                 "Require 'transaction_id' to construct a Transaction"
@@ -238,9 +124,6 @@ class Transaction(JsonEncodedObject):
         for p in pdus:
             p.transaction_id = kwargs["transaction_id"]
 
-        kwargs["pdus"] = [p.get_dict() for p in pdus]
+        kwargs["pdus"] = [p.get_pdu_json() for p in pdus]
 
         return Transaction(**kwargs)
-
-
-

synapse/handlers/_base.py

@@ -14,7 +14,18 @@
 # limitations under the License.
 
 from twisted.internet import defer
+
 from synapse.api.errors import LimitExceededError
+from synapse.util.async import run_on_reactor
+from synapse.crypto.event_signing import add_hashes_and_signatures
+from synapse.api.events.room import RoomMemberEvent
+from synapse.api.constants import Membership
+
+import logging
+
+
+logger = logging.getLogger(__name__)
+
 
 class BaseHandler(object):
 
@@ -30,6 +41,9 @@ class BaseHandler(object):
         self.clock = hs.get_clock()
         self.hs = hs
 
+        self.signing_key = hs.config.signing_key[0]
+        self.server_name = hs.hostname
+
     def ratelimit(self, user_id):
         time_now = self.clock.time()
         allowed, time_allowed = self.ratelimiter.send_message(
@@ -44,19 +58,61 @@ class BaseHandler(object):
 
     @defer.inlineCallbacks
     def _on_new_room_event(self, event, snapshot, extra_destinations=[],
-                           extra_users=[]):
+                           extra_users=[], suppress_auth=False,
+                           do_invite_host=None):
+        yield run_on_reactor()
+
         snapshot.fill_out_prev_events(event)
 
+        yield self.state_handler.annotate_event_with_state(event)
+
+        yield self.auth.add_auth_events(event)
+
+        logger.debug("Signing event...")
+
+        add_hashes_and_signatures(
+            event, self.server_name, self.signing_key
+        )
+
+        logger.debug("Signed event.")
+
+        if not suppress_auth:
+            logger.debug("Authing...")
+            self.auth.check(event, auth_events=event.old_state_events)
+            logger.debug("Authed")
+        else:
+            logger.debug("Suppressed auth.")
+
+        if do_invite_host:
+            federation_handler = self.hs.get_handlers().federation_handler
+            invite_event = yield federation_handler.send_invite(
+                do_invite_host,
+                event
+            )
+
+            # FIXME: We need to check if the remote changed anything else
+            event.signatures = invite_event.signatures
+
         yield self.store.persist_event(event)
 
         destinations = set(extra_destinations)
         # Send a PDU to all hosts who have joined the room.
-        destinations.update((yield self.store.get_joined_hosts_for_room(
-            event.room_id
-        )))
+
+        for k, s in event.state_events.items():
+            try:
+                if k[0] == RoomMemberEvent.TYPE:
+                    if s.content["membership"] == Membership.JOIN:
+                        destinations.add(
+                            self.hs.parse_userid(s.state_key).domain
+                        )
+            except:
+                logger.warn(
+                    "Failed to get destination from event %s", s.event_id
+                )
+
         event.destinations = list(destinations)
 
-        self.notifier.on_new_room_event(event, extra_users=extra_users)
+        yield self.notifier.on_new_room_event(event, extra_users=extra_users)
 
         federation_handler = self.hs.get_handlers().federation_handler
         yield federation_handler.handle_new_event(event, snapshot)

synapse/handlers/directory.py

@@ -17,12 +17,10 @@
 from twisted.internet import defer
 from ._base import BaseHandler
 
-from synapse.api.errors import SynapseError
+from synapse.api.errors import SynapseError, Codes, CodeMessageException
 from synapse.api.events.room import RoomAliasesEvent
 
 import logging
-import sqlite3
-
 
 logger = logging.getLogger(__name__)
 
@@ -56,17 +54,11 @@ class DirectoryHandler(BaseHandler):
         if not servers:
             raise SynapseError(400, "Failed to get server list")
 
-        try:
-            yield self.store.create_room_alias_association(
-                room_alias,
-                room_id,
-                servers
-            )
-        except sqlite3.IntegrityError:
-            defer.returnValue("Already exists")
-
-        # TODO: Send the room event.
-        yield self._update_room_alias_events(user_id, room_id)
+        yield self.store.create_room_alias_association(
+            room_alias,
+            room_id,
+            servers
+        )
 
     @defer.inlineCallbacks
     def delete_association(self, user_id, room_alias):
@@ -92,22 +84,32 @@ class DirectoryHandler(BaseHandler):
                 room_id = result.room_id
                 servers = result.servers
         else:
-            result = yield self.federation.make_query(
-                destination=room_alias.domain,
-                query_type="directory",
-                args={
-                    "room_alias": room_alias.to_string(),
-                },
-                retry_on_dns_fail=False,
-            )
+            try:
+                result = yield self.federation.make_query(
+                    destination=room_alias.domain,
+                    query_type="directory",
+                    args={
+                        "room_alias": room_alias.to_string(),
+                    },
+                    retry_on_dns_fail=False,
+                )
+            except CodeMessageException as e:
+                logging.warn("Error retrieving alias")
+                if e.code == 404:
+                    result = None
+                else:
+                    raise
 
             if result and "room_id" in result and "servers" in result:
                 room_id = result["room_id"]
                 servers = result["servers"]
 
         if not room_id:
-            defer.returnValue({})
-            return
+            raise SynapseError(
+                404,
+                "Room alias %r not found" % (room_alias.to_string(),),
+                Codes.NOT_FOUND
+            )
 
         extra_servers = yield self.store.get_joined_hosts_for_room(room_id)
         servers = list(set(extra_servers) | set(servers))
@@ -130,13 +132,20 @@ class DirectoryHandler(BaseHandler):
             room_alias
         )
 
-        defer.returnValue({
-            "room_id": result.room_id,
-            "servers": result.servers,
-        })
+        if result is not None:
+            defer.returnValue({
+                "room_id": result.room_id,
+                "servers": result.servers,
+            })
+        else:
+            raise SynapseError(
+                404,
+                "Room alias %r not found" % (room_alias.to_string(),),
+                Codes.NOT_FOUND
+            )
 
     @defer.inlineCallbacks
-    def _update_room_alias_events(self, user_id, room_id):
+    def send_room_alias_update_event(self, user_id, room_id):
         aliases = yield self.store.get_aliases_for_room(room_id)
 
         event = self.event_factory.create_event(
@@ -147,10 +156,8 @@ class DirectoryHandler(BaseHandler):
             content={"aliases": aliases},
         )
 
-        snapshot = yield self.store.snapshot_room(
-            room_id=room_id,
-            user_id=user_id,
-        )
+        snapshot = yield self.store.snapshot_room(event)
 
-        yield self.state_handler.handle_new_event(event, snapshot)
-        yield self._on_new_room_event(event, snapshot, extra_users=[user_id])
+        yield self._on_new_room_event(
+            event, snapshot, extra_users=[user_id], suppress_auth=True
+        )

synapse/handlers/events.py

@@ -15,6 +15,7 @@
 
 from twisted.internet import defer
 
+from synapse.util.logcontext import PreserveLoggingContext
 from synapse.util.logutils import log_function
 
 from ._base import BaseHandler
@@ -52,10 +53,14 @@ class EventStreamHandler(BaseHandler):
             if auth_user not in self._streams_per_user:
                 self._streams_per_user[auth_user] = 0
                 if auth_user in self._stop_timer_per_user:
-                    self.clock.cancel_call_later(
-                        self._stop_timer_per_user.pop(auth_user))
+                    try:
+                        self.clock.cancel_call_later(
+                            self._stop_timer_per_user.pop(auth_user)
+                        )
+                    except:
+                        logger.exception("Failed to cancel event timer")
                 else:
-                    self.distributor.fire(
+                    yield self.distributor.fire(
                         "started_user_eventstream", auth_user
                     )
             self._streams_per_user[auth_user] += 1
@@ -64,11 +69,14 @@ class EventStreamHandler(BaseHandler):
                 pagin_config.from_token = None
 
             rm_handler = self.hs.get_handlers().room_member_handler
+            logger.debug("BETA")
             room_ids = yield rm_handler.get_rooms_for_user(auth_user)
 
-            events, tokens = yield self.notifier.get_events_for(
-                auth_user, room_ids, pagin_config, timeout
-            )
+            logger.debug("ALPHA")
+            with PreserveLoggingContext():
+                events, tokens = yield self.notifier.get_events_for(
+                    auth_user, room_ids, pagin_config, timeout
+                )
 
             chunks = [self.hs.serialize_event(e) for e in events]
 
@@ -91,10 +99,12 @@ class EventStreamHandler(BaseHandler):
                     logger.debug(
                         "_later stopped_user_eventstream %s", auth_user
                     )
-                    self.distributor.fire(
+
+                    self._stop_timer_per_user.pop(auth_user, None)
+
+                    yield self.distributor.fire(
                         "stopped_user_eventstream", auth_user
                     )
-                    del self._stop_timer_per_user[auth_user]
 
                 logger.debug("Scheduling _later: for %s", auth_user)
                 self._stop_timer_per_user[auth_user] = (

synapse/handlers/federation.py

@@ -17,13 +17,21 @@
 
 from ._base import BaseHandler
 
-from synapse.api.events.room import InviteJoinEvent, RoomMemberEvent
+from synapse.api.events.utils import prune_event
+from synapse.api.errors import (
+    AuthError, FederationError, SynapseError, StoreError,
+)
+from synapse.api.events.room import RoomMemberEvent, RoomCreateEvent
 from synapse.api.constants import Membership
 from synapse.util.logutils import log_function
-from synapse.federation.pdu_codec import PduCodec
-from synapse.api.errors import SynapseError
+from synapse.util.async import run_on_reactor
+from synapse.crypto.event_signing import (
+    compute_event_signature, check_event_content_hash,
+    add_hashes_and_signatures,
+)
+from syutil.jsonutil import encode_canonical_json
 
-from twisted.internet import defer, reactor
+from twisted.internet import defer
 
 import logging
 
@@ -38,6 +46,8 @@ class FederationHandler(BaseHandler):
         of the home server (including auth and state conflict resoultion)
         b) converting events that were produced by local clients that may need
         to be sent to remote home servers.
+        c) doing the necessary dances to invite remote users and join remote
+        rooms.
     """
 
     def __init__(self, hs):
@@ -55,12 +65,14 @@ class FederationHandler(BaseHandler):
         self.state_handler = hs.get_state_handler()
         # self.auth_handler = gs.get_auth_handler()
         self.server_name = hs.hostname
+        self.keyring = hs.get_keyring()
 
         self.lock_manager = hs.get_room_lock_manager()
 
         self.replication_layer.set_handler(self)
 
-        self.pdu_codec = PduCodec(hs)
+        # When joining a room we need to queue any events for that room up
+        self.room_queues = {}
 
     @log_function
     @defer.inlineCallbacks
@@ -78,7 +90,9 @@ class FederationHandler(BaseHandler):
             processing.
         """
 
-        pdu = self.pdu_codec.pdu_from_event(event)
+        yield run_on_reactor()
+
+        pdu = event
 
         if not hasattr(pdu, "destinations") or not pdu.destinations:
             pdu.destinations = []
@@ -87,181 +101,617 @@ class FederationHandler(BaseHandler):
 
     @log_function
     @defer.inlineCallbacks
-    def on_receive_pdu(self, pdu, backfilled):
+    def on_receive_pdu(self, origin, pdu, backfilled, state=None):
         """ Called by the ReplicationLayer when we have a new pdu. We need to
-        do auth checks and put it throught the StateHandler.
+        do auth checks and put it through the StateHandler.
         """
-        event = self.pdu_codec.event_from_pdu(pdu)
+        event = pdu
 
         logger.debug("Got event: %s", event.event_id)
 
-        with (yield self.lock_manager.lock(pdu.context)):
-            if event.is_state and not backfilled:
-                is_new_state = yield self.state_handler.handle_new_state(
-                    pdu
-                )
-            else:
-                is_new_state = False
-        # TODO: Implement something in federation that allows us to
-        # respond to PDU.
+        # If we are currently in the process of joining this room, then we
+        # queue up events for later processing.
+        if event.room_id in self.room_queues:
+            self.room_queues[event.room_id].append((pdu, origin))
+            return
 
-        target_is_mine = False
-        if hasattr(event, "target_host"):
-            target_is_mine = event.target_host == self.hs.hostname
+        logger.debug("Processing event: %s", event.event_id)
 
-        if event.type == InviteJoinEvent.TYPE:
-            if not target_is_mine:
-                logger.debug("Ignoring invite/join event %s", event)
-                return
+        redacted_event = prune_event(event)
 
-            # If we receive an invite/join event then we need to join the
-            # sender to the given room.
-            # TODO: We should probably auth this or some such
-            content = event.content
-            content.update({"membership": Membership.JOIN})
-            new_event = self.event_factory.create_event(
-                etype=RoomMemberEvent.TYPE,
-                state_key=event.user_id,
-                room_id=event.room_id,
-                user_id=event.user_id,
-                membership=Membership.JOIN,
-                content=content
+        redacted_pdu_json = redacted_event.get_pdu_json()
+        try:
+            yield self.keyring.verify_json_for_server(
+                event.origin, redacted_pdu_json
+            )
+        except SynapseError as e:
+            logger.warn(
+                "Signature check failed for %s redacted to %s",
+                encode_canonical_json(pdu.get_pdu_json()),
+                encode_canonical_json(redacted_pdu_json),
+            )
+            raise FederationError(
+                "ERROR",
+                e.code,
+                e.msg,
+                affected=event.event_id,
             )
 
-            yield self.hs.get_handlers().room_member_handler.change_membership(
-                new_event,
-                do_auth=False,
+        if not check_event_content_hash(event):
+            logger.warn(
+                "Event content has been tampered, redacting %s, %s",
+                event.event_id, encode_canonical_json(event.get_full_dict())
+            )
+            event = redacted_event
+
+        logger.debug("Event: %s", event)
+
+        # FIXME (erikj): Awful hack to make the case where we are not currently
+        # in the room work
+        current_state = None
+        is_in_room = yield self.auth.check_host_in_room(
+            event.room_id,
+            self.server_name
+        )
+        if not is_in_room and not event.outlier:
+            logger.debug("Got event for room we're not in.")
+
+            replication_layer = self.replication_layer
+            auth_chain = yield replication_layer.get_event_auth(
+                origin,
+                context=event.room_id,
+                event_id=event.event_id,
             )
 
-        else:
-            with (yield self.room_lock.lock(event.room_id)):
-                yield self.store.persist_event(
-                    event,
-                    backfilled,
-                    is_new_state=is_new_state
-                )
-
-            room = yield self.store.get_room(event.room_id)
-
-            if not room:
-                # Huh, let's try and get the current state
+            for e in auth_chain:
+                e.outlier = True
                 try:
-                    yield self.replication_layer.get_state_for_context(
-                        event.origin, event.room_id
-                    )
-
-                    hosts = yield self.store.get_joined_hosts_for_room(
-                        event.room_id
-                    )
-                    if self.hs.hostname in hosts:
-                        try:
-                            yield self.store.store_room(
-                                room_id=event.room_id,
-                                room_creator_user_id="",
-                                is_public=False,
-                            )
-                        except:
-                            pass
+                    yield self._handle_new_event(e, fetch_missing=False)
                 except:
                     logger.exception(
-                        "Failed to get current state for room %s",
-                        event.room_id
+                        "Failed to parse auth event %s",
+                        e.event_id,
                     )
 
-            if not backfilled:
-                extra_users = []
-                if event.type == RoomMemberEvent.TYPE:
-                    target_user_id = event.state_key
-                    target_user = self.hs.parse_userid(target_user_id)
-                    extra_users.append(target_user)
-
-                yield self.notifier.on_new_room_event(
-                    event, extra_users=extra_users
+            if not state:
+                state = yield replication_layer.get_state_for_context(
+                    origin,
+                    context=event.room_id,
+                    event_id=event.event_id,
                 )
 
+            current_state = state
+
+        if state:
+            for e in state:
+                e.outlier = True
+                try:
+                    yield self._handle_new_event(e)
+                except:
+                    logger.exception(
+                        "Failed to parse state event %s",
+                        e.event_id,
+                    )
+
+        try:
+            yield self._handle_new_event(
+                event,
+                state=state,
+                backfilled=backfilled,
+                current_state=current_state,
+            )
+        except AuthError as e:
+            raise FederationError(
+                "ERROR",
+                e.code,
+                e.msg,
+                affected=event.event_id,
+            )
+
+        room = yield self.store.get_room(event.room_id)
+
+        if not room:
+            try:
+                yield self.store.store_room(
+                    room_id=event.room_id,
+                    room_creator_user_id="",
+                    is_public=False,
+                )
+            except StoreError:
+                logger.exception("Failed to store room.")
+
+        if not backfilled:
+            extra_users = []
+            if event.type == RoomMemberEvent.TYPE:
+                target_user_id = event.state_key
+                target_user = self.hs.parse_userid(target_user_id)
+                extra_users.append(target_user)
+
+            yield self.notifier.on_new_room_event(
+                event, extra_users=extra_users
+            )
+
         if event.type == RoomMemberEvent.TYPE:
             if event.membership == Membership.JOIN:
                 user = self.hs.parse_userid(event.state_key)
-                self.distributor.fire(
+                yield self.distributor.fire(
                     "user_joined_room", user=user, room_id=event.room_id
                 )
 
     @log_function
     @defer.inlineCallbacks
     def backfill(self, dest, room_id, limit):
-        pdus = yield self.replication_layer.backfill(dest, room_id, limit)
+        """ Trigger a backfill request to `dest` for the given `room_id`
+        """
+        extremities = yield self.store.get_oldest_events_in_room(room_id)
+
+        pdus = yield self.replication_layer.backfill(
+            dest,
+            room_id,
+            limit,
+            extremities=extremities,
+        )
 
         events = []
 
         for pdu in pdus:
-            event = self.pdu_codec.event_from_pdu(pdu)
+            event = pdu
+
+            # FIXME (erikj): Not sure this actually works :/
+            yield self.state_handler.annotate_event_with_state(event)
+
             events.append(event)
+
             yield self.store.persist_event(event, backfilled=True)
 
         defer.returnValue(events)
 
+    @defer.inlineCallbacks
+    def send_invite(self, target_host, event):
+        """ Sends the invite to the remote server for signing.
+
+        Invites must be signed by the invitee's server before distribution.
+        """
+        pdu = yield self.replication_layer.send_invite(
+            destination=target_host,
+            context=event.room_id,
+            event_id=event.event_id,
+            pdu=event
+        )
+
+
+
+        defer.returnValue(pdu)
+
+    @defer.inlineCallbacks
+    def on_event_auth(self, event_id):
+        auth = yield self.store.get_auth_chain(event_id)
+
+        for event in auth:
+            event.signatures.update(
+                compute_event_signature(
+                    event,
+                    self.hs.hostname,
+                    self.hs.config.signing_key[0]
+                )
+            )
+
+        defer.returnValue([e for e in auth])
+
     @log_function
     @defer.inlineCallbacks
     def do_invite_join(self, target_host, room_id, joinee, content, snapshot):
+        """ Attempts to join the `joinee` to the room `room_id` via the
+        server `target_host`.
 
-        hosts = yield self.store.get_joined_hosts_for_room(room_id)
-        if self.hs.hostname in hosts:
-            # We are already in the room.
-            logger.debug("We're already in the room apparently")
-            defer.returnValue(False)
+        This first triggers a /make_join/ request that returns a partial
+        event that we can fill out and sign. This is then sent to the
+        remote server via /send_join/ which responds with the state at that
+        event and the auth_chains.
 
-        # First get current state to see if we are already joined.
-        try:
-            yield self.replication_layer.get_state_for_context(
-                target_host, room_id
-            )
+        We suspend processing of any received events from this room until we
+        have finished processing the join.
+        """
+        logger.debug("Joining %s to %s", joinee, room_id)
 
-            hosts = yield self.store.get_joined_hosts_for_room(room_id)
-            if self.hs.hostname in hosts:
-                # Oh, we were actually in the room already.
-                logger.debug("We're already in the room apparently")
-                defer.returnValue(False)
-        except Exception:
-            logger.exception("Failed to get current state")
-
-        new_event = self.event_factory.create_event(
-            etype=InviteJoinEvent.TYPE,
-            target_host=target_host,
-            room_id=room_id,
-            user_id=joinee,
-            content=content
+        pdu = yield self.replication_layer.make_join(
+            target_host,
+            room_id,
+            joinee
         )
 
-        new_event.destinations = [target_host]
+        logger.debug("Got response to make_join: %s", pdu)
 
-        snapshot.fill_out_prev_events(new_event)
-        yield self.handle_new_event(new_event, snapshot)
+        event = pdu
 
-        # TODO (erikj): Time out here.
-        d = defer.Deferred()
-        self.waiting_for_join_list.setdefault((joinee, room_id), []).append(d)
-        reactor.callLater(10, d.cancel)
+        # We should assert some things.
+        assert(event.type == RoomMemberEvent.TYPE)
+        assert(event.user_id == joinee)
+        assert(event.state_key == joinee)
+        assert(event.room_id == room_id)
+
+        event.outlier = False
+
+        self.room_queues[room_id] = []
 
         try:
-            yield d
-        except defer.CancelledError:
-            raise SynapseError(500, "Unable to join remote room")
+            event.event_id = self.event_factory.create_event_id()
+            event.origin = self.hs.hostname
+            event.content = content
 
-        try:
-            yield self.store.store_room(
-                room_id=room_id,
-                room_creator_user_id="",
-                is_public=False
+            if not hasattr(event, "signatures"):
+                event.signatures = {}
+
+            add_hashes_and_signatures(
+                event,
+                self.hs.hostname,
+                self.hs.config.signing_key[0],
             )
-        except:
-            pass
 
+            ret = yield self.replication_layer.send_join(
+                target_host,
+                event
+            )
+
+            state = ret["state"]
+            auth_chain = ret["auth_chain"]
+            auth_chain.sort(key=lambda e: e.depth)
+
+            logger.debug("do_invite_join auth_chain: %s", auth_chain)
+            logger.debug("do_invite_join state: %s", state)
+
+            logger.debug("do_invite_join event: %s", event)
+
+            try:
+                yield self.store.store_room(
+                    room_id=room_id,
+                    room_creator_user_id="",
+                    is_public=False
+                )
+            except:
+                # FIXME
+                pass
+
+            for e in auth_chain:
+                e.outlier = True
+                try:
+                    yield self._handle_new_event(e, fetch_missing=False)
+                except:
+                    logger.exception(
+                        "Failed to parse auth event %s",
+                        e.event_id,
+                    )
+
+            for e in state:
+                # FIXME: Auth these.
+                e.outlier = True
+                try:
+                    yield self._handle_new_event(
+                        e,
+                        fetch_missing=True
+                    )
+                except:
+                    logger.exception(
+                        "Failed to parse state event %s",
+                        e.event_id,
+                    )
+
+            yield self._handle_new_event(
+                event,
+                state=state,
+                current_state=state,
+            )
+
+            yield self.notifier.on_new_room_event(
+                event, extra_users=[joinee]
+            )
+
+            logger.debug("Finished joining %s to %s", joinee, room_id)
+        finally:
+            room_queue = self.room_queues[room_id]
+            del self.room_queues[room_id]
+
+            for p, origin in room_queue:
+                try:
+                    self.on_receive_pdu(origin, p, backfilled=False)
+                except:
+                    logger.exception("Couldn't handle pdu")
 
         defer.returnValue(True)
 
+    @defer.inlineCallbacks
+    @log_function
+    def on_make_join_request(self, context, user_id):
+        """ We've received a /make_join/ request, so we create a partial
+        join event for the room and return that. We don *not* persist or
+        process it until the other server has signed it and sent it back.
+        """
+        event = self.event_factory.create_event(
+            etype=RoomMemberEvent.TYPE,
+            content={"membership": Membership.JOIN},
+            room_id=context,
+            user_id=user_id,
+            state_key=user_id,
+        )
+
+        snapshot = yield self.store.snapshot_room(event)
+        snapshot.fill_out_prev_events(event)
+
+        yield self.state_handler.annotate_event_with_state(event)
+        yield self.auth.add_auth_events(event)
+        self.auth.check(event, auth_events=event.old_state_events)
+
+        pdu = event
+
+        defer.returnValue(pdu)
+
+    @defer.inlineCallbacks
+    @log_function
+    def on_send_join_request(self, origin, pdu):
+        """ We have received a join event for a room. Fully process it and
+        respond with the current state and auth chains.
+        """
+        event = pdu
+
+        event.outlier = False
+
+        yield self._handle_new_event(event)
+
+        extra_users = []
+        if event.type == RoomMemberEvent.TYPE:
+            target_user_id = event.state_key
+            target_user = self.hs.parse_userid(target_user_id)
+            extra_users.append(target_user)
+
+        yield self.notifier.on_new_room_event(
+            event, extra_users=extra_users
+        )
+
+        if event.type == RoomMemberEvent.TYPE:
+            if event.content["membership"] == Membership.JOIN:
+                user = self.hs.parse_userid(event.state_key)
+                yield self.distributor.fire(
+                    "user_joined_room", user=user, room_id=event.room_id
+                )
+
+        new_pdu = event
+
+        destinations = set()
+
+        for k, s in event.state_events.items():
+            try:
+                if k[0] == RoomMemberEvent.TYPE:
+                    if s.content["membership"] == Membership.JOIN:
+                        destinations.add(
+                            self.hs.parse_userid(s.state_key).domain
+                        )
+            except:
+                logger.warn(
+                    "Failed to get destination from event %s", s.event_id
+                )
+
+        new_pdu.destinations = list(destinations)
+
+        yield self.replication_layer.send_pdu(new_pdu)
+
+        auth_chain = yield self.store.get_auth_chain(event.event_id)
+
+        defer.returnValue({
+            "state": event.state_events.values(),
+            "auth_chain": auth_chain,
+        })
+
+    @defer.inlineCallbacks
+    def on_invite_request(self, origin, pdu):
+        """ We've got an invite event. Process and persist it. Sign it.
+
+        Respond with the now signed event.
+        """
+        event = pdu
+
+        event.outlier = True
+
+        event.signatures.update(
+            compute_event_signature(
+                event,
+                self.hs.hostname,
+                self.hs.config.signing_key[0]
+            )
+        )
+
+        yield self.state_handler.annotate_event_with_state(event)
+
+        yield self.store.persist_event(
+            event,
+            backfilled=False,
+        )
+
+        target_user = self.hs.parse_userid(event.state_key)
+        yield self.notifier.on_new_room_event(
+            event, extra_users=[target_user],
+        )
+
+        defer.returnValue(event)
+
+    @defer.inlineCallbacks
+    def get_state_for_pdu(self, origin, room_id, event_id):
+        yield run_on_reactor()
+
+        in_room = yield self.auth.check_host_in_room(room_id, origin)
+        if not in_room:
+            raise AuthError(403, "Host not in room.")
+
+        state_groups = yield self.store.get_state_groups(
+            [event_id]
+        )
+
+        if state_groups:
+            _, state = state_groups.items().pop()
+            results = {
+                (e.type, e.state_key): e for e in state
+            }
+
+            event = yield self.store.get_event(event_id)
+            if hasattr(event, "state_key"):
+                # Get previous state
+                if hasattr(event, "replaces_state") and event.replaces_state:
+                    prev_event = yield self.store.get_event(
+                        event.replaces_state
+                    )
+                    results[(event.type, event.state_key)] = prev_event
+                else:
+                    del results[(event.type, event.state_key)]
+
+            res = results.values()
+            for event in res:
+                event.signatures.update(
+                    compute_event_signature(
+                        event,
+                        self.hs.hostname,
+                        self.hs.config.signing_key[0]
+                    )
+                )
+
+            defer.returnValue(res)
+        else:
+            defer.returnValue([])
+
+    @defer.inlineCallbacks
+    @log_function
+    def on_backfill_request(self, origin, context, pdu_list, limit):
+        in_room = yield self.auth.check_host_in_room(context, origin)
+        if not in_room:
+            raise AuthError(403, "Host not in room.")
+
+        events = yield self.store.get_backfill_events(
+            context,
+            pdu_list,
+            limit
+        )
+
+        defer.returnValue(events)
+
+    @defer.inlineCallbacks
+    @log_function
+    def get_persisted_pdu(self, origin, event_id, do_auth=True):
+        """ Get a PDU from the database with given origin and id.
+
+        Returns:
+            Deferred: Results in a `Pdu`.
+        """
+        event = yield self.store.get_event(
+            event_id,
+            allow_none=True,
+        )
+
+        if event:
+            # FIXME: This is a temporary work around where we occasionally
+            # return events slightly differently than when they were
+            # originally signed
+            event.signatures.update(
+                compute_event_signature(
+                    event,
+                    self.hs.hostname,
+                    self.hs.config.signing_key[0]
+                )
+            )
+
+            if do_auth:
+                in_room = yield self.auth.check_host_in_room(
+                    event.room_id,
+                    origin
+                )
+                if not in_room:
+                    raise AuthError(403, "Host not in room.")
+
+            defer.returnValue(event)
+        else:
+            defer.returnValue(None)
+
+    @log_function
+    def get_min_depth_for_context(self, context):
+        return self.store.get_min_depth(context)
 
     @log_function
     def _on_user_joined(self, user, room_id):
-        waiters = self.waiting_for_join_list.get((user.to_string(), room_id), [])
+        waiters = self.waiting_for_join_list.get(
+            (user.to_string(), room_id),
+            []
+        )
         while waiters:
             waiters.pop().callback(None)
+
+    @defer.inlineCallbacks
+    def _handle_new_event(self, event, state=None, backfilled=False,
+                          current_state=None, fetch_missing=True):
+        is_new_state = yield self.state_handler.annotate_event_with_state(
+            event,
+            old_state=state
+        )
+
+        if event.old_state_events:
+            known_ids = set(
+                [s.event_id for s in event.old_state_events.values()]
+            )
+            for e_id, _ in event.auth_events:
+                if e_id not in known_ids:
+                    e = yield self.store.get_event(
+                        e_id,
+                        allow_none=True,
+                    )
+
+                    if not e:
+                        # TODO: Do some conflict res to make sure that we're
+                        # not the ones who are wrong.
+                        logger.info(
+                            "Rejecting %s as %s not in %s",
+                            event.event_id, e_id, known_ids,
+                        )
+                        raise AuthError(403, "Auth events are stale")
+
+            auth_events = event.old_state_events
+        else:
+            # We need to get the auth events from somewhere.
+
+            # TODO: Don't just hit the DBs?
+
+            auth_events = {}
+            for e_id, _ in event.auth_events:
+                e = yield self.store.get_event(
+                    e_id,
+                    allow_none=True,
+                )
+
+                if not e:
+                    e = yield self.replication_layer.get_pdu(
+                        event.origin, e_id, outlier=True
+                    )
+
+                    if e and fetch_missing:
+                        try:
+                            yield self.on_receive_pdu(event.origin, e, False)
+                        except:
+                            logger.exception(
+                                "Failed to parse auth event %s",
+                                e_id,
+                            )
+
+                if not e:
+                    logger.warn("Can't find auth event %s.", e_id)
+
+                auth_events[(e.type, e.state_key)] = e
+
+            if event.type == RoomMemberEvent.TYPE and not event.auth_events:
+                if len(event.prev_events) == 1:
+                    c = yield self.store.get_event(event.prev_events[0][0])
+                    if c.type == RoomCreateEvent.TYPE:
+                        auth_events[(c.type, c.state_key)] = c
+
+        self.auth.check(event, auth_events=auth_events)
+
+        yield self.store.persist_event(
+            event,
+            backfilled=backfilled,
+            is_new_state=(is_new_state and not backfilled),
+            current_state=current_state,
+        )

synapse/handlers/login.py

@@ -17,13 +17,12 @@ from twisted.internet import defer
 
 from ._base import BaseHandler
 from synapse.api.errors import LoginError, Codes
-from synapse.http.client import IdentityServerHttpClient
+from synapse.http.client import SimpleHttpClient
 from synapse.util.emailutils import EmailException
 import synapse.util.emailutils as emailutils
 
 import bcrypt
 import logging
-import urllib
 
 logger = logging.getLogger(__name__)
 
@@ -54,7 +53,7 @@ class LoginHandler(BaseHandler):
         # pull out the hash for this user if they exist
         user_info = yield self.store.get_user_by_id(user_id=user)
         if not user_info:
-            logger.warn("Attempted to login as %s but they do not exist.", user)
+            logger.warn("Attempted to login as %s but they do not exist", user)
             raise LoginError(403, "", errcode=Codes.FORBIDDEN)
 
         stored_hash = user_info[0]["password_hash"]
@@ -97,10 +96,16 @@ class LoginHandler(BaseHandler):
 
     @defer.inlineCallbacks
     def _query_email(self, email):
-        httpCli = IdentityServerHttpClient(self.hs)
+        httpCli = SimpleHttpClient(self.hs)
         data = yield httpCli.get_json(
-            'matrix.org:8090',  # TODO FIXME This should be configurable.
-            "/_matrix/identity/api/v1/lookup?medium=email&address=" +
-            "%s" % urllib.quote(email)
+            # TODO FIXME This should be configurable.
+            # XXX: ID servers need to use HTTPS
+            "http://%s%s" % (
+                "matrix.org:8090", "/_matrix/identity/api/v1/lookup"
+            ),
+            {
+                'medium': 'email',
+                'address': email
+            }
         )
         defer.returnValue(data)

synapse/handlers/message.py

@@ -16,9 +16,9 @@
 from twisted.internet import defer
 
 from synapse.api.constants import Membership
-from synapse.api.events.room import RoomTopicEvent
 from synapse.api.errors import RoomError
 from synapse.streams.config import PaginationConfig
+from synapse.util.logcontext import PreserveLoggingContext
 from ._base import BaseHandler
 
 import logging
@@ -26,7 +26,6 @@ import logging
 logger = logging.getLogger(__name__)
 
 
-
 class MessageHandler(BaseHandler):
 
     def __init__(self, hs):
@@ -59,7 +58,8 @@ class MessageHandler(BaseHandler):
 #            user_id=sender_id
 #        )
 
-        # TODO (erikj): Once we work out the correct c-s api we need to think on how to do this.
+        # TODO (erikj): Once we work out the correct c-s api we need to think
+        # on how to do this.
 
         defer.returnValue(None)
 
@@ -81,17 +81,17 @@ class MessageHandler(BaseHandler):
         user = self.hs.parse_userid(event.user_id)
         assert user.is_mine, "User must be our own: %s" % (user,)
 
-        snapshot = yield self.store.snapshot_room(event.room_id, event.user_id)
+        snapshot = yield self.store.snapshot_room(event)
 
-        if not suppress_auth:
-            yield self.auth.check(event, snapshot, raises=True)
-
-        yield self._on_new_room_event(event, snapshot)
-
-        self.hs.get_handlers().presence_handler.bump_presence_active_time(
-            user
+        yield self._on_new_room_event(
+            event, snapshot, suppress_auth=suppress_auth
         )
 
+        with PreserveLoggingContext():
+            self.hs.get_handlers().presence_handler.bump_presence_active_time(
+                user
+            )
+
     @defer.inlineCallbacks
     def get_messages(self, user_id=None, room_id=None, pagin_config=None,
                      feedback=False):
@@ -111,12 +111,18 @@ class MessageHandler(BaseHandler):
         data_source = self.hs.get_event_sources().sources["room"]
 
         if not pagin_config.from_token:
-            pagin_config.from_token = yield self.hs.get_event_sources().get_current_token()
+            pagin_config.from_token = (
+                yield self.hs.get_event_sources().get_current_token()
+            )
 
         user = self.hs.parse_userid(user_id)
 
-        events, next_token = yield data_source.get_pagination_rows(
-            user, pagin_config, room_id
+        events, next_key = yield data_source.get_pagination_rows(
+            user, pagin_config.get_source_config("room"), room_id
+        )
+
+        next_token = pagin_config.from_token.copy_and_replace(
+            "room_key", next_key
         )
 
         chunk = {
@@ -138,66 +144,27 @@ class MessageHandler(BaseHandler):
             SynapseError if something went wrong.
         """
 
-        snapshot = yield self.store.snapshot_room(
-            event.room_id,
-            event.user_id,
-            state_type=event.type,
-            state_key=event.state_key,
-        )
-
-        yield self.auth.check(event, snapshot, raises=True)
-
-        yield self.state_handler.handle_new_event(event, snapshot)
+        snapshot = yield self.store.snapshot_room(event)
 
         yield self._on_new_room_event(event, snapshot)
 
     @defer.inlineCallbacks
     def get_room_data(self, user_id=None, room_id=None,
-                      event_type=None, state_key="",
-                      public_room_rules=[],
-                      private_room_rules=["join"]):
+                      event_type=None, state_key=""):
         """ Get data from a room.
 
         Args:
             event : The room path event
-            public_room_rules : A list of membership states the user can be in,
-            in order to read this data IN A PUBLIC ROOM. An empty list means
-            'any state'.
-            private_room_rules : A list of membership states the user can be
-            in, in order to read this data IN A PRIVATE ROOM. An empty list
-            means 'any state'.
         Returns:
             The path data content.
         Raises:
             SynapseError if something went wrong.
         """
-        if event_type == RoomTopicEvent.TYPE:
-            # anyone invited/joined can read the topic
-            private_room_rules = ["invite", "join"]
+        have_joined = yield self.auth.check_joined_room(room_id, user_id)
+        if not have_joined:
+            raise RoomError(403, "User not in room.")
 
-        # does this room exist
-        room = yield self.store.get_room(room_id)
-        if not room:
-            raise RoomError(403, "Room does not exist.")
-
-        # does this user exist in this room
-        member = yield self.store.get_room_member(
-            room_id=room_id,
-            user_id="" if not user_id else user_id)
-
-        member_state = member.membership if member else None
-
-        if room.is_public and public_room_rules:
-            # make sure the user meets public room rules
-            if member_state not in public_room_rules:
-                raise RoomError(403, "Member does not meet public room rules.")
-        elif not room.is_public and private_room_rules:
-            # make sure the user meets private room rules
-            if member_state not in private_room_rules:
-                raise RoomError(
-                    403, "Member does not meet private room rules.")
-
-        data = yield self.store.get_current_state(
+        data = yield self.state_handler.get_current_state(
             room_id, event_type, state_key
         )
         defer.returnValue(data)
@@ -215,9 +182,7 @@ class MessageHandler(BaseHandler):
 
     @defer.inlineCallbacks
     def send_feedback(self, event):
-        snapshot = yield self.store.snapshot_room(event.room_id, event.user_id)
-
-        yield self.auth.check(event, snapshot, raises=True)
+        snapshot = yield self.store.snapshot_room(event)
 
         # store message in db
         yield self._on_new_room_event(event, snapshot)
@@ -235,7 +200,7 @@ class MessageHandler(BaseHandler):
         yield self.auth.check_joined_room(room_id, user_id)
 
         # TODO: This is duplicating logic from snapshot_all_rooms
-        current_state = yield self.store.get_current_state(room_id)
+        current_state = yield self.state_handler.get_current_state(room_id)
         defer.returnValue([self.hs.serialize_event(c) for c in current_state])
 
     @defer.inlineCallbacks
@@ -271,22 +236,24 @@ class MessageHandler(BaseHandler):
         presence_stream = self.hs.get_event_sources().sources["presence"]
         pagination_config = PaginationConfig(from_token=now_token)
         presence, _ = yield presence_stream.get_pagination_rows(
-            user, pagination_config, None
+            user, pagination_config.get_source_config("presence"), None
         )
 
         public_rooms = yield self.store.get_rooms(is_public=True)
         public_room_ids = [r["room_id"] for r in public_rooms]
 
         limit = pagin_config.limit
-        if not limit:
+        if limit is None:
             limit = 10
 
         for event in room_list:
             d = {
                 "room_id": event.room_id,
                 "membership": event.membership,
-                "visibility": ("public" if event.room_id in
-                              public_room_ids else "private"),
+                "visibility": (
+                    "public" if event.room_id in public_room_ids
+                    else "private"
+                ),
             }
 
             if event.membership == Membership.INVITE:
@@ -312,10 +279,12 @@ class MessageHandler(BaseHandler):
                     "end": end_token.to_string(),
                 }
 
-                current_state = yield self.store.get_current_state(
+                current_state = yield self.state_handler.get_current_state(
                     event.room_id
                 )
-                d["state"] = [self.hs.serialize_event(c) for c in current_state]
+                d["state"] = [
+                    self.hs.serialize_event(c) for c in current_state
+                ]
             except:
                 logger.exception("Failed to get snapshot")
 
@@ -327,4 +296,64 @@ class MessageHandler(BaseHandler):
 
         defer.returnValue(ret)
 
+    @defer.inlineCallbacks
+    def room_initial_sync(self, user_id, room_id, pagin_config=None,
+                          feedback=False):
+        yield self.auth.check_joined_room(room_id, user_id)
 
+        # TODO(paul): I wish I was called with user objects not user_id
+        #   strings...
+        auth_user = self.hs.parse_userid(user_id)
+
+        # TODO: These concurrently
+        state_tuples = yield self.state_handler.get_current_state(room_id)
+        state = [self.hs.serialize_event(x) for x in state_tuples]
+
+        member_event = (yield self.store.get_room_member(
+            user_id=user_id,
+            room_id=room_id
+        ))
+
+        now_token = yield self.hs.get_event_sources().get_current_token()
+
+        limit = pagin_config.limit if pagin_config else None
+        if limit is None:
+            limit = 10
+
+        messages, token = yield self.store.get_recent_events_for_room(
+            room_id,
+            limit=limit,
+            end_token=now_token.room_key,
+        )
+
+        start_token = now_token.copy_and_replace("room_key", token[0])
+        end_token = now_token.copy_and_replace("room_key", token[1])
+
+        room_members = yield self.store.get_room_members(room_id)
+
+        presence_handler = self.hs.get_handlers().presence_handler
+        presence = []
+        for m in room_members:
+            try:
+                member_presence = yield presence_handler.get_state(
+                    target_user=self.hs.parse_userid(m.user_id),
+                    auth_user=auth_user,
+                    as_event=True,
+                )
+                presence.append(member_presence)
+            except Exception:
+                logger.exception(
+                    "Failed to get member presence of %r", m.user_id
+                )
+
+        defer.returnValue({
+            "membership": member_event.membership,
+            "room_id": room_id,
+            "messages": {
+                "chunk": [self.hs.serialize_event(m) for m in messages],
+                "start": start_token.to_string(),
+                "end": end_token.to_string(),
+            },
+            "state": state,
+            "presence": presence
+        })

synapse/handlers/presence.py

@@ -19,6 +19,7 @@ from synapse.api.errors import SynapseError, AuthError
 from synapse.api.constants import PresenceState
 
 from synapse.util.logutils import log_function
+from synapse.util.logcontext import PreserveLoggingContext
 
 from ._base import BaseHandler
 
@@ -76,9 +77,7 @@ class PresenceHandler(BaseHandler):
             "stopped_user_eventstream", self.stopped_user_eventstream
         )
 
-        distributor.observe("user_joined_room",
-            self.user_joined_room
-        )
+        distributor.observe("user_joined_room", self.user_joined_room)
 
         distributor.declare("collect_presencelike_data")
 
@@ -141,12 +140,10 @@ class PresenceHandler(BaseHandler):
         if user in self._user_cachemap:
             return self._user_cachemap[user]
         else:
-            statuscache = UserPresenceCache()
-            statuscache.update({"presence": PresenceState.OFFLINE}, user)
-            return statuscache
+            return UserPresenceCache()
 
     def registered_user(self, user):
-        self.store.create_presence(user.localpart)
+        return self.store.create_presence(user.localpart)
 
     @defer.inlineCallbacks
     def is_presence_visible(self, observer_user, observed_user):
@@ -156,22 +153,21 @@ class PresenceHandler(BaseHandler):
             defer.returnValue(True)
 
         if (yield self.store.user_rooms_intersect(
-            [u.to_string() for u in observer_user, observed_user]
-        )):
+                [u.to_string() for u in observer_user, observed_user])):
             defer.returnValue(True)
 
         if (yield self.store.is_presence_visible(
-            observed_localpart=observed_user.localpart,
-            observer_userid=observer_user.to_string(),
-        )):
+                observed_localpart=observed_user.localpart,
+                observer_userid=observer_user.to_string())):
             defer.returnValue(True)
 
         defer.returnValue(False)
 
     @defer.inlineCallbacks
-    def get_state(self, target_user, auth_user):
+    def get_state(self, target_user, auth_user, as_event=False):
         if target_user.is_mine:
-            visible = yield self.is_presence_visible(observer_user=auth_user,
+            visible = yield self.is_presence_visible(
+                observer_user=auth_user,
                 observed_user=target_user
             )
 
@@ -183,9 +179,9 @@ class PresenceHandler(BaseHandler):
             state["presence"] = state.pop("state")
 
             if target_user in self._user_cachemap:
-                state["last_active"] = (
-                    self._user_cachemap[target_user].get_state()["last_active"]
-                )
+                cached_state = self._user_cachemap[target_user].get_state()
+                if "last_active" in cached_state:
+                    state["last_active"] = cached_state["last_active"]
         else:
             # TODO(paul): Have remote server send us permissions set
             state = self._get_or_offline_usercache(target_user).get_state()
@@ -194,7 +190,20 @@ class PresenceHandler(BaseHandler):
             state["last_active_ago"] = int(
                 self.clock.time_msec() - state.pop("last_active")
             )
-        defer.returnValue(state)
+
+        if as_event:
+            content = state
+
+            content["user_id"] = target_user.to_string()
+
+            if "last_active" in content:
+                content["last_active_ago"] = int(
+                    self._clock.time_msec() - content.pop("last_active")
+                )
+
+            defer.returnValue({"type": "m.presence", "content": content})
+        else:
+            defer.returnValue(state)
 
     @defer.inlineCallbacks
     @log_function
@@ -219,9 +228,9 @@ class PresenceHandler(BaseHandler):
                 )
 
         if state["presence"] not in self.STATE_LEVELS:
-            raise SynapseError(400, "'%s' is not a valid presence state" %
-                state["presence"]
-            )
+            raise SynapseError(400, "'%s' is not a valid presence state" % (
+                state["presence"],
+            ))
 
         logger.debug("Updating presence state of %s to %s",
                      target_user.localpart, state["presence"])
@@ -229,18 +238,16 @@ class PresenceHandler(BaseHandler):
         state_to_store = dict(state)
         state_to_store["state"] = state_to_store.pop("presence")
 
-        statuscache=self._get_or_offline_usercache(target_user)
+        statuscache = self._get_or_offline_usercache(target_user)
         was_level = self.STATE_LEVELS[statuscache.get_state()["presence"]]
         now_level = self.STATE_LEVELS[state["presence"]]
 
-        yield defer.DeferredList([
-            self.store.set_presence_state(
-                target_user.localpart, state_to_store
-            ),
-            self.distributor.fire(
-                "collect_presencelike_data", target_user, state
-            ),
-        ])
+        yield self.store.set_presence_state(
+            target_user.localpart, state_to_store
+        )
+        yield self.distributor.fire(
+            "collect_presencelike_data", target_user, state
+        )
 
         if now_level > was_level:
             state["last_active"] = self.clock.time_msec()
@@ -248,14 +255,15 @@ class PresenceHandler(BaseHandler):
         now_online = state["presence"] != PresenceState.OFFLINE
         was_polling = target_user in self._user_cachemap
 
-        if now_online and not was_polling:
-            self.start_polling_presence(target_user, state=state)
-        elif not now_online and was_polling:
-            self.stop_polling_presence(target_user)
+        with PreserveLoggingContext():
+            if now_online and not was_polling:
+                self.start_polling_presence(target_user, state=state)
+            elif not now_online and was_polling:
+                self.stop_polling_presence(target_user)
 
-        # TODO(paul): perform a presence push as part of start/stop poll so
-        #   we don't have to do this all the time
-        self.changed_presencelike_data(target_user, state)
+            # TODO(paul): perform a presence push as part of start/stop poll so
+            #   we don't have to do this all the time
+            self.changed_presencelike_data(target_user, state)
 
     def bump_presence_active_time(self, user, now=None):
         if now is None:
@@ -269,7 +277,7 @@ class PresenceHandler(BaseHandler):
         self._user_cachemap_latest_serial += 1
         statuscache.update(state, serial=self._user_cachemap_latest_serial)
 
-        self.push_presence(user, statuscache=statuscache)
+        return self.push_presence(user, statuscache=statuscache)
 
     @log_function
     def started_user_eventstream(self, user):
@@ -373,8 +381,10 @@ class PresenceHandler(BaseHandler):
         yield self.store.set_presence_list_accepted(
             observer_user.localpart, observed_user.to_string()
         )
-
-        self.start_polling_presence(observer_user, target_user=observed_user)
+        with PreserveLoggingContext():
+            self.start_polling_presence(
+                observer_user, target_user=observed_user
+            )
 
     @defer.inlineCallbacks
     def deny_presence(self, observed_user, observer_user):
@@ -393,7 +403,10 @@ class PresenceHandler(BaseHandler):
             observer_user.localpart, observed_user.to_string()
         )
 
-        self.stop_polling_presence(observer_user, target_user=observed_user)
+        with PreserveLoggingContext():
+            self.stop_polling_presence(
+                observer_user, target_user=observed_user
+            )
 
     @defer.inlineCallbacks
     def get_presence_list(self, observer_user, accepted=None):
@@ -649,8 +662,9 @@ class PresenceHandler(BaseHandler):
             del state["user_id"]
 
             if "presence" not in state:
-                logger.warning("Received a presence 'push' EDU from %s without"
-                    + " a 'presence' key", origin
+                logger.warning(
+                    "Received a presence 'push' EDU from %s without a"
+                    " 'presence' key", origin
                 )
                 continue
 
@@ -701,7 +715,8 @@ class PresenceHandler(BaseHandler):
                 if not self._remote_sendmap[user]:
                     del self._remote_sendmap[user]
 
-        yield defer.DeferredList(deferreds)
+        with PreserveLoggingContext():
+            yield defer.DeferredList(deferreds)
 
     @defer.inlineCallbacks
     def push_update_to_local_and_remote(self, observed_user, statuscache,
@@ -745,7 +760,7 @@ class PresenceHandler(BaseHandler):
         defer.returnValue((localusers, remote_domains))
 
     def push_update_to_clients(self, observed_user, users_to_push=[],
-                                 room_ids=[], statuscache=None):
+                               room_ids=[], statuscache=None):
         self.notifier.on_new_user_event(
             users_to_push,
             room_ids,
@@ -765,8 +780,7 @@ class PresenceEventSource(object):
         presence = self.hs.get_handlers().presence_handler
 
         if (yield presence.store.user_rooms_intersect(
-            [u.to_string() for u in observer_user, observed_user]
-        )):
+                [u.to_string() for u in observer_user, observed_user])):
             defer.returnValue(True)
 
         if observed_user.is_mine:
@@ -823,15 +837,12 @@ class PresenceEventSource(object):
     def get_pagination_rows(self, user, pagination_config, key):
         # TODO (erikj): Does this make sense? Ordering?
 
-        from_token = pagination_config.from_token
-        to_token = pagination_config.to_token
-
         observer_user = user
 
-        from_key = int(from_token.presence_key)
+        from_key = int(pagination_config.from_key)
 
-        if to_token:
-            to_key = int(to_token.presence_key)
+        if pagination_config.to_key:
+            to_key = int(pagination_config.to_key)
         else:
             to_key = -1
 
@@ -841,7 +852,7 @@ class PresenceEventSource(object):
         updates = []
         # TODO(paul): use a DeferredList ? How to limit concurrency.
         for observed_user in cachemap.keys():
-            if not (to_key < cachemap[observed_user].serial < from_key):
+            if not (to_key < cachemap[observed_user].serial <= from_key):
                 continue
 
             if (yield self.is_visible(observer_user, observed_user)):
@@ -849,30 +860,15 @@ class PresenceEventSource(object):
 
         # TODO(paul): limit
 
-        updates = [(k, cachemap[k]) for k in cachemap
-                   if to_key < cachemap[k].serial < from_key]
-
         if updates:
             clock = self.clock
 
             earliest_serial = max([x[1].serial for x in updates])
             data = [x[1].make_event(user=x[0], clock=clock) for x in updates]
 
-            if to_token:
-                next_token = to_token
-            else:
-                next_token = from_token
-
-            next_token = next_token.copy_and_replace(
-                "presence_key", earliest_serial
-            )
-            defer.returnValue((data, next_token))
+            defer.returnValue((data, earliest_serial))
         else:
-            if not to_token:
-                to_token = from_token.copy_and_replace(
-                    "presence_key", 0
-                )
-            defer.returnValue(([], to_token))
+            defer.returnValue(([], 0))
 
 
 class UserPresenceCache(object):
@@ -881,7 +877,7 @@ class UserPresenceCache(object):
     Includes the update timestamp.
     """
     def __init__(self):
-        self.state = {}
+        self.state = {"presence": PresenceState.OFFLINE}
         self.serial = None
 
     def update(self, state, serial):

synapse/handlers/profile.py

@@ -17,7 +17,7 @@ from twisted.internet import defer
 
 from synapse.api.errors import SynapseError, AuthError, CodeMessageException
 from synapse.api.constants import Membership
-from synapse.api.events.room import RoomMemberEvent
+from synapse.util.logcontext import PreserveLoggingContext
 
 from ._base import BaseHandler
 
@@ -47,7 +47,7 @@ class ProfileHandler(BaseHandler):
         )
 
     def registered_user(self, user):
-        self.store.create_profile(user.localpart)
+        return self.store.create_profile(user.localpart)
 
     @defer.inlineCallbacks
     def get_displayname(self, target_user):
@@ -153,10 +153,14 @@ class ProfileHandler(BaseHandler):
         if not user.is_mine:
             defer.returnValue(None)
 
-        (displayname, avatar_url) = yield defer.gatherResults([
-            self.store.get_profile_displayname(user.localpart),
-            self.store.get_profile_avatar_url(user.localpart),
-        ])
+        with PreserveLoggingContext():
+            (displayname, avatar_url) = yield defer.gatherResults(
+                [
+                    self.store.get_profile_displayname(user.localpart),
+                    self.store.get_profile_avatar_url(user.localpart),
+                ],
+                consumeErrors=True
+            )
 
         state["displayname"] = displayname
         state["avatar_url"] = avatar_url
@@ -196,14 +200,10 @@ class ProfileHandler(BaseHandler):
         )
 
         for j in joins:
-            snapshot = yield self.store.snapshot_room(
-                j.room_id, j.state_key, RoomMemberEvent.TYPE,
-                j.state_key
-            )
+            snapshot = yield self.store.snapshot_room(j)
 
             content = {
                 "membership": j.content["membership"],
-                "prev": j.content["membership"],
             }
 
             yield self.distributor.fire(
@@ -218,5 +218,6 @@ class ProfileHandler(BaseHandler):
                 user_id=j.state_key,
             )
 
-            yield self.state_handler.handle_new_event(new_event, snapshot)
-            yield self._on_new_room_event(new_event, snapshot)
+            yield self._on_new_room_event(
+                new_event, snapshot, suppress_auth=True
+            )

synapse/handlers/register.py

@@ -22,7 +22,7 @@ from synapse.api.errors import (
 )
 from ._base import BaseHandler
 import synapse.util.stringutils as stringutils
-from synapse.http.client import IdentityServerHttpClient
+from synapse.http.client import SimpleHttpClient
 from synapse.http.client import CaptchaServerHttpClient
 
 import base64
@@ -63,11 +63,13 @@ class RegistrationHandler(BaseHandler):
             user_id = user.to_string()
 
             token = self._generate_token(user_id)
-            yield self.store.register(user_id=user_id,
+            yield self.store.register(
+                user_id=user_id,
                 token=token,
-                password_hash=password_hash)
+                password_hash=password_hash
+            )
 
-            self.distributor.fire("registered_user", user)
+            yield self.distributor.fire("registered_user", user)
         else:
             # autogen a random user ID
             attempts = 0
@@ -126,12 +128,12 @@ class RegistrationHandler(BaseHandler):
             try:
                 threepid = yield self._threepid_from_creds(c)
             except:
-                logger.err()
+                logger.exception("Couldn't validate 3pid")
                 raise RegistrationError(400, "Couldn't validate 3pid")
 
             if not threepid:
                 raise RegistrationError(400, "Couldn't validate 3pid")
-            logger.info("got threepid medium %s address %s",
+            logger.info("got threepid with medium '%s' and address '%s'",
                         threepid['medium'], threepid['address'])
 
     @defer.inlineCallbacks
@@ -157,7 +159,7 @@ class RegistrationHandler(BaseHandler):
     def _threepid_from_creds(self, creds):
         # TODO: get this from the homeserver rather than creating a new one for
         # each request
-        httpCli = IdentityServerHttpClient(self.hs)
+        httpCli = SimpleHttpClient(self.hs)
         # XXX: make this configurable!
         trustedIdServers = ['matrix.org:8090']
         if not creds['idServer'] in trustedIdServers:
@@ -165,8 +167,11 @@ class RegistrationHandler(BaseHandler):
                         'credentials', creds['idServer'])
             defer.returnValue(None)
         data = yield httpCli.get_json(
-            creds['idServer'],
-            "/_matrix/identity/api/v1/3pid/getValidated3pid",
+            # XXX: This should be HTTPS
+            "http://%s%s" % (
+                creds['idServer'],
+                "/_matrix/identity/api/v1/3pid/getValidated3pid"
+            ),
             {'sid': creds['sid'], 'clientSecret': creds['clientSecret']}
         )
 
@@ -176,13 +181,21 @@ class RegistrationHandler(BaseHandler):
 
     @defer.inlineCallbacks
     def _bind_threepid(self, creds, mxid):
-        httpCli = IdentityServerHttpClient(self.hs)
+        yield
+        logger.debug("binding threepid")
+        httpCli = SimpleHttpClient(self.hs)
         data = yield httpCli.post_urlencoded_get_json(
-            creds['idServer'],
-            "/_matrix/identity/api/v1/3pid/bind",
-            {'sid': creds['sid'], 'clientSecret': creds['clientSecret'],
-            'mxid': mxid}
+            # XXX: Change when ID servers are all HTTPS
+            "http://%s%s" % (
+                creds['idServer'], "/_matrix/identity/api/v1/3pid/bind"
+            ),
+            {
+                'sid': creds['sid'],
+                'clientSecret': creds['clientSecret'],
+                'mxid': mxid,
+            }
         )
+        logger.debug("bound threepid")
         defer.returnValue(data)
 
     @defer.inlineCallbacks
@@ -210,10 +223,7 @@ class RegistrationHandler(BaseHandler):
         # each request
         client = CaptchaServerHttpClient(self.hs)
         data = yield client.post_urlencoded_get_raw(
-            "www.google.com:80",
-            "/recaptcha/api/verify",
-            # twisted dislikes google's response, no content length.
-            accept_partial=True,
+            "http://www.google.com:80/recaptcha/api/verify",
             args={
                 'privatekey': private_key,
                 'remoteip': ip_addr,
@@ -222,5 +232,3 @@ class RegistrationHandler(BaseHandler):
             }
         )
         defer.returnValue(data)
-
-

synapse/handlers/room.py

@@ -21,10 +21,10 @@ from synapse.api.constants import Membership, JoinRules
 from synapse.api.errors import StoreError, SynapseError
 from synapse.api.events.room import (
     RoomMemberEvent, RoomCreateEvent, RoomPowerLevelsEvent,
-    RoomJoinRulesEvent, RoomAddStateLevelEvent, RoomTopicEvent,
-    RoomSendEventLevelEvent, RoomOpsPowerLevelsEvent, RoomNameEvent,
+    RoomTopicEvent, RoomNameEvent, RoomJoinRulesEvent,
 )
 from synapse.util import stringutils
+from synapse.util.async import run_on_reactor
 from ._base import BaseHandler
 
 import logging
@@ -106,11 +106,6 @@ class RoomCreationHandler(BaseHandler):
             if not room_id:
                 raise StoreError(500, "Couldn't generate a room ID.")
 
-        user = self.hs.parse_userid(user_id)
-        creation_events = self._create_events_for_new_room(
-            user, room_id, is_public=is_public
-        )
-
         if room_alias:
             directory_handler = self.hs.get_handlers().directory_handler
             yield directory_handler.create_association(
@@ -120,17 +115,28 @@ class RoomCreationHandler(BaseHandler):
                 servers=[self.hs.hostname],
             )
 
+        user = self.hs.parse_userid(user_id)
+        creation_events = self._create_events_for_new_room(
+            user, room_id, is_public=is_public
+        )
+
+        room_member_handler = self.hs.get_handlers().room_member_handler
+
         @defer.inlineCallbacks
         def handle_event(event):
-            snapshot = yield self.store.snapshot_room(
-                room_id=room_id,
-                user_id=user_id,
-            )
+            snapshot = yield self.store.snapshot_room(event)
 
             logger.debug("Event: %s", event)
 
-            yield self.state_handler.handle_new_event(event, snapshot)
-            yield self._on_new_room_event(event, snapshot, extra_users=[user])
+            if event.type == RoomMemberEvent.TYPE:
+                yield room_member_handler.change_membership(
+                    event,
+                    do_auth=True
+                )
+            else:
+                yield self._on_new_room_event(
+                    event, snapshot, extra_users=[user], suppress_auth=True
+                )
 
         for event in creation_events:
             yield handle_event(event)
@@ -141,7 +147,6 @@ class RoomCreationHandler(BaseHandler):
                 etype=RoomNameEvent.TYPE,
                 room_id=room_id,
                 user_id=user_id,
-                required_power_level=50,
                 content={"name": name},
             )
 
@@ -153,27 +158,11 @@ class RoomCreationHandler(BaseHandler):
                 etype=RoomTopicEvent.TYPE,
                 room_id=room_id,
                 user_id=user_id,
-                required_power_level=50,
                 content={"topic": topic},
             )
 
             yield handle_event(topic_event)
 
-        content = {"membership": Membership.JOIN}
-        join_event = self.event_factory.create_event(
-            etype=RoomMemberEvent.TYPE,
-            state_key=user_id,
-            room_id=room_id,
-            user_id=user_id,
-            membership=Membership.JOIN,
-            content=content
-        )
-
-        yield self.hs.get_handlers().room_member_handler.change_membership(
-            join_event,
-            do_auth=False
-        )
-
         content = {"membership": Membership.INVITE}
         for invitee in invite_list:
             invite_event = self.event_factory.create_event(
@@ -183,27 +172,24 @@ class RoomCreationHandler(BaseHandler):
                 user_id=user_id,
                 content=content
             )
+            yield handle_event(invite_event)
 
-            yield self.hs.get_handlers().room_member_handler.change_membership(
-                invite_event,
-                do_auth=False
-            )
-
-        yield self.hs.get_handlers().room_member_handler.change_membership(
-            join_event,
-            do_auth=False
-        )
         result = {"room_id": room_id}
+
         if room_alias:
             result["room_alias"] = room_alias.to_string()
+            yield directory_handler.send_room_alias_update_event(
+                user_id, room_id
+            )
 
         defer.returnValue(result)
 
     def _create_events_for_new_room(self, creator, room_id, is_public=False):
+        creator_id = creator.to_string()
+
         event_keys = {
             "room_id": room_id,
-            "user_id": creator.to_string(),
-            "required_power_level": 100,
+            "user_id": creator_id,
         }
 
         def create(etype, **content):
@@ -218,9 +204,32 @@ class RoomCreationHandler(BaseHandler):
             creator=creator.to_string(),
         )
 
+        join_event = self.event_factory.create_event(
+            etype=RoomMemberEvent.TYPE,
+            state_key=creator_id,
+            content={
+                "membership": Membership.JOIN,
+            },
+            **event_keys
+        )
+
         power_levels_event = self.event_factory.create_event(
             etype=RoomPowerLevelsEvent.TYPE,
-            content={creator.to_string(): 100, "default": 0},
+            content={
+                "users": {
+                    creator.to_string(): 100,
+                },
+                "users_default": 0,
+                "events": {
+                    RoomNameEvent.TYPE: 100,
+                    RoomPowerLevelsEvent.TYPE: 100,
+                },
+                "events_default": 0,
+                "state_default": 50,
+                "ban": 50,
+                "kick": 50,
+                "redact": 50
+            },
             **event_keys
         )
 
@@ -230,30 +239,11 @@ class RoomCreationHandler(BaseHandler):
             join_rule=join_rule,
         )
 
-        add_state_event = create(
-            etype=RoomAddStateLevelEvent.TYPE,
-            level=100,
-        )
-
-        send_event = create(
-            etype=RoomSendEventLevelEvent.TYPE,
-            level=0,
-        )
-
-        ops = create(
-            etype=RoomOpsPowerLevelsEvent.TYPE,
-            ban_level=50,
-            kick_level=50,
-            redact_level=50,
-        )
-
         return [
             creation_event,
+            join_event,
             power_levels_event,
             join_rules_event,
-            add_state_event,
-            send_event,
-            ops,
         ]
 
 
@@ -368,25 +358,13 @@ class RoomMemberHandler(BaseHandler):
         """
         target_user_id = event.state_key
 
-        snapshot = yield self.store.snapshot_room(
-            event.room_id, event.user_id,
-            RoomMemberEvent.TYPE, target_user_id
-        )
+        snapshot = yield self.store.snapshot_room(event)
+
         ## TODO(markjh): get prev state from snapshot.
         prev_state = yield self.store.get_room_member(
             target_user_id, event.room_id
         )
 
-        if prev_state:
-            event.content["prev"] = prev_state.membership
-
-#        if prev_state and prev_state.membership == event.membership:
-#            # treat this event as a NOOP.
-#            if do_auth:  # This is mainly to fix a unit test.
-#                yield self.auth.check(event, raises=True)
-#            defer.returnValue({})
-#            return
-
         room_id = event.room_id
 
         # If we're trying to join a room then we have to do this differently
@@ -396,29 +374,17 @@ class RoomMemberHandler(BaseHandler):
             yield self._do_join(event, snapshot, do_auth=do_auth)
         else:
             # This is not a JOIN, so we can handle it normally.
-            if do_auth:
-                yield self.auth.check(event, snapshot, raises=True)
-
-            # If we're banning someone, set a req power level
-            if event.membership == Membership.BAN:
-                if not hasattr(event, "required_power_level") or event.required_power_level is None:
-                    # Add some default required_power_level
-                    user_level = yield self.store.get_power_level(
-                        event.room_id,
-                        event.user_id,
-                    )
-                    event.required_power_level = user_level
 
             if prev_state and prev_state.membership == event.membership:
                 # double same action, treat this event as a NOOP.
                 defer.returnValue({})
                 return
 
-            yield self.state_handler.handle_new_event(event, snapshot)
             yield self._do_local_membership_update(
                 event,
                 membership=event.content["membership"],
                 snapshot=snapshot,
+                do_auth=do_auth,
             )
 
         defer.returnValue({"room_id": room_id})
@@ -448,10 +414,7 @@ class RoomMemberHandler(BaseHandler):
             content=content,
         )
 
-        snapshot = yield self.store.snapshot_room(
-            room_id, joinee.to_string(), RoomMemberEvent.TYPE,
-            joinee.to_string()
-        )
+        snapshot = yield self.store.snapshot_room(new_event)
 
         yield self._do_join(new_event, snapshot, room_host=host, do_auth=True)
 
@@ -473,9 +436,12 @@ class RoomMemberHandler(BaseHandler):
         # that we are allowed to join when we decide whether or not we
         # need to do the invite/join dance.
 
-        hosts = yield self.store.get_joined_hosts_for_room(room_id)
+        is_host_in_room = yield self.auth.check_host_in_room(
+            event.room_id,
+            self.hs.hostname
+        )
 
-        if self.hs.hostname in hosts:
+        if is_host_in_room:
             should_do_dance = False
         elif room_host:
             should_do_dance = True
@@ -507,18 +473,15 @@ class RoomMemberHandler(BaseHandler):
         if not have_joined:
             logger.debug("Doing normal join")
 
-            if do_auth:
-                yield self.auth.check(event, snapshot, raises=True)
-
-            yield self.state_handler.handle_new_event(event, snapshot)
             yield self._do_local_membership_update(
                 event,
                 membership=event.content["membership"],
                 snapshot=snapshot,
+                do_auth=do_auth,
             )
 
         user = self.hs.parse_userid(event.user_id)
-        self.distributor.fire(
+        yield self.distributor.fire(
             "user_joined_room", user=user, room_id=room_id
         )
 
@@ -558,26 +521,29 @@ class RoomMemberHandler(BaseHandler):
 
         defer.returnValue([r.room_id for r in rooms])
 
-    def _do_local_membership_update(self, event, membership, snapshot):
-        destinations = []
+    @defer.inlineCallbacks
+    def _do_local_membership_update(self, event, membership, snapshot,
+                                    do_auth):
+        yield run_on_reactor()
 
         # If we're inviting someone, then we should also send it to that
         # HS.
         target_user_id = event.state_key
         target_user = self.hs.parse_userid(target_user_id)
-        if membership == Membership.INVITE:
-            host = target_user.domain
-            destinations.append(host)
+        if membership == Membership.INVITE and not target_user.is_mine:
+            do_invite_host = target_user.domain
+        else:
+            do_invite_host = None
 
-        # Always include target domain
-        host = target_user.domain
-        destinations.append(host)
-
-        return self._on_new_room_event(
-            event, snapshot, extra_destinations=destinations,
-            extra_users=[target_user]
+        yield self._on_new_room_event(
+            event,
+            snapshot,
+            extra_users=[target_user],
+            suppress_auth=(not do_auth),
+            do_invite_host=do_invite_host,
         )
 
+
 class RoomListHandler(BaseHandler):
 
     @defer.inlineCallbacks
@@ -617,23 +583,14 @@ class RoomEventSource(object):
         return self.store.get_room_events_max_id()
 
     @defer.inlineCallbacks
-    def get_pagination_rows(self, user, pagination_config, key):
-        from_token = pagination_config.from_token
-        to_token = pagination_config.to_token
-        limit = pagination_config.limit
-        direction = pagination_config.direction
-
-        to_key = to_token.room_key if to_token else None
-
+    def get_pagination_rows(self, user, config, key):
         events, next_key = yield self.store.paginate_room_events(
             room_id=key,
-            from_key=from_token.room_key,
-            to_key=to_key,
-            direction=direction,
-            limit=limit,
+            from_key=config.from_key,
+            to_key=config.to_key,
+            direction=config.direction,
+            limit=config.limit,
             with_feedback=True
         )
 
-        next_token = from_token.copy_and_replace("room_key", next_key)
-
-        defer.returnValue((events, next_token))
+        defer.returnValue((events, next_key))

synapse/handlers/typing.py

@@ -96,9 +96,10 @@ class TypingNotificationHandler(BaseHandler):
         remotedomains = set()
 
         rm_handler = self.homeserver.get_handlers().room_member_handler
-        yield rm_handler.fetch_room_distributions_into(room_id,
-                localusers=localusers, remotedomains=remotedomains,
-                ignore_user=user)
+        yield rm_handler.fetch_room_distributions_into(
+            room_id, localusers=localusers, remotedomains=remotedomains,
+            ignore_user=user
+        )
 
         for u in localusers:
             self.push_update_to_clients(
@@ -130,8 +131,9 @@ class TypingNotificationHandler(BaseHandler):
         localusers = set()
 
         rm_handler = self.homeserver.get_handlers().room_member_handler
-        yield rm_handler.fetch_room_distributions_into(room_id,
-                localusers=localusers)
+        yield rm_handler.fetch_room_distributions_into(
+            room_id, localusers=localusers
+        )
 
         for u in localusers:
             self.push_update_to_clients(
@@ -142,7 +144,7 @@ class TypingNotificationHandler(BaseHandler):
             )
 
     def push_update_to_clients(self, room_id, observer_user, observed_user,
-            typing):
+                               typing):
         # TODO(paul) steal this from presence.py
         pass
 
@@ -158,4 +160,4 @@ class TypingNotificationEventSource(object):
         return 0
 
     def get_pagination_rows(self, user, pagination_config, key):
-        return ([], pagination_config.from_token)
+        return ([], pagination_config.from_key)

synapse/http/__init__.py

@@ -12,4 +12,3 @@
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
-

synapse/http/client.py

@@ -15,232 +15,57 @@
 
 
 from twisted.internet import defer, reactor
-from twisted.internet.error import DNSLookupError
-from twisted.web.client import _AgentBase, _URI, readBody, FileBodyProducer, PartialDownloadError
+from twisted.web.client import (
+    Agent, readBody, FileBodyProducer, PartialDownloadError
+)
 from twisted.web.http_headers import Headers
 
-from synapse.http.endpoint import matrix_endpoint
-from synapse.util.async import sleep
-
-from syutil.jsonutil import encode_canonical_json
-
-from synapse.api.errors import CodeMessageException, SynapseError
-
-from syutil.crypto.jsonsign import sign_json
-
 from StringIO import StringIO
 
 import json
 import logging
 import urllib
-import urlparse
 
 
 logger = logging.getLogger(__name__)
 
 
-class MatrixHttpAgent(_AgentBase):
-
-    def __init__(self, reactor, pool=None):
-        _AgentBase.__init__(self, reactor, pool)
-
-    def request(self, destination, endpoint, method, path, params, query,
-                headers, body_producer):
-
-        host = b""
-        port = 0
-        fragment = b""
-
-        parsed_URI = _URI(b"http", destination, host, port, path, params,
-                          query, fragment)
-
-        # Set the connection pool key to be the destination.
-        key = destination
-
-        return self._requestWithEndpoint(key, endpoint, method, parsed_URI,
-                                         headers, body_producer,
-                                         parsed_URI.originForm)
-
-
-class BaseHttpClient(object):
-    """Base class for HTTP clients using twisted.
+class SimpleHttpClient(object):
+    """
+    A simple, no-frills HTTP client with methods that wrap up common ways of
+    using HTTP in Matrix
     """
-
     def __init__(self, hs):
-        self.agent = MatrixHttpAgent(reactor)
         self.hs = hs
+        # The default context factory in Twisted 14.0.0 (which we require) is
+        # BrowserLikePolicyForHTTPS which will do regular cert validation
+        # 'like a browser'
+        self.agent = Agent(reactor)
 
     @defer.inlineCallbacks
-    def _create_request(self, destination, method, path_bytes,
-                        body_callback, headers_dict={}, param_bytes=b"",
-                        query_bytes=b"", retry_on_dns_fail=True):
-        """ Creates and sends a request to the given url
-        """
-        headers_dict[b"User-Agent"] = [b"Synapse"]
-        headers_dict[b"Host"] = [destination]
+    def post_urlencoded_get_json(self, uri, args={}):
+        logger.debug("post_urlencoded_get_json args: %s", args)
+        query_bytes = urllib.urlencode(args, True)
 
-        url_bytes = urlparse.urlunparse(
-            ("", "", path_bytes, param_bytes, query_bytes, "",)
+        response = yield self.agent.request(
+            "POST",
+            uri.encode("ascii"),
+            headers=Headers({
+                "Content-Type": ["application/x-www-form-urlencoded"]
+            }),
+            bodyProducer=FileBodyProducer(StringIO(query_bytes))
         )
 
-        logger.debug("Sending request to %s: %s %s",
-                     destination, method, url_bytes)
-
-        logger.debug(
-            "Types: %s",
-            [
-                type(destination), type(method), type(path_bytes),
-                type(param_bytes),
-                type(query_bytes)
-            ]
-        )
-
-        retries_left = 5
-
-        endpoint = self._getEndpoint(reactor, destination);
-
-        while True:
-
-            producer = body_callback(method, url_bytes, headers_dict)
-
-            try:
-                response = yield self.agent.request(
-                    destination,
-                    endpoint,
-                    method,
-                    path_bytes,
-                    param_bytes,
-                    query_bytes,
-                    Headers(headers_dict),
-                    producer
-                )
-
-                logger.debug("Got response to %s", method)
-                break
-            except Exception as e:
-                if not retry_on_dns_fail and isinstance(e, DNSLookupError):
-                    logger.warn("DNS Lookup failed to %s with %s", destination,
-                                e)
-                    raise SynapseError(400, "Domain specified not found.")
-
-                logger.exception("Got error in _create_request")
-                _print_ex(e)
-
-                if retries_left:
-                    yield sleep(2 ** (5 - retries_left))
-                    retries_left -= 1
-                else:
-                    raise
-
-        if 200 <= response.code < 300:
-            # We need to update the transactions table to say it was sent?
-            pass
-        else:
-            # :'(
-            # Update transactions table?
-            logger.error(
-                "Got response %d %s", response.code, response.phrase
-            )
-            raise CodeMessageException(
-                response.code, response.phrase
-            )
-
-        defer.returnValue(response)
-
-
-class MatrixHttpClient(BaseHttpClient):
-    """ Wrapper around the twisted HTTP client api. Implements
-
-    Attributes:
-        agent (twisted.web.client.Agent): The twisted Agent used to send the
-            requests.
-    """
-
-    RETRY_DNS_LOOKUP_FAILURES = "__retry_dns"
-
-    def __init__(self, hs):
-        self.signing_key = hs.config.signing_key[0]
-        self.server_name = hs.hostname
-        BaseHttpClient.__init__(self, hs)
-
-    def sign_request(self, destination, method, url_bytes, headers_dict,
-                     content=None):
-        request = {
-            "method": method,
-            "uri": url_bytes,
-            "origin": self.server_name,
-            "destination": destination,
-        }
-
-        if content is not None:
-            request["content"] = content
-
-        request = sign_json(request, self.server_name, self.signing_key)
-
-        auth_headers = []
-
-        for key,sig in request["signatures"][self.server_name].items():
-            auth_headers.append(bytes(
-                "X-Matrix origin=%s,key=\"%s\",sig=\"%s\"" % (
-                    self.server_name, key, sig,
-                )
-            ))
-
-        headers_dict[b"Authorization"] = auth_headers
-
-    @defer.inlineCallbacks
-    def put_json(self, destination, path, data={}, json_data_callback=None):
-        """ Sends the specifed json data using PUT
-
-        Args:
-            destination (str): The remote server to send the HTTP request
-                to.
-            path (str): The HTTP path.
-            data (dict): A dict containing the data that will be used as
-                the request body. This will be encoded as JSON.
-            json_data_callback (callable): A callable returning the dict to
-                use as the request body.
-
-        Returns:
-            Deferred: Succeeds when we get a 2xx HTTP response. The result
-            will be the decoded JSON body. On a 4xx or 5xx error response a
-            CodeMessageException is raised.
-        """
-
-        if not json_data_callback:
-            def json_data_callback():
-                return data
-
-        def body_callback(method, url_bytes, headers_dict):
-            json_data = json_data_callback()
-            self.sign_request(
-                destination, method, url_bytes, headers_dict, json_data
-            )
-            producer = _JsonProducer(json_data)
-            return producer
-
-        response = yield self._create_request(
-            destination.encode("ascii"),
-            "PUT",
-            path.encode("ascii"),
-            body_callback=body_callback,
-            headers_dict={"Content-Type": ["application/json"]},
-        )
-
-        logger.debug("Getting resp body")
         body = yield readBody(response)
-        logger.debug("Got resp body")
 
-        defer.returnValue((response.code, body))
+        defer.returnValue(json.loads(body))
 
     @defer.inlineCallbacks
-    def get_json(self, destination, path, args={}, retry_on_dns_fail=True):
-        """ Get's some json from the given host homeserver and path
+    def get_json(self, uri, args={}):
+        """ Get's some json from the given host and path
 
         Args:
-            destination (str): The remote server to send the HTTP request
-                to.
-            path (str): The HTTP path.
+            uri (str): The URI to request, not including query parameters
             args (dict): A dictionary used to create query strings, defaults to
                 None.
                 **Note**: The value of each key is assumed to be an iterable
@@ -252,22 +77,15 @@ class MatrixHttpClient(BaseHttpClient):
             The result of the deferred is a tuple of `(code, response)`,
             where `response` is a dict representing the decoded JSON body.
         """
-        logger.debug("get_json args: %s", args)
 
-        query_bytes = urllib.urlencode(args, True)
-        logger.debug("Query bytes: %s Retry DNS: %s", args, retry_on_dns_fail)
+        yield
+        if len(args):
+            query_bytes = urllib.urlencode(args, True)
+            uri = "%s?%s" % (uri, query_bytes)
 
-        def body_callback(method, url_bytes, headers_dict):
-            self.sign_request(destination, method, url_bytes, headers_dict)
-            return None
-
-        response = yield self._create_request(
-            destination.encode("ascii"),
+        response = yield self.agent.request(
             "GET",
-            path.encode("ascii"),
-            query_bytes=query_bytes,
-            body_callback=body_callback,
-            retry_on_dns_fail=retry_on_dns_fail
+            uri.encode("ascii"),
         )
 
         body = yield readBody(response)
@@ -275,76 +93,32 @@ class MatrixHttpClient(BaseHttpClient):
         defer.returnValue(json.loads(body))
 
 
-    def _getEndpoint(self, reactor, destination):
-        return matrix_endpoint(
-            reactor, destination, timeout=10,
-            ssl_context_factory=self.hs.tls_context_factory
-        )
-
-
-class IdentityServerHttpClient(BaseHttpClient):
-    """Separate HTTP client for talking to the Identity servers since they
-    don't use SRV records and talk x-www-form-urlencoded rather than JSON.
+class CaptchaServerHttpClient(SimpleHttpClient):
+    """
+    Separate HTTP client for talking to google's captcha servers
+    Only slightly special because accepts partial download responses
     """
-    def _getEndpoint(self, reactor, destination):
-        #TODO: This should be talking TLS
-        return matrix_endpoint(reactor, destination, timeout=10)
 
     @defer.inlineCallbacks
-    def post_urlencoded_get_json(self, destination, path, args={}):
-        logger.debug("post_urlencoded_get_json args: %s", args)
+    def post_urlencoded_get_raw(self, url, args={}):
         query_bytes = urllib.urlencode(args, True)
 
-        def body_callback(method, url_bytes, headers_dict):
-            return FileBodyProducer(StringIO(query_bytes))
-
-        response = yield self._create_request(
-            destination.encode("ascii"),
+        response = yield self.agent.request(
             "POST",
-            path.encode("ascii"),
-            body_callback=body_callback,
-            headers_dict={
+            url.encode("ascii"),
+            bodyProducer=FileBodyProducer(StringIO(query_bytes)),
+            headers=Headers({
                 "Content-Type": ["application/x-www-form-urlencoded"]
-            }
-        )
-
-        body = yield readBody(response)
-
-        defer.returnValue(json.loads(body))
-
-
-class CaptchaServerHttpClient(MatrixHttpClient):
-    """Separate HTTP client for talking to google's captcha servers"""
-
-    def _getEndpoint(self, reactor, destination):
-        return matrix_endpoint(reactor, destination, timeout=10)
-
-    @defer.inlineCallbacks
-    def post_urlencoded_get_raw(self, destination, path, accept_partial=False,
-                                args={}):
-        query_bytes = urllib.urlencode(args, True)
-
-        def body_callback(method, url_bytes, headers_dict):
-            return FileBodyProducer(StringIO(query_bytes))
-
-        response = yield self._create_request(
-            destination.encode("ascii"),
-            "POST",
-            path.encode("ascii"),
-            body_callback=body_callback,
-            headers_dict={
-                "Content-Type": ["application/x-www-form-urlencoded"]
-            }
+            })
         )
 
         try:
             body = yield readBody(response)
             defer.returnValue(body)
         except PartialDownloadError as e:
-            if accept_partial:
-                defer.returnValue(e.response)
-            else:
-                raise e
+            # twisted dislikes google's response, no content length.
+            defer.returnValue(e.response)
+
 
 def _print_ex(e):
     if hasattr(e, "reasons") and e.reasons:
@@ -352,24 +126,3 @@ def _print_ex(e):
             _print_ex(ex)
     else:
         logger.exception(e)
-
-
-class _JsonProducer(object):
-    """ Used by the twisted http client to create the HTTP body from json
-    """
-    def __init__(self, jsn):
-        self.reset(jsn)
-
-    def reset(self, jsn):
-        self.body = encode_canonical_json(jsn)
-        self.length = len(self.body)
-
-    def startProducing(self, consumer):
-        consumer.write(self.body)
-        return defer.succeed(None)
-
-    def pauseProducing(self):
-        pass
-
-    def stopProducing(self):
-        pass

synapse/http/content_repository.py

@@ -38,8 +38,8 @@ class ContentRepoResource(resource.Resource):
 
     Uploads are POSTed to wherever this Resource is linked to. This resource
     returns a "content token" which can be used to GET this content again. The
-    token is typically a path, but it may not be. Tokens can expire, be one-time
-    uses, etc.
+    token is typically a path, but it may not be. Tokens can expire, be
+    one-time uses, etc.
 
     In this case, the token is a path to the file and contains 3 interesting
     sections:
@@ -129,6 +129,16 @@ class ContentRepoResource(resource.Resource):
             logger.info("Sending file %s", file_path)
             f = open(file_path, 'rb')
             request.setHeader('Content-Type', content_type)
+
+            # cache for at least a day.
+            # XXX: we might want to turn this off for data we don't want to
+            # recommend caching as it's sensitive or private - or at least
+            # select private. don't bother setting Expires as all our matrix
+            # clients are smart enough to be happy with Cache-Control (right?)
+            request.setHeader(
+                "Cache-Control", "public,max-age=86400,s-maxage=86400"
+            )
+
             d = FileSender().beginFileTransfer(f, request)
 
             # after the file has been sent, clean up and finish the request
@@ -171,17 +181,16 @@ class ContentRepoResource(resource.Resource):
 
             fname = yield self.map_request_to_name(request)
 
-            # TODO I have a suspcious feeling this is just going to block
+            # TODO I have a suspicious feeling this is just going to block
             with open(fname, "wb") as f:
                 f.write(request.content.read())
 
-
             # FIXME (erikj): These should use constants.
             file_name = os.path.basename(fname)
-            # FIXME: we can't assume what the public mounted path of the repo is
+            # FIXME: we can't assume what the repo's public mounted path is
             # ...plus self-signed SSL won't work to remote clients anyway
             # ...and we can't assume that it's SSL anyway, as we might want to
-            # server it via the non-SSL listener...
+            # serve it via the non-SSL listener...
             url = "%s/_matrix/content/%s" % (
                 self.external_addr, file_name
             )
@@ -201,6 +210,3 @@ class ContentRepoResource(resource.Resource):
                 500,
                 json.dumps({"error": "Internal server error"}),
                 send_cors=True)
-
-
-

synapse/http/endpoint.py

@@ -27,8 +27,8 @@ import random
 logger = logging.getLogger(__name__)
 
 
-def matrix_endpoint(reactor, destination, ssl_context_factory=None,
-                    timeout=None):
+def matrix_federation_endpoint(reactor, destination, ssl_context_factory=None,
+                               timeout=None):
     """Construct an endpoint for the given matrix destination.
 
     Args:

synapse/http/matrixfederationclient.py

@@ -0,0 +1,308 @@
+# -*- coding: utf-8 -*-
+# Copyright 2014 OpenMarket Ltd
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+from twisted.internet import defer, reactor
+from twisted.internet.error import DNSLookupError
+from twisted.web.client import readBody, _AgentBase, _URI
+from twisted.web.http_headers import Headers
+
+from synapse.http.endpoint import matrix_federation_endpoint
+from synapse.util.async import sleep
+from synapse.util.logcontext import PreserveLoggingContext
+
+from syutil.jsonutil import encode_canonical_json
+
+from synapse.api.errors import CodeMessageException, SynapseError
+
+from syutil.crypto.jsonsign import sign_json
+
+import json
+import logging
+import urllib
+import urlparse
+
+
+logger = logging.getLogger(__name__)
+
+
+class MatrixFederationHttpAgent(_AgentBase):
+
+    def __init__(self, reactor, pool=None):
+        _AgentBase.__init__(self, reactor, pool)
+
+    def request(self, destination, endpoint, method, path, params, query,
+                headers, body_producer):
+
+        host = b""
+        port = 0
+        fragment = b""
+
+        parsed_URI = _URI(b"http", destination, host, port, path, params,
+                          query, fragment)
+
+        # Set the connection pool key to be the destination.
+        key = destination
+
+        return self._requestWithEndpoint(key, endpoint, method, parsed_URI,
+                                         headers, body_producer,
+                                         parsed_URI.originForm)
+
+
+class MatrixFederationHttpClient(object):
+    """HTTP client used to talk to other homeservers over the federation
+    protocol. Send client certificates and signs requests.
+
+    Attributes:
+        agent (twisted.web.client.Agent): The twisted Agent used to send the
+            requests.
+    """
+
+    def __init__(self, hs):
+        self.hs = hs
+        self.signing_key = hs.config.signing_key[0]
+        self.server_name = hs.hostname
+        self.agent = MatrixFederationHttpAgent(reactor)
+
+    @defer.inlineCallbacks
+    def _create_request(self, destination, method, path_bytes,
+                        body_callback, headers_dict={}, param_bytes=b"",
+                        query_bytes=b"", retry_on_dns_fail=True):
+        """ Creates and sends a request to the given url
+        """
+        headers_dict[b"User-Agent"] = [b"Synapse"]
+        headers_dict[b"Host"] = [destination]
+
+        url_bytes = urlparse.urlunparse(
+            ("", "", path_bytes, param_bytes, query_bytes, "",)
+        )
+
+        logger.debug("Sending request to %s: %s %s",
+                     destination, method, url_bytes)
+
+        logger.debug(
+            "Types: %s",
+            [
+                type(destination), type(method), type(path_bytes),
+                type(param_bytes),
+                type(query_bytes)
+            ]
+        )
+
+        retries_left = 5
+
+        endpoint = self._getEndpoint(reactor, destination)
+
+        while True:
+            producer = None
+            if body_callback:
+                producer = body_callback(method, url_bytes, headers_dict)
+
+            try:
+                with PreserveLoggingContext():
+                    response = yield self.agent.request(
+                        destination,
+                        endpoint,
+                        method,
+                        path_bytes,
+                        param_bytes,
+                        query_bytes,
+                        Headers(headers_dict),
+                        producer
+                    )
+
+                logger.debug("Got response to %s", method)
+                break
+            except Exception as e:
+                if not retry_on_dns_fail and isinstance(e, DNSLookupError):
+                    logger.warn("DNS Lookup failed to %s with %s", destination,
+                                e)
+                    raise SynapseError(400, "Domain specified not found.")
+
+                logger.exception("Got error in _create_request")
+                _print_ex(e)
+
+                if retries_left:
+                    yield sleep(2 ** (5 - retries_left))
+                    retries_left -= 1
+                else:
+                    raise
+
+        if 200 <= response.code < 300:
+            # We need to update the transactions table to say it was sent?
+            pass
+        else:
+            # :'(
+            # Update transactions table?
+            logger.error(
+                "Got response %d %s", response.code, response.phrase
+            )
+            raise CodeMessageException(
+                response.code, response.phrase
+            )
+
+        defer.returnValue(response)
+
+    def sign_request(self, destination, method, url_bytes, headers_dict,
+                     content=None):
+        request = {
+            "method": method,
+            "uri": url_bytes,
+            "origin": self.server_name,
+            "destination": destination,
+        }
+
+        if content is not None:
+            request["content"] = content
+
+        request = sign_json(request, self.server_name, self.signing_key)
+
+        auth_headers = []
+
+        for key, sig in request["signatures"][self.server_name].items():
+            auth_headers.append(bytes(
+                "X-Matrix origin=%s,key=\"%s\",sig=\"%s\"" % (
+                    self.server_name, key, sig,
+                )
+            ))
+
+        headers_dict[b"Authorization"] = auth_headers
+
+    @defer.inlineCallbacks
+    def put_json(self, destination, path, data={}, json_data_callback=None):
+        """ Sends the specifed json data using PUT
+
+        Args:
+            destination (str): The remote server to send the HTTP request
+                to.
+            path (str): The HTTP path.
+            data (dict): A dict containing the data that will be used as
+                the request body. This will be encoded as JSON.
+            json_data_callback (callable): A callable returning the dict to
+                use as the request body.
+
+        Returns:
+            Deferred: Succeeds when we get a 2xx HTTP response. The result
+            will be the decoded JSON body. On a 4xx or 5xx error response a
+            CodeMessageException is raised.
+        """
+
+        if not json_data_callback:
+            def json_data_callback():
+                return data
+
+        def body_callback(method, url_bytes, headers_dict):
+            json_data = json_data_callback()
+            self.sign_request(
+                destination, method, url_bytes, headers_dict, json_data
+            )
+            producer = _JsonProducer(json_data)
+            return producer
+
+        response = yield self._create_request(
+            destination.encode("ascii"),
+            "PUT",
+            path.encode("ascii"),
+            body_callback=body_callback,
+            headers_dict={"Content-Type": ["application/json"]},
+        )
+
+        logger.debug("Getting resp body")
+        body = yield readBody(response)
+        logger.debug("Got resp body")
+
+        defer.returnValue((response.code, body))
+
+    @defer.inlineCallbacks
+    def get_json(self, destination, path, args={}, retry_on_dns_fail=True):
+        """ Get's some json from the given host homeserver and path
+
+        Args:
+            destination (str): The remote server to send the HTTP request
+                to.
+            path (str): The HTTP path.
+            args (dict): A dictionary used to create query strings, defaults to
+                None.
+                **Note**: The value of each key is assumed to be an iterable
+                and *not* a string.
+
+        Returns:
+            Deferred: Succeeds when we get *any* HTTP response.
+
+            The result of the deferred is a tuple of `(code, response)`,
+            where `response` is a dict representing the decoded JSON body.
+        """
+        logger.debug("get_json args: %s", args)
+
+        encoded_args = {}
+        for k, vs in args.items():
+            if isinstance(vs, basestring):
+                vs = [vs]
+            encoded_args[k] = [v.encode("UTF-8") for v in vs]
+
+        query_bytes = urllib.urlencode(encoded_args, True)
+        logger.debug("Query bytes: %s Retry DNS: %s", args, retry_on_dns_fail)
+
+        def body_callback(method, url_bytes, headers_dict):
+            self.sign_request(destination, method, url_bytes, headers_dict)
+            return None
+
+        response = yield self._create_request(
+            destination.encode("ascii"),
+            "GET",
+            path.encode("ascii"),
+            query_bytes=query_bytes,
+            body_callback=body_callback,
+            retry_on_dns_fail=retry_on_dns_fail
+        )
+
+        body = yield readBody(response)
+
+        defer.returnValue(json.loads(body))
+
+    def _getEndpoint(self, reactor, destination):
+        return matrix_federation_endpoint(
+            reactor, destination, timeout=10,
+            ssl_context_factory=self.hs.tls_context_factory
+        )
+
+
+def _print_ex(e):
+    if hasattr(e, "reasons") and e.reasons:
+        for ex in e.reasons:
+            _print_ex(ex)
+    else:
+        logger.exception(e)
+
+
+class _JsonProducer(object):
+    """ Used by the twisted http client to create the HTTP body from json
+    """
+    def __init__(self, jsn):
+        self.reset(jsn)
+
+    def reset(self, jsn):
+        self.body = encode_canonical_json(jsn)
+        self.length = len(self.body)
+
+    def startProducing(self, consumer):
+        consumer.write(self.body)
+        return defer.succeed(None)
+
+    def pauseProducing(self):
+        pass
+
+    def stopProducing(self):
+        pass

synapse/http/server.py

@@ -20,6 +20,7 @@ from syutil.jsonutil import (
 from synapse.api.errors import (
     cs_exception, SynapseError, CodeMessageException
 )
+from synapse.util.logcontext import LoggingContext
 
 from twisted.internet import defer, reactor
 from twisted.web import server, resource
@@ -88,9 +89,19 @@ class JsonResource(HttpServer, resource.Resource):
     def render(self, request):
         """ This get's called by twisted every time someone sends us a request.
         """
-        self._async_render(request)
+        self._async_render_with_logging_context(request)
         return server.NOT_DONE_YET
 
+    _request_id = 0
+
+    @defer.inlineCallbacks
+    def _async_render_with_logging_context(self, request):
+        request_id = "%s-%s" % (request.method, JsonResource._request_id)
+        JsonResource._request_id += 1
+        with LoggingContext(request_id) as request_context:
+            request_context.request = request_id
+            yield self._async_render(request)
+
     @defer.inlineCallbacks
     def _async_render(self, request):
         """ This get's called by twisted every time someone sends us a request.
@@ -127,8 +138,7 @@ class JsonResource(HttpServer, resource.Resource):
             )
         except CodeMessageException as e:
             if isinstance(e, SynapseError):
-                logger.error("%s SynapseError: %s - %s", request, e.code,
-                             e.msg)
+                logger.info("%s SynapseError: %s - %s", request, e.code, e.msg)
             else:
                 logger.exception(e)
             self._send_response(

synapse/notifier.py

@@ -13,9 +13,11 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-from twisted.internet import defer, reactor
+from twisted.internet import defer
 
 from synapse.util.logutils import log_function
+from synapse.util.logcontext import PreserveLoggingContext
+from synapse.util.async import run_on_reactor
 
 import logging
 
@@ -79,6 +81,8 @@ class Notifier(object):
 
         self.event_sources = hs.get_event_sources()
 
+        self.clock = hs.get_clock()
+
         hs.get_distributor().observe(
             "user_joined_room", self._user_joined_room
         )
@@ -93,6 +97,7 @@ class Notifier(object):
         listening to the room, and any listeners for the users in the
         `extra_users` param.
         """
+        yield run_on_reactor()
         room_id = event.room_id
 
         room_source = self.event_sources.sources["room"]
@@ -127,9 +132,10 @@ class Notifier(object):
         def eb(failure):
             logger.exception("Failed to notify listener", failure)
 
-        yield defer.DeferredList(
-            [notify(l).addErrback(eb) for l in listeners]
-        )
+        with PreserveLoggingContext():
+            yield defer.DeferredList(
+                [notify(l).addErrback(eb) for l in listeners]
+            )
 
     @defer.inlineCallbacks
     @log_function
@@ -139,6 +145,7 @@ class Notifier(object):
 
         Will wake up all listeners for the given users and rooms.
         """
+        yield run_on_reactor()
         presence_source = self.event_sources.sources["presence"]
 
         listeners = set()
@@ -167,16 +174,18 @@ class Notifier(object):
                 )
 
         def eb(failure):
-            logger.error("Failed to notify listener",
+            logger.error(
+                "Failed to notify listener",
                 exc_info=(
                     failure.type,
                     failure.value,
                     failure.getTracebackObject())
             )
 
-        yield defer.DeferredList(
-            [notify(l).addErrback(eb) for l in listeners]
-        )
+        with PreserveLoggingContext():
+            yield defer.DeferredList(
+                [notify(l).addErrback(eb) for l in listeners]
+            )
 
     def get_events_for(self, user, rooms, pagination_config, timeout):
         """ For the given user and rooms, return any new events for them. If
@@ -206,28 +215,28 @@ class Notifier(object):
             deferred,
         )
 
+        def _timeout_listener():
+            # TODO (erikj): We should probably set to_token to the current
+            # max rather than reusing from_token.
+            listener.notify(
+                self,
+                [],
+                listener.from_token,
+                listener.from_token,
+            )
+
         if timeout:
-            reactor.callLater(timeout/1000, self._timeout_listener, listener)
+            self.clock.call_later(timeout/1000.0, _timeout_listener)
 
             self._register_with_keys(listener)
 
         yield self._check_for_updates(listener)
 
         if not timeout:
-            self._timeout_listener(listener)
+            _timeout_listener()
 
         return
 
-    def _timeout_listener(self, listener):
-        # TODO (erikj): We should probably set to_token to the current max
-        # rather than reusing from_token.
-        listener.notify(
-            self,
-            [],
-            listener.from_token,
-            listener.from_token,
-        )
-
     @log_function
     def _register_with_keys(self, listener):
         for room in listener.rooms:

synapse/rest/base.py

@@ -18,6 +18,11 @@ from synapse.api.urls import CLIENT_PREFIX
 from synapse.rest.transactions import HttpTransactionStore
 import re
 
+import logging
+
+
+logger = logging.getLogger(__name__)
+
 
 def client_path_pattern(path_regex):
     """Creates a regex compiled client path with the correct client path
@@ -62,6 +67,8 @@ class RestServlet(object):
         self.auth = hs.get_auth()
         self.txns = HttpTransactionStore()
 
+        self.validator = hs.get_event_validator()
+
     def register(self, http_server):
         """ Register this servlet with the given HTTP server. """
         if hasattr(self, "PATTERN"):

synapse/rest/directory.py

@@ -36,7 +36,9 @@ class ClientDirectoryServer(RestServlet):
 
     @defer.inlineCallbacks
     def on_GET(self, request, room_alias):
-        room_alias = self.hs.parse_roomalias(urllib.unquote(room_alias))
+        room_alias = self.hs.parse_roomalias(
+            urllib.unquote(room_alias).decode("utf-8")
+        )
 
         dir_handler = self.handlers.directory_handler
         res = yield dir_handler.get_association(room_alias)
@@ -54,7 +56,9 @@ class ClientDirectoryServer(RestServlet):
 
         logger.debug("Got content: %s", content)
 
-        room_alias = self.hs.parse_roomalias(urllib.unquote(room_alias))
+        room_alias = self.hs.parse_roomalias(
+            urllib.unquote(room_alias).decode("utf-8")
+        )
 
         logger.debug("Got room name: %s", room_alias.to_string())
 
@@ -70,9 +74,11 @@ class ClientDirectoryServer(RestServlet):
         dir_handler = self.handlers.directory_handler
 
         try:
+            user_id = user.to_string()
             yield dir_handler.create_association(
-                user.to_string(), room_alias, room_id, servers
+                user_id, room_alias, room_id, servers
             )
+            yield dir_handler.send_room_alias_update_event(user_id, room_id)
         except SynapseError as e:
             raise e
         except:
@@ -91,7 +97,9 @@ class ClientDirectoryServer(RestServlet):
 
         dir_handler = self.handlers.directory_handler
 
-        room_alias = self.hs.parse_roomalias(urllib.unquote(room_alias))
+        room_alias = self.hs.parse_roomalias(
+            urllib.unquote(room_alias).decode("utf-8")
+        )
 
         yield dir_handler.delete_association(
             user.to_string(), room_alias

synapse/rest/events.py

@@ -20,6 +20,11 @@ from synapse.api.errors import SynapseError
 from synapse.streams.config import PaginationConfig
 from synapse.rest.base import RestServlet, client_path_pattern
 
+import logging
+
+
+logger = logging.getLogger(__name__)
+
 
 class EventStreamRestServlet(RestServlet):
     PATTERN = client_path_pattern("/events$")
@@ -29,18 +34,22 @@ class EventStreamRestServlet(RestServlet):
     @defer.inlineCallbacks
     def on_GET(self, request):
         auth_user = yield self.auth.get_user_by_req(request)
+        try:
+            handler = self.handlers.event_stream_handler
+            pagin_config = PaginationConfig.from_request(request)
+            timeout = EventStreamRestServlet.DEFAULT_LONGPOLL_TIME_MS
+            if "timeout" in request.args:
+                try:
+                    timeout = int(request.args["timeout"][0])
+                except ValueError:
+                    raise SynapseError(400, "timeout must be in milliseconds.")
 
-        handler = self.handlers.event_stream_handler
-        pagin_config = PaginationConfig.from_request(request)
-        timeout = EventStreamRestServlet.DEFAULT_LONGPOLL_TIME_MS
-        if "timeout" in request.args:
-            try:
-                timeout = int(request.args["timeout"][0])
-            except ValueError:
-                raise SynapseError(400, "timeout must be in milliseconds.")
-
-        chunk = yield handler.get_stream(auth_user.to_string(), pagin_config,
-                                         timeout=timeout)
+            chunk = yield handler.get_stream(
+                auth_user.to_string(), pagin_config, timeout=timeout
+            )
+        except:
+            logger.exception("Event stream failed")
+            raise
 
         defer.returnValue((200, chunk))