Version bump for bug fix.

Conflicts:
	webclient/room/room-controller.js

.gitignore

@@ -10,10 +10,17 @@ docs/build/
 *.egg-info
 
 cmdclient_config.json
-homeserver.db
+homeserver*.db
 
 .coverage
 htmlcov
 
 demo/*.db
 demo/*.log
+demo/*.pid
+
+graph/*.svg
+graph/*.png
+graph/*.dot
+
+uploads

CHANGES.rst

@@ -0,0 +1,57 @@
+Changes in synapse 0.1.1 (2014-08-29)
+=====================================
+
+Homeserver:
+    * Fix bug that caused the event stream to not notify some clients about
+      changes.
+
+Changes in synapse 0.1.0 (2014-08-29)
+=====================================
+Presence has been reenabled in this release.
+
+Homeserver:
+ * Update client to server API, including:
+    - Use a more consistent url scheme.
+    - Provide more useful information in the initial sync api.
+ * Change the presence handling to be much more efficient.
+ * Change the presence server to server API to not require explicit polling of
+   all users who share a room with a user.
+ * Fix races in the event streaming logic.
+
+Webclient:
+ * Update to use new client to server API.
+ * Add basic VOIP support.
+ * Add idle timers that change your status to away.
+ * Add recent rooms column when viewing a room.
+ * Various network efficiency improvements.
+ * Add basic mobile browser support.
+ * Add a settings page.
+
+Changes in synapse 0.0.1 (2014-08-22)
+=====================================
+Presence has been disabled in this release due to a bug that caused the
+homeserver to spam other remote homeservers.
+
+Homeserver:
+ * Completely change the database schema to support generic event types.
+ * Improve presence reliability.
+ * Improve reliability of joining remote rooms.
+ * Fix bug where room join events were duplicated.
+ * Improve initial sync API to return more information to the client.
+ * Stop generating fake messages for room membership events.
+
+Webclient:
+ * Add tab completion of names.
+ * Add ability to upload and send images.
+ * Add profile pages.
+ * Improve CSS layout of room.
+ * Disambiguate identical display names.
+ * Don't get remote users display names and avatars individually.
+ * Use the new initial sync API to reduce number of round trips to the homeserver.
+ * Change url scheme to use room aliases instead of room ids where known.
+ * Increase longpoll timeout.
+
+Changes in synapse 0.0.0 (2014-08-13)
+=====================================
+
+ * Initial alpha release

README.rst

@@ -24,11 +24,8 @@ To get up and running:
     
     - To run your own **private** homeserver on localhost:8080, install synapse 
       with ``python setup.py develop --user`` and then run one with
-      ``python synapse/app/homeserver.py``
-      
-    - To run your own webclient:
-      ``cd webclient; python -m SimpleHTTPServer`` and hit http://localhost:8000
-      in your web browser (a recent Chrome, Safari or Firefox for now,
+      ``python synapse/app/homeserver.py`` - you will find a webclient running
+      at http://localhost:8080 (use a recent Chrome, Safari or Firefox for now,
       please...)
              
     - To make the homeserver **public** and let it exchange messages with 
@@ -36,7 +33,12 @@ To get up and running:
       up port 8080 and run ``python synapse/app/homeserver.py --host 
       machine.my.domain.name``.  Then come join ``#matrix:matrix.org`` and
       say hi! :)
-    
+
+For more detailed setup instructions, please see further down this document.
+
+[1] VoIP currently in development
+
+   
 About Matrix
 ============
 
@@ -87,8 +89,6 @@ https://github.com/matrix-org/synapse/issues or at matrix@matrix.org.
 
 Thanks for trying Matrix!
 
-[1] VoIP currently in development
-
 [2] Cryptographic signing of messages isn't turned on yet
 
 [3] End-to-end encryption is currently in development
@@ -120,6 +120,10 @@ may need to also run:
     $ sudo apt-get install python-pip
     $ sudo pip install --upgrade setuptools
     
+If you don't have access to github, then you may need to install ``syutil``
+manually by checking it out and running ``python setup.py develop --user`` on it
+too.
+    
 If you get errors about ``sodium.h`` being missing, you may also need to
 manually install a newer PyNaCl via pip as setuptools installs an old one. Or
 you can check PyNaCl out of git directly (https://github.com/pyca/pynacl) and
@@ -142,6 +146,13 @@ This should end with a 'PASSED' result::
     PASSED (successes=143)
 
 
+Upgrading an existing homeserver
+================================
+
+Before upgrading an existing homeserver to a new version, please refer to
+UPGRADE.rst for any additional instructions.
+ 
+
 Setting up Federation
 =====================
 
@@ -189,22 +200,15 @@ Running a Demo Federation of Homeservers
 
 If you want to get up and running quickly with a trio of homeservers in a
 private federation (``localhost:8080``, ``localhost:8081`` and
-``localhost:8082``) which you can then access through the webclient running at http://localhost:8080.  Simply run::
+``localhost:8082``) which you can then access through the webclient running at
+http://localhost:8080. Simply run::
 
     $ demo/start.sh
 
 Running The Demo Web Client
 ===========================
 
-At the present time, the web client is not directly served by the homeserver's
-HTTP server. To serve this in a form the web browser can reach, arrange for the
-'webclient' sub-directory to be made available by any sort of HTTP server that
-can serve static files. For example, python's SimpleHTTPServer will suffice::
-
-    $ cd webclient
-    $ python -m SimpleHTTPServer
-
-You can now point your browser at  http://localhost:8000/  to find the client.
+The homeserver runs a web client by default at http://localhost:8080.
 
 If this is the first time you have used the client from that browser (it uses
 HTML5 local storage to remember its config), you will need to log in to your

UPGRADE.rst

@@ -0,0 +1,24 @@
+Upgrading to v0.0.1
+===================
+
+This release completely changes the database schema and so requires upgrading
+it before starting the new version of the homeserver.
+
+The script "database-prepare-for-0.0.1.sh" should be used to upgrade the
+database. This will save all user information, such as logins and profiles, 
+but will otherwise purge the database. This includes messages, which
+rooms the home server was a member of and room alias mappings.
+
+Before running the command the homeserver should be first completely 
+shutdown. To run it, simply specify the location of the database, e.g.:
+
+  ./database-prepare-for-0.0.1.sh "homeserver.db"
+
+Once this has successfully completed it will be safe to restart the 
+homeserver. You may notice that the homeserver takes a few seconds longer to 
+restart than usual as it reinitializes the database.
+
+On startup of the new version, users can either rejoin remote rooms using room
+aliases or by being reinvited. Alternatively, if any other homeserver sends a
+message to a room that the homeserver was previously in the local HS will 
+automatically rejoin the room.

VERSION

@@ -0,0 +1 @@
+0.1.1

cmdclient/console.py

@@ -61,7 +61,7 @@ class SynapseCmd(cmd.Cmd):
             "send_delivery_receipts": "on"
         }
         self.path_prefix = "/matrix/client/api/v1"
-        self.event_stream_token = "START"
+        self.event_stream_token = "END"
         self.prompt = ">>> "
 
     def do_EOF(self, line):  # allows CTRL+D quitting
@@ -225,64 +225,81 @@ class SynapseCmd(cmd.Cmd):
         json_res = yield self.http_client.do_request("GET", url)
         print json_res
 
-        if ("type" not in json_res or "m.login.password" != json_res["type"] or
-                "stages" in json_res):
+        if "flows" not in json_res:
+            print "Failed to find any login flows."
+            defer.returnValue(False)
+
+        flow = json_res["flows"][0] # assume first is the one we want.
+        if ("type" not in flow or "m.login.password" != flow["type"] or
+                "stages" in flow):
             fallback_url = self._url() + "/login/fallback"
             print ("Unable to login via the command line client. Please visit "
                 "%s to login." % fallback_url)
             defer.returnValue(False)
         defer.returnValue(True)
 
-    def do_3pidrequest(self, line):
+    def do_emailrequest(self, line):
         """Requests the association of a third party identifier
-        <medium> The medium of the identifer (currently only 'email')
-        <address> The address of the identifer (ie. the email address)
+        <address> The email address)
+        <clientSecret> A string of characters generated when requesting an email that you'll supply in subsequent calls to identify yourself
+        <sendAttempt> The number of times the user has requested an email. Leave this the same between requests to retry the request at the transport level. Increment it to request that the email be sent again.
         """
-        args = self._parse(line, ['medium', 'address'])
+        args = self._parse(line, ['address', 'clientSecret', 'sendAttempt'])
 
-        if not args['medium'] == 'email':
-            print "Only email is supported currently"
-            return
+        postArgs = {'email': args['address'], 'clientSecret': args['clientSecret'], 'sendAttempt': args['sendAttempt']}
 
-        postArgs = {'email': args['address'], 'clientSecret': '____'}
-
-        reactor.callFromThread(self._do_3pidrequest, postArgs)
+        reactor.callFromThread(self._do_emailrequest, postArgs)
 
     @defer.inlineCallbacks
-    def _do_3pidrequest(self, args):
+    def _do_emailrequest(self, args):
         url = self._identityServerUrl()+"/matrix/identity/api/v1/validate/email/requestToken"
 
         json_res = yield self.http_client.do_request("POST", url, data=urllib.urlencode(args), jsonreq=False,
                                                      headers={'Content-Type': ['application/x-www-form-urlencoded']})
         print json_res
-        if 'tokenId' in json_res:
-            print "Token ID %s sent" % (json_res['tokenId'])
+        if 'sid' in json_res:
+            print "Token sent. Your session ID is %s" % (json_res['sid'])
 
-    def do_3pidvalidate(self, line):
+    def do_emailvalidate(self, line):
         """Validate and associate a third party ID
-        <medium> The medium of the identifer (currently only 'email')
-        <tokenId> The identifier iof the token given in 3pidrequest
+        <sid> The session ID (sid) given to you in the response to requestToken
         <token> The token sent to your third party identifier address
+        <clientSecret> The same clientSecret you supplied in requestToken
         """
-        args = self._parse(line, ['medium', 'tokenId', 'token'])
+        args = self._parse(line, ['sid', 'token', 'clientSecret'])
 
-        if not args['medium'] == 'email':
-            print "Only email is supported currently"
-            return
+        postArgs = { 'sid' : args['sid'], 'token' : args['token'], 'clientSecret': args['clientSecret'] }
 
-        postArgs = { 'tokenId' : args['tokenId'], 'token' : args['token'] }
-        postArgs['mxId'] = self.config["user"]
-
-        reactor.callFromThread(self._do_3pidvalidate, postArgs)
+        reactor.callFromThread(self._do_emailvalidate, postArgs)
 
     @defer.inlineCallbacks
-    def _do_3pidvalidate(self, args):
+    def _do_emailvalidate(self, args):
         url = self._identityServerUrl()+"/matrix/identity/api/v1/validate/email/submitToken"
 
         json_res = yield self.http_client.do_request("POST", url, data=urllib.urlencode(args), jsonreq=False,
                                                      headers={'Content-Type': ['application/x-www-form-urlencoded']})
         print json_res
 
+    def do_3pidbind(self, line):
+        """Validate and associate a third party ID
+        <sid> The session ID (sid) given to you in the response to requestToken
+        <clientSecret> The same clientSecret you supplied in requestToken
+        """
+        args = self._parse(line, ['sid', 'clientSecret'])
+
+        postArgs = { 'sid' : args['sid'], 'clientSecret': args['clientSecret'] }
+        postArgs['mxid'] = self.config["user"]
+
+        reactor.callFromThread(self._do_3pidbind, postArgs)
+
+    @defer.inlineCallbacks
+    def _do_3pidbind(self, args):
+        url = self._identityServerUrl()+"/matrix/identity/api/v1/3pid/bind"
+
+        json_res = yield self.http_client.do_request("POST", url, data=urllib.urlencode(args), jsonreq=False,
+                                                     headers={'Content-Type': ['application/x-www-form-urlencoded']})
+        print json_res
+
     def do_join(self, line):
         """Joins a room: "join <roomid>" """
         try:
@@ -390,19 +407,16 @@ class SynapseCmd(cmd.Cmd):
         """Leaves a room: "leave <roomid>" """
         try:
             args = self._parse(line, ["roomid"], force_keys=True)
-            path = ("/rooms/%s/members/%s/state" %
-                    (urllib.quote(args["roomid"]), self._usr()))
-            reactor.callFromThread(self._run_and_pprint, "DELETE", path)
+            self._do_membership_change(args["roomid"], "leave", self._usr())
         except Exception as e:
             print e
 
     def do_send(self, line):
         """Sends a message. "send <roomid> <body>" """
         args = self._parse(line, ["roomid", "body"])
-        msg_id = "m%s" % int(time.time())
-        path = "/rooms/%s/messages/%s/%s" % (urllib.quote(args["roomid"]),
-                                             self._usr(),
-                                             msg_id)
+        txn_id = "txn%s" % int(time.time())
+        path = "/rooms/%s/send/m.room.message/%s" % (urllib.quote(args["roomid"]),
+                                             txn_id)
         body_json = {
             "msgtype": "m.text",
             "body": args["body"]
@@ -426,7 +440,7 @@ class SynapseCmd(cmd.Cmd):
             print "Unrecognised type: %s" % args["type"]
             return
         room_id = args["roomid"]
-        path = "/rooms/%s/%s/list" % (urllib.quote(room_id), args["type"])
+        path = "/rooms/%s/%s" % (urllib.quote(room_id), args["type"])
 
         qp = {"access_token": self._tok()}
         if "qp" in args:
@@ -462,7 +476,7 @@ class SynapseCmd(cmd.Cmd):
             room_name = args["vis"]
             body["room_alias_name"] = room_name
 
-        reactor.callFromThread(self._run_and_pprint, "POST", "/rooms", body)
+        reactor.callFromThread(self._run_and_pprint, "POST", "/createRoom", body)
 
     def do_raw(self, line):
         """Directly send a JSON object: "raw <method> <path> <data> <notoken>"
@@ -555,7 +569,7 @@ class SynapseCmd(cmd.Cmd):
                                alt_text="Sent receipt for %s" % event["msg_id"])
 
     def _do_membership_change(self, roomid, membership, userid):
-        path = "/rooms/%s/members/%s/state" % (urllib.quote(roomid), userid)
+        path = "/rooms/%s/state/m.room.member/%s" % (urllib.quote(roomid), urllib.quote(userid))
         data = {
             "membership": membership
         }

database-prepare-for-0.0.1.sh

@@ -0,0 +1,21 @@
+#!/bin/bash
+
+# This is will prepare a synapse database for running with v0.0.1 of synapse. 
+# It will store all the user information, but will *delete* all messages and
+# room data.
+
+set -e
+
+cp "$1" "$1.bak"
+
+DUMP=$(sqlite3 "$1" << 'EOF'
+.dump users
+.dump access_tokens
+.dump presence
+.dump profiles
+EOF
+)
+
+rm "$1"
+
+sqlite3 "$1" <<< "$DUMP" 

database-save.sh

@@ -8,7 +8,7 @@
 #
 #   $ sqlite3 homeserver.db < table-save.sql
 
-sqlite3 homeserver.db <<'EOF' >table-save.sql
+sqlite3 "$1" <<'EOF' >table-save.sql
 .dump users
 .dump access_tokens
 .dump presence

demo/start.sh

@@ -6,7 +6,7 @@ CWD=$(pwd)
 
 cd "$DIR/.."
 
-for port in "8080" "8081" "8082"; do
+for port in 8080 8081 8082; do
     echo "Starting server on port $port... "
 
     python -m synapse.app.homeserver \
@@ -15,7 +15,8 @@ for port in "8080" "8081" "8082"; do
         -f "$DIR/$port.log" \
         -d "$DIR/$port.db" \
         -vv \
-        -D --pid-file "$DIR/$port.pid"
+        -D --pid-file "$DIR/$port.pid" \
+        --manhole $((port + 1000))
 done
 
 echo "Starting webclient on port 8000..."

demo/webserver.py

@@ -2,9 +2,32 @@ import argparse
 import BaseHTTPServer
 import os
 import SimpleHTTPServer
+import cgi, logging
 
 from daemonize import Daemonize
 
+class SimpleHTTPRequestHandlerWithPOST(SimpleHTTPServer.SimpleHTTPRequestHandler):
+    UPLOAD_PATH = "upload"
+
+    """
+    Accept all post request as file upload
+    """
+    def do_POST(self):
+
+        path = os.path.join(self.UPLOAD_PATH, os.path.basename(self.path))
+        length = self.headers['content-length']
+        data = self.rfile.read(int(length))
+
+        with open(path, 'wb') as fh:
+            fh.write(data)
+
+        self.send_response(200)
+        self.send_header('Content-Type', 'application/json')
+        self.end_headers()
+
+        # Return the absolute path of the uploaded file
+        self.wfile.write('{"url":"/%s"}' % path)
+
 
 def setup():
     parser = argparse.ArgumentParser()
@@ -19,7 +42,7 @@ def setup():
 
     httpd = BaseHTTPServer.HTTPServer(
         ('', args.port),
-        SimpleHTTPServer.SimpleHTTPRequestHandler
+        SimpleHTTPRequestHandlerWithPOST
     )
 
     def run():

docs/client-server/howto.rst

@@ -0,0 +1,303 @@
+TODO(kegan): Tweak joinalias API keys/path? Event stream historical > live needs
+a token (currently doesn't). im/sync responses include outdated event formats
+(room membership change messages). Room config (specifically: message history,
+public rooms). /register seems super simplistic compared to /login, maybe it
+would be better if /register used the same technique as /login? /register should
+be "user" not "user_id".
+
+
+How to use the client-server API
+================================
+
+This guide focuses on how the client-server APIs *provided by the reference 
+home server* can be used. Since this is specific to a home server 
+implementation, there may be variations in relation to registering/logging in
+which are not covered in extensive detail in this guide.
+
+If you haven't already, get a home server up and running on 
+``http://localhost:8080``.
+
+
+Accounts
+========
+Before you can send and receive messages, you must **register** for an account. 
+If you already have an account, you must **login** into it.
+
+**Try out the fiddle: http://jsfiddle.net/jrf1h02d/**
+
+Registration
+------------
+The aim of registration is to get a user ID and access token which you will need
+when accessing other APIs::
+
+    curl -XPOST -d '{"user_id":"example", "password":"wordpass"}' "http://localhost:8080/matrix/client/api/v1/register"
+
+    {
+        "access_token": "QGV4YW1wbGU6bG9jYWxob3N0.AqdSzFmFYrLrTmteXc", 
+        "home_server": "localhost", 
+        "user_id": "@example:localhost"
+    }
+
+NB: If a ``user_id`` is not specified, one will be randomly generated for you. 
+If you do not specify a ``password``, you will be unable to login to the account
+if you forget the ``access_token``.
+
+Implementation note: The matrix specification does not enforce how users 
+register with a server. It just specifies the URL path and absolute minimum 
+keys. The reference home server uses a username/password to authenticate user,
+but other home servers may use different methods.
+
+Login
+-----
+The aim when logging in is to get an access token for your existing user ID::
+
+    curl -XGET "http://localhost:8080/matrix/client/api/v1/login"
+
+    {
+        "type": "m.login.password"
+    }
+
+    curl -XPOST -d '{"type":"m.login.password", "user":"example", "password":"wordpass"}' "http://localhost:8080/matrix/client/api/v1/login"
+
+    {
+        "access_token": "QGV4YW1wbGU6bG9jYWxob3N0.vRDLTgxefmKWQEtgGd", 
+        "home_server": "localhost", 
+        "user_id": "@example:localhost"
+    }
+    
+Implementation note: Different home servers may implement different methods for 
+logging in to an existing account. In order to check that you know how to login 
+to this home server, you must perform a ``GET`` first and make sure you 
+recognise the login type. If you do not know how to login, you can 
+``GET /login/fallback`` which will return a basic webpage which you can use to 
+login. The reference home server implementation support username/password login,
+but other home servers may support different login methods (e.g. OAuth2).
+
+
+Communicating
+=============
+
+In order to communicate with another user, you must **create a room** with that 
+user and **send a message** to that room. 
+
+**Try out the fiddle: http://jsfiddle.net/jnwqcshc/**
+
+Creating a room
+---------------
+If you want to send a message to someone, you have to be in a room with them. To
+create a room::
+
+    curl -XPOST -d '{"room_alias_name":"tutorial"}' "http://localhost:8080/matrix/client/api/v1/rooms?access_token=QGV4YW1wbGU6bG9jYWxob3N0.vRDLTgxefmKWQEtgGd"
+
+    {
+        "room_alias": "#tutorial:localhost", 
+        "room_id": "!CvcvRuDYDzTOzfKKgh:localhost"
+    }
+    
+The "room alias" is a human-readable string which can be shared with other users
+so they can join a room, rather than the room ID which is a randomly generated
+string. You can have multiple room aliases per room.
+
+TODO(kegan): How to add/remove aliases from an existing room.
+    
+
+Sending messages
+----------------
+You can now send messages to this room::
+
+    curl -XPUT -d '{"msgtype":"m.text", "body":"hello"}' "http://localhost:8080/matrix/client/api/v1/rooms/%21CvcvRuDYDzTOzfKKgh:localhost/messages/%40example%3Alocalhost/msgid1?access_token=QGV4YW1wbGU6bG9jYWxob3N0.vRDLTgxefmKWQEtgGd"
+    
+NB: There are no limitations to the types of messages which can be exchanged.
+The only requirement is that ``"msgtype"`` is specified.
+
+NB: Depending on the room config, users who join the room may be able to see
+message history from before they joined.
+
+Users and rooms
+===============
+
+Each room can be configured to allow or disallow certain rules. In particular,
+these rules may specify if you require an **invitation** from someone already in
+the room in order to **join the room**. In addition, you may also be able to 
+join a room **via a room alias** if one was set up.
+
+**Try out the fiddle: http://jsfiddle.net/og1xokcr/**
+
+Inviting a user to a room
+-------------------------
+You can directly invite a user to a room like so::
+
+    curl -XPUT -d '{"membership":"invite"}' "http://localhost:8080/matrix/client/api/v1/rooms/%21CvcvRuDYDzTOzfKKgh:localhost/members/%40myfriend%3Alocalhost/state?access_token=QGV4YW1wbGU6bG9jYWxob3N0.vRDLTgxefmKWQEtgGd"
+    
+This informs ``@myfriend:localhost`` of the room ID 
+``!CvcvRuDYDzTOzfKKgh:localhost`` and allows them to join the room.
+
+Joining a room via an invite
+----------------------------
+If you receive an invite, you can join the room by changing the membership to
+join::
+
+    curl -XPUT -d '{"membership":"join"}' "http://localhost:8080/matrix/client/api/v1/rooms/%21CvcvRuDYDzTOzfKKgh:localhost/members/%40myfriend%3Alocalhost/state?access_token=QG15ZnJpZW5kOmxvY2FsaG9zdA...XKuGdVsovHmwMyDDvK"
+    
+NB: Only the person invited (``@myfriend:localhost``) can change the membership
+state to ``"join"``.
+
+Joining a room via an alias
+---------------------------
+Alternatively, if you know the room alias for this room and the room config 
+allows it, you can directly join a room via the alias::
+
+    curl -XPUT -d '{}' "http://localhost:8080/matrix/client/api/v1/join/%23tutorial%3Alocalhost?access_token=QG15ZnJpZW5kOmxvY2FsaG9zdA...XKuGdVsovHmwMyDDvK"
+    
+    {
+        "room_id": "!CvcvRuDYDzTOzfKKgh:localhost"
+    }
+    
+You will need to use the room ID when sending messages, not the room alias.
+
+NB: If the room is configured to be an invite-only room, you will still require
+an invite in order to join the room even though you know the room alias. As a
+result, it is more common to see a room alias in relation to a public room, 
+which do not require invitations.
+
+Getting events
+==============
+An event is some interesting piece of data that a client may be interested in. 
+It can be a message in a room, a room invite, etc. There are many different ways
+of getting events, depending on what the client already knows.
+
+**Try out the fiddle: http://jsfiddle.net/5uk4dqe2/**
+
+Getting all state
+-----------------
+If the client doesn't know any information on the rooms the user is 
+invited/joined on, they can get all the user's state for all rooms::
+
+    curl -XGET "http://localhost:8080/matrix/client/api/v1/im/sync?access_token=QG15ZnJpZW5kOmxvY2FsaG9zdA...XKuGdVsovHmwMyDDvK"
+    
+    [
+        {
+            "membership": "join", 
+            "messages": {
+                "chunk": [
+                    {
+                        "content": {
+                            "body": "@example:localhost joined the room.", 
+                            "hsob_ts": 1408444664249, 
+                            "membership": "join", 
+                            "membership_source": "@example:localhost", 
+                            "membership_target": "@example:localhost", 
+                            "msgtype": "m.text"
+                        }, 
+                        "event_id": "lZjmmlrEvo", 
+                        "msg_id": "m1408444664249", 
+                        "room_id": "!CvcvRuDYDzTOzfKKgh:localhost", 
+                        "type": "m.room.message", 
+                        "user_id": "_homeserver_"
+                    }, 
+                    {
+                        "content": {
+                            "body": "hello", 
+                            "hsob_ts": 1408445405672, 
+                            "msgtype": "m.text"
+                        }, 
+                        "event_id": "BiBJqamISg", 
+                        "msg_id": "msgid1", 
+                        "room_id": "!CvcvRuDYDzTOzfKKgh:localhost", 
+                        "type": "m.room.message", 
+                        "user_id": "@example:localhost"
+                    }, 
+                    [...]
+                    {
+                        "content": {
+                            "body": "@myfriend:localhost joined the room.", 
+                            "hsob_ts": 1408446501661, 
+                            "membership": "join", 
+                            "membership_source": "@myfriend:localhost", 
+                            "membership_target": "@myfriend:localhost", 
+                            "msgtype": "m.text"
+                        }, 
+                        "event_id": "IMmXbOzFAa", 
+                        "msg_id": "m1408446501661", 
+                        "room_id": "!CvcvRuDYDzTOzfKKgh:localhost", 
+                        "type": "m.room.message", 
+                        "user_id": "_homeserver_"
+                    }
+                ], 
+                "end": "20", 
+                "start": "0"
+            }, 
+            "room_id": "!CvcvRuDYDzTOzfKKgh:localhost"
+        }
+    ]
+    
+This returns all the room IDs of rooms the user is invited/joined on, as well as
+all of the messages and feedback for these rooms. This can be a LOT of data. You
+may just want the most recent message for each room. This can be achieved by 
+applying pagination stream parameters to this request::
+
+    curl -XGET "http://localhost:8080/matrix/client/api/v1/im/sync?access_token=QG15ZnJpZW5kOmxvY2FsaG9zdA...XKuGdVsovHmwMyDDvK&from=END&to=START&limit=1"
+    
+    [
+        {
+            "membership": "join", 
+            "messages": {
+                "chunk": [
+                    {
+                        "content": {
+                            "body": "@myfriend:localhost joined the room.", 
+                            "hsob_ts": 1408446501661, 
+                            "membership": "join", 
+                            "membership_source": "@myfriend:localhost", 
+                            "membership_target": "@myfriend:localhost", 
+                            "msgtype": "m.text"
+                        }, 
+                        "event_id": "IMmXbOzFAa", 
+                        "msg_id": "m1408446501661", 
+                        "room_id": "!CvcvRuDYDzTOzfKKgh:localhost", 
+                        "type": "m.room.message", 
+                        "user_id": "_homeserver_"
+                    }
+                ], 
+                "end": "20", 
+                "start": "21"
+            }, 
+            "room_id": "!CvcvRuDYDzTOzfKKgh:localhost"
+        }
+    ]
+
+Getting live state
+------------------
+Once you know which rooms the client has previously interacted with, you need to
+listen for incoming events. This can be done like so::
+
+    curl -XGET "http://localhost:8080/matrix/client/api/v1/events?access_token=QG15ZnJpZW5kOmxvY2FsaG9zdA...XKuGdVsovHmwMyDDvK&from=END"
+    
+    {
+        "chunk": [], 
+        "end": "215", 
+        "start": "215"
+    }
+    
+This will block waiting for an incoming event, timing out after several seconds.
+Even if there are no new events (as in the example above), there will be some
+pagination stream response keys. The client should make subsequent requests 
+using the value of the ``"end"`` key (in this case ``215``) as the ``from`` 
+query parameter. This value should be stored so when the client reopens your app
+after a period of inactivity, you can resume from where you got up to in the 
+event stream. If it has been a long period of inactivity, there may be LOTS of 
+events waiting for the user. In this case, you may wish to get all state instead
+and then resume getting live state from a newer end token.
+
+NB: The timeout can be changed by adding a ``timeout`` query parameter, which is
+in milliseconds. A timeout of 0 will not block.
+
+
+Example application
+-------------------
+The following example demonstrates registration and login, live event streaming,
+creating and joining rooms, sending messages, getting member lists and getting 
+historical messages for a room. This covers most functionality of a messaging
+application.
+
+**Try out the fiddle: http://jsfiddle.net/L8r3o1wr/**

docs/client-server/specification.rst

@@ -1,6 +1,6 @@
-=========================
-Synapse Client-Server API
-=========================
+========================
+Matrix Client-Server API
+========================
 
 The following specification outlines how a client can send and receive data from 
 a home server.
@@ -262,7 +262,10 @@ the error, but the keys 'error' and 'errcode' will always be present.
 Some standard error codes are below:
 
 M_FORBIDDEN:
-Forbidden access, e.g. bad access token, failed login.
+Forbidden access, e.g. joining a room without permission, failed login.
+
+M_UNKNOWN_TOKEN:
+The access token specified was not recognised.
 
 M_BAD_JSON:
 Request contained valid JSON, but it was malformed in some way, e.g. missing
@@ -411,6 +414,9 @@ The server checks this, finds it is valid, and returns:
 {
   "access_token": "abcdef0123456789"
 }
+The server may optionally return "user_id" to confirm or change the user's ID.
+This is particularly useful if the home server wishes to support localpart entry
+of usernames (e.g. "bob" rather than "@bob:matrix.org").
 
 OAuth2-based
 ------------
@@ -688,6 +694,16 @@ Invite/Joining/Leaving a room
   Required keys:
     membership : [join|invite] - The membership state of $user_id in room 
                                  $room_id.
+  Optional keys:
+    displayname,
+    avatar_url : String fields from the member user's profile
+    state,
+    status_msg,
+    mtime_age : Presence information
+
+  These optional keys provide extra information that the client is likely to
+  be interested in so it doesn't have to perform an additional profile or
+  presence information fetch.
 
 Where:
   join - Indicate you ($user_id) are joining the room $room_id.

docs/client-server/swagger_matrix/api-docs

@@ -0,0 +1,46 @@
+{
+  "apiVersion": "1.0.0",
+  "swaggerVersion": "1.2",
+  "apis": [
+    {
+      "path": "/login",
+      "description": "Login operations"
+    },
+    {
+      "path": "/registration",
+      "description": "Registration operations"
+    },
+    {
+      "path": "/rooms",
+      "description": "Room operations"
+    },
+    {
+      "path": "/profile",
+      "description": "Profile operations"
+    },
+    {
+      "path": "/presence",
+      "description": "Presence operations"
+    },
+    {
+      "path": "/events",
+      "description": "Event operations"
+    },
+    {
+      "path": "/directory",
+      "description": "Directory operations"
+    }
+  ],
+  "authorizations": {
+    "token": {
+      "scopes": []
+    }
+  },
+  "info": {
+    "title": "Matrix Client-Server API Reference",
+    "description": "This contains the client-server API for the reference implementation of the home server",
+    "termsOfServiceUrl": "http://matrix.org",
+    "license": "Apache 2.0",
+    "licenseUrl": "http://www.apache.org/licenses/LICENSE-2.0.html"
+  }
+}

docs/client-server/swagger_matrix/directory

@@ -0,0 +1,83 @@
+{
+  "apiVersion": "1.0.0",
+  "swaggerVersion": "1.2",
+  "basePath": "http://localhost:8080/matrix/client/api/v1",
+  "resourcePath": "/directory",
+  "produces": [
+    "application/json"
+  ],
+  "apis": [
+    {
+      "path": "/directory/room/{roomAlias}",
+      "operations": [
+        {
+          "method": "GET",
+          "summary": "Get the room ID corresponding to this room alias.",
+          "type": "DirectoryResponse",
+          "nickname": "get_room_id_for_alias",
+          "parameters": [
+            {
+              "name": "roomAlias",
+              "description": "The room alias.",
+              "required": true,
+              "type": "string",
+              "paramType": "path"
+            }
+          ]
+        },
+        {
+          "method": "PUT",
+          "summary": "Create a new mapping from room alias to room ID.",
+          "type": "void",
+          "nickname": "add_room_alias",
+          "parameters": [
+            {
+              "name": "roomAlias",
+              "description": "The room alias to set.",
+              "required": true,
+              "type": "string",
+              "paramType": "path"
+            },
+            {
+              "name": "body",
+              "description": "The room ID to set.",
+              "required": true,
+              "type": "RoomAliasRequest",
+              "paramType": "body"
+            }
+          ]
+        }
+      ]
+    }
+  ],
+  "models": {
+    "DirectoryResponse": {
+      "id": "DirectoryResponse",
+      "properties": {
+        "room_id": {
+          "type": "string",
+          "description": "The fully-qualified room ID.",
+          "required": true
+        },
+        "servers": {
+          "type": "array",
+          "items": {
+            "$ref": "string"
+          },
+          "description": "A list of servers that know about this room.",
+          "required": true
+        }
+      }
+    },
+    "RoomAliasRequest": {
+      "id": "RoomAliasRequest",
+      "properties": {
+        "room_id": {
+          "type": "string",
+          "description": "The room ID to map the alias to.",
+          "required": true
+        }
+      }
+    }
+  }
+}

docs/client-server/swagger_matrix/events

@@ -0,0 +1,246 @@
+{
+  "apiVersion": "1.0.0",
+  "swaggerVersion": "1.2",
+  "basePath": "http://localhost:8080/matrix/client/api/v1",
+  "resourcePath": "/events",
+  "produces": [
+    "application/json"
+  ],
+  "apis": [
+    {
+      "path": "/events",
+      "operations": [
+        {
+          "method": "GET",
+          "summary": "Listen on the event stream",
+          "notes": "This can only be done by the logged in user. This will block until an event is received, or until the timeout is reached.",
+          "type": "PaginationChunk",
+          "nickname": "get_event_stream"
+        }
+      ],
+      "parameters": [
+        {
+          "name": "from",
+          "description": "The token to stream from.",
+          "required": false,
+          "type": "string",
+          "paramType": "query"
+        },
+        {
+          "name": "timeout",
+          "description": "The maximum time in milliseconds to wait for an event.",
+          "required": false,
+          "type": "integer",
+          "paramType": "query"
+        }
+      ],
+      "responseMessages": [
+        {
+          "code": 400,
+          "message": "Bad pagination token."
+        }
+      ]
+    },
+    {
+      "path": "/events/{eventId}",
+      "operations": [
+        {
+          "method": "GET",
+          "summary": "Get information about a single event.",
+          "notes": "Get information about a single event.",
+          "type": "Event",
+          "nickname": "get_event",
+          "parameters": [
+            {
+              "name": "eventId",
+              "description": "The event ID to get.",
+              "required": true,
+              "type": "string",
+              "paramType": "path"
+            }
+          ],
+          "responseMessages": [
+            {
+              "code": 404,
+              "message": "Event not found."
+            }
+          ]
+        }
+      ]
+    },
+    {
+      "path": "/initialSync",
+      "operations": [
+        {
+          "method": "GET",
+          "summary": "Get this user's current state.",
+          "notes": "Get this user's current state.",
+          "type": "InitialSyncResponse",
+          "nickname": "initial_sync",
+          "parameters": [
+            {
+              "name": "limit",
+              "description": "The maximum number of messages to return for each room.",
+              "type": "integer",
+              "paramType": "query",
+              "required": false
+            }
+          ]
+        }
+      ]
+    },
+    {
+      "path": "/publicRooms",
+      "operations": [
+        {
+          "method": "GET",
+          "summary": "Get a list of publicly visible rooms.",
+          "type": "PublicRoomsPaginationChunk",
+          "nickname": "get_public_room_list"
+        }
+      ]
+    }
+  ],
+  "models": {
+    "PaginationChunk": {
+      "id": "PaginationChunk",
+      "properties": {
+        "start": {
+          "type": "string",
+          "description": "A token which correlates to the first value in \"chunk\" for paginating.",
+          "required": true
+        },
+        "end": {
+          "type": "string",
+          "description": "A token which correlates to the last value in \"chunk\" for paginating.",
+          "required": true
+        },
+        "chunk": {
+          "type": "array",
+          "description": "An array of events.",
+          "required": true,
+          "items": {
+            "$ref": "Event"
+          }
+        }
+      }
+    },
+    "Event": {
+      "id": "Event",
+      "properties": {
+        "event_id": {
+          "type": "string",
+          "description": "An ID which uniquely identifies this event.",
+          "required": true
+        },
+        "room_id": {
+          "type": "string",
+          "description": "The room in which this event occurred.",
+          "required": true
+        }
+      }
+    },
+    "PublicRoomInfo": {
+      "id": "PublicRoomInfo",
+      "properties": {
+        "aliases": {
+          "type": "array",
+          "description": "A list of room aliases for this room.",
+          "items": {
+            "$ref": "string"
+          }
+        },
+        "name": {
+          "type": "string",
+          "description": "The name of the room, as given by the m.room.name state event."
+        },
+        "room_id": {
+          "type": "string",
+          "description": "The room ID for this public room.",
+          "required": true
+        },
+        "topic": {
+          "type": "string",
+          "description": "The topic of this room, as given by the m.room.topic state event."
+        }
+      }
+    },
+    "PublicRoomsPaginationChunk": {
+      "id": "PublicRoomsPaginationChunk",
+      "properties": {
+        "start": {
+          "type": "string",
+          "description": "A token which correlates to the first value in \"chunk\" for paginating.",
+          "required": true
+        },
+        "end": {
+          "type": "string",
+          "description": "A token which correlates to the last value in \"chunk\" for paginating.",
+          "required": true
+        },
+        "chunk": {
+          "type": "array",
+          "description": "A list of public room data.",
+          "required": true,
+          "items": {
+            "$ref": "PublicRoomInfo"
+          }
+        }
+      }
+    },
+    "InitialSyncResponse": {
+      "id": "InitialSyncResponse",
+      "properties": {
+        "end": {
+          "type": "string",
+          "description": "A streaming token which can be used with /events to continue from this snapshot of data.",
+          "required": true
+        },
+        "presence": {
+          "type": "array",
+          "description": "A list of presence events.",
+          "items": {
+            "$ref": "Event"
+          },
+          "required": false
+        },
+        "rooms": {
+          "type": "array",
+          "description": "A list of initial sync room data.",
+          "required": false,
+          "items": {
+            "$ref": "InitialSyncRoomData"
+          }
+        }
+      }
+    },
+    "InitialSyncRoomData": {
+      "id": "InitialSyncRoomData",
+      "properties": {
+        "membership": {
+          "type": "string",
+          "description": "This user's membership state in this room.",
+          "required": true
+        },
+        "room_id": {
+          "type": "string",
+          "description": "The ID of this room.",
+          "required": true
+        },
+        "messages": {
+          "type": "PaginationChunk",
+          "description": "The most recent messages for this room, governed by the limit parameter.",
+          "required": false
+        },
+        "state": {
+          "type": "array",
+          "description": "A list of state events representing the current state of the room.",
+          "required": false,
+          "items": {
+            "$ref": "Event"
+          }
+        }
+      }
+    }
+  }
+}

docs/client-server/swagger_matrix/login

@@ -0,0 +1,102 @@
+{
+  "apiVersion": "1.0.0", 
+  "apis": [
+    {
+      "operations": [
+        {
+          "method": "GET", 
+          "nickname": "get_login_info", 
+          "notes": "All login stages MUST be mentioned if there is >1 login type.", 
+          "summary": "Get the login mechanism to use when logging in.", 
+          "type": "LoginInfo"
+        }, 
+        {
+          "method": "POST", 
+          "nickname": "submit_login", 
+          "notes": "If this is part of a multi-stage login, there MUST be a 'session' key.", 
+          "parameters": [
+            {
+              "description": "A login submission", 
+              "name": "body", 
+              "paramType": "body", 
+              "required": true, 
+              "type": "LoginSubmission"
+            }
+          ], 
+          "responseMessages": [
+            {
+              "code": 400, 
+              "message": "Bad login type"
+            }, 
+            {
+              "code": 400, 
+              "message": "Missing JSON keys"
+            }
+          ], 
+          "summary": "Submit a login action.", 
+          "type": "LoginResult"
+        }
+      ], 
+      "path": "/login"
+    }
+  ], 
+  "basePath": "http://localhost:8080/matrix/client/api/v1", 
+  "consumes": [
+    "application/json"
+  ], 
+  "models": {
+    "LoginInfo": {
+      "id": "LoginInfo", 
+      "properties": {
+        "stages": {
+          "description": "Multi-stage login only: An array of all the login types required to login.", 
+          "format": "string", 
+          "type": "array"
+        }, 
+        "type": {
+          "description": "The login type that must be used when logging in.", 
+          "type": "string"
+        }
+      }
+    }, 
+    "LoginResult": {
+      "id": "LoginResult", 
+      "properties": {
+        "access_token": {
+          "description": "The access token for this user's login if this is the final stage of the login process.", 
+          "type": "string"
+        }, 
+        "next": {
+          "description": "Multi-stage login only: The next login type to submit.", 
+          "type": "string"
+        },
+        "session": {
+          "description": "Multi-stage login only: The session token to send when submitting the next login type.",
+          "type": "string"
+        }
+      }
+    }, 
+    "LoginSubmission": {
+      "id": "LoginSubmission", 
+      "properties": {
+        "type": {
+          "description": "The type of login being submitted.", 
+          "type": "string"
+        },
+        "session": {
+          "description": "Multi-stage login only: The session token from an earlier login stage.",
+          "type": "string"
+        },
+        "_login_type_defined_keys_": {
+          "description": "Keys as defined by the specified login type, e.g. \"user\", \"password\""
+        }
+      }
+    }
+  }, 
+  "produces": [
+    "application/json"
+  ], 
+  "resourcePath": "/login", 
+  "swaggerVersion": "1.2"
+}
+

docs/client-server/swagger_matrix/presence

@@ -0,0 +1,164 @@
+{
+  "apiVersion": "1.0.0",
+  "swaggerVersion": "1.2",
+  "basePath": "http://localhost:8080/matrix/client/api/v1",
+  "resourcePath": "/presence",
+  "produces": [
+    "application/json"
+  ],
+  "consumes": [
+    "application/json"
+  ],
+  "apis": [
+    {
+      "path": "/presence/{userId}/status",
+      "operations": [
+        {
+          "method": "PUT",
+          "summary": "Update this user's presence state.",
+          "notes": "This can only be done by the logged in user.",
+          "type": "void",
+          "nickname": "update_presence",
+          "parameters": [
+            {
+              "name": "body",
+              "description": "The new presence state",
+              "required": true,
+              "type": "PresenceUpdate",
+              "paramType": "body"
+            },
+            {
+              "name": "userId",
+              "description": "The user whose presence to set.",
+              "required": true,
+              "type": "string",
+              "paramType": "path"
+            }
+          ]
+        },
+        {
+          "method": "GET",
+          "summary": "Get this user's presence state.",
+          "notes": "Get this user's presence state.",
+          "type": "PresenceUpdate",
+          "nickname": "get_presence",
+          "parameters": [
+            {
+              "name": "userId",
+              "description": "The user whose presence to get.",
+              "required": true,
+              "type": "string",
+              "paramType": "path"
+            }
+          ]
+        }
+      ]
+    },
+    {
+      "path": "/presence/list/{userId}",
+      "operations": [
+        {
+          "method": "GET",
+          "summary": "Retrieve a list of presences for all of this user's friends.",
+          "notes": "",
+          "type": "array",
+          "items": {
+            "$ref": "Presence"
+          },
+          "nickname": "get_presence_list",
+          "parameters": [
+            {
+              "name": "userId",
+              "description": "The user whose presence list to get.",
+              "required": true,
+              "type": "string",
+              "paramType": "path"
+            }
+          ]
+        },
+        {
+          "method": "POST",
+          "summary": "Add or remove users from this presence list.",
+          "notes": "Add or remove users from this presence list.",
+          "type": "void",
+          "nickname": "modify_presence_list",
+          "parameters": [
+            {
+              "name": "userId",
+              "description": "The user whose presence list is being modified.",
+              "required": true,
+              "type": "string",
+              "paramType": "path"
+            },
+            {
+              "name": "body",
+              "description": "The modifications to make to this presence list.",
+              "required": true,
+              "type": "PresenceListModifications",
+              "paramType": "body"
+            }
+          ]
+        }
+      ]
+    }
+  ],
+  "models": {
+    "PresenceUpdate": {
+      "id": "PresenceUpdate",
+      "properties": {
+        "state": {
+          "type": "string",
+          "description": "Enum: The presence state.",
+          "enum": [
+            "offline",
+            "unavailable",
+            "online",
+            "free_for_chat"
+          ]
+        },
+        "status_msg": {
+          "type": "string",
+          "description": "The user-defined message associated with this presence state."
+        }
+      },
+      "subTypes": [
+        "Presence"
+      ]
+    },
+    "Presence": {
+      "id": "Presence",
+      "properties": {
+        "mtime_age": {
+          "type": "integer",
+          "format": "int64",
+          "description": "The last time this user's presence state changed, in milliseconds."
+        },
+        "user_id": {
+          "type": "string",
+          "description": "The fully qualified user ID"
+        }
+      }
+    },
+    "PresenceListModifications": {
+      "id": "PresenceListModifications",
+      "properties": {
+        "invite": {
+          "type": "array",
+          "description": "A list of user IDs to add to the list.",
+          "items": {
+            "type": "string",
+            "description": "A fully qualified user ID."
+          }
+        },
+        "drop": {
+          "type": "array",
+          "description": "A list of user IDs to remove from the list.",
+          "items": {
+            "type": "string",
+            "description": "A fully qualified user ID."
+          }
+        }
+      }
+    }
+  }
+}

docs/client-server/swagger_matrix/profile

@@ -0,0 +1,122 @@
+{
+  "apiVersion": "1.0.0",
+  "swaggerVersion": "1.2",
+  "basePath": "http://localhost:8080/matrix/client/api/v1",
+  "resourcePath": "/profile",
+  "produces": [
+    "application/json"
+  ],
+  "consumes": [
+    "application/json"
+  ],
+  "apis": [
+    {
+      "path": "/profile/{userId}/displayname",
+      "operations": [
+        {
+          "method": "PUT",
+          "summary": "Set a display name.",
+          "notes": "This can only be done by the logged in user.",
+          "type": "void",
+          "nickname": "set_display_name",
+          "parameters": [
+            {
+              "name": "body",
+              "description": "The new display name for this user.",
+              "required": true,
+              "type": "DisplayName",
+              "paramType": "body"
+            },
+            {
+              "name": "userId",
+              "description": "The user whose display name to set.",
+              "required": true,
+              "type": "string",
+              "paramType": "path"
+            }
+          ]
+        },
+        {
+          "method": "GET",
+          "summary": "Get a display name.",
+          "notes": "This can be done by anyone.",
+          "type": "DisplayName",
+          "nickname": "get_display_name",
+          "parameters": [
+            {
+              "name": "userId",
+              "description": "The user whose display name to get.",
+              "required": true,
+              "type": "string",
+              "paramType": "path"
+            }
+          ]
+        }
+      ]
+    },
+    {
+      "path": "/profile/{userId}/avatar_url",
+      "operations": [
+        {
+          "method": "PUT",
+          "summary": "Set an avatar URL.",
+          "notes": "This can only be done by the logged in user.",
+          "type": "void",
+          "nickname": "set_avatar_url",
+          "parameters": [
+            {
+              "name": "body",
+              "description": "The new avatar url for this user.",
+              "required": true,
+              "type": "AvatarUrl",
+              "paramType": "body"
+            },
+            {
+              "name": "userId",
+              "description": "The user whose avatar url to set.",
+              "required": true,
+              "type": "string",
+              "paramType": "path"
+            }
+          ]
+        },
+        {
+          "method": "GET",
+          "summary": "Get an avatar url.",
+          "notes": "This can be done by anyone.",
+          "type": "AvatarUrl",
+          "nickname": "get_avatar_url",
+          "parameters": [
+            {
+              "name": "userId",
+              "description": "The user whose avatar url to get.",
+              "required": true,
+              "type": "string",
+              "paramType": "path"
+            }
+          ]
+        }
+      ]
+    }
+  ],
+  "models": {
+    "DisplayName": {
+      "id": "DisplayName",
+      "properties": {
+        "displayname": {
+          "type": "string",
+          "description": "The textual display name"
+        }
+      }
+    },
+    "AvatarUrl": {
+      "id": "AvatarUrl",
+      "properties": {
+        "avatar_url": {
+          "type": "string",
+          "description": "A url to an image representing an avatar."
+        }
+      }
+    }
+  }
+}

docs/client-server/swagger_matrix/registration

@@ -0,0 +1,75 @@
+{
+  "apiVersion": "1.0.0", 
+  "apis": [
+    {
+      "operations": [
+        {
+          "method": "POST", 
+          "nickname": "register", 
+          "notes": "Volatile: This API is likely to change.", 
+          "parameters": [
+            {
+              "description": "A registration request", 
+              "name": "body", 
+              "paramType": "body", 
+              "required": true, 
+              "type": "RegistrationRequest"
+            }
+          ], 
+          "responseMessages": [
+            {
+              "code": 400, 
+              "message": "No JSON object."
+            }, 
+            {
+              "code": 400, 
+              "message": "User ID must only contain characters which do not require url encoding."
+            },
+            {
+              "code": 400, 
+              "message": "User ID already taken."
+            }
+          ], 
+          "summary": "Register with the home server.", 
+          "type": "RegistrationResponse"
+        }
+      ], 
+      "path": "/register"
+    }
+  ], 
+  "basePath": "http://localhost:8080/matrix/client/api/v1", 
+  "consumes": [
+    "application/json"
+  ], 
+  "models": {
+    "RegistrationResponse": {
+      "id": "RegistrationResponse", 
+      "properties": {
+        "access_token": {
+          "description": "The access token for this user.", 
+          "type": "string"
+        }, 
+        "user_id": {
+          "description": "The fully-qualified user ID.", 
+          "type": "string"
+        }
+      }
+    }, 
+    "RegistrationRequest": {
+      "id": "RegistrationRequest", 
+      "properties": {
+        "user_id": {
+          "description": "The desired user ID. If not specified, a random user ID will be allocated.", 
+          "type": "string",
+          "required": false
+        }
+      }
+    }
+  }, 
+  "produces": [
+    "application/json"
+  ], 
+  "resourcePath": "/register", 
+  "swaggerVersion": "1.2"
+}
+

docs/client-server/swagger_matrix/rooms

@@ -0,0 +1,863 @@
+{
+  "apiVersion": "1.0.0",
+  "swaggerVersion": "1.2",
+  "basePath": "http://localhost:8080/matrix/client/api/v1", 
+  "resourcePath": "/rooms",
+  "produces": [
+    "application/json"
+  ],
+  "consumes": [
+    "application/json"
+  ],
+  "authorizations": {
+    "token": []
+  },
+  "apis": [
+    {
+      "path": "/rooms/{roomId}/send/{eventType}/{txnId}",
+      "operations": [
+        {
+          "method": "PUT",
+          "summary": "Send a generic non-state event to this room.",
+          "notes": "This operation can also be done as a POST to /rooms/{roomId}/send/{eventType}",
+          "type": "EventId",
+          "nickname": "send_non_state_event",
+          "consumes": [
+            "application/json"
+          ],
+          "parameters": [
+            {
+              "name": "body",
+              "description": "The event contents",
+              "required": true,
+              "type": "EventContent",
+              "paramType": "body"
+            },
+            {
+              "name": "roomId",
+              "description": "The room to send the message in.",
+              "required": true,
+              "type": "string",
+              "paramType": "path"
+            },
+            {
+              "name": "eventType",
+              "description": "The type of event to send.",
+              "required": true,
+              "type": "string",
+              "paramType": "path"
+            },
+            {
+              "name": "txnId",
+              "description": "A client transaction ID to ensure idempotency. This can only be omitted if the HTTP method becomes a POST.",
+              "required": true,
+              "type": "string",
+              "paramType": "path"
+            }
+          ]
+        }
+      ]
+    },
+    {
+      "path": "/rooms/{roomId}/state/{eventType}/{stateKey}",
+      "operations": [
+        {
+          "method": "PUT",
+          "summary": "Send a generic state event to this room.",
+          "notes": "The state key can be omitted, such that you can PUT to /rooms/{roomId}/state/{eventType}. The state key defaults to a 0 length string in this case.",
+          "type": "void",
+          "nickname": "send_state_event",
+          "consumes": [
+            "application/json"
+          ],
+          "parameters": [
+            {
+              "name": "body",
+              "description": "The event contents",
+              "required": true,
+              "type": "EventContent",
+              "paramType": "body"
+            },
+            {
+              "name": "roomId",
+              "description": "The room to send the message in.",
+              "required": true,
+              "type": "string",
+              "paramType": "path"
+            },
+            {
+              "name": "eventType",
+              "description": "The type of event to send.",
+              "required": true,
+              "type": "string",
+              "paramType": "path"
+            },
+            {
+              "name": "stateKey",
+              "description": "An identifier used to specify clobbering semantics. State events with the same (roomId, eventType, stateKey) will be replaced.",
+              "required": true,
+              "type": "string",
+              "paramType": "path"
+            }
+          ]
+        }
+      ]
+    },
+    {
+      "path": "/rooms/{roomId}/send/m.room.message/{txnId}",
+      "operations": [
+        {
+          "method": "PUT",
+          "summary": "Send a message in this room.",
+          "notes": "This operation can also be done as a POST to /rooms/{roomId}/send/m.room.message",
+          "type": "EventId",
+          "nickname": "send_message",
+          "consumes": [
+            "application/json"
+          ],
+          "parameters": [
+            {
+              "name": "body",
+              "description": "The message contents",
+              "required": true,
+              "type": "Message",
+              "paramType": "body"
+            },
+            {
+              "name": "roomId",
+              "description": "The room to send the message in.",
+              "required": true,
+              "type": "string",
+              "paramType": "path"
+            },
+            {
+              "name": "txnId",
+              "description": "A client transaction ID to ensure idempotency. This can only be omitted if the HTTP method becomes a POST.",
+              "required": true,
+              "type": "string",
+              "paramType": "path"
+            }
+          ]
+        }
+      ]
+    },
+    {
+      "path": "/rooms/{roomId}/state/m.room.topic",
+      "operations": [
+        {
+          "method": "PUT",
+          "summary": "Set the topic for this room.",
+          "notes": "Set the topic for this room.",
+          "type": "void",
+          "nickname": "set_topic",
+          "consumes": [
+            "application/json"
+          ],
+          "parameters": [
+            {
+              "name": "body",
+              "description": "The topic contents",
+              "required": true,
+              "type": "Topic",
+              "paramType": "body"
+            },
+            {
+              "name": "roomId",
+              "description": "The room to set the topic in.",
+              "required": true,
+              "type": "string",
+              "paramType": "path"
+            }
+          ]
+        },
+        {
+          "method": "GET",
+          "summary": "Get the topic for this room.",
+          "notes": "Get the topic for this room.",
+          "type": "Topic",
+          "nickname": "get_topic",
+          "parameters": [
+            {
+              "name": "roomId",
+              "description": "The room to get topic in.",
+              "required": true,
+              "type": "string",
+              "paramType": "path"
+            }
+          ],
+          "responseMessages": [
+            {
+              "code": 404,
+              "message": "Topic not found."
+            }
+          ]
+        }
+      ]
+    },
+    {
+      "path": "/rooms/{roomId}/send/m.room.message.feedback/{txnId}",
+      "operations": [
+        {
+          "method": "PUT",
+          "summary": "Send feedback to a message.",
+          "notes": "This operation can also be done as a POST to /rooms/{roomId}/send/m.room.message.feedback",
+          "type": "EventId",
+          "nickname": "send_feedback",
+          "consumes": [
+            "application/json"
+          ],
+          "parameters": [
+            {
+              "name": "body",
+              "description": "The feedback contents",
+              "required": true,
+              "type": "Feedback",
+              "paramType": "body"
+            },
+            {
+              "name": "roomId",
+              "description": "The room to send the feedback in.",
+              "required": true,
+              "type": "string",
+              "paramType": "path"
+            },
+            {
+              "name": "txnId",
+              "description": "A client transaction ID to ensure idempotency. This can only be omitted if the HTTP method becomes a POST.",
+              "required": true,
+              "type": "string",
+              "paramType": "path"
+            }
+          ],
+          "responseMessages": [
+            {
+              "code": 400,
+              "message": "Bad feedback type."
+            }
+          ]
+        }
+      ]
+    },
+    {
+      "path": "/rooms/{roomId}/invite/{txnId}",
+      "operations": [
+        {
+          "method": "PUT",
+          "summary": "Invite a user to this room.",
+          "notes": "This operation can also be done as a POST to /rooms/{roomId}/invite",
+          "type": "void",
+          "nickname": "invite",
+          "consumes": [
+            "application/json"
+          ],
+          "parameters": [
+            {
+              "name": "roomId",
+              "description": "The room which has this user.",
+              "required": true,
+              "type": "string",
+              "paramType": "path"
+            },
+            {
+              "name": "txnId",
+              "description": "A client transaction ID for this PUT to ensure idempotency. This can only be omitted if the HTTP method becomes a POST. ",
+              "required": false,
+              "type": "string",
+              "paramType": "path"
+            },
+            {
+              "name": "body",
+              "description": "The user to invite.",
+              "required": true,
+              "type": "InviteRequest",
+              "paramType": "body"
+            }
+          ]
+        }  
+      ]
+    },
+    {
+      "path": "/rooms/{roomId}/join/{txnId}",
+      "operations": [
+        {
+          "method": "PUT",
+          "summary": "Join this room.",
+          "notes": "This operation can also be done as a POST to /rooms/{roomId}/join",
+          "type": "void",
+          "nickname": "join_room",
+          "consumes": [
+            "application/json"
+          ],
+          "parameters": [
+            {
+              "name": "roomId",
+              "description": "The room to join.",
+              "required": true,
+              "type": "string",
+              "paramType": "path"
+            },
+            {
+              "name": "txnId",
+              "description": "A client transaction ID for this PUT to ensure idempotency. This can only be omitted if the HTTP method becomes a POST. ",
+              "required": false,
+              "type": "string",
+              "paramType": "path"
+            }
+          ]
+        }  
+      ]
+    },
+    {
+      "path": "/rooms/{roomId}/leave/{txnId}",
+      "operations": [
+        {
+          "method": "PUT",
+          "summary": "Leave this room.",
+          "notes": "This operation can also be done as a POST to /rooms/{roomId}/leave",
+          "type": "void",
+          "nickname": "leave",
+          "consumes": [
+            "application/json"
+          ],
+          "parameters": [
+            {
+              "name": "roomId",
+              "description": "The room to leave.",
+              "required": true,
+              "type": "string",
+              "paramType": "path"
+            },
+            {
+              "name": "txnId",
+              "description": "A client transaction ID for this PUT to ensure idempotency. This can only be omitted if the HTTP method becomes a POST. ",
+              "required": false,
+              "type": "string",
+              "paramType": "path"
+            }
+          ]
+        }  
+      ]
+    },
+    {
+      "path": "/rooms/{roomId}/state/m.room.member/{userId}",
+      "operations": [
+        {
+          "method": "PUT",
+          "summary": "Change the membership state for a user in a room.",
+          "notes": "Change the membership state for a user in a room.",
+          "type": "void",
+          "nickname": "set_membership",
+          "consumes": [
+            "application/json"
+          ],
+          "parameters": [
+            {
+              "name": "body",
+              "description": "The new membership state",
+              "required": true,
+              "type": "Member",
+              "paramType": "body"
+            },
+            {
+              "name": "userId",
+              "description": "The user whose membership is being changed.",
+              "required": true,
+              "type": "string",
+              "paramType": "path"
+            },
+            {
+              "name": "roomId",
+              "description": "The room which has this user.",
+              "required": true,
+              "type": "string",
+              "paramType": "path"
+            }
+          ],
+          "responseMessages": [
+            {
+              "code": 400,
+              "message": "No membership key."
+            },
+            {
+              "code": 400,
+              "message": "Bad membership value."
+            },
+            {
+              "code": 403,
+              "message": "When inviting: You are not in the room."
+            },
+            {
+              "code": 403,
+              "message": "When inviting: <target> is already in the room."
+            },
+            {
+              "code": 403,
+              "message": "When joining: Cannot force another user to join."
+            },
+            {
+              "code": 403,
+              "message": "When joining: You are not invited to this room."
+            }
+          ]
+        },
+        {
+          "method": "GET",
+          "summary": "Get the membership state of a user in a room.",
+          "notes": "Get the membership state of a user in a room.",
+          "type": "Member",
+          "nickname": "get_membership",
+          "parameters": [
+            {
+              "name": "userId",
+              "description": "The user whose membership state you want to get.",
+              "required": true,
+              "type": "string",
+              "paramType": "path"
+            },
+            {
+              "name": "roomId",
+              "description": "The room which has this user.",
+              "required": true,
+              "type": "string",
+              "paramType": "path"
+            }
+          ],
+          "responseMessages": [
+            {
+              "code": 404,
+              "message": "Member not found."
+            }
+          ]
+        }
+      ]
+    },
+    {
+      "path": "/join/{roomAliasOrId}",
+      "operations": [
+        {
+          "method": "PUT",
+          "summary": "Join a room via a room alias or room ID.",
+          "notes": "Join a room via a room alias or room ID.",
+          "type": "RoomInfo",
+          "nickname": "join",
+          "consumes": [
+            "application/json"
+          ],
+          "parameters": [
+            {
+              "name": "roomAliasOrId",
+              "description": "The room alias or room ID to join.",
+              "required": true,
+              "type": "string",
+              "paramType": "path"
+            }
+          ],
+          "responseMessages": [
+            {
+              "code": 400,
+              "message": "Bad room alias."
+            }
+          ]
+        }
+      ]
+    },
+    {
+      "path": "/createRoom",
+      "operations": [
+        {
+          "method": "POST",
+          "summary": "Create a room.",
+          "notes": "Create a room.",
+          "type": "RoomInfo",
+          "nickname": "create_room",
+          "consumes": [
+            "application/json"
+          ],
+          "parameters": [
+            {
+              "name": "body",
+              "description": "The desired configuration for the room.",
+              "required": true,
+              "type": "RoomConfig",
+              "paramType": "body"
+            }
+          ],
+          "responseMessages": [
+            {
+              "code": 400,
+              "message": "Body must be JSON."
+            },
+            {
+              "code": 400,
+              "message": "Room alias already taken."
+            }
+          ]
+        }
+      ]
+    },
+    {
+      "path": "/rooms/{roomId}/messages",
+      "operations": [
+        {
+          "method": "GET",
+          "summary": "Get a list of messages for this room.",
+          "notes": "Get a list of messages for this room.",
+          "type": "MessagePaginationChunk",
+          "nickname": "get_messages",
+          "parameters": [
+            {
+              "name": "roomId",
+              "description": "The room to get messages in.",
+              "required": true,
+              "type": "string",
+              "paramType": "path"
+            },
+            {
+              "name": "from",
+              "description": "The token to start getting results from.",
+              "required": false,
+              "type": "string",
+              "paramType": "query"
+            },
+            {
+              "name": "to",
+              "description": "The token to stop getting results at.",
+              "required": false,
+              "type": "string",
+              "paramType": "query"
+            },
+            {
+              "name": "limit",
+              "description": "The maximum number of messages to return.",
+              "required": false,
+              "type": "integer",
+              "paramType": "query"
+            }
+          ]
+        }
+      ]
+    },
+    {
+      "path": "/rooms/{roomId}/members",
+      "operations": [
+        {
+          "method": "GET",
+          "summary": "Get a list of members for this room.",
+          "notes": "Get a list of members for this room.",
+          "type": "MemberPaginationChunk",
+          "nickname": "get_members",
+          "parameters": [
+            {
+              "name": "roomId",
+              "description": "The room to get a list of members from.",
+              "required": true,
+              "type": "string",
+              "paramType": "path"
+            },
+            {
+              "name": "from",
+              "description": "The token to start getting results from.",
+              "required": false,
+              "type": "string",
+              "paramType": "query"
+            },
+            {
+              "name": "to",
+              "description": "The token to stop getting results at.",
+              "required": false,
+              "type": "string",
+              "paramType": "query"
+            },
+            {
+              "name": "limit",
+              "description": "The maximum number of members to return.",
+              "required": false,
+              "type": "integer",
+              "paramType": "query"
+            }
+          ]
+        }
+      ]
+    },
+    {
+      "path": "/rooms/{roomId}/state",
+      "operations": [
+        {
+          "method": "GET",
+          "summary": "Get a list of all the current state events for this room.",
+          "notes": "Get a list of all the current state events for this room.",
+          "type": "array",
+          "items": {
+            "$ref": "Event"
+          },
+          "nickname": "get_state_events",
+          "parameters": [
+            {
+              "name": "roomId",
+              "description": "The room to get a list of current state events from.",
+              "required": true,
+              "type": "string",
+              "paramType": "path"
+            }
+          ]
+        }
+      ]
+    },
+    {
+      "path": "/rooms/{roomId}/initialSync",
+      "operations": [
+        {
+          "method": "GET",
+          "summary": "Get all the current information for this room, including messages and state events.",
+          "notes": "Get all the current information for this room, including messages and state events.",
+          "type": "InitialSyncRoomData",
+          "nickname": "get_room_sync_data",
+          "parameters": [
+            {
+              "name": "roomId",
+              "description": "The room to get information for.",
+              "required": true,
+              "type": "string",
+              "paramType": "path"
+            }
+          ]
+        }
+      ]
+    }
+  ],
+  "models": {
+    "Topic": {
+      "id": "Topic",
+      "properties": {
+        "topic": {
+          "type": "string",
+          "description": "The topic text"
+        }
+      }
+    },
+    "Message": {
+      "id": "Message",
+      "properties": {
+        "msgtype": {
+          "type": "string",
+          "description": "The type of message being sent, e.g. \"m.text\"",
+          "required": true
+        },
+        "_msgtype_defined_keys_": {
+          "description": "Additional keys as defined by the msgtype, e.g. \"body\""
+        }
+      }
+    },
+    "Feedback": {
+      "id": "Feedback",
+      "properties": {
+      }
+    },
+    "Member": {
+      "id": "Member",
+      "properties": {
+        "membership": {
+          "type": "string",
+          "description": "Enum: The membership state of this member.",
+          "enum": [
+            "invite",
+            "join",
+            "leave",
+            "knock"
+          ]
+        }
+      }
+    },
+    "RoomInfo": {
+      "id": "RoomInfo",
+      "properties": {
+        "room_id": {
+          "type": "string",
+          "description": "The allocated room ID.",
+          "required": true
+        },
+        "room_alias": {
+          "type": "string",
+          "description": "The alias for the room.",
+          "required": false
+        }
+      }
+    },
+    "RoomConfig": {
+      "id": "RoomConfig",
+      "properties": {
+        "visibility": {
+          "type": "string",
+          "description": "Enum: The room visibility.",
+          "required": false,
+          "enum": [
+            "public",
+            "private"
+          ]
+        },
+        "room_alias_name": {
+          "type": "string",
+          "description": "The alias to give the new room.",
+          "required": false
+        }
+      }
+    },
+    "PaginationRequest": {
+      "id": "PaginationRequest",
+      "properties": {
+        "from": {
+          "type": "string",
+          "description": "The token to start getting results from."
+        },
+        "to": {
+          "type": "string",
+          "description": "The token to stop getting results at."
+        },
+        "limit": {
+          "type": "integer",
+          "description": "The maximum number of entries to return."
+        }
+      }
+    },
+    "PaginationChunk": {
+      "id": "PaginationChunk",
+      "properties": {
+        "start": {
+          "type": "string",
+          "description": "A token which correlates to the first value in \"chunk\" for paginating.",
+          "required": true
+        },
+        "end": {
+          "type": "string",
+          "description": "A token which correlates to the last value in \"chunk\" for paginating.",
+          "required": true
+        }
+      },
+      "subTypes": [
+        "MessagePaginationChunk"
+      ]
+    },
+    "MessagePaginationChunk": {
+      "id": "MessagePaginationChunk",
+      "properties": {
+        "chunk": {
+          "type": "array",
+          "description": "A list of message events.",
+          "items": {
+            "$ref": "MessageEvent"
+          },
+          "required": true
+        }
+      }
+    },
+    "MemberPaginationChunk": {
+      "id": "MemberPaginationChunk",
+      "properties": {
+        "chunk": {
+          "type": "array",
+          "description": "A list of member events.",
+          "items": {
+            "$ref": "MemberEvent"
+          },
+          "required": true
+        }
+      }
+    },
+    "Event": {
+      "id": "Event",
+      "properties": {
+        "event_id": {
+          "type": "string",
+          "description": "An ID which uniquely identifies this event. This is automatically set by the server.",
+          "required": true
+        },
+        "room_id": {
+          "type": "string",
+          "description": "The room in which this event occurred. This is automatically set by the server.",
+          "required": true
+        },
+        "type": {
+          "type": "string",
+          "description": "The event type.",
+          "required": true
+        }
+      },
+      "subTypes": [
+        "MessageEvent"
+      ]
+    },
+    "EventId": {
+      "id": "EventId",
+      "properties": {
+        "event_id": {
+          "type": "string",
+          "description": "The allocated event ID for this event.",
+          "required": true
+        }
+      }
+    },
+    "EventContent": {
+      "id": "EventContent",
+      "properties": {
+        "__event_content_keys__": {
+          "type": "string",
+          "description": "Event-specific content keys and values.",
+          "required": false
+        }
+      }
+    },
+    "MessageEvent": {
+      "id": "MessageEvent",
+      "properties": {
+        "content": {
+          "type": "Message"
+        }
+      }
+    },
+    "MemberEvent": {
+      "id": "MemberEvent",
+      "properties": {
+        "content": {
+          "type": "Member"
+        }
+      }
+    },
+    "InviteRequest": {
+      "id": "InviteRequest",
+      "properties": {
+        "user_id": {
+          "type": "string",
+          "description": "The fully-qualified user ID."
+        }
+      }
+    },
+    "InitialSyncRoomData": {
+      "id": "InitialSyncRoomData",
+      "properties": {
+        "membership": {
+          "type": "string",
+          "description": "This user's membership state in this room.",
+          "required": true
+        },
+        "room_id": {
+          "type": "string",
+          "description": "The ID of this room.",
+          "required": true
+        },
+        "messages": {
+          "type": "MessagePaginationChunk",
+          "description": "The most recent messages for this room, governed by the limit parameter.",
+          "required": false
+        },
+        "state": {
+          "type": "array",
+          "description": "A list of state events representing the current state of the room.",
+          "required": false,
+          "items": {
+            "$ref": "Event"
+          }
+        }
+      }
+    }
+  }
+}

docs/client-server/urls.rst

@@ -1,92 +0,0 @@
-=========================
-Client-Server URL Summary
-=========================
-
-A brief overview of the URL scheme involved in the Synapse Client-Server API.
-
-
-URLs
-====
-
-Fetch events:
-  GET /events
-
-Registering an account
-  POST /register
-
-Unregistering an account
-  POST /unregister
-
-Rooms
------
-
-Creating a room by ID
-  PUT /rooms/$roomid
-
-Creating an anonymous room
-  POST /rooms
-
-Room topic
-  GET /rooms/$roomid/topic
-  PUT /rooms/$roomid/topic
-
-List rooms
-  GET /rooms/list
-
-Invite/Join/Leave
-  GET    /rooms/$roomid/members/$userid/state
-  PUT    /rooms/$roomid/members/$userid/state
-  DELETE /rooms/$roomid/members/$userid/state
-
-List members
-  GET  /rooms/$roomid/members/list
-
-Sending/reading messages
-  PUT /rooms/$roomid/messages/$sender/$msgid
-
-Feedback
-  GET /rooms/$roomid/messages/$sender/$msgid/feedback/$feedbackuser/$feedback
-  PUT /rooms/$roomid/messages/$sender/$msgid/feedback/$feedbackuser/$feedback
-
-Paginating messages
-  GET /rooms/$roomid/messages/list
-
-Profiles
---------
-
-Display name
-  GET /profile/$userid/displayname
-  PUT /profile/$userid/displayname
-
-Avatar URL
-  GET /profile/$userid/avatar_url
-  PUT /profile/$userid/avatar_url
-
-Metadata
-  GET  /profile/$userid/metadata
-  POST /profile/$userid/metadata
-
-Presence
---------
-
-My state or status message
-  GET /presence/$userid/status
-  PUT /presence/$userid/status
-    also 'GET' for fetching others
-
-TODO(paul): per-device idle time, device type; similar to above
-
-My presence list
-  GET  /presence_list/$myuserid
-  POST /presence_list/$myuserid
-    body is JSON-encoded dict of keys:
-      invite: list of UserID strings to invite
-      drop: list of UserID strings to remove
-      TODO(paul): define other ops: accept, group management, ordering?
-
-Presence polling start/stop
-  POST /presence_list/$myuserid?op=start
-  POST /presence_list/$myuserid?op=stop
-
-Presence invite
-  POST /presence_list/$myuserid/invite/$targetuserid

docs/server-server/specification.rst

@@ -1,8 +1,8 @@
-============================
-Synapse Server-to-Server API
-============================
+===========================
+Matrix Server-to-Server API
+===========================
 
-A description of the protocol used to communicate between Synapse home servers;
+A description of the protocol used to communicate between Matrix home servers;
 also known as Federation.
 
 
@@ -10,7 +10,7 @@ Overview
 ========
 
 The server-server API is a mechanism by which two home servers can exchange
-Synapse event messages, both as a real-time push of current events, and as a
+Matrix event messages, both as a real-time push of current events, and as a
 historic fetching mechanism to synchronise past history for clients to view. It
 uses HTTP connections between each pair of servers involved as the underlying
 transport. Messages are exchanged between servers in real-time by active pushing
@@ -19,7 +19,7 @@ historic data for the purpose of back-filling scrollback buffers and the like
 can also be performed.
 
 
- { Synapse entities }                            { Synapse entities }
+  { Matrix clients }                              { Matrix clients }
      ^          |                                    ^          |
      |  events  |                                    |  events  |
      |          V                                    |          V
@@ -29,27 +29,53 @@ can also be performed.
  |                  |<--------( HTTP )-----------|                  |
  +------------------+                            +------------------+
 
+There are three main kinds of communication that occur between home servers:
 
-Transactions and PDUs
-=====================
+ * Queries
+   These are single request/response interactions between a given pair of
+   servers, initiated by one side sending an HTTP request to obtain some
+   information, and responded by the other. They are not persisted and contain
+   no long-term significant history. They simply request a snapshot state at the
+   instant the query is made.
 
-The communication between home servers is performed by a bidirectional exchange
-of messages. These messages are called Transactions, and are encoded as JSON
-objects with a dict as the top-level element, passed over HTTP. A Transaction is
-meaningful only to the pair of home servers that exchanged it; they are not
-globally-meaningful.
+ * EDUs - Ephemeral Data Units
+   These are notifications of events that are pushed from one home server to
+   another. They are not persisted and contain no long-term significant history,
+   nor does the receiving home server have to reply to them.
 
-Each transaction has an opaque ID and timestamp (UNIX epoch time in miliseconds)
-generated by its origin server, an origin and destination server name, a list of
-"previous IDs", and a list of PDUs - the actual message payload that the
-Transaction carries.
+ * PDUs - Persisted Data Units
+   These are notifications of events that are broadcast from one home server to
+   any others that are interested in the same "context" (namely, a Room ID).
+   They are persisted to long-term storage and form the record of history for
+   that context.
+
+Where Queries are presented directly across the HTTP connection as GET requests
+to specific URLs, EDUs and PDUs are further wrapped in an envelope called a
+Transaction, which is transferred from the origin to the destination home server
+using a PUT request.
+
+
+Transactions and EDUs/PDUs
+==========================
+
+The transfer of EDUs and PDUs between home servers is performed by an exchange
+of Transaction messages, which are encoded as JSON objects with a dict as the
+top-level element, passed over an HTTP PUT request. A Transaction is meaningful
+only to the pair of home servers that exchanged it; they are not globally-
+meaningful.
+
+Each transaction has an opaque ID and timestamp (UNIX epoch time in
+milliseconds) generated by its origin server, an origin and destination server
+name, a list of "previous IDs", and a list of PDUs - the actual message payload
+that the Transaction carries.
 
  {"transaction_id":"916d630ea616342b42e98a3be0b74113",
   "ts":1404835423000,
   "origin":"red",
   "destination":"blue",
   "prev_ids":["e1da392e61898be4d2009b9fecce5325"],
-  "pdus":[...]}
+  "pdus":[...],
+  "edus":[...]}
 
 The "previous IDs" field will contain a list of previous transaction IDs that
 the origin server has sent to this destination. Its purpose is to act as a
@@ -58,7 +84,9 @@ successfully received that Transaction, or ask for a retransmission if not.
 
 The "pdus" field of a transaction is a list, containing zero or more PDUs.[*]
 Each PDU is itself a dict containing a number of keys, the exact details of
-which will vary depending on the type of PDU.
+which will vary depending on the type of PDU. Similarly, the "edus" field is
+another list containing the EDUs. This key may be entirely absent if there are
+no EDUs to transfer.
 
 (* Normally the PDU list will be non-empty, but the server should cope with
 receiving an "empty" transaction, as this is useful for informing peers of other
@@ -86,7 +114,7 @@ field of a PDU refers to PDUs that any origin server has sent, rather than
 previous IDs that this origin has sent. This list may refer to other PDUs sent
 by the same origin as the current one, or other origins.
 
-Because of the distributed nature of participants in a Synapse conversation, it
+Because of the distributed nature of participants in a Matrix conversation, it
 is impossible to establish a globally-consistent total ordering on the events.
 However, by annotating each outbound PDU at its origin with IDs of other PDUs it
 has received, a partial ordering can be constructed allowing causallity
@@ -112,6 +140,15 @@ so on. This part needs refining. And writing in its own document as the details
 relate to the server/system as a whole, not specifically to server-server
 federation.]]
 
+EDUs, by comparison to PDUs, do not have an ID, a context, or a list of
+"previous" IDs. The only mandatory fields for these are the type, origin and
+destination home server names, and the actual nested content.
+
+ {"edu_type":"m.presence",
+  "origin":"blue",
+  "destination":"orange",
+  "content":...}
+
 
 Protocol URLs
 =============
@@ -179,3 +216,16 @@ To stream events all the events:
   Retrieves all of the transactions later than any version given by the "v"
   arguments. [[TODO(paul): I'm not sure what the "origin" argument does because
   I think at some point in the code it's got swapped around.]]
+
+
+To make a query:
+
+  GET .../query/:query_type
+    Query args: as specified by the individual query types
+
+  Response: JSON encoding of a response object
+
+  Performs a single query request on the receiving home server. The Query Type
+  part of the path specifies the kind of query being made, and its query
+  arguments have a meaning specific to that kind of query. The response is a
+  JSON-encoded object whose meaning also depends on the kind of query.

docs/specification.rst

@@ -0,0 +1,839 @@
+Matrix Specification
+====================
+
+TODO(Introduction) : Matthew
+ - Similar to intro paragraph from README.
+ - Explaining the overall mission, what this spec describes...
+ - "What is Matrix?"
+ - Draw parallels with email?
+
+Architecture
+============
+
+Clients transmit data to other clients through home servers (HSes). Clients do not communicate with each
+other directly.
+
+::
+
+                         How data flows between clients
+                         ==============================
+
+       { Matrix client A }                             { Matrix client B }
+           ^          |                                    ^          |
+           |  events  |                                    |  events  |
+           |          V                                    |          V
+       +------------------+                            +------------------+
+       |                  |---------( HTTP )---------->|                  |
+       |   Home Server    |                            |   Home Server    |
+       |                  |<--------( HTTP )-----------|                  |
+       +------------------+        Federation          +------------------+
+
+A "Client" is an end-user, typically a human using a web application or mobile app. Clients use the
+"Client-to-Server" (C-S) API to communicate with their home server. A single Client is usually
+responsible for a single user account. A user account is represented by their "User ID". This ID is
+namespaced to the home server which allocated the account and looks like::
+
+  @localpart:domain
+
+The ``localpart`` of a user ID may be a user name, or an opaque ID identifying this user.
+
+
+A "Home Server" is a server which provides C-S APIs and has the ability to federate with other HSes.
+It is typically responsible for multiple clients. "Federation" is the term used to describe the
+sharing of data between two or more home servers.
+
+Data in Matrix is encapsulated in an "Event". An event is an action within the system. Typically each
+action (e.g. sending a message) correlates with exactly one event. Each event has a ``type`` which is
+used to differentiate different kinds of data. ``type`` values SHOULD be namespaced according to standard
+Java package naming conventions, e.g. ``com.example.myapp.event``. Events are usually sent in the context
+of a "Room".
+
+Room structure
+--------------
+
+A room is a conceptual place where users can send and receive events. Rooms 
+can be created, joined and left. Events are sent to a room, and all 
+participants in that room will receive the event. Rooms are uniquely 
+identified via a "Room ID", which look like::
+
+  !opaque_id:domain
+
+There is exactly one room ID for each room. Whilst the room ID does contain a
+domain, it is simply for namespacing room IDs. The room does NOT reside on the
+domain specified. Room IDs are not meant to be human readable.
+
+The following diagram shows an ``m.room.message`` event being sent in the room 
+``!qporfwt:matrix.org``::
+
+       { @alice:matrix.org }                             { @bob:domain.com }
+               |                                                 ^
+               |                                                 |
+      Room ID: !qporfwt:matrix.org                 Room ID: !qporfwt:matrix.org
+      Event type: m.room.message                   Event type: m.room.message
+      Content: { JSON object }                     Content: { JSON object }
+               |                                                 |
+               V                                                 |
+       +------------------+                          +------------------+
+       |   Home Server    |                          |   Home Server    |
+       |   matrix.org     |<-------Federation------->|   domain.com     |
+       +------------------+                          +------------------+
+                |       .................................        |
+                |______|     Partially Shared State      |_______|
+                       | Room ID: !qporfwt:matrix.org    |
+                       | Servers: matrix.org, domain.com |
+                       | Members:                        |
+                       |  - @alice:matrix.org            |
+                       |  - @bob:domain.com              |
+                       |.................................|
+
+Federation maintains shared state between multiple home servers, such that when an event is
+sent to a room, the home server knows where to forward the event on to, and how to process
+the event. Home servers do not need to have completely shared state in order to participate 
+in a room. State is scoped to a single room, and federation ensures that all home servers 
+have the information they need, even if that means the home server has to request more 
+information from another home server before processing the event.
+
+Room Aliases
+------------
+
+Each room can also have multiple "Room Aliases", which looks like::
+
+  #room_alias:domain
+
+A room alias "points" to a room ID. The room ID the alias is pointing to can be obtained
+by visiting the domain specified. Room aliases are designed to be human readable strings
+which can be used to publicise rooms. Note that the mapping from a room alias to a 
+room ID is not fixed, and may change over time to point to a different room ID. For this
+reason, Clients SHOULD resolve the room alias to a room ID once and then use that ID on
+subsequent requests.
+
+::
+
+          GET    
+   #matrix:domain.com      !aaabaa:matrix.org
+           |                    ^
+           |                    |
+    _______V____________________|____
+   |          domain.com            |
+   | Mappings:                      |
+   | #matrix >> !aaabaa:matrix.org  |
+   | #golf >> !wfeiofh:sport.com    |
+   | #bike >> !4rguxf:matrix.org    |
+   |________________________________|
+
+       
+Identity
+--------
+- Identity in relation to 3PIDs. Discovery of users based on 3PIDs.
+- Identity servers; trusted clique of servers which replicate content.
+- They govern the mapping of 3PIDs to user IDs and the creation of said mappings.
+- Not strictly required in order to communicate.
+
+
+API Standards
+-------------
+- All HTTP[S]
+- Uses JSON as HTTP bodies
+- Standard error response format { errcode: M_WHATEVER, error: "some message" }
+- C-S API provides POST for operations, or PUT with txn IDs. Explain txn IDs.
+
+Receiving live updates on a client
+----------------------------------
+- C-S longpoll event stream
+- Concept of start/end tokens.
+- Mention /initialSync to get token.
+
+
+Rooms
+=====
+- How are they created? PDU anchor point: "root of the tree".
+- Adding / removing aliases.
+- Invite/join dance
+- State and non-state data (+extensibility)
+
+TODO : Room permissions / config / power levels.
+
+Messages
+========
+
+This specification outlines several standard event types, all of which are
+prefixed with ``m.``
+
+State messages
+--------------
+- m.room.name
+- m.room.topic
+- m.room.member
+- m.room.config
+- m.room.invite_join
+
+What are they, when are they used, what do they contain, how should they be used
+
+Non-state messages
+------------------
+- m.room.message
+- m.room.message.feedback (and compressed format)
+
+What are they, when are they used, what do they contain, how should they be used
+
+m.room.message msgtypes
+-----------------------
+Each ``m.room.message`` MUST have a ``msgtype`` key which identifies the type of
+message being sent. Each type has their own required and optional keys, as outlined
+below:
+
+``m.text``
+  Required keys:
+    - ``body`` : "string" - The body of the message.
+  Optional keys:
+    None.
+  Example:
+    ``{ "msgtype": "m.text", "body": "I am a fish" }``
+
+``m.emote``
+  Required keys:
+    - ``body`` : "string" - The emote action to perform.
+  Optional keys:
+    None.
+  Example:
+    ``{ "msgtype": "m.emote", "body": "tries to come up with a witty explanation" }``
+
+``m.image``
+  Required keys:
+    - ``url`` : "string" - The URL to the image.
+  Optional keys:
+    - ``info`` : "string" - info : JSON object (ImageInfo) - The image info for image 
+      referred to in ``url``.
+    - ``thumbnail_url`` : "string" - The URL to the thumbnail.
+    - ``thumbnail_info`` : JSON object (ImageInfo) - The image info for the image 
+      referred to in ``thumbnail_url``.
+    - ``body`` : "string" - The alt text of the image, or some kind of content 
+      description for accessibility e.g. "image attachment".
+
+  ImageInfo: 
+    Information about an image::
+    
+      { 
+        "size" : integer (size of image in bytes),
+        "w" : integer (width of image in pixels),
+        "h" : integer (height of image in pixels),
+        "mimetype" : "string (e.g. image/jpeg)",
+      }
+
+``m.audio``
+  Required keys:
+    - ``url`` : "string" - The URL to the audio.
+  Optional keys:
+    - ``info`` : JSON object (AudioInfo) - The audio info for the audio referred to in 
+      ``url``.
+    - ``body`` : "string" - A description of the audio e.g. "Bee Gees - 
+      Stayin' Alive", or some kind of content description for accessibility e.g. 
+      "audio attachment".
+  AudioInfo: 
+    Information about a piece of audio::
+
+      {
+        "mimetype" : "string (e.g. audio/aac)",
+        "size" : integer (size of audio in bytes),
+        "duration" : integer (duration of audio in milliseconds),
+      }
+
+``m.video``
+  Required keys:
+    - ``url`` : "string" - The URL to the video.
+  Optional keys:
+    - ``info`` : JSON object (VideoInfo) - The video info for the video referred to in 
+      ``url``.
+    - ``body`` : "string" - A description of the video e.g. "Gangnam style", 
+      or some kind of content description for accessibility e.g. "video attachment".
+
+  VideoInfo: 
+    Information about a video::
+
+      {
+        "mimetype" : "string (e.g. video/mp4)",
+        "size" : integer (size of video in bytes),
+        "duration" : integer (duration of video in milliseconds),
+        "w" : integer (width of video in pixels),
+        "h" : integer (height of video in pixels),
+        "thumbnail_url" : "string (URL to image)",
+        "thumbanil_info" : JSON object (ImageInfo)
+      }
+
+``m.location``
+  Required keys:
+    - ``geo_uri`` : "string" - The geo URI representing the location.
+  Optional keys:
+    - ``thumbnail_url`` : "string" - The URL to a thumnail of the location being 
+      represented.
+    - ``thumbnail_info`` : JSON object (ImageInfo) - The image info for the image 
+      referred to in ``thumbnail_url``.
+    - ``body`` : "string" - A description of the location e.g. "Big Ben, 
+      London, UK", or some kind of content description for accessibility e.g. 
+      "location attachment".
+
+The following keys can be attached to any ``m.room.message``:
+
+  Optional keys:
+    - ``sender_ts`` : integer - A timestamp (ms resolution) representing the 
+      wall-clock time when the message was sent from the client.
+
+Presence
+========
+
+Each user has the concept of presence information. This encodes the
+"availability" of that user, suitable for display on other user's clients. This
+is transmitted as an ``m.presence`` event and is one of the few events which
+are sent *outside the context of a room*. The basic piece of presence information 
+is represented by the ``state`` key, which is an enum of one of the following:
+
+  - ``online`` : The default state when the user is connected to an event stream.
+  - ``unavailable`` : The user is not reachable at this time.
+  - ``offline`` : The user is not connected to an event stream.
+  - ``free_for_chat`` : The user is generally willing to receive messages 
+    moreso than default.
+  - ``hidden`` : TODO. Behaves as offline, but allows the user to see the client 
+    state anyway and generally interact with client features.
+
+This basic ``state`` field applies to the user as a whole, regardless of how many
+client devices they have connected. The home server should synchronise this
+status choice among multiple devices to ensure the user gets a consistent
+experience.
+
+Idle Time
+---------
+As well as the basic ``state`` field, the presence information can also show a sense
+of an "idle timer". This should be maintained individually by the user's
+clients, and the home server can take the highest reported time as that to
+report. When a user is offline, the home server can still report when the user was last
+seen online.
+
+Transmission
+------------
+- Transmitted as an EDU.
+- Presence lists determine who to send to.
+
+Presence List
+-------------
+Each user's home server stores a "presence list" for that user. This stores a
+list of other user IDs the user has chosen to add to it. To be added to this 
+list, the user being added must receive permission from the list owner. Once
+granted, both user's HS(es) store this information. Since such subscriptions
+are likely to be bidirectional, HSes may wish to automatically accept requests
+when a reverse subscription already exists.
+
+Presence and Permissions
+------------------------
+For a viewing user to be allowed to see the presence information of a target
+user, either:
+
+ - The target user has allowed the viewing user to add them to their presence
+   list, or
+ - The two users share at least one room in common
+
+In the latter case, this allows for clients to display some minimal sense of
+presence information in a user list for a room.
+
+Typing notifications
+====================
+
+TODO : Leo
+
+Voice over IP
+=============
+
+TODO : Dave
+
+Profiles
+========
+
+Internally within Matrix users are referred to by their user ID, which is not a
+human-friendly string. Profiles grant users the ability to see human-readable 
+names for other users that are in some way meaningful to them. Additionally, 
+profiles can publish additional information, such as the user's age or location.
+
+A Profile consists of a display name, an avatar picture, and a set of other 
+metadata fields that the user may wish to publish (email address, phone
+numbers, website URLs, etc...). This specification puts no requirements on the 
+display name other than it being a valid unicode string.
+
+- Metadata extensibility
+- Bundled with which events? e.g. m.room.member
+- Generate own events? What type?
+
+Registration and login
+======================
+
+Clients must register with a home server in order to use Matrix. After 
+registering, the client will be given an access token which must be used in ALL
+requests to that home server as a query parameter 'access_token'.
+
+- TODO Kegan : Make registration like login (just omit the "user" key on the 
+  initial request?)
+
+If the client has already registered, they need to be able to login to their
+account. The home server may provide many different ways of logging in, such
+as user/password auth, login via a social network (OAuth2), login by confirming 
+a token sent to their email address, etc. This specification does not define how
+home servers should authorise their users who want to login to their existing 
+accounts, but instead defines the standard interface which implementations 
+should follow so that ANY client can login to ANY home server.
+
+The login process breaks down into the following:
+  1. Determine the requirements for logging in.
+  2. Submit the login stage credentials.
+  3. Get credentials or be told the next stage in the login process and repeat 
+     step 2.
+     
+As each home server may have different ways of logging in, the client needs to know how
+they should login. All distinct login stages MUST have a corresponding ``type``.
+A ``type`` is a namespaced string which details the mechanism for logging in.
+
+A client may be able to login via multiple valid login flows, and should choose a single
+flow when logging in. A flow is a series of login stages. The home server MUST respond 
+with all the valid login flows when requested::
+
+  The client can login via 3 paths: 1a and 1b, 2a and 2b, or 3. The client should
+  select one of these paths.
+  
+  {
+    "flows": [
+      {
+        "type": "<login type1a>",
+        "stages": [ "<login type 1a>", "<login type 1b>" ]
+      },
+      {
+        "type": "<login type2a>",
+        "stages": [ "<login type 2a>", "<login type 2b>" ]
+      },
+      {
+        "type": "<login type3>"
+      }
+    ]
+  }
+
+After the login is completed, the client's fully-qualified user ID and a new access 
+token MUST be returned::
+
+  {
+    "user_id": "@user:matrix.org",
+    "access_token": "abcdef0123456789"
+  }
+
+The ``user_id`` key is particularly useful if the home server wishes to support 
+localpart entry of usernames (e.g. "user" rather than "@user:matrix.org"), as the
+client may not be able to determine its ``user_id`` in this case.
+
+If a login has multiple requests, the home server may wish to create a session. If
+a home server responds with a 'session' key to a request, clients MUST submit it in 
+subsequent requests until the login is completed::
+
+  {
+    "session": "<session id>"
+  }
+
+This specification defines the following login types:
+ - ``m.login.password``
+ - ``m.login.oauth2``
+ - ``m.login.email.code``
+ - ``m.login.email.url``
+
+
+Password-based
+--------------
+:Type: 
+  m.login.password
+:Description: 
+  Login is supported via a username and password.
+
+To respond to this type, reply with::
+
+  {
+    "type": "m.login.password",
+    "user": "<user_id or user localpart>",
+    "password": "<password>"
+  }
+
+The home server MUST respond with either new credentials, the next stage of the login
+process, or a standard error response.
+
+OAuth2-based
+------------
+:Type: 
+  m.login.oauth2
+:Description:
+  Login is supported via OAuth2 URLs. This login consists of multiple requests.
+
+To respond to this type, reply with::
+
+  {
+    "type": "m.login.oauth2",
+    "user": "<user_id or user localpart>"
+  }
+
+The server MUST respond with::
+
+  {
+    "uri": <Authorization Request URI OR service selection URI>
+  }
+
+The home server acts as a 'confidential' client for the purposes of OAuth2.
+If the uri is a ``sevice selection URI``, it MUST point to a webpage which prompts the 
+user to choose which service to authorize with. On selection of a service, this
+MUST link through to an ``Authorization Request URI``. If there is only 1 service which the
+home server accepts when logging in, this indirection can be skipped and the
+"uri" key can be the ``Authorization Request URI``. 
+
+The client then visits the ``Authorization Request URI``, which then shows the OAuth2 
+Allow/Deny prompt. Hitting 'Allow' returns the ``redirect URI`` with the auth code. 
+Home servers can choose any path for the ``redirect URI``. The client should visit 
+the ``redirect URI``, which will then finish the OAuth2 login process, granting the 
+home server an access token for the chosen service. When the home server gets 
+this access token, it verifies that the cilent has authorised with the 3rd party, and 
+can now complete the login. The OAuth2 ``redirect URI`` (with auth code) MUST respond 
+with either new credentials, the next stage of the login process, or a standard error 
+response.
+    
+For example, if a home server accepts OAuth2 from Google, it would return the 
+Authorization Request URI for Google::
+
+  {
+    "uri": "https://accounts.google.com/o/oauth2/auth?response_type=code&
+    client_id=CLIENT_ID&redirect_uri=REDIRECT_URI&scope=photos"
+  }
+
+The client then visits this URI and authorizes the home server. The client then
+visits the REDIRECT_URI with the auth code= query parameter which returns::
+
+  {
+    "user_id": "@user:matrix.org",
+    "access_token": "0123456789abcdef"
+  }
+
+Email-based (code)
+------------------
+:Type: 
+  m.login.email.code
+:Description:
+  Login is supported by typing in a code which is sent in an email. This login 
+  consists of multiple requests.
+
+To respond to this type, reply with::
+
+  {
+    "type": "m.login.email.code",
+    "user": "<user_id or user localpart>",
+    "email": "<email address>"
+  }
+
+After validating the email address, the home server MUST send an email containing
+an authentication code and return::
+
+  {
+    "type": "m.login.email.code",
+    "session": "<session id>"
+  }
+
+The second request in this login stage involves sending this authentication code::
+
+  {
+    "type": "m.login.email.code",
+    "session": "<session id>",
+    "code": "<code in email sent>"
+  }
+
+The home server MUST respond to this with either new credentials, the next stage of 
+the login process, or a standard error response.
+
+Email-based (url)
+-----------------
+:Type: 
+  m.login.email.url
+:Description:
+  Login is supported by clicking on a URL in an email. This login consists of 
+  multiple requests.
+
+To respond to this type, reply with::
+
+  {
+    "type": "m.login.email.url",
+    "user": "<user_id or user localpart>",
+    "email": "<email address>"
+  }
+
+After validating the email address, the home server MUST send an email containing
+an authentication URL and return::
+
+  {
+    "type": "m.login.email.url",
+    "session": "<session id>"
+  }
+
+The email contains a URL which must be clicked. After it has been clicked, the
+client should perform another request::
+
+  {
+    "type": "m.login.email.url",
+    "session": "<session id>"
+  }
+
+The home server MUST respond to this with either new credentials, the next stage of 
+the login process, or a standard error response. 
+
+A common client implementation will be to periodically poll until the link is clicked.
+If the link has not been visited yet, a standard error response with an errcode of 
+``M_LOGIN_EMAIL_URL_NOT_YET`` should be returned.
+
+
+N-Factor Authentication
+-----------------------
+Multiple login stages can be combined to create N-factor authentication during login.
+
+This can be achieved by responding with the ``next`` login type on completion of a 
+previous login stage::
+
+  {
+    "next": "<next login type>"
+  }
+
+If a home server implements N-factor authentication, it MUST respond with all 
+``stages`` when initially queried for their login requirements::
+
+  {
+    "type": "<1st login type>",
+    "stages": [ <1st login type>, <2nd login type>, ... , <Nth login type> ]
+  }
+
+This can be represented conceptually as::
+
+   _______________________
+  |    Login Stage 1      |
+  | type: "<login type1>" |
+  |  ___________________  |
+  | |_Request_1_________| | <-- Returns "session" key which is used throughout.
+  |  ___________________  |     
+  | |_Request_2_________| | <-- Returns a "next" value of "login type2"
+  |_______________________|
+            |
+            |
+   _________V_____________
+  |    Login Stage 2      |
+  | type: "<login type2>" |
+  |  ___________________  |
+  | |_Request_1_________| |
+  |  ___________________  |
+  | |_Request_2_________| |
+  |  ___________________  |
+  | |_Request_3_________| | <-- Returns a "next" value of "login type3"
+  |_______________________|
+            |
+            |
+   _________V_____________
+  |    Login Stage 3      |
+  | type: "<login type3>" |
+  |  ___________________  |
+  | |_Request_1_________| | <-- Returns user credentials
+  |_______________________|
+
+Fallback
+--------
+Clients cannot be expected to be able to know how to process every single
+login type. If a client determines it does not know how to handle a given
+login type, it should request a login fallback page::
+
+  GET matrix/client/api/v1/login/fallback
+
+This MUST return an HTML page which can perform the entire login process.
+
+Identity
+========
+
+TODO : Dave
+- 3PIDs and identity server, functions
+
+Federation
+==========
+
+Federation is the term used to describe how to communicate between Matrix home 
+servers. Federation is a mechanism by which two home servers can exchange
+Matrix event messages, both as a real-time push of current events, and as a
+historic fetching mechanism to synchronise past history for clients to view. It
+uses HTTP connections between each pair of servers involved as the underlying
+transport. Messages are exchanged between servers in real-time by active pushing
+from each server's HTTP client into the server of the other. Queries to fetch
+historic data for the purpose of back-filling scrollback buffers and the like
+can also be performed.
+
+There are three main kinds of communication that occur between home servers:
+
+:Queries:
+   These are single request/response interactions between a given pair of
+   servers, initiated by one side sending an HTTP GET request to obtain some
+   information, and responded by the other. They are not persisted and contain
+   no long-term significant history. They simply request a snapshot state at the
+   instant the query is made.
+
+:Ephemeral Data Units (EDUs):
+   These are notifications of events that are pushed from one home server to
+   another. They are not persisted and contain no long-term significant history,
+   nor does the receiving home server have to reply to them.
+
+:Persisted Data Units (PDUs):
+   These are notifications of events that are broadcast from one home server to
+   any others that are interested in the same "context" (namely, a Room ID).
+   They are persisted to long-term storage and form the record of history for
+   that context.
+
+EDUs and PDUs are further wrapped in an envelope called a Transaction, which is 
+transferred from the origin to the destination home server using an HTTP PUT request.
+
+
+Transactions
+------------
+The transfer of EDUs and PDUs between home servers is performed by an exchange
+of Transaction messages, which are encoded as JSON objects, passed over an 
+HTTP PUT request. A Transaction is meaningful only to the pair of home servers that 
+exchanged it; they are not globally-meaningful.
+
+Each transaction has:
+ - An opaque transaction ID.
+ - A timestamp (UNIX epoch time in milliseconds) generated by its origin server.
+ - An origin and destination server name.
+ - A list of "previous IDs".
+ - A list of PDUs and EDUs - the actual message payload that the Transaction carries.
+
+::
+
+ {
+  "transaction_id":"916d630ea616342b42e98a3be0b74113",
+  "ts":1404835423000,
+  "origin":"red",
+  "destination":"blue",
+  "prev_ids":["e1da392e61898be4d2009b9fecce5325"],
+  "pdus":[...],
+  "edus":[...]
+ }
+
+The ``prev_ids`` field contains a list of previous transaction IDs that
+the ``origin`` server has sent to this ``destination``. Its purpose is to act as a
+sequence checking mechanism - the destination server can check whether it has
+successfully received that Transaction, or ask for a retransmission if not.
+
+The ``pdus`` field of a transaction is a list, containing zero or more PDUs.[*]
+Each PDU is itself a JSON object containing a number of keys, the exact details of
+which will vary depending on the type of PDU. Similarly, the ``edus`` field is
+another list containing the EDUs. This key may be entirely absent if there are
+no EDUs to transfer.
+
+(* Normally the PDU list will be non-empty, but the server should cope with
+receiving an "empty" transaction, as this is useful for informing peers of other
+transaction IDs they should be aware of. This effectively acts as a push
+mechanism to encourage peers to continue to replicate content.)
+
+PDUs and EDUs
+-------------
+
+All PDUs have:
+ - An ID
+ - A context
+ - A declaration of their type
+ - A list of other PDU IDs that have been seen recently on that context (regardless of which origin
+   sent them)
+
+[[TODO(paul): Update this structure so that 'pdu_id' is a two-element
+[origin,ref] pair like the prev_pdus are]]
+
+::
+
+ {
+  "pdu_id":"a4ecee13e2accdadf56c1025af232176",
+  "context":"#example.green",
+  "origin":"green",
+  "ts":1404838188000,
+  "pdu_type":"m.text",
+  "prev_pdus":[["blue","99d16afbc857975916f1d73e49e52b65"]],
+  "content":...
+  "is_state":false
+ }
+
+In contrast to Transactions, it is important to note that the ``prev_pdus``
+field of a PDU refers to PDUs that any origin server has sent, rather than
+previous IDs that this ``origin`` has sent. This list may refer to other PDUs sent
+by the same origin as the current one, or other origins.
+
+Because of the distributed nature of participants in a Matrix conversation, it
+is impossible to establish a globally-consistent total ordering on the events.
+However, by annotating each outbound PDU at its origin with IDs of other PDUs it
+has received, a partial ordering can be constructed allowing causallity
+relationships to be preserved. A client can then display these messages to the
+end-user in some order consistent with their content and ensure that no message
+that is semantically in reply of an earlier one is ever displayed before it.
+
+PDUs fall into two main categories: those that deliver Events, and those that
+synchronise State. For PDUs that relate to State synchronisation, additional
+keys exist to support this:
+
+::
+
+ {...,
+  "is_state":true,
+  "state_key":TODO
+  "power_level":TODO
+  "prev_state_id":TODO
+  "prev_state_origin":TODO}
+
+[[TODO(paul): At this point we should probably have a long description of how
+State management works, with descriptions of clobbering rules, power levels, etc
+etc... But some of that detail is rather up-in-the-air, on the whiteboard, and
+so on. This part needs refining. And writing in its own document as the details
+relate to the server/system as a whole, not specifically to server-server
+federation.]]
+
+EDUs, by comparison to PDUs, do not have an ID, a context, or a list of
+"previous" IDs. The only mandatory fields for these are the type, origin and
+destination home server names, and the actual nested content.
+
+::
+
+ {"edu_type":"m.presence",
+  "origin":"blue",
+  "destination":"orange",
+  "content":...}
+
+Backfilling
+-----------
+- What it is, when is it used, how is it done
+
+SRV Records
+-----------
+- Why it is needed
+
+Security
+========
+- rate limiting
+- crypto (s-s auth)
+- E2E
+- Lawful intercept + Key Escrow
+
+TODO Mark
+
+Policy Servers
+==============
+TODO
+
+Content repository
+==================
+- thumbnail paths
+
+Address book repository
+=======================
+- format
+
+
+Glossary
+========
+- domain specific words/acronyms with definitions
+
+User ID:
+  An opaque ID which identifies an end-user, which consists of some opaque 
+  localpart combined with the domain name of their home server. 

graph/graph.py

@@ -113,7 +113,7 @@ def make_graph(pdus, room, filename_prefix):
                 graph.add_edge(state_edge)
 
     graph.write('%s.dot' % filename_prefix, format='raw', prog='dot')
-    graph.write_png("%s.png" % filename_prefix, prog='dot')
+#    graph.write_png("%s.png" % filename_prefix, prog='dot')
     graph.write_svg("%s.svg" % filename_prefix, prog='dot')
 
 

jsfiddles/create_room_send_msg/demo.css

@@ -0,0 +1,17 @@
+.loggedin {
+    visibility: hidden;
+}
+
+p {
+    font-family: monospace;
+}
+
+table
+{
+    border-spacing:5px;
+}
+
+th,td
+{
+    padding:5px;
+}

jsfiddles/create_room_send_msg/demo.html

@@ -0,0 +1,30 @@
+<div>
+    <p>This room creation / message sending demo requires a home server to be running on http://localhost:8080</p>
+</div>
+<form class="loginForm">
+    <input type="text" id="userLogin" placeholder="Username"></input>
+    <input type="password" id="passwordLogin" placeholder="Password"></input>
+    <input type="button" class="login" value="Login"></input>
+</form>
+<div class="loggedin">
+    <form class="createRoomForm">
+        <input type="text" id="roomAlias" placeholder="Room alias (optional)"></input>
+        <input type="button" class="createRoom" value="Create Room"></input>
+    </form>
+    <form class="sendMessageForm">
+        <input type="text" id="roomId" placeholder="Room ID"></input>
+        <input type="text" id="messageBody" placeholder="Message body"></input>
+        <input type="button" class="sendMessage" value="Send Message"></input>
+    </form>
+    <table id="rooms">
+        <tbody>
+            <tr>
+                <th>Room ID</th>
+                <th>My state</th>
+                <th>Room Alias</th>
+                <th>Latest message</th>
+            </tr>
+        </tbody>
+    </table>
+</div>
+    

jsfiddles/create_room_send_msg/demo.js

@@ -0,0 +1,109 @@
+var accountInfo = {};
+
+var showLoggedIn = function(data) {
+    accountInfo = data;
+    getCurrentRoomList();
+    $(".loggedin").css({visibility: "visible"});
+};
+
+$('.login').live('click', function() {
+    var user = $("#userLogin").val();
+    var password = $("#passwordLogin").val();
+    $.ajax({
+        url: "http://localhost:8080/matrix/client/api/v1/login",
+        type: "POST",
+        contentType: "application/json; charset=utf-8",
+        data: JSON.stringify({ user: user, password: password, type: "m.login.password" }),
+        dataType: "json",
+        success: function(data) {
+            showLoggedIn(data);
+        },
+        error: function(err) {
+            alert(JSON.stringify($.parseJSON(err.responseText)));  
+        }
+    }); 
+});
+
+var getCurrentRoomList = function() {
+    var url = "http://localhost:8080/matrix/client/api/v1/im/sync?access_token=" + accountInfo.access_token + "&from=END&to=START&limit=1";
+    $.getJSON(url, function(data) {
+        for (var i=0; i<data.length; ++i) {
+            data[i].latest_message = data[i].messages.chunk[0].content.body;
+            addRoom(data[i]);   
+        }
+    }).fail(function(err) {
+        alert(JSON.stringify($.parseJSON(err.responseText)));
+    });
+};
+
+$('.createRoom').live('click', function() {
+    var roomAlias = $("#roomAlias").val();
+    var data = {};
+    if (roomAlias.length > 0) {
+        data.room_alias_name = roomAlias;   
+    }
+    $.ajax({
+        url: "http://localhost:8080/matrix/client/api/v1/rooms?access_token="+accountInfo.access_token,
+        type: "POST",
+        contentType: "application/json; charset=utf-8",
+        data: JSON.stringify(data),
+        dataType: "json",
+        success: function(data) {
+            data.membership = "join"; // you are automatically joined into every room you make.
+            data.latest_message = "";
+            addRoom(data);
+        },
+        error: function(err) {
+            alert(JSON.stringify($.parseJSON(err.responseText)));  
+        }
+    }); 
+});
+
+var addRoom = function(data) {
+    row = "<tr>" +
+        "<td>"+data.room_id+"</td>" +
+        "<td>"+data.membership+"</td>" +
+        "<td>"+data.room_alias+"</td>" +
+        "<td>"+data.latest_message+"</td>" +
+        "</tr>";
+    $("#rooms").append(row);
+};
+
+$('.sendMessage').live('click', function() {
+    var roomId = $("#roomId").val();
+    var body = $("#messageBody").val();
+    var msgId = $.now();
+    
+    if (roomId.length === 0 || body.length === 0) {
+        return;
+    }
+    
+    var url = "http://localhost:8080/matrix/client/api/v1/rooms/$roomid/messages/$user/$msgid?access_token=$token";
+    url = url.replace("$token", accountInfo.access_token);
+    url = url.replace("$roomid", encodeURIComponent(roomId));
+    url = url.replace("$user", encodeURIComponent(accountInfo.user_id));
+    url = url.replace("$msgid", msgId);
+    
+    var data = {
+        msgtype: "m.text",
+        body: body
+    };
+    
+    $.ajax({
+        url: url,
+        type: "PUT",
+        contentType: "application/json; charset=utf-8",
+        data: JSON.stringify(data),
+        dataType: "json",
+        success: function(data) {
+            $("#messageBody").val("");
+            // wipe the table and reload it. Using the event stream would be the best
+            // solution but that is out of scope of this fiddle.
+            $("#rooms").find("tr:gt(0)").remove();
+            getCurrentRoomList();
+        },
+        error: function(err) {
+            alert(JSON.stringify($.parseJSON(err.responseText)));  
+        }
+    }); 
+});

jsfiddles/event_stream/demo.css

@@ -0,0 +1,17 @@
+.loggedin {
+    visibility: hidden;
+}
+
+p {
+    font-family: monospace;
+}
+
+table
+{
+    border-spacing:5px;
+}
+
+th,td
+{
+    padding:5px;
+}

jsfiddles/event_stream/demo.html

@@ -0,0 +1,23 @@
+<div>
+    <p>This event stream demo requires a home server to be running on http://localhost:8080</p>
+</div>
+<form class="loginForm">
+    <input type="text" id="userLogin" placeholder="Username"></input>
+    <input type="password" id="passwordLogin" placeholder="Password"></input>
+    <input type="button" class="login" value="Login"></input>
+</form>
+<div class="loggedin">
+    <form class="sendMessageForm">
+        <input type="button" class="sendMessage" value="Send random message"></input>
+    </form>
+    <p id="streamErrorText"></p>
+    <table id="rooms">
+        <tbody>
+            <tr>
+                <th>Room ID</th>
+                <th>Latest message</th>
+            </tr>
+        </tbody>
+    </table>
+</div>
+    

jsfiddles/event_stream/demo.js

@@ -0,0 +1,142 @@
+var accountInfo = {};
+
+var eventStreamInfo = {
+    from: "END"
+};
+
+var roomInfo = [];
+
+var longpollEventStream = function() {
+    var url = "http://localhost:8080/matrix/client/api/v1/events?access_token=$token&from=$from";
+    url = url.replace("$token", accountInfo.access_token);
+    url = url.replace("$from", eventStreamInfo.from);
+    
+    $.getJSON(url, function(data) {
+        eventStreamInfo.from = data.end;
+        
+        var hasNewLatestMessage = false;
+        for (var i=0; i<data.chunk.length; ++i) {
+            if (data.chunk[i].type === "m.room.message") {
+                for (var j=0; j<roomInfo.length; ++j) {
+                    if (roomInfo[j].room_id === data.chunk[i].room_id) {
+                        roomInfo[j].latest_message = data.chunk[i].content.body;
+                        hasNewLatestMessage = true;
+                    }
+                }
+            }
+        }
+        
+        if (hasNewLatestMessage) {
+           setRooms(roomInfo);
+        }
+        $("#streamErrorText").text("");
+        longpollEventStream();
+    }).fail(function(err) {
+        $("#streamErrorText").text("Event stream error: "+JSON.stringify($.parseJSON(err.responseText)));
+        setTimeout(longpollEventStream, 5000);
+    });
+};
+
+var showLoggedIn = function(data) {
+    accountInfo = data;
+    longpollEventStream();
+    getCurrentRoomList();
+    $(".loggedin").css({visibility: "visible"});
+};
+
+$('.login').live('click', function() {
+    var user = $("#userLogin").val();
+    var password = $("#passwordLogin").val();
+    $.ajax({
+        url: "http://localhost:8080/matrix/client/api/v1/login",
+        type: "POST",
+        contentType: "application/json; charset=utf-8",
+        data: JSON.stringify({ user: user, password: password, type: "m.login.password" }),
+        dataType: "json",
+        success: function(data) {
+            $("#rooms").find("tr:gt(0)").remove();
+            showLoggedIn(data);
+        },
+        error: function(err) {
+            alert(JSON.stringify($.parseJSON(err.responseText)));  
+        }
+    }); 
+});
+
+var getCurrentRoomList = function() {
+    $("#roomId").val("");
+    var url = "http://localhost:8080/matrix/client/api/v1/im/sync?access_token=" + accountInfo.access_token + "&from=END&to=START&limit=1";
+    $.getJSON(url, function(data) {
+        for (var i=0; i<data.length; ++i) {
+            if ("messages" in data[i]) {
+                data[i].latest_message = data[i].messages.chunk[0].content.body;   
+            }
+        }
+        roomInfo = data;
+        setRooms(roomInfo);  
+    }).fail(function(err) {
+        alert(JSON.stringify($.parseJSON(err.responseText)));
+    });
+};
+
+$('.sendMessage').live('click', function() {
+    if (roomInfo.length === 0) {
+        alert("There is no room to send a message to!");
+        return;
+    }
+    
+    var index = Math.floor(Math.random() * roomInfo.length);
+    
+    sendMessage(roomInfo[index].room_id);
+});
+
+var sendMessage = function(roomId) {
+    var body = "jsfiddle message @" + $.now();
+    var msgId = $.now();
+    
+    if (roomId.length === 0) {
+        return;
+    }
+    
+    var url = "http://localhost:8080/matrix/client/api/v1/rooms/$roomid/messages/$user/$msgid?access_token=$token";
+    url = url.replace("$token", accountInfo.access_token);
+    url = url.replace("$roomid", encodeURIComponent(roomId));
+    url = url.replace("$user", encodeURIComponent(accountInfo.user_id));
+    url = url.replace("$msgid", msgId);
+    
+    var data = {
+        msgtype: "m.text",
+        body: body
+    };
+    
+    $.ajax({
+        url: url,
+        type: "PUT",
+        contentType: "application/json; charset=utf-8",
+        data: JSON.stringify(data),
+        dataType: "json",
+        success: function(data) {
+            $("#messageBody").val("");
+        },
+        error: function(err) {
+            alert(JSON.stringify($.parseJSON(err.responseText)));  
+        }
+    }); 
+};
+
+var setRooms = function(roomList) {
+    // wipe existing entries
+    $("#rooms").find("tr:gt(0)").remove();
+    
+    var rows = "";
+    for (var i=0; i<roomList.length; ++i) {
+        row = "<tr>" +
+            "<td>"+roomList[i].room_id+"</td>" +
+            "<td>"+roomList[i].latest_message+"</td>" +
+            "</tr>";  
+        rows += row;
+    }
+    
+    $("#rooms").append(rows);
+};
+

jsfiddles/example_app/demo.css

@@ -0,0 +1,43 @@
+.roomListDashboard, .roomContents, .sendMessageForm {
+    visibility: hidden;
+}
+
+.roomList {
+    background-color: #909090;
+}
+
+.messageWrapper {
+    background-color: #EEEEEE;
+    height: 400px;
+    overflow: scroll;
+}
+
+.membersWrapper {
+    background-color: #EEEEEE;
+    height: 200px;
+    width: 50%;
+    overflow: scroll;
+}
+
+.textEntry {
+    width: 100%
+}
+
+p {
+    font-family: monospace;
+}
+
+table
+{
+    border-spacing:5px;
+}
+
+th,td
+{
+    padding:5px;
+}
+
+.roomList tr:not(:first-child):hover {
+    background-color: orange;
+    cursor: pointer;
+}

jsfiddles/example_app/demo.html

@@ -0,0 +1,56 @@
+<div class="signUp">
+    <p>Matrix example application: Requires a local home server running at http://localhost:8080</p>
+    <form class="registrationForm">
+        <p>No account? Register:</p>
+        <input type="text" id="userReg" placeholder="Username"></input>
+        <input type="password" id="passwordReg" placeholder="Password"></input>
+        <input type="button" class="register" value="Register"></input>
+    </form>
+    <form class="loginForm">
+        <p>Got an account? Login:</p>
+        <input type="text" id="userLogin" placeholder="Username"></input>
+        <input type="password" id="passwordLogin" placeholder="Password"></input>
+        <input type="button" class="login" value="Login"></input>
+    </form>
+</div>
+
+<div class="roomListDashboard">
+    <form class="createRoomForm">
+        <input type="text" id="roomAlias" placeholder="Room alias"></input>
+        <input type="button" class="createRoom" value="Create Room"></input>
+    </form>
+    <table id="rooms" class="roomList">
+        <tbody>
+            <tr>
+                <th>Room</th>
+                <th>My state</th>
+                <th>Latest message</th>
+            </tr>
+        </tbody>
+    </table>
+</div>
+
+<div class="roomContents">
+    <p id="roomName">Select a room</p>
+    <div class="messageWrapper">
+        <table id="messages">
+            <tbody>
+            </tbody>
+        </table>
+    </div>
+    <form class="sendMessageForm">
+        <input type="text" class="textEntry" id="body" placeholder="Enter text here..." onkeydown="javascript:if (event.keyCode == 13) document.getElementById('sendMsg').focus()"></input>
+        <input type="button" class="sendMessage" id="sendMsg" value="Send"></input>
+    </form>
+</div>
+
+<div>
+    <p>Member list:</p>
+    <div class="membersWrapper">
+        <table id="members">
+            <tbody>
+            </tbody>
+        </table>
+    </div>
+</div>
+    

jsfiddles/example_app/demo.js

@@ -0,0 +1,303 @@
+var accountInfo = {};
+
+var eventStreamInfo = {
+    from: "END"
+};
+
+var roomInfo = [];
+var memberInfo = [];
+var viewingRoomId;
+
+// ************** Event Streaming **************
+var longpollEventStream = function() {
+    var url = "http://localhost:8080/matrix/client/api/v1/events?access_token=$token&from=$from";
+    url = url.replace("$token", accountInfo.access_token);
+    url = url.replace("$from", eventStreamInfo.from);
+
+    $.getJSON(url, function(data) {
+        eventStreamInfo.from = data.end;
+        
+        var hasNewLatestMessage = false;
+        var updatedMemberList = false;
+        var i=0;
+        var j=0;
+        for (i=0; i<data.chunk.length; ++i) {
+            if (data.chunk[i].type === "m.room.message") {
+                console.log("Got new message: " + JSON.stringify(data.chunk[i]));
+                if (viewingRoomId === data.chunk[i].room_id) {
+                    addMessage(data.chunk[i]);
+                }
+                
+                for (j=0; j<roomInfo.length; ++j) {
+                    if (roomInfo[j].room_id === data.chunk[i].room_id) {
+                        roomInfo[j].latest_message = data.chunk[i].content.body;
+                        hasNewLatestMessage = true;
+                    }
+                }
+            }
+            else if (data.chunk[i].type === "m.room.member") {
+                if (viewingRoomId === data.chunk[i].room_id) {
+                    console.log("Got new member: " + JSON.stringify(data.chunk[i]));
+                    for (j=0; j<memberInfo.length; ++j) {
+                        if (memberInfo[j].target_user_id === data.chunk[i].target_user_id) {
+                            memberInfo[j] = data.chunk[i];
+                            updatedMemberList = true;
+                            break;
+                        }
+                    }
+                    if (!updatedMemberList) {
+                        memberInfo.push(data.chunk[i]);  
+                        updatedMemberList = true;
+                    }
+                }
+                if (data.chunk[i].target_user_id === accountInfo.user_id) {
+                    getCurrentRoomList(); // update our join/invite list
+                }
+            }
+            else {
+                console.log("Discarding: " + JSON.stringify(data.chunk[i]));
+            }
+        }
+        
+        if (hasNewLatestMessage) {
+           setRooms(roomInfo);
+        }
+        if (updatedMemberList) {
+            $("#members").empty();
+            for (i=0; i<memberInfo.length; ++i) { 
+                addMember(memberInfo[i]);
+            }
+        }
+        longpollEventStream();
+    }).fail(function(err) {
+        setTimeout(longpollEventStream, 5000);
+    });
+};
+
+// ************** Registration and Login **************
+var onLoggedIn = function(data) {
+    accountInfo = data;
+    longpollEventStream();
+    getCurrentRoomList();
+    $(".roomListDashboard").css({visibility: "visible"});
+    $(".roomContents").css({visibility: "visible"});
+    $(".signUp").css({display: "none"});
+};
+
+$('.login').live('click', function() {
+    var user = $("#userLogin").val();
+    var password = $("#passwordLogin").val();
+    $.ajax({
+        url: "http://localhost:8080/matrix/client/api/v1/login",
+        type: "POST",
+        contentType: "application/json; charset=utf-8",
+        data: JSON.stringify({ user: user, password: password, type: "m.login.password" }),
+        dataType: "json",
+        success: function(data) {
+            onLoggedIn(data);
+        },
+        error: function(err) {
+            alert("Unable to login: is the home server running?");  
+        }
+    }); 
+});
+
+$('.register').live('click', function() {
+    var user = $("#userReg").val();
+    var password = $("#passwordReg").val();
+    $.ajax({
+        url: "http://localhost:8080/matrix/client/api/v1/register",
+        type: "POST",
+        contentType: "application/json; charset=utf-8",
+        data: JSON.stringify({ user_id: user, password: password }),
+        dataType: "json",
+        success: function(data) {
+            onLoggedIn(data);
+        },
+        error: function(err) {
+            var msg = "Is the home server running?";
+            var errJson = $.parseJSON(err.responseText);
+            if (errJson !== null) {
+                msg = errJson.error;   
+            }
+            alert("Unable to register: "+msg);  
+        }
+    });
+});
+
+// ************** Creating a room ******************
+$('.createRoom').live('click', function() {
+    var roomAlias = $("#roomAlias").val();
+    var data = {};
+    if (roomAlias.length > 0) {
+        data.room_alias_name = roomAlias;   
+    }
+    $.ajax({
+        url: "http://localhost:8080/matrix/client/api/v1/rooms?access_token="+accountInfo.access_token,
+        type: "POST",
+        contentType: "application/json; charset=utf-8",
+        data: JSON.stringify(data),
+        dataType: "json",
+        success: function(response) {
+            $("#roomAlias").val("");
+            response.membership = "join"; // you are automatically joined into every room you make.
+            response.latest_message = "";
+            
+            roomInfo.push(response);
+            setRooms(roomInfo);
+        },
+        error: function(err) {
+            alert(JSON.stringify($.parseJSON(err.responseText)));  
+        }
+    }); 
+});
+
+// ************** Getting current state **************
+var getCurrentRoomList = function() {
+    var url = "http://localhost:8080/matrix/client/api/v1/im/sync?access_token=" + accountInfo.access_token + "&from=END&to=START&limit=1";
+    $.getJSON(url, function(data) {
+        for (var i=0; i<data.length; ++i) {
+            if ("messages" in data[i]) {
+                data[i].latest_message = data[i].messages.chunk[0].content.body;   
+            }
+        }
+        roomInfo = data;
+        setRooms(roomInfo);  
+    }).fail(function(err) {
+        alert(JSON.stringify($.parseJSON(err.responseText)));
+    });
+};
+
+var loadRoomContent = function(roomId) {
+    console.log("loadRoomContent " + roomId);
+    viewingRoomId = roomId;
+    $("#roomName").text("Room: "+roomId);
+    $(".sendMessageForm").css({visibility: "visible"});
+    getMessages(roomId);
+    getMemberList(roomId);
+};
+
+var getMessages = function(roomId) {
+    $("#messages").empty();
+    var url = "http://localhost:8080/matrix/client/api/v1/rooms/" + roomId + "/messages/list?access_token=" + accountInfo.access_token + "&from=END&to=START&limit=10";
+    $.getJSON(url, function(data) {
+        for (var i=data.chunk.length-1; i>=0; --i) {
+            addMessage(data.chunk[i]);   
+        }
+    });
+};
+
+var getMemberList = function(roomId) {
+    $("#members").empty();
+    memberInfo = [];
+    var url = "http://localhost:8080/matrix/client/api/v1/rooms/" + roomId + "/members/list?access_token=" + accountInfo.access_token;
+    $.getJSON(url, function(data) {
+        for (var i=0; i<data.chunk.length; ++i) {
+            memberInfo.push(data.chunk[i]);
+            addMember(data.chunk[i]);   
+        }
+    });
+};
+
+// ************** Sending messages **************
+$('.sendMessage').live('click', function() {
+    if (viewingRoomId === undefined) {
+        alert("There is no room to send a message to!");
+        return;
+    }
+    var body = $("#body").val();
+    sendMessage(viewingRoomId, body);
+});
+
+var sendMessage = function(roomId, body) {
+    var msgId = $.now();
+    
+    var url = "http://localhost:8080/matrix/client/api/v1/rooms/$roomid/messages/$user/$msgid?access_token=$token";
+    url = url.replace("$token", accountInfo.access_token);
+    url = url.replace("$roomid", encodeURIComponent(roomId));
+    url = url.replace("$user", encodeURIComponent(accountInfo.user_id));
+    url = url.replace("$msgid", msgId);
+    
+    var data = {
+        msgtype: "m.text",
+        body: body
+    };
+    
+    $.ajax({
+        url: url,
+        type: "PUT",
+        contentType: "application/json; charset=utf-8",
+        data: JSON.stringify(data),
+        dataType: "json",
+        success: function(data) {
+            $("#body").val("");
+        },
+        error: function(err) {
+            alert(JSON.stringify($.parseJSON(err.responseText)));  
+        }
+    });
+};
+
+// ************** Navigation and DOM manipulation **************
+var setRooms = function(roomList) {
+    // wipe existing entries
+    $("#rooms").find("tr:gt(0)").remove();
+    
+    var rows = "";
+    for (var i=0; i<roomList.length; ++i) {
+        row = "<tr>" +
+              "<td>"+roomList[i].room_id+"</td>" +
+              "<td>"+roomList[i].membership+"</td>" +
+              "<td>"+roomList[i].latest_message+"</td>" +
+              "</tr>";  
+        rows += row;
+    }
+    
+    $("#rooms").append(rows);
+    
+    $('#rooms').find("tr").click(function(){
+        var roomId = $(this).find('td:eq(0)').text();
+        var membership = $(this).find('td:eq(1)').text();
+        if (membership !== "join") {
+            console.log("Joining room " + roomId); 
+            var url = "http://localhost:8080/matrix/client/api/v1/rooms/$roomid/members/$user/state?access_token=$token";
+            url = url.replace("$token", accountInfo.access_token);
+            url = url.replace("$roomid", encodeURIComponent(roomId));
+            url = url.replace("$user", encodeURIComponent(accountInfo.user_id));
+            $.ajax({
+                url: url,
+                type: "PUT",
+                contentType: "application/json; charset=utf-8",
+                data: JSON.stringify({membership: "join"}),
+                dataType: "json",
+                success: function(data) {
+                    loadRoomContent(roomId);
+                    getCurrentRoomList();
+                },
+                error: function(err) {
+                    alert(JSON.stringify($.parseJSON(err.responseText)));  
+                }
+            });
+        }
+        else {
+            loadRoomContent(roomId);
+        }
+    });
+};
+
+var addMessage = function(data) {
+    var row = "<tr>" +
+              "<td>"+data.user_id+"</td>" +
+              "<td>"+data.content.body+"</td>" +
+              "</tr>"; 
+    $("#messages").append(row);
+};
+
+var addMember = function(data) {
+    var row = "<tr>" +
+              "<td>"+data.target_user_id+"</td>" +
+              "<td>"+data.content.membership+"</td>" +
+              "</tr>"; 
+    $("#members").append(row);
+};
+

jsfiddles/register_login/demo.css

@@ -0,0 +1,7 @@
+.loggedin {
+    visibility: hidden;
+}
+
+p {
+    font-family: monospace;
+}

jsfiddles/register_login/demo.html

@@ -0,0 +1,20 @@
+<div>
+    <p>This registration/login demo requires a home server to be running on http://localhost:8080</p>
+</div>
+<form class="registrationForm">
+    <input type="text" id="user" placeholder="Username"></input>
+    <input type="password" id="password" placeholder="Password"></input>
+    <input type="button" class="register" value="Register"></input>
+</form>
+<form class="loginForm">
+    <input type="text" id="userLogin" placeholder="Username"></input>
+    <input type="password" id="passwordLogin" placeholder="Password"></input>
+    <input type="button" class="login" value="Login"></input>
+</form>
+<div class="loggedin">
+    <p id="welcomeText"></p>
+    <input type="button" class="testToken" value="Test token"></input>
+    <input type="button" class="logout" value="Logout"></input>
+    <p id="imSyncText"></p>
+</div>
+    

jsfiddles/register_login/demo.js

@@ -0,0 +1,69 @@
+var accountInfo = {};
+
+var showLoggedIn = function(data) {
+    accountInfo = data;
+    $(".loggedin").css({visibility: "visible"});
+    $("#welcomeText").text("Welcome " + accountInfo.user_id+". Your access token is:  " +
+                           accountInfo.access_token);    
+};
+
+$('.register').live('click', function() {
+    var user = $("#user").val();
+    var password = $("#password").val();
+    $.ajax({
+        url: "http://localhost:8080/matrix/client/api/v1/register",
+        type: "POST",
+        contentType: "application/json; charset=utf-8",
+        data: JSON.stringify({ user_id: user, password: password }),
+        dataType: "json",
+        success: function(data) {
+            showLoggedIn(data);
+        },
+        error: function(err) {
+            alert(JSON.stringify($.parseJSON(err.responseText)));  
+        }
+    });
+});
+
+var login = function(user, password) {
+    $.ajax({
+        url: "http://localhost:8080/matrix/client/api/v1/login",
+        type: "POST",
+        contentType: "application/json; charset=utf-8",
+        data: JSON.stringify({ user: user, password: password, type: "m.login.password" }),
+        dataType: "json",
+        success: function(data) {
+            showLoggedIn(data);
+        },
+        error: function(err) {
+            alert(JSON.stringify($.parseJSON(err.responseText)));  
+        }
+    });  
+};
+
+$('.login').live('click', function() {
+    var user = $("#userLogin").val();
+    var password = $("#passwordLogin").val();
+    $.getJSON("http://localhost:8080/matrix/client/api/v1/login", function(data) {
+        if (data.type !== "m.login.password") {
+            alert("I don't know how to login with this type: " + data.type);
+            return;
+        }
+        login(user, password);
+    });
+});
+
+$('.logout').live('click', function() {
+    accountInfo = {};
+    $("#imSyncText").text("");
+    $(".loggedin").css({visibility: "hidden"});
+});
+
+$('.testToken').live('click', function() {
+    var url = "http://localhost:8080/matrix/client/api/v1/im/sync?access_token=" + accountInfo.access_token + "&from=END&to=START&limit=1";
+    $.getJSON(url, function(data) {
+         $("#imSyncText").text(JSON.stringify(data, undefined, 2));
+    }).fail(function(err) {
+        $("#imSyncText").text(JSON.stringify($.parseJSON(err.responseText)));
+    });
+});

jsfiddles/room_memberships/demo.css

@@ -0,0 +1,17 @@
+.loggedin {
+    visibility: hidden;
+}
+
+p {
+    font-family: monospace;
+}
+
+table
+{
+    border-spacing:5px;
+}
+
+th,td
+{
+    padding:5px;
+}

jsfiddles/room_memberships/demo.html

@@ -0,0 +1,37 @@
+<div>
+    <p>This room membership demo requires a home server to be running on http://localhost:8080</p>
+</div>
+<form class="loginForm">
+    <input type="text" id="userLogin" placeholder="Username"></input>
+    <input type="password" id="passwordLogin" placeholder="Password"></input>
+    <input type="button" class="login" value="Login"></input>
+</form>
+<div class="loggedin">
+    <form class="createRoomForm">
+        <input type="button" class="createRoom" value="Create Room"></input>
+    </form>
+    <form class="changeMembershipForm">
+        <input type="text" id="roomId" placeholder="Room ID"></input>
+        <input type="text" id="targetUser" placeholder="Target User ID"></input>
+        <select id="membership">
+            <option value="invite">Invite</option>
+            <option value="join">Join</option>
+            <option value="leave">Leave</option>
+        </select>
+        <input type="button" class="changeMembership" value="Change Membership"></input>
+    </form>
+    <form class="joinAliasForm">
+        <input type="text" id="roomAlias" placeholder="Room Alias (#name:domain)"></input>
+        <input type="button" class="joinAlias" value="Join via Alias"></input>
+    </form>
+    <table id="rooms">
+        <tbody>
+            <tr>
+                <th>Room ID</th>
+                <th>My state</th>
+                <th>Room Alias</th>
+            </tr>
+        </tbody>
+    </table>
+</div>
+    

jsfiddles/room_memberships/demo.js

@@ -0,0 +1,139 @@
+var accountInfo = {};
+
+var showLoggedIn = function(data) {
+    accountInfo = data;
+    getCurrentRoomList();
+    $(".loggedin").css({visibility: "visible"});
+};
+
+$('.login').live('click', function() {
+    var user = $("#userLogin").val();
+    var password = $("#passwordLogin").val();
+    $.ajax({
+        url: "http://localhost:8080/matrix/client/api/v1/login",
+        type: "POST",
+        contentType: "application/json; charset=utf-8",
+        data: JSON.stringify({ user: user, password: password, type: "m.login.password" }),
+        dataType: "json",
+        success: function(data) {
+            $("#rooms").find("tr:gt(0)").remove();
+            showLoggedIn(data);
+        },
+        error: function(err) {
+            alert(JSON.stringify($.parseJSON(err.responseText)));  
+        }
+    }); 
+});
+
+var getCurrentRoomList = function() {
+    $("#roomId").val("");
+    // wipe the table and reload it. Using the event stream would be the best
+    // solution but that is out of scope of this fiddle.
+    $("#rooms").find("tr:gt(0)").remove();
+    
+    var url = "http://localhost:8080/matrix/client/api/v1/im/sync?access_token=" + accountInfo.access_token + "&from=END&to=START&limit=1";
+    $.getJSON(url, function(data) {
+        for (var i=0; i<data.length; ++i) {
+            addRoom(data[i]);   
+        }
+    }).fail(function(err) {
+        alert(JSON.stringify($.parseJSON(err.responseText)));
+    });
+};
+
+$('.createRoom').live('click', function() {
+    var data = {};
+    $.ajax({
+        url: "http://localhost:8080/matrix/client/api/v1/rooms?access_token="+accountInfo.access_token,
+        type: "POST",
+        contentType: "application/json; charset=utf-8",
+        data: JSON.stringify(data),
+        dataType: "json",
+        success: function(data) {
+            data.membership = "join"; // you are automatically joined into every room you make.
+            data.latest_message = "";
+            addRoom(data);
+        },
+        error: function(err) {
+            alert(JSON.stringify($.parseJSON(err.responseText)));  
+        }
+    }); 
+});
+
+var addRoom = function(data) {
+    row = "<tr>" +
+        "<td>"+data.room_id+"</td>" +
+        "<td>"+data.membership+"</td>" +
+        "<td>"+data.room_alias+"</td>" +
+        "</tr>";
+    $("#rooms").append(row);
+};
+
+$('.changeMembership').live('click', function() {
+    var roomId = $("#roomId").val();
+    var member = $("#targetUser").val();
+    var membership = $("#membership").val();
+    
+    if (roomId.length === 0) {
+        return;
+    }
+    
+    var url = "http://localhost:8080/matrix/client/api/v1/rooms/$roomid/members/$user/state?access_token=$token";
+    url = url.replace("$token", accountInfo.access_token);
+    url = url.replace("$roomid", encodeURIComponent(roomId));
+    url = url.replace("$user", encodeURIComponent(member));
+    
+    if (membership === "leave") {
+        $.ajax({
+            url: url,
+            type: "DELETE",
+            contentType: "application/json; charset=utf-8",
+            dataType: "json",
+            success: function(data) {
+                getCurrentRoomList();
+            },
+            error: function(err) {
+                alert(JSON.stringify($.parseJSON(err.responseText)));  
+            }
+        });
+    }
+    else {
+        var data = {
+            membership: membership
+        };
+        
+        $.ajax({
+            url: url,
+            type: "PUT",
+            contentType: "application/json; charset=utf-8",
+            data: JSON.stringify(data),
+            dataType: "json",
+            success: function(data) {
+                getCurrentRoomList();
+            },
+            error: function(err) {
+                alert(JSON.stringify($.parseJSON(err.responseText)));  
+            }
+        }); 
+    }
+});
+
+$('.joinAlias').live('click', function() {
+    var roomAlias = $("#roomAlias").val();
+    var url = "http://localhost:8080/matrix/client/api/v1/join/$roomalias?access_token=$token";
+    url = url.replace("$token", accountInfo.access_token);
+    url = url.replace("$roomalias", encodeURIComponent(roomAlias));
+    $.ajax({
+        url: url,
+        type: "PUT",
+        contentType: "application/json; charset=utf-8",
+        data: JSON.stringify({}),
+        dataType: "json",
+        success: function(data) {
+            getCurrentRoomList();
+        },
+        error: function(err) {
+            alert(JSON.stringify($.parseJSON(err.responseText)));  
+        }
+    }); 
+});

nuke-room-from-db.sh

@@ -0,0 +1,24 @@
+#!/bin/bash
+
+## CAUTION:
+## This script will remove (hopefully) all trace of the given room ID from
+## your homeserver.db
+
+## Do not run it lightly.
+
+ROOMID="$1"
+
+sqlite3 homeserver.db <<EOF
+DELETE FROM context_depth WHERE context = '$ROOMID';
+DELETE FROM current_state WHERE context = '$ROOMID';
+DELETE FROM feedback WHERE room_id = '$ROOMID';
+DELETE FROM messages WHERE room_id = '$ROOMID';
+DELETE FROM pdu_backward_extremities WHERE context = '$ROOMID';
+DELETE FROM pdu_edges WHERE context = '$ROOMID';
+DELETE FROM pdu_forward_extremities WHERE context = '$ROOMID';
+DELETE FROM pdus WHERE context = '$ROOMID';
+DELETE FROM room_data WHERE room_id = '$ROOMID';
+DELETE FROM room_memberships WHERE room_id = '$ROOMID';
+DELETE FROM rooms WHERE room_id = '$ROOMID';
+DELETE FROM state_pdus WHERE context = '$ROOMID';
+EOF

setup.py

@@ -1,3 +1,5 @@
+#!/usr/bin/env python
+
 # Copyright 2014 matrix.org
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
@@ -25,7 +27,7 @@ def read(fname):
 
 setup(
     name="SynapseHomeServer",
-    version="0.1",
+    version="0.0.1",
     packages=find_packages(exclude=["tests"]),
     description="Reference Synapse Home Server",
     install_requires=[

synapse/__init__.py

@@ -15,3 +15,5 @@
 
 """ This is a reference implementation of a synapse home server.
 """
+
+__version__ = "0.1.1"

synapse/api/auth.py

@@ -18,9 +18,8 @@
 from twisted.internet import defer
 
 from synapse.api.constants import Membership
-from synapse.api.errors import AuthError, StoreError
-from synapse.api.events.room import (RoomTopicEvent, RoomMemberEvent,
-                                     MessageEvent, FeedbackEvent)
+from synapse.api.errors import AuthError, StoreError, Codes
+from synapse.api.events.room import RoomMemberEvent
 
 import logging
 
@@ -34,7 +33,7 @@ class Auth(object):
         self.store = hs.get_datastore()
 
     @defer.inlineCallbacks
-    def check(self, event, raises=False):
+    def check(self, event, snapshot, raises=False):
         """ Checks if this event is correctly authed.
 
         Returns:
@@ -44,15 +43,19 @@ class Auth(object):
             be raised only if raises=True.
         """
         try:
-            if event.type in [RoomTopicEvent.TYPE, MessageEvent.TYPE,
-                              FeedbackEvent.TYPE]:
-                yield self.check_joined_room(event.room_id, event.user_id)
-                defer.returnValue(True)
-            elif event.type == RoomMemberEvent.TYPE:
-                allowed = yield self.is_membership_change_allowed(event)
-                defer.returnValue(allowed)
+            if hasattr(event, "room_id"):
+                if event.type == RoomMemberEvent.TYPE:
+                    allowed = yield self.is_membership_change_allowed(event)
+                    defer.returnValue(allowed)
+                else:
+                    self._check_joined_room(
+                        member=snapshot.membership_state,
+                        user_id=snapshot.user_id,
+                        room_id=snapshot.room_id,
+                    )
+                    defer.returnValue(True)
             else:
-                raise AuthError(500, "Unknown event type %s" % event.type)
+                raise AuthError(500, "Unknown event: %s" % event)
         except AuthError as e:
             logger.info("Event auth check failed on event %s with msg: %s",
                         event, e.msg)
@@ -67,16 +70,22 @@ class Auth(object):
                 room_id=room_id,
                 user_id=user_id
             )
-            if not member or member.membership != Membership.JOIN:
-                raise AuthError(403, "User %s not in room %s" %
-                                (user_id, room_id))
+            self._check_joined_room(member, user_id, room_id)
             defer.returnValue(member)
         except AttributeError:
             pass
         defer.returnValue(None)
 
+    def _check_joined_room(self, member, user_id, room_id):
+        if not member or member.membership != Membership.JOIN:
+            raise AuthError(403, "User %s not in room %s (%s)" % (
+                user_id, room_id, repr(member)
+            ))
+
     @defer.inlineCallbacks
     def is_membership_change_allowed(self, event):
+        target_user_id = event.state_key
+
         # does this room even exist
         room = yield self.store.get_room(event.room_id)
         if not room:
@@ -94,7 +103,7 @@ class Auth(object):
         # get info about the target
         try:
             target = yield self.store.get_room_member(
-                user_id=event.target_user_id,
+                user_id=target_user_id,
                 room_id=event.room_id)
         except:
             target = None
@@ -108,12 +117,12 @@ class Auth(object):
                 raise AuthError(403, "You are not in room %s." % event.room_id)
             elif target_in_room:  # the target is already in the room.
                 raise AuthError(403, "%s is already in the room." %
-                                     event.target_user_id)
+                                     target_user_id)
         elif Membership.JOIN == membership:
             # Joins are valid iff caller == target and they were:
             # invited: They are accepting the invitation
             # joined: It's a NOOP
-            if event.user_id != event.target_user_id:
+            if event.user_id != target_user_id:
                 raise AuthError(403, "Cannot force another user to join.")
             elif room.is_public:
                 pass  # anyone can join public rooms.
@@ -123,10 +132,10 @@ class Auth(object):
         elif Membership.LEAVE == membership:
             if not caller_in_room:  # trying to leave a room you aren't joined
                 raise AuthError(403, "You are not in room %s." % event.room_id)
-            elif event.target_user_id != event.user_id:
+            elif target_user_id != event.user_id:
                 # trying to force another user to leave
                 raise AuthError(403, "Cannot force %s to leave." %
-                                event.target_user_id)
+                                target_user_id)
         else:
             raise AuthError(500, "Unknown membership %s" % membership)
 
@@ -161,6 +170,9 @@ class Auth(object):
         """
         try:
             user_id = yield self.store.get_user_by_token(token=token)
+            if not user_id:
+                raise StoreError()
             defer.returnValue(self.hs.parse_userid(user_id))
         except StoreError:
-            raise AuthError(403, "Unrecognised access token.")
+            raise AuthError(403, "Unrecognised access token.",
+                            errcode=Codes.UNKNOWN_TOKEN)

synapse/api/constants.py

@@ -23,6 +23,7 @@ class Membership(object):
     JOIN = u"join"
     KNOCK = u"knock"
     LEAVE = u"leave"
+    LIST = (INVITE, JOIN, KNOCK, LEAVE)
 
 
 class Feedback(object):
@@ -30,8 +31,8 @@ class Feedback(object):
     """Represents the types of feedback a user can send in response to a
     message."""
 
-    DELIVERED = u"d"
-    READ = u"r"
+    DELIVERED = u"delivered"
+    READ = u"read"
     LIST = (DELIVERED, READ)
 
 

synapse/api/errors.py

@@ -27,6 +27,7 @@ class Codes(object):
     BAD_PAGINATION = "M_BAD_PAGINATION"
     UNKNOWN = "M_UNKNOWN"
     NOT_FOUND = "M_NOT_FOUND"
+    UNKNOWN_TOKEN = "M_UNKNOWN_TOKEN"
 
 
 class CodeMessageException(Exception):
@@ -74,7 +75,10 @@ class AuthError(SynapseError):
 
 class EventStreamError(SynapseError):
     """An error raised when there a problem with the event stream."""
-    pass
+    def __init__(self, *args, **kwargs):
+        if "errcode" not in kwargs:
+            kwargs["errcode"] = Codes.BAD_PAGINATION
+        super(EventStreamError, self).__init__(*args, **kwargs)
 
 
 class LoginError(SynapseError):

synapse/api/events/__init__.py

@@ -41,16 +41,17 @@ class SynapseEvent(JsonEncodedObject):
         "room_id",
         "user_id",  # sender/initiator
         "content",  # HTTP body, JSON
+        "state_key",
     ]
 
     internal_keys = [
         "is_state",
-        "state_key",
         "prev_events",
         "prev_state",
         "depth",
         "destinations",
         "origin",
+        "outlier",
     ]
 
     required_keys = [

synapse/api/events/factory.py

@@ -15,7 +15,7 @@
 
 from synapse.api.events.room import (
     RoomTopicEvent, MessageEvent, RoomMemberEvent, FeedbackEvent,
-    InviteJoinEvent, RoomConfigEvent
+    InviteJoinEvent, RoomConfigEvent, RoomNameEvent, GenericEvent,
 )
 
 from synapse.util.stringutils import random_string
@@ -25,6 +25,7 @@ class EventFactory(object):
 
     _event_classes = [
         RoomTopicEvent,
+        RoomNameEvent,
         MessageEvent,
         RoomMemberEvent,
         FeedbackEvent,
@@ -32,20 +33,24 @@ class EventFactory(object):
         RoomConfigEvent
     ]
 
-    def __init__(self):
+    def __init__(self, hs):
         self._event_list = {}  # dict of TYPE to event class
         for event_class in EventFactory._event_classes:
             self._event_list[event_class.TYPE] = event_class
 
+        self.clock = hs.get_clock()
+
     def create_event(self, etype=None, **kwargs):
         kwargs["type"] = etype
         if "event_id" not in kwargs:
             kwargs["event_id"] = random_string(10)
 
-        try:
+        if "ts" not in kwargs:
+            kwargs["ts"] = int(self.clock.time_msec())
+
+        if etype in self._event_list:
             handler = self._event_list[etype]
-        except KeyError:  # unknown event type
-            # TODO allow custom event types.
-            raise NotImplementedError("Unknown etype=%s" % etype)
+        else:
+            handler = GenericEvent
 
         return handler(**kwargs)

synapse/api/events/room.py

@@ -13,31 +13,63 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
+from synapse.api.constants import Feedback, Membership
+from synapse.api.errors import SynapseError
 from . import SynapseEvent
 
 
+class GenericEvent(SynapseEvent):
+    def get_content_template(self):
+        return {}
+
+
 class RoomTopicEvent(SynapseEvent):
     TYPE = "m.room.topic"
 
+    internal_keys = SynapseEvent.internal_keys + [
+        "topic",
+    ]
+
     def __init__(self, **kwargs):
         kwargs["state_key"] = ""
+        if "topic" in kwargs["content"]:
+            kwargs["topic"] = kwargs["content"]["topic"]
         super(RoomTopicEvent, self).__init__(**kwargs)
 
     def get_content_template(self):
         return {"topic": u"string"}
 
 
+class RoomNameEvent(SynapseEvent):
+    TYPE = "m.room.name"
+
+    internal_keys = SynapseEvent.internal_keys + [
+        "name",
+    ]
+
+    def __init__(self, **kwargs):
+        kwargs["state_key"] = ""
+        if "name" in kwargs["content"]:
+            kwargs["name"] = kwargs["content"]["name"]
+        super(RoomNameEvent, self).__init__(**kwargs)
+
+    def get_content_template(self):
+        return {"name": u"string"}
+
+
 class RoomMemberEvent(SynapseEvent):
     TYPE = "m.room.member"
 
     valid_keys = SynapseEvent.valid_keys + [
-        "target_user_id",  # target
+        # target is the state_key
         "membership",  # action
     ]
 
     def __init__(self, **kwargs):
-        if "target_user_id" in kwargs:
-            kwargs["state_key"] = kwargs["target_user_id"]
+        if "membership" not in kwargs:
+            kwargs["membership"] = kwargs.get("content", {}).get("membership")
+        if not kwargs["membership"] in Membership.LIST:
+            raise SynapseError(400, "Bad membership value.")
         super(RoomMemberEvent, self).__init__(**kwargs)
 
     def get_content_template(self):
@@ -61,24 +93,26 @@ class MessageEvent(SynapseEvent):
 class FeedbackEvent(SynapseEvent):
     TYPE = "m.room.message.feedback"
 
-    valid_keys = SynapseEvent.valid_keys + [
-        "msg_id",  # the message ID being acknowledged
-        "msg_sender_id",  # person who is sending the feedback is 'user_id'
-        "feedback_type",  # the type of feedback (delivery, read, etc)
-    ]
+    valid_keys = SynapseEvent.valid_keys
 
     def __init__(self, **kwargs):
         super(FeedbackEvent, self).__init__(**kwargs)
+        if not kwargs["content"]["type"] in Feedback.LIST:
+            raise SynapseError(400, "Bad feedback value.")
 
     def get_content_template(self):
-        return {}
+        return {
+            "type": u"string",
+            "target_event_id": u"string",
+            "msg_sender_id": u"string"
+        }
 
 
 class InviteJoinEvent(SynapseEvent):
     TYPE = "m.room.invite_join"
 
     valid_keys = SynapseEvent.valid_keys + [
-        "target_user_id",
+        # target_user_id is the state_key
         "target_host",
     ]
 

synapse/api/notifier.py

@@ -1,191 +0,0 @@
-# -*- coding: utf-8 -*-
-# Copyright 2014 matrix.org
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-from synapse.api.constants import Membership
-from synapse.api.events.room import RoomMemberEvent
-
-from twisted.internet import defer
-from twisted.internet import reactor
-
-import logging
-
-logger = logging.getLogger(__name__)
-
-
-class Notifier(object):
-
-    def __init__(self, hs):
-        self.store = hs.get_datastore()
-        self.hs = hs
-        self.stored_event_listeners = {}
-
-    @defer.inlineCallbacks
-    def on_new_room_event(self, event, store_id):
-        """Called when there is a new room event which may potentially be sent
-        down listening users' event streams.
-
-        This function looks for interested *users* who may want to be notified
-        for this event. This is different to users requesting from the event
-        stream which looks for interested *events* for this user.
-
-        Args:
-            event (SynapseEvent): The new event, which must have a room_id
-            store_id (int): The ID of this event after it was stored with the
-            data store.
-        '"""
-        member_list = yield self.store.get_room_members(room_id=event.room_id,
-                                                        membership="join")
-        if not member_list:
-            member_list = []
-
-        member_list = [u.user_id for u in member_list]
-
-        # invites MUST prod the person being invited, who won't be in the room.
-        if (event.type == RoomMemberEvent.TYPE and
-                event.content["membership"] == Membership.INVITE):
-            member_list.append(event.target_user_id)
-
-        for user_id in member_list:
-            if user_id in self.stored_event_listeners:
-                self._notify_and_callback(
-                    user_id=user_id,
-                    event_data=event.get_dict(),
-                    stream_type=event.type,
-                    store_id=store_id)
-
-    def on_new_user_event(self, user_id, event_data, stream_type, store_id):
-        if user_id in self.stored_event_listeners:
-            self._notify_and_callback(
-                user_id=user_id,
-                event_data=event_data,
-                stream_type=stream_type,
-                store_id=store_id
-            )
-
-    def _notify_and_callback(self, user_id, event_data, stream_type, store_id):
-        logger.debug(
-            "Notifying %s of a new event.",
-            user_id
-        )
-
-        stream_ids = list(self.stored_event_listeners[user_id])
-        for stream_id in stream_ids:
-            self._notify_and_callback_stream(user_id, stream_id, event_data,
-                                             stream_type, store_id)
-
-        if not self.stored_event_listeners[user_id]:
-            del self.stored_event_listeners[user_id]
-
-    def _notify_and_callback_stream(self, user_id, stream_id, event_data,
-                                    stream_type, store_id):
-
-        event_listener = self.stored_event_listeners[user_id].pop(stream_id)
-        return_event_object = {
-            k: event_listener[k] for k in ["start", "chunk", "end"]
-        }
-
-        # work out the new end token
-        token = event_listener["start"]
-        end = self._next_token(stream_type, store_id, token)
-        return_event_object["end"] = end
-
-        # add the event to the chunk
-        chunk = event_listener["chunk"]
-        chunk.append(event_data)
-
-        # callback the defer. We know this can't have been resolved before as
-        # we always remove the event_listener from the map before resolving.
-        event_listener["defer"].callback(return_event_object)
-
-    def _next_token(self, stream_type, store_id, current_token):
-        stream_handler = self.hs.get_handlers().event_stream_handler
-        return stream_handler.get_event_stream_token(
-            stream_type,
-            store_id,
-            current_token
-        )
-
-    def store_events_for(self, user_id=None, stream_id=None, from_tok=None):
-        """Store all incoming events for this user. This should be paired with
-        get_events_for to return chunked data.
-
-        Args:
-            user_id (str): The user to monitor incoming events for.
-            stream (object): The stream that is receiving events
-            from_tok (str): The token to monitor incoming events from.
-        """
-        event_listener = {
-            "start": from_tok,
-            "chunk": [],
-            "end": from_tok,
-            "defer": defer.Deferred(),
-        }
-
-        if user_id not in self.stored_event_listeners:
-            self.stored_event_listeners[user_id] = {stream_id: event_listener}
-        else:
-            self.stored_event_listeners[user_id][stream_id] = event_listener
-
-    def purge_events_for(self, user_id=None, stream_id=None):
-        """Purges any stored events for this user.
-
-        Args:
-            user_id (str): The user to purge stored events for.
-        """
-        try:
-            del self.stored_event_listeners[user_id][stream_id]
-            if not self.stored_event_listeners[user_id]:
-                del self.stored_event_listeners[user_id]
-        except KeyError:
-            pass
-
-    def get_events_for(self, user_id=None, stream_id=None, timeout=0):
-        """Retrieve stored events for this user, waiting if necessary.
-
-        It is advisable to wrap this call in a maybeDeferred.
-
-        Args:
-            user_id (str): The user to get events for.
-            timeout (int): The time in seconds to wait before giving up.
-        Returns:
-            A Deferred or a dict containing the chunk data, depending on if
-            there was data to return yet. The Deferred callback may be None if
-            there were no events before the timeout expired.
-        """
-        logger.debug("%s is listening for events.", user_id)
-
-        try:
-            streams = self.stored_event_listeners[user_id][stream_id]["chunk"]
-            if streams:
-                logger.debug("%s returning existing chunk.", user_id)
-                return streams
-        except KeyError:
-            return None
-
-        reactor.callLater(
-            (timeout / 1000.0), self._timeout, user_id, stream_id
-        )
-        return self.stored_event_listeners[user_id][stream_id]["defer"]
-
-    def _timeout(self, user_id, stream_id):
-        try:
-            # We remove the event_listener from the map so that we can't
-            # resolve the deferred twice.
-            event_listeners = self.stored_event_listeners[user_id]
-            event_listener = event_listeners.pop(stream_id)
-            event_listener["defer"].callback(None)
-            logger.debug("%s event listening timed out.", user_id)
-        except KeyError:
-            pass

synapse/api/streams/__init__.py

@@ -1,97 +0,0 @@
-# -*- coding: utf-8 -*-
-# Copyright 2014 matrix.org
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-from synapse.api.errors import SynapseError
-
-
-class PaginationConfig(object):
-
-    """A configuration object which stores pagination parameters."""
-
-    def __init__(self, from_tok=None, to_tok=None, limit=0):
-        self.from_tok = from_tok
-        self.to_tok = to_tok
-        self.limit = limit
-
-    @classmethod
-    def from_request(cls, request, raise_invalid_params=True):
-        params = {
-            "from_tok": PaginationStream.TOK_START,
-            "to_tok": PaginationStream.TOK_END,
-            "limit": 0
-        }
-
-        query_param_mappings = [  # 3-tuple of qp_key, attribute, rules
-            ("from", "from_tok", lambda x: type(x) == str),
-            ("to", "to_tok", lambda x: type(x) == str),
-            ("limit", "limit", lambda x: x.isdigit())
-        ]
-
-        for qp, attr, is_valid in query_param_mappings:
-            if qp in request.args:
-                if is_valid(request.args[qp][0]):
-                    params[attr] = request.args[qp][0]
-                elif raise_invalid_params:
-                    raise SynapseError(400, "%s parameter is invalid." % qp)
-
-        return PaginationConfig(**params)
-
-
-class PaginationStream(object):
-
-    """ An interface for streaming data as chunks. """
-
-    TOK_START = "START"
-    TOK_END = "END"
-
-    def get_chunk(self, config=None):
-        """ Return the next chunk in the stream.
-
-        Args:
-            config (PaginationConfig): The config to aid which chunk to get.
-        Returns:
-            A dict containing the new start token "start", the new end token
-            "end" and the data "chunk" as a list.
-        """
-        raise NotImplementedError()
-
-
-class StreamData(object):
-
-    """ An interface for obtaining streaming data from a table. """
-
-    def __init__(self, hs):
-        self.hs = hs
-        self.store = hs.get_datastore()
-
-    def get_rows(self, user_id, from_pkey, to_pkey, limit):
-        """ Get event stream data between the specified pkeys.
-
-        Args:
-            user_id : The user's ID
-            from_pkey : The starting pkey.
-            to_pkey : The end pkey. May be -1 to mean "latest".
-            limit: The max number of results to return.
-        Returns:
-            A tuple containing the list of event stream data and the last pkey.
-        """
-        raise NotImplementedError()
-
-    def max_token(self):
-        """ Get the latest currently-valid token.
-
-        Returns:
-            The latest token."""
-        raise NotImplementedError()

synapse/api/streams/event.py

@@ -1,248 +0,0 @@
-# -*- coding: utf-8 -*-
-# Copyright 2014 matrix.org
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-"""This module contains classes for streaming from the event stream: /events.
-"""
-from twisted.internet import defer
-
-from synapse.api.errors import EventStreamError
-from synapse.api.events.room import (
-    RoomMemberEvent, MessageEvent, FeedbackEvent, RoomTopicEvent
-)
-from synapse.api.streams import PaginationStream, StreamData
-
-import logging
-
-logger = logging.getLogger(__name__)
-
-
-class MessagesStreamData(StreamData):
-    EVENT_TYPE = MessageEvent.TYPE
-
-    def __init__(self, hs, room_id=None, feedback=False):
-        super(MessagesStreamData, self).__init__(hs)
-        self.room_id = room_id
-        self.with_feedback = feedback
-
-    @defer.inlineCallbacks
-    def get_rows(self, user_id, from_key, to_key, limit):
-        (data, latest_ver) = yield self.store.get_message_stream(
-            user_id=user_id,
-            from_key=from_key,
-            to_key=to_key,
-            limit=limit,
-            room_id=self.room_id,
-            with_feedback=self.with_feedback
-        )
-        defer.returnValue((data, latest_ver))
-
-    @defer.inlineCallbacks
-    def max_token(self):
-        val = yield self.store.get_max_message_id()
-        defer.returnValue(val)
-
-
-class RoomMemberStreamData(StreamData):
-    EVENT_TYPE = RoomMemberEvent.TYPE
-
-    @defer.inlineCallbacks
-    def get_rows(self, user_id, from_key, to_key, limit):
-        (data, latest_ver) = yield self.store.get_room_member_stream(
-            user_id=user_id,
-            from_key=from_key,
-            to_key=to_key
-        )
-
-        defer.returnValue((data, latest_ver))
-
-    @defer.inlineCallbacks
-    def max_token(self):
-        val = yield self.store.get_max_room_member_id()
-        defer.returnValue(val)
-
-
-class FeedbackStreamData(StreamData):
-    EVENT_TYPE = FeedbackEvent.TYPE
-
-    def __init__(self, hs, room_id=None):
-        super(FeedbackStreamData, self).__init__(hs)
-        self.room_id = room_id
-
-    @defer.inlineCallbacks
-    def get_rows(self, user_id, from_key, to_key, limit):
-        (data, latest_ver) = yield self.store.get_feedback_stream(
-            user_id=user_id,
-            from_key=from_key,
-            to_key=to_key,
-            limit=limit,
-            room_id=self.room_id
-        )
-        defer.returnValue((data, latest_ver))
-
-    @defer.inlineCallbacks
-    def max_token(self):
-        val = yield self.store.get_max_feedback_id()
-        defer.returnValue(val)
-
-
-class RoomDataStreamData(StreamData):
-    EVENT_TYPE = RoomTopicEvent.TYPE  # TODO need multiple event types
-
-    def __init__(self, hs, room_id=None):
-        super(RoomDataStreamData, self).__init__(hs)
-        self.room_id = room_id
-
-    @defer.inlineCallbacks
-    def get_rows(self, user_id, from_key, to_key, limit):
-        (data, latest_ver) = yield self.store.get_room_data_stream(
-            user_id=user_id,
-            from_key=from_key,
-            to_key=to_key,
-            limit=limit,
-            room_id=self.room_id
-        )
-        defer.returnValue((data, latest_ver))
-
-    @defer.inlineCallbacks
-    def max_token(self):
-        val = yield self.store.get_max_room_data_id()
-        defer.returnValue(val)
-
-
-class EventStream(PaginationStream):
-
-    SEPARATOR = '_'
-
-    def __init__(self, user_id, stream_data_list):
-        super(EventStream, self).__init__()
-        self.user_id = user_id
-        self.stream_data = stream_data_list
-
-    @defer.inlineCallbacks
-    def fix_tokens(self, pagination_config):
-        pagination_config.from_tok = yield self.fix_token(
-            pagination_config.from_tok)
-        pagination_config.to_tok = yield self.fix_token(
-            pagination_config.to_tok)
-        defer.returnValue(pagination_config)
-
-    @defer.inlineCallbacks
-    def fix_token(self, token):
-        """Fixes unknown values in a token to known values.
-
-        Args:
-            token (str): The token to fix up.
-        Returns:
-            The fixed-up token, which may == token.
-        """
-        # replace TOK_START and TOK_END with 0_0_0 or -1_-1_-1 depending.
-        replacements = [
-            (PaginationStream.TOK_START, "0"),
-            (PaginationStream.TOK_END, "-1")
-        ]
-        for magic_token, key in replacements:
-            if magic_token == token:
-                token = EventStream.SEPARATOR.join(
-                    [key] * len(self.stream_data)
-                )
-
-        # replace -1 values with an actual pkey
-        token_segments = self._split_token(token)
-        for i, tok in enumerate(token_segments):
-            if tok == -1:
-                # add 1 to the max token because results are EXCLUSIVE from the
-                # latest version.
-                token_segments[i] = 1 + (yield self.stream_data[i].max_token())
-        defer.returnValue(EventStream.SEPARATOR.join(
-            str(x) for x in token_segments
-        ))
-
-    @defer.inlineCallbacks
-    def get_chunk(self, config=None):
-        # no support for limit on >1 streams, makes no sense.
-        if config.limit and len(self.stream_data) > 1:
-            raise EventStreamError(
-                400, "Limit not supported on multiplexed streams."
-            )
-
-        (chunk_data, next_tok) = yield self._get_chunk_data(config.from_tok,
-                                                            config.to_tok,
-                                                            config.limit)
-
-        defer.returnValue({
-            "chunk": chunk_data,
-            "start": config.from_tok,
-            "end": next_tok
-        })
-
-    @defer.inlineCallbacks
-    def _get_chunk_data(self, from_tok, to_tok, limit):
-        """ Get event data between the two tokens.
-
-        Tokens are SEPARATOR separated values representing pkey values of
-        certain tables, and the position determines the StreamData invoked
-        according to the STREAM_DATA list.
-
-        The magic value '-1' can be used to get the latest value.
-
-        Args:
-            from_tok - The token to start from.
-            to_tok - The token to end at. Must have values > from_tok or be -1.
-        Returns:
-            A list of event data.
-        Raises:
-            EventStreamError if something went wrong.
-        """
-        # sanity check
-        if (from_tok.count(EventStream.SEPARATOR) !=
-                to_tok.count(EventStream.SEPARATOR) or
-                (from_tok.count(EventStream.SEPARATOR) + 1) !=
-                len(self.stream_data)):
-            raise EventStreamError(400, "Token lengths don't match.")
-
-        chunk = []
-        next_ver = []
-        for i, (from_pkey, to_pkey) in enumerate(zip(
-            self._split_token(from_tok),
-            self._split_token(to_tok)
-        )):
-            if from_pkey == to_pkey:
-                # tokens are the same, we have nothing to do.
-                next_ver.append(str(to_pkey))
-                continue
-
-            (event_chunk, max_pkey) = yield self.stream_data[i].get_rows(
-                self.user_id, from_pkey, to_pkey, limit
-            )
-
-            chunk += event_chunk
-            next_ver.append(str(max_pkey))
-
-        defer.returnValue((chunk, EventStream.SEPARATOR.join(next_ver)))
-
-    def _split_token(self, token):
-        """Splits the given token into a list of pkeys.
-
-        Args:
-            token (str): The token with SEPARATOR values.
-        Returns:
-            A list of ints.
-        """
-        segments = token.split(EventStream.SEPARATOR)
-        try:
-            int_segments = [int(x) for x in segments]
-        except ValueError:
-            raise EventStreamError(400, "Bad token: %s" % token)
-        return int_segments

synapse/api/urls.py

@@ -13,21 +13,9 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-"""This module contains REST servlets to do with public paths: /public"""
-from twisted.internet import defer
+"""Contains the URL paths to prefix various aspects of the server with. """
 
-from base import RestServlet, client_path_pattern
-
-
-class PublicRoomListRestServlet(RestServlet):
-    PATTERN = client_path_pattern("/public/rooms$")
-
-    @defer.inlineCallbacks
-    def on_GET(self, request):
-        handler = self.handlers.room_list_handler
-        data = yield handler.get_public_room_list()
-        defer.returnValue((200, data))
-
-
-def register_servlets(hs, http_server):
-    PublicRoomListRestServlet(hs).register(http_server)
+CLIENT_PREFIX = "/matrix/client/api/v1"
+FEDERATION_PREFIX = "/matrix/federation/v1"
+WEB_CLIENT_PREFIX = "/matrix/client"
+CONTENT_REPO_PREFIX = "/matrix/content"

synapse/app/homeserver.py

@@ -21,58 +21,195 @@ from synapse.server import HomeServer
 from twisted.internet import reactor
 from twisted.enterprise import adbapi
 from twisted.python.log import PythonLoggingObserver
-from synapse.http.server import TwistedHttpServer
+from twisted.web.resource import Resource
+from twisted.web.static import File
+from twisted.web.server import Site
+from synapse.http.server import JsonResource, RootRedirect, ContentRepoResource
 from synapse.http.client import TwistedHttpClient
+from synapse.api.urls import (
+    CLIENT_PREFIX, FEDERATION_PREFIX, WEB_CLIENT_PREFIX, CONTENT_REPO_PREFIX
+)
 
 from daemonize import Daemonize
+import twisted.manhole.telnet
 
 import argparse
 import logging
 import logging.config
 import sqlite3
+import os
+import re
 
 logger = logging.getLogger(__name__)
 
 
+SCHEMAS = [
+    "transactions",
+    "pdu",
+    "users",
+    "profiles",
+    "presence",
+    "im",
+    "room_aliases",
+]
+
+
+# Remember to update this number every time an incompatible change is made to
+# database schema files, so the users will be informed on server restarts.
+SCHEMA_VERSION = 1
+
+
 class SynapseHomeServer(HomeServer):
-    def build_http_server(self):
-        return TwistedHttpServer()
 
     def build_http_client(self):
         return TwistedHttpClient()
 
+    def build_resource_for_client(self):
+        return JsonResource()
+
+    def build_resource_for_federation(self):
+        return JsonResource()
+
+    def build_resource_for_web_client(self):
+        return File("webclient")  # TODO configurable?
+
+    def build_resource_for_content_repo(self):
+        return ContentRepoResource(self, self.upload_dir, self.auth)
+
     def build_db_pool(self):
         """ Set up all the dbs. Since all the *.sql have IF NOT EXISTS, so we
         don't have to worry about overwriting existing content.
         """
         logging.info("Preparing database: %s...", self.db_name)
+
+        with sqlite3.connect(self.db_name) as db_conn:
+            c = db_conn.cursor()
+            c.execute("PRAGMA user_version")
+            row = c.fetchone()
+
+            if row and row[0]:
+                user_version = row[0]
+
+                if user_version < SCHEMA_VERSION:
+                    # TODO(paul): add some kind of intelligent fixup here
+                    raise ValueError("Cannot use this database as the " +
+                        "schema version (%d) does not match (%d)" %
+                        (user_version, SCHEMA_VERSION)
+                    )
+
+            else:
+                for sql_loc in SCHEMAS:
+                    sql_script = read_schema(sql_loc)
+
+                    c.executescript(sql_script)
+                    db_conn.commit()
+
+                c.execute("PRAGMA user_version = %d" % SCHEMA_VERSION)
+
+            c.close()
+
+        logging.info("Database prepared in %s.", self.db_name)
+
         pool = adbapi.ConnectionPool(
             'sqlite3', self.db_name, check_same_thread=False,
             cp_min=1, cp_max=1)
 
-        schemas = [
-            "transactions",
-            "pdu",
-            "users",
-            "profiles",
-            "presence",
-            "im",
-            "room_aliases",
-        ]
-
-        for sql_loc in schemas:
-            sql_script = read_schema(sql_loc)
-
-            with sqlite3.connect(self.db_name) as db_conn:
-                c = db_conn.cursor()
-                c.executescript(sql_script)
-                c.close()
-                db_conn.commit()
-
-        logging.info("Database prepared in %s.", self.db_name)
-
         return pool
 
+    def create_resource_tree(self, web_client, redirect_root_to_web_client):
+        """Create the resource tree for this Home Server.
+
+        This in unduly complicated because Twisted does not support putting
+        child resources more than 1 level deep at a time.
+
+        Args:
+            web_client (bool): True to enable the web client.
+            redirect_root_to_web_client (bool): True to redirect '/' to the
+            location of the web client. This does nothing if web_client is not
+            True.
+        """
+        # list containing (path_str, Resource) e.g:
+        # [ ("/aaa/bbb/cc", Resource1), ("/aaa/dummy", Resource2) ]
+        desired_tree = [
+            (CLIENT_PREFIX, self.get_resource_for_client()),
+            (FEDERATION_PREFIX, self.get_resource_for_federation()),
+            (CONTENT_REPO_PREFIX, self.get_resource_for_content_repo())
+        ]
+        if web_client:
+            logger.info("Adding the web client.")
+            desired_tree.append((WEB_CLIENT_PREFIX,
+                                self.get_resource_for_web_client()))
+
+        if web_client and redirect_root_to_web_client:
+            self.root_resource = RootRedirect(WEB_CLIENT_PREFIX)
+        else:
+            self.root_resource = Resource()
+
+        # ideally we'd just use getChild and putChild but getChild doesn't work
+        # unless you give it a Request object IN ADDITION to the name :/ So
+        # instead, we'll store a copy of this mapping so we can actually add
+        # extra resources to existing nodes. See self._resource_id for the key.
+        resource_mappings = {}
+        for (full_path, resource) in desired_tree:
+            logging.info("Attaching %s to path %s", resource, full_path)
+            last_resource = self.root_resource
+            for path_seg in full_path.split('/')[1:-1]:
+                if not path_seg in last_resource.listNames():
+                    # resource doesn't exist, so make a "dummy resource"
+                    child_resource = Resource()
+                    last_resource.putChild(path_seg, child_resource)
+                    res_id = self._resource_id(last_resource, path_seg)
+                    resource_mappings[res_id] = child_resource
+                    last_resource = child_resource
+                else:
+                    # we have an existing Resource, use that instead.
+                    res_id = self._resource_id(last_resource, path_seg)
+                    last_resource = resource_mappings[res_id]
+
+            # ===========================
+            # now attach the actual desired resource
+            last_path_seg = full_path.split('/')[-1]
+
+            # if there is already a resource here, thieve its children and
+            # replace it
+            res_id = self._resource_id(last_resource, last_path_seg)
+            if res_id in resource_mappings:
+                # there is a dummy resource at this path already, which needs
+                # to be replaced with the desired resource.
+                existing_dummy_resource = resource_mappings[res_id]
+                for child_name in existing_dummy_resource.listNames():
+                    child_res_id = self._resource_id(existing_dummy_resource,
+                                                     child_name)
+                    child_resource = resource_mappings[child_res_id]
+                    # steal the children
+                    resource.putChild(child_name, child_resource)
+
+            # finally, insert the desired resource in the right place
+            last_resource.putChild(last_path_seg, resource)
+            res_id = self._resource_id(last_resource, last_path_seg)
+            resource_mappings[res_id] = resource
+
+        return self.root_resource
+
+    def _resource_id(self, resource, path_seg):
+        """Construct an arbitrary resource ID so you can retrieve the mapping
+        later.
+
+        If you want to represent resource A putChild resource B with path C,
+        the mapping should looks like _resource_id(A,C) = B.
+
+        Args:
+            resource (Resource): The *parent* Resource
+            path_seg (str): The name of the child Resource to be attached.
+        Returns:
+            str: A unique string which can be a key to the child Resource.
+        """
+        return "%s-%s" % (resource, path_seg)
+
+    def start_listening(self, port):
+        reactor.listenTCP(port, Site(self.root_resource))
+        logger.info("Synapse now listening on port %d", port)
+
 
 def setup_logging(verbosity=0, filename=None, config_path=None):
     """ Sets up logging with verbosity levels.
@@ -125,34 +262,55 @@ def setup():
     parser.add_argument('--pid-file', dest="pid", help="When running as a "
                         "daemon, the file to store the pid in",
                         default="hs.pid")
-    parser.add_argument("-w", "--webclient", dest="webclient",
-                        action="store_true", help="Host the web client.")
+    parser.add_argument("-W", "--webclient", dest="webclient", default=True,
+                        action="store_false", help="Don't host a web client.")
+    parser.add_argument("--manhole", dest="manhole", type=int, default=None,
+                        help="Turn on the twisted telnet manhole service.")
     args = parser.parse_args()
 
     verbosity = int(args.verbose) if args.verbose else None
 
+    # Because if/when we daemonize we change to root dir.
+    db_name = os.path.abspath(args.db)
+    log_file = args.log_file
+    if log_file:
+        log_file = os.path.abspath(log_file)
+
     setup_logging(
         verbosity=verbosity,
-        filename=args.log_file,
+        filename=log_file,
         config_path=args.log_config,
     )
 
     logger.info("Server hostname: %s", args.host)
 
+    if re.search(":[0-9]+$", args.host):
+        domain_with_port = args.host
+    else:
+        domain_with_port = "%s:%s" % (args.host, args.port)
+
     hs = SynapseHomeServer(
         args.host,
-        db_name=args.db
+        domain_with_port=domain_with_port,
+        upload_dir=os.path.abspath("uploads"),
+        db_name=db_name,
     )
 
-    # This object doesn't need to be saved because it's set as the handler for
-    # the replication layer
-    hs.get_federation()
-
     hs.register_servlets()
 
-    hs.get_http_server().start_listening(args.port)
+    hs.create_resource_tree(
+        web_client=args.webclient,
+        redirect_root_to_web_client=True)
+    hs.start_listening(args.port)
 
-    hs.build_db_pool()
+    hs.get_db_pool()
+
+    if args.manhole:
+        f = twisted.manhole.telnet.ShellFactory()
+        f.username = "matrix"
+        f.password = "rabbithole"
+        f.namespace['hs'] = hs
+        reactor.listenTCP(args.manhole, f, interface='127.0.0.1')
 
     if args.daemonize:
         daemon = Daemonize(

synapse/federation/__init__.py

@@ -23,7 +23,7 @@ from .transport import TransportLayer
 def initialize_http_replication(homeserver):
     transport = TransportLayer(
         homeserver.hostname,
-        server=homeserver.get_http_server(),
+        server=homeserver.get_resource_for_federation(),
         client=homeserver.get_http_client()
     )
 

synapse/federation/handler.py

@@ -1,149 +0,0 @@
-# -*- coding: utf-8 -*-
-# Copyright 2014 matrix.org
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-
-from twisted.internet import defer
-
-from .pdu_codec import PduCodec
-
-from synapse.api.errors import AuthError
-from synapse.util.logutils import log_function
-
-import logging
-
-
-logger = logging.getLogger(__name__)
-
-
-class FederationEventHandler(object):
-    """ Responsible for:
-        a) handling received Pdus before handing them on as Events to the rest
-        of the home server (including auth and state conflict resoultion)
-        b) converting events that were produced by local clients that may need
-        to be sent to remote home servers.
-    """
-
-    def __init__(self, hs):
-        self.store = hs.get_datastore()
-        self.replication_layer = hs.get_replication_layer()
-        self.state_handler = hs.get_state_handler()
-        # self.auth_handler = gs.get_auth_handler()
-        self.event_handler = hs.get_handlers().federation_handler
-        self.server_name = hs.hostname
-
-        self.lock_manager = hs.get_room_lock_manager()
-
-        self.replication_layer.set_handler(self)
-
-        self.pdu_codec = PduCodec(hs)
-
-    @log_function
-    @defer.inlineCallbacks
-    def handle_new_event(self, event):
-        """ Takes in an event from the client to server side, that has already
-        been authed and handled by the state module, and sends it to any
-        remote home servers that may be interested.
-
-        Args:
-            event
-
-        Returns:
-            Deferred: Resolved when it has successfully been queued for
-            processing.
-        """
-        yield self._fill_out_prev_events(event)
-
-        pdu = self.pdu_codec.pdu_from_event(event)
-
-        if not hasattr(pdu, "destinations") or not pdu.destinations:
-            pdu.destinations = []
-
-        yield self.replication_layer.send_pdu(pdu)
-
-    @log_function
-    @defer.inlineCallbacks
-    def backfill(self, room_id, limit):
-        # TODO: Work out which destinations to ask for backfill
-        # self.replication_layer.backfill(dest, room_id, limit)
-        pass
-
-    @log_function
-    def get_state_for_room(self, destination, room_id):
-        return self.replication_layer.get_state_for_context(
-            destination, room_id
-        )
-
-    @log_function
-    @defer.inlineCallbacks
-    def on_receive_pdu(self, pdu):
-        """ Called by the ReplicationLayer when we have a new pdu. We need to
-        do auth checks and put it throught the StateHandler.
-        """
-        event = self.pdu_codec.event_from_pdu(pdu)
-
-        try:
-            with (yield self.lock_manager.lock(pdu.context)):
-                if event.is_state:
-                    is_new_state = yield self.state_handler.handle_new_state(
-                        pdu
-                    )
-                    if not is_new_state:
-                        return
-                else:
-                    is_new_state = False
-
-            yield self.event_handler.on_receive(event, is_new_state)
-
-        except AuthError:
-            # TODO: Implement something in federation that allows us to
-            # respond to PDU.
-            raise
-
-        return
-
-    @defer.inlineCallbacks
-    def _on_new_state(self, pdu, new_state_event):
-        # TODO: Do any store stuff here. Notifiy C2S about this new
-        # state.
-
-        yield self.store.update_current_state(
-            pdu_id=pdu.pdu_id,
-            origin=pdu.origin,
-            context=pdu.context,
-            pdu_type=pdu.pdu_type,
-            state_key=pdu.state_key
-        )
-
-        yield self.event_handler.on_receive(new_state_event)
-
-    @defer.inlineCallbacks
-    def _fill_out_prev_events(self, event):
-        if hasattr(event, "prev_events"):
-            return
-
-        results = yield self.store.get_latest_pdus_in_context(
-            event.room_id
-        )
-
-        es = [
-            "%s@%s" % (p_id, origin) for p_id, origin, _ in results
-        ]
-
-        event.prev_events = [e for e in es if e != event.event_id]
-
-        if results:
-            event.depth = max([int(v) for _, _, v in results]) + 1
-        else:
-            event.depth = 0

synapse/federation/persistence.py

@@ -25,7 +25,6 @@ from .units import Pdu
 
 from synapse.util.logutils import log_function
 
-import copy
 import json
 import logging
 
@@ -40,28 +39,6 @@ class PduActions(object):
     def __init__(self, datastore):
         self.store = datastore
 
-    @log_function
-    def persist_received(self, pdu):
-        """ Persists the given `Pdu` that was received from a remote home
-        server.
-
-        Returns:
-            Deferred
-        """
-        return self._persist(pdu)
-
-    @defer.inlineCallbacks
-    @log_function
-    def persist_outgoing(self, pdu):
-        """ Persists the given `Pdu` that this home server created.
-
-        Returns:
-            Deferred
-        """
-        ret = yield self._persist(pdu)
-
-        defer.returnValue(ret)
-
     @log_function
     def mark_as_processed(self, pdu):
         """ Persist the fact that we have fully processed the given `Pdu`
@@ -71,25 +48,6 @@ class PduActions(object):
         """
         return self.store.mark_pdu_as_processed(pdu.pdu_id, pdu.origin)
 
-    @defer.inlineCallbacks
-    @log_function
-    def populate_previous_pdus(self, pdu):
-        """ Given an outgoing `Pdu` fill out its `prev_ids` key with the `Pdu`s
-        that we have received.
-
-        Returns:
-            Deferred
-        """
-        results = yield self.store.get_latest_pdus_in_context(pdu.context)
-
-        pdu.prev_pdus = [(p_id, origin) for p_id, origin, _ in results]
-
-        vs = [int(v) for _, _, v in results]
-        if vs:
-            pdu.depth = max(vs) + 1
-        else:
-            pdu.depth = 0
-
     @defer.inlineCallbacks
     @log_function
     def after_transaction(self, transaction_id, destination, origin):
@@ -143,28 +101,6 @@ class PduActions(object):
             depth=pdu.depth
         )
 
-    @defer.inlineCallbacks
-    @log_function
-    def _persist(self, pdu):
-        kwargs = copy.copy(pdu.__dict__)
-        unrec_keys = copy.copy(pdu.unrecognized_keys)
-        del kwargs["content"]
-        kwargs["content_json"] = json.dumps(pdu.content)
-        kwargs["unrecognized_keys"] = json.dumps(unrec_keys)
-
-        logger.debug("Persisting: %s", repr(kwargs))
-
-        if pdu.is_state:
-            ret = yield self.store.persist_state(**kwargs)
-        else:
-            ret = yield self.store.persist_pdu(**kwargs)
-
-        yield self.store.update_min_depth_for_context(
-            pdu.context, pdu.depth
-        )
-
-        defer.returnValue(ret)
-
 
 class TransactionActions(object):
     """ Defines persistence actions that relate to handling Transactions.

synapse/federation/replication.py

@@ -134,10 +134,8 @@ class ReplicationLayer(object):
 
         logger.debug("[%s] Persisting PDU", pdu.pdu_id)
 
-        #yield self.pdu_actions.populate_previous_pdus(pdu)
-
         # Save *before* trying to send
-        yield self.pdu_actions.persist_outgoing(pdu)
+        yield self.store.persist_event(pdu=pdu)
 
         logger.debug("[%s] Persisted PDU", pdu.pdu_id)
         logger.debug("[%s] transaction_layer.enqueue_pdu... ", pdu.pdu_id)
@@ -158,6 +156,7 @@ class ReplicationLayer(object):
 
         # TODO, add errback, etc.
         self._transaction_queue.enqueue_edu(edu)
+        return defer.succeed(None)
 
     @log_function
     def make_query(self, destination, query_type, args):
@@ -208,7 +207,7 @@ class ReplicationLayer(object):
 
         pdus = [Pdu(outlier=False, **p) for p in transaction.pdus]
         for pdu in pdus:
-            yield self._handle_new_pdu(pdu)
+            yield self._handle_new_pdu(pdu, backfilled=True)
 
         defer.returnValue(pdus)
 
@@ -415,7 +414,7 @@ class ReplicationLayer(object):
 
     @defer.inlineCallbacks
     @log_function
-    def _handle_new_pdu(self, pdu):
+    def _handle_new_pdu(self, pdu, backfilled=False):
         # We reprocess pdus when we have seen them only as outliers
         existing = yield self._get_persisted_pdu(pdu.pdu_id, pdu.origin)
 
@@ -449,9 +448,12 @@ class ReplicationLayer(object):
                             logger.exception("Failed to get PDU")
 
         # Persist the Pdu, but don't mark it as processed yet.
-        yield self.pdu_actions.persist_received(pdu)
+        yield self.store.persist_event(pdu=pdu)
 
-        ret = yield self.handler.on_receive_pdu(pdu)
+        if not backfilled:
+            ret = yield self.handler.on_receive_pdu(pdu, backfilled=backfilled)
+        else:
+            ret = None
 
         yield self.pdu_actions.mark_as_processed(pdu)
 
@@ -505,10 +507,10 @@ class _TransactionQueue(object):
         # a transaction in progress. If we do, stick it in the pending_pdus
         # table and we'll get back to it later.
 
-        destinations = [
+        destinations = set([
             d for d in pdu.destinations
             if d != self.server_name
-        ]
+        ])
 
         logger.debug("Sending to: %s", str(destinations))
 
@@ -539,7 +541,10 @@ class _TransactionQueue(object):
         )
 
         def eb(failure):
-            deferred.errback(failure)
+            if not deferred.called:
+                deferred.errback(failure)
+            else:
+                logger.exception("Failed to send edu", failure)
         self._attempt_new_transaction(destination).addErrback(eb)
 
         return deferred

synapse/federation/transport.py

@@ -23,6 +23,7 @@ over a different (albeit still reliable) protocol.
 
 from twisted.internet import defer
 
+from synapse.api.urls import FEDERATION_PREFIX as PREFIX
 from synapse.util.logutils import log_function
 
 import logging
@@ -33,9 +34,6 @@ import re
 logger = logging.getLogger(__name__)
 
 
-PREFIX = "/matrix/federation/v1"
-
-
 class TransportLayer(object):
     """This is a basic implementation of the transport layer that translates
     transactions and other requests to/from HTTP.

synapse/handlers/__init__.py

@@ -15,14 +15,16 @@
 
 from .register import RegistrationHandler
 from .room import (
-    MessageHandler, RoomCreationHandler, RoomMemberHandler, RoomListHandler
+    RoomCreationHandler, RoomMemberHandler, RoomListHandler
 )
-from .events import EventStreamHandler
+from .message import MessageHandler
+from .events import EventStreamHandler, EventHandler
 from .federation import FederationHandler
 from .login import LoginHandler
 from .profile import ProfileHandler
 from .presence import PresenceHandler
 from .directory import DirectoryHandler
+from .typing import TypingNotificationHandler
 
 
 class Handlers(object):
@@ -39,9 +41,11 @@ class Handlers(object):
         self.room_creation_handler = RoomCreationHandler(hs)
         self.room_member_handler = RoomMemberHandler(hs)
         self.event_stream_handler = EventStreamHandler(hs)
+        self.event_handler = EventHandler(hs)
         self.federation_handler = FederationHandler(hs)
         self.profile_handler = ProfileHandler(hs)
         self.presence_handler = PresenceHandler(hs)
         self.room_list_handler = RoomListHandler(hs)
         self.login_handler = LoginHandler(hs)
         self.directory_handler = DirectoryHandler(hs)
+        self.typing_notification_handler = TypingNotificationHandler(hs)

synapse/handlers/_base.py

@@ -13,7 +13,7 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-
+from twisted.internet import defer
 
 class BaseHandler(object):
 
@@ -24,4 +24,27 @@ class BaseHandler(object):
         self.notifier = hs.get_notifier()
         self.room_lock = hs.get_room_lock_manager()
         self.state_handler = hs.get_state_handler()
+        self.distributor = hs.get_distributor()
         self.hs = hs
+
+
+class BaseRoomHandler(BaseHandler):
+
+    @defer.inlineCallbacks
+    def _on_new_room_event(self, event, snapshot, extra_destinations=[],
+                           extra_users=[]):
+        snapshot.fill_out_prev_events(event)
+
+        yield self.store.persist_event(event)
+
+        destinations = set(extra_destinations)
+        # Send a PDU to all hosts who have joined the room.
+        destinations.update((yield self.store.get_joined_hosts_for_room(
+            event.room_id
+        )))
+        event.destinations = list(destinations)
+
+        self.notifier.on_new_room_event(event, extra_users=extra_users)
+
+        federation_handler = self.hs.get_handlers().federation_handler
+        yield federation_handler.handle_new_event(event, snapshot)

synapse/handlers/directory.py

@@ -20,17 +20,11 @@ from ._base import BaseHandler
 from synapse.api.errors import SynapseError
 
 import logging
-import json
-import urllib
 
 
 logger = logging.getLogger(__name__)
 
 
-# TODO(erikj): This needs to be factored out somewere
-PREFIX = "/matrix/client/api/v1"
-
-
 class DirectoryHandler(BaseHandler):
 
     def __init__(self, hs):

synapse/handlers/events.py

@@ -15,24 +15,19 @@
 
 from twisted.internet import defer
 
+from synapse.api.events import SynapseEvent
+from synapse.util.logutils import log_function
+
 from ._base import BaseHandler
-from synapse.api.streams.event import (
-    EventStream, MessagesStreamData, RoomMemberStreamData, FeedbackStreamData,
-    RoomDataStreamData
-)
-from synapse.handlers.presence import PresenceStreamData
+
+import logging
+
+
+logger = logging.getLogger(__name__)
 
 
 class EventStreamHandler(BaseHandler):
 
-    stream_data_classes = [
-        MessagesStreamData,
-        RoomMemberStreamData,
-        FeedbackStreamData,
-        RoomDataStreamData,
-        PresenceStreamData,
-    ]
-
     def __init__(self, hs):
         super(EventStreamHandler, self).__init__(hs)
 
@@ -47,45 +42,13 @@ class EventStreamHandler(BaseHandler):
 
         self.clock = hs.get_clock()
 
-    def get_event_stream_token(self, stream_type, store_id, start_token):
-        """Return the next token after this event.
-
-        Args:
-            stream_type (str): The StreamData.EVENT_TYPE
-            store_id (int): The new storage ID assigned from the data store.
-            start_token (str): The token the user started with.
-        Returns:
-            str: The end token.
-        """
-        for i, stream_cls in enumerate(EventStreamHandler.stream_data_classes):
-            if stream_cls.EVENT_TYPE == stream_type:
-                # this is the stream for this event, so replace this part of
-                # the token
-                store_ids = start_token.split(EventStream.SEPARATOR)
-                store_ids[i] = str(store_id)
-                return EventStream.SEPARATOR.join(store_ids)
-        raise RuntimeError("Didn't find a stream type %s" % stream_type)
+        self.notifier = hs.get_notifier()
 
     @defer.inlineCallbacks
+    @log_function
     def get_stream(self, auth_user_id, pagin_config, timeout=0):
-        """Gets events as an event stream for this user.
-
-        This function looks for interesting *events* for this user. This is
-        different from the notifier, which looks for interested *users* who may
-        want to know about a single event.
-
-        Args:
-            auth_user_id (str): The user requesting their event stream.
-            pagin_config (synapse.api.streams.PaginationConfig): The config to
-            use when obtaining the stream.
-            timeout (int): The max time to wait for an incoming event in ms.
-        Returns:
-            A pagination stream API dict
-        """
         auth_user = self.hs.parse_userid(auth_user_id)
 
-        stream_id = object()
-
         try:
             if auth_user not in self._streams_per_user:
                 self._streams_per_user[auth_user] = 0
@@ -98,41 +61,30 @@ class EventStreamHandler(BaseHandler):
                     )
             self._streams_per_user[auth_user] += 1
 
-            # construct an event stream with the correct data ordering
-            stream_data_list = []
-            for stream_class in EventStreamHandler.stream_data_classes:
-                stream_data_list.append(stream_class(self.hs))
-            event_stream = EventStream(auth_user_id, stream_data_list)
+            if pagin_config.from_token is None:
+                pagin_config.from_token = None
 
-            # fix unknown tokens to known tokens
-            pagin_config = yield event_stream.fix_tokens(pagin_config)
+            rm_handler = self.hs.get_handlers().room_member_handler
+            room_ids = yield rm_handler.get_rooms_for_user(auth_user)
 
-            # register interest in receiving new events
-            self.notifier.store_events_for(user_id=auth_user_id,
-                                           stream_id=stream_id,
-                                           from_tok=pagin_config.from_tok)
+            events, tokens = yield self.notifier.get_events_for(
+                auth_user, room_ids, pagin_config, timeout
+            )
 
-            # see if we can grab a chunk now
-            data_chunk = yield event_stream.get_chunk(config=pagin_config)
+            chunks = [
+                e.get_dict() if isinstance(e, SynapseEvent) else e
+                for e in events
+            ]
 
-            # if there are previous events, return those. If not, wait on the
-            # new events for 'timeout' seconds.
-            if len(data_chunk["chunk"]) == 0 and timeout != 0:
-                results = yield defer.maybeDeferred(
-                    self.notifier.get_events_for,
-                    user_id=auth_user_id,
-                    stream_id=stream_id,
-                    timeout=timeout
-                )
-                if results:
-                    defer.returnValue(results)
+            chunk = {
+                "chunk": chunks,
+                "start": tokens[0].to_string(),
+                "end": tokens[1].to_string(),
+            }
+
+            defer.returnValue(chunk)
 
-            defer.returnValue(data_chunk)
         finally:
-            # cleanup
-            self.notifier.purge_events_for(user_id=auth_user_id,
-                                           stream_id=stream_id)
-
             self._streams_per_user[auth_user] -= 1
             if not self._streams_per_user[auth_user]:
                 del self._streams_per_user[auth_user]
@@ -140,11 +92,39 @@ class EventStreamHandler(BaseHandler):
                 # 10 seconds of grace to allow the client to reconnect again
                 #   before we think they're gone
                 def _later():
+                    logger.debug("_later stopped_user_eventstream %s", auth_user)
                     self.distributor.fire(
                         "stopped_user_eventstream", auth_user
                     )
                     del self._stop_timer_per_user[auth_user]
 
+                logger.debug("Scheduling _later: for %s", auth_user)
                 self._stop_timer_per_user[auth_user] = (
-                    self.clock.call_later(5, _later)
+                    self.clock.call_later(30, _later)
                 )
+
+
+class EventHandler(BaseHandler):
+
+    @defer.inlineCallbacks
+    def get_event(self, user, event_id):
+        """Retrieve a single specified event.
+
+        Args:
+            user (synapse.types.UserID): The user requesting the event
+            event_id (str): The event ID to obtain.
+        Returns:
+            dict: An event, or None if there is no event matching this ID.
+        Raises:
+            SynapseError if there was a problem retrieving this event, or
+            AuthError if the user does not have the rights to inspect this
+            event.
+        """
+        event = yield self.store.get_event(event_id)
+
+        if not event:
+            defer.returnValue(None)
+            return
+
+        yield self.auth.check(event, raises=True)
+        defer.returnValue(event)

synapse/handlers/federation.py

@@ -20,6 +20,7 @@ from ._base import BaseHandler
 from synapse.api.events.room import InviteJoinEvent, RoomMemberEvent
 from synapse.api.constants import Membership
 from synapse.util.logutils import log_function
+from synapse.federation.pdu_codec import PduCodec
 
 from twisted.internet import defer
 
@@ -30,12 +31,79 @@ logger = logging.getLogger(__name__)
 
 
 class FederationHandler(BaseHandler):
+    """Handles events that originated from federation.
+        Responsible for:
+        a) handling received Pdus before handing them on as Events to the rest
+        of the home server (including auth and state conflict resoultion)
+        b) converting events that were produced by local clients that may need
+        to be sent to remote home servers.
+    """
 
-    """Handles events that originated from federation."""
+    def __init__(self, hs):
+        super(FederationHandler, self).__init__(hs)
+
+        self.distributor.observe(
+            "user_joined_room",
+            self._on_user_joined
+        )
+
+        self.waiting_for_join_list = {}
+
+        self.store = hs.get_datastore()
+        self.replication_layer = hs.get_replication_layer()
+        self.state_handler = hs.get_state_handler()
+        # self.auth_handler = gs.get_auth_handler()
+        self.server_name = hs.hostname
+
+        self.lock_manager = hs.get_room_lock_manager()
+
+        self.replication_layer.set_handler(self)
+
+        self.pdu_codec = PduCodec(hs)
 
     @log_function
     @defer.inlineCallbacks
-    def on_receive(self, event, is_new_state):
+    def handle_new_event(self, event, snapshot):
+        """ Takes in an event from the client to server side, that has already
+        been authed and handled by the state module, and sends it to any
+        remote home servers that may be interested.
+
+        Args:
+            event
+            snapshot (.storage.Snapshot): THe snapshot the event happened after
+
+        Returns:
+            Deferred: Resolved when it has successfully been queued for
+            processing.
+        """
+
+        pdu = self.pdu_codec.pdu_from_event(event)
+
+        if not hasattr(pdu, "destinations") or not pdu.destinations:
+            pdu.destinations = []
+
+        yield self.replication_layer.send_pdu(pdu)
+
+    @log_function
+    @defer.inlineCallbacks
+    def on_receive_pdu(self, pdu, backfilled):
+        """ Called by the ReplicationLayer when we have a new pdu. We need to
+        do auth checks and put it throught the StateHandler.
+        """
+        event = self.pdu_codec.event_from_pdu(pdu)
+
+        with (yield self.lock_manager.lock(pdu.context)):
+            if event.is_state and not backfilled:
+                is_new_state = yield self.state_handler.handle_new_state(
+                    pdu
+                )
+                if not is_new_state:
+                    return
+            else:
+                is_new_state = False
+        # TODO: Implement something in federation that allows us to
+        # respond to PDU.
+
         if hasattr(event, "state_key") and not is_new_state:
             logger.debug("Ignoring old state.")
             return
@@ -56,7 +124,7 @@ class FederationHandler(BaseHandler):
             content.update({"membership": Membership.JOIN})
             new_event = self.event_factory.create_event(
                 etype=RoomMemberEvent.TYPE,
-                target_user_id=event.user_id,
+                state_key=event.user_id,
                 room_id=event.room_id,
                 user_id=event.user_id,
                 membership=Membership.JOIN,
@@ -65,11 +133,121 @@ class FederationHandler(BaseHandler):
 
             yield self.hs.get_handlers().room_member_handler.change_membership(
                 new_event,
-                True
+                do_auth=True
             )
 
         else:
             with (yield self.room_lock.lock(event.room_id)):
-                store_id = yield self.store.persist_event(event)
+                yield self.store.persist_event(event, backfilled)
 
-            yield self.notifier.on_new_room_event(event, store_id)
+            room = yield self.store.get_room(event.room_id)
+
+            if not room:
+                # Huh, let's try and get the current state
+                try:
+                    yield self.replication_layer.get_state_for_context(
+                        event.origin, event.room_id
+                    )
+
+                    hosts = yield self.store.get_joined_hosts_for_room(
+                        event.room_id
+                    )
+                    if self.hs.hostname in hosts:
+                        try:
+                            yield self.store.store_room(
+                                room_id=event.room_id,
+                                room_creator_user_id="",
+                                is_public=False,
+                            )
+                        except:
+                            pass
+                except:
+                    logger.exception(
+                        "Failed to get current state for room %s",
+                        event.room_id
+                    )
+
+            if not backfilled:
+                yield self.notifier.on_new_room_event(event)
+
+        if event.type == RoomMemberEvent.TYPE:
+            if event.membership == Membership.JOIN:
+                user = self.hs.parse_userid(event.state_key)
+                self.distributor.fire(
+                    "user_joined_room", user=user, room_id=event.room_id
+                )
+
+    @log_function
+    @defer.inlineCallbacks
+    def backfill(self, dest, room_id, limit):
+        pdus = yield self.replication_layer.backfill(dest, room_id, limit)
+
+        events = []
+
+        for pdu in pdus:
+            event = self.pdu_codec.event_from_pdu(pdu)
+            events.append(event)
+            yield self.store.persist_event(event, backfilled=True)
+
+        defer.returnValue(events)
+
+    @log_function
+    @defer.inlineCallbacks
+    def do_invite_join(self, target_host, room_id, joinee, content, snapshot):
+
+        hosts = yield self.store.get_joined_hosts_for_room(room_id)
+        if self.hs.hostname in hosts:
+            # We are already in the room.
+            logger.debug("We're already in the room apparently")
+            defer.returnValue(False)
+
+        # First get current state to see if we are already joined.
+        try:
+            yield self.replication_layer.get_state_for_context(
+                target_host, room_id
+            )
+
+            hosts = yield self.store.get_joined_hosts_for_room(room_id)
+            if self.hs.hostname in hosts:
+                # Oh, we were actually in the room already.
+                logger.debug("We're already in the room apparently")
+                defer.returnValue(False)
+        except Exception:
+            logger.exception("Failed to get current state")
+
+        new_event = self.event_factory.create_event(
+            etype=InviteJoinEvent.TYPE,
+            target_host=target_host,
+            room_id=room_id,
+            user_id=joinee,
+            content=content
+        )
+
+        new_event.destinations = [target_host]
+
+        snapshot.fill_out_prev_events(new_event)
+        yield self.handle_new_event(new_event, snapshot)
+
+        # TODO (erikj): Time out here.
+        d = defer.Deferred()
+        self.waiting_for_join_list.setdefault((joinee, room_id), []).append(d)
+        yield d
+
+        try:
+            yield self.store.store_room(
+                room_id=room_id,
+                room_creator_user_id="",
+                is_public=False
+            )
+        except:
+            pass
+
+
+        defer.returnValue(True)
+
+
+    @log_function
+    def _on_user_joined(self, user, room_id):
+        waiters = self.waiting_for_join_list.get((user.to_string(), room_id), [])
+        while waiters:
+            waiters.pop().callback(None)

synapse/handlers/login.py

@@ -16,7 +16,7 @@
 from twisted.internet import defer
 
 from ._base import BaseHandler
-from synapse.api.errors import LoginError
+from synapse.api.errors import LoginError, Codes
 
 import bcrypt
 import logging
@@ -51,7 +51,7 @@ class LoginHandler(BaseHandler):
         user_info = yield self.store.get_user_by_id(user_id=user)
         if not user_info:
             logger.warn("Attempted to login as %s but they do not exist.", user)
-            raise LoginError(403, "")
+            raise LoginError(403, "", errcode=Codes.FORBIDDEN)
 
         stored_hash = user_info[0]["password_hash"]
         if bcrypt.checkpw(password, stored_hash):
@@ -62,4 +62,4 @@ class LoginHandler(BaseHandler):
             defer.returnValue(token)
         else:
             logger.warn("Failed password login for user %s", user)
-            raise LoginError(403, "")
+            raise LoginError(403, "", errcode=Codes.FORBIDDEN)

synapse/handlers/message.py

@@ -0,0 +1,304 @@
+# -*- coding: utf-8 -*-
+# Copyright 2014 matrix.org
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from twisted.internet import defer
+
+from synapse.api.constants import Membership
+from synapse.api.events.room import RoomTopicEvent
+from synapse.api.errors import RoomError
+from synapse.streams.config import PaginationConfig
+from ._base import BaseRoomHandler
+
+import logging
+
+logger = logging.getLogger(__name__)
+
+
+
+class MessageHandler(BaseRoomHandler):
+
+    def __init__(self, hs):
+        super(MessageHandler, self).__init__(hs)
+        self.hs = hs
+        self.clock = hs.get_clock()
+        self.event_factory = hs.get_event_factory()
+
+    @defer.inlineCallbacks
+    def get_message(self, msg_id=None, room_id=None, sender_id=None,
+                    user_id=None):
+        """ Retrieve a message.
+
+        Args:
+            msg_id (str): The message ID to obtain.
+            room_id (str): The room where the message resides.
+            sender_id (str): The user ID of the user who sent the message.
+            user_id (str): The user ID of the user making this request.
+        Returns:
+            The message, or None if no message exists.
+        Raises:
+            SynapseError if something went wrong.
+        """
+        yield self.auth.check_joined_room(room_id, user_id)
+
+        # Pull out the message from the db
+#        msg = yield self.store.get_message(
+#            room_id=room_id,
+#            msg_id=msg_id,
+#            user_id=sender_id
+#        )
+
+        # TODO (erikj): Once we work out the correct c-s api we need to think on how to do this.
+
+        defer.returnValue(None)
+
+    @defer.inlineCallbacks
+    def send_message(self, event=None, suppress_auth=False, stamp_event=True):
+        """ Send a message.
+
+        Args:
+            event : The message event to store.
+            suppress_auth (bool) : True to suppress auth for this message. This
+            is primarily so the home server can inject messages into rooms at
+            will.
+            stamp_event (bool) : True to stamp event content with server keys.
+        Raises:
+            SynapseError if something went wrong.
+        """
+        if stamp_event:
+            event.content["hsob_ts"] = int(self.clock.time_msec())
+
+        snapshot = yield self.store.snapshot_room(event.room_id, event.user_id)
+
+        if not suppress_auth:
+            yield self.auth.check(event, snapshot, raises=True)
+
+        yield self._on_new_room_event(event, snapshot)
+
+    @defer.inlineCallbacks
+    def get_messages(self, user_id=None, room_id=None, pagin_config=None,
+                     feedback=False):
+        """Get messages in a room.
+
+        Args:
+            user_id (str): The user requesting messages.
+            room_id (str): The room they want messages from.
+            pagin_config (synapse.api.streams.PaginationConfig): The pagination
+            config rules to apply, if any.
+            feedback (bool): True to get compressed feedback with the messages
+        Returns:
+            dict: Pagination API results
+        """
+        yield self.auth.check_joined_room(room_id, user_id)
+
+        data_source = self.hs.get_event_sources().sources["room"]
+
+        if not pagin_config.from_token:
+            pagin_config.from_token = yield self.hs.get_event_sources().get_current_token()
+
+        user = self.hs.parse_userid(user_id)
+
+        events, next_token = yield data_source.get_pagination_rows(
+            user, pagin_config, room_id
+        )
+
+        chunk = {
+            "chunk": [e.get_dict() for e in events],
+            "start": pagin_config.from_token.to_string(),
+            "end": next_token.to_string(),
+        }
+
+        defer.returnValue(chunk)
+
+    @defer.inlineCallbacks
+    def store_room_data(self, event=None, stamp_event=True):
+        """ Stores data for a room.
+
+        Args:
+            event : The room path event
+            stamp_event (bool) : True to stamp event content with server keys.
+        Raises:
+            SynapseError if something went wrong.
+        """
+
+        snapshot = yield self.store.snapshot_room(event.room_id, event.user_id)
+
+        yield self.auth.check(event, snapshot, raises=True)
+
+        if stamp_event:
+            event.content["hsob_ts"] = int(self.clock.time_msec())
+
+        yield self.state_handler.handle_new_event(event, snapshot)
+
+        yield self._on_new_room_event(event, snapshot)
+
+    @defer.inlineCallbacks
+    def get_room_data(self, user_id=None, room_id=None,
+                      event_type=None, state_key="",
+                      public_room_rules=[],
+                      private_room_rules=["join"]):
+        """ Get data from a room.
+
+        Args:
+            event : The room path event
+            public_room_rules : A list of membership states the user can be in,
+            in order to read this data IN A PUBLIC ROOM. An empty list means
+            'any state'.
+            private_room_rules : A list of membership states the user can be
+            in, in order to read this data IN A PRIVATE ROOM. An empty list
+            means 'any state'.
+        Returns:
+            The path data content.
+        Raises:
+            SynapseError if something went wrong.
+        """
+        if event_type == RoomTopicEvent.TYPE:
+            # anyone invited/joined can read the topic
+            private_room_rules = ["invite", "join"]
+
+        # does this room exist
+        room = yield self.store.get_room(room_id)
+        if not room:
+            raise RoomError(403, "Room does not exist.")
+
+        # does this user exist in this room
+        member = yield self.store.get_room_member(
+            room_id=room_id,
+            user_id="" if not user_id else user_id)
+
+        member_state = member.membership if member else None
+
+        if room.is_public and public_room_rules:
+            # make sure the user meets public room rules
+            if member_state not in public_room_rules:
+                raise RoomError(403, "Member does not meet public room rules.")
+        elif not room.is_public and private_room_rules:
+            # make sure the user meets private room rules
+            if member_state not in private_room_rules:
+                raise RoomError(
+                    403, "Member does not meet private room rules.")
+
+        data = yield self.store.get_current_state(
+            room_id, event_type, state_key
+        )
+        defer.returnValue(data)
+
+    @defer.inlineCallbacks
+    def get_feedback(self, event_id):
+        # yield self.auth.check_joined_room(room_id, user_id)
+
+        # Pull out the feedback from the db
+        fb = yield self.store.get_feedback(event_id)
+
+        if fb:
+            defer.returnValue(fb)
+        defer.returnValue(None)
+
+    @defer.inlineCallbacks
+    def send_feedback(self, event, stamp_event=True):
+        if stamp_event:
+            event.content["hsob_ts"] = int(self.clock.time_msec())
+
+        snapshot = yield self.store.snapshot_room(event.room_id, event.user_id)
+
+        yield self.auth.check(event, snapshot, raises=True)
+
+        # store message in db
+        yield self._on_new_room_event(event, snapshot)
+
+    @defer.inlineCallbacks
+    def snapshot_all_rooms(self, user_id=None, pagin_config=None,
+                           feedback=False):
+        """Retrieve a snapshot of all rooms the user is invited or has joined.
+
+        This snapshot may include messages for all rooms where the user is
+        joined, depending on the pagination config.
+
+        Args:
+            user_id (str): The ID of the user making the request.
+            pagin_config (synapse.api.streams.PaginationConfig): The pagination
+            config used to determine how many messages *PER ROOM* to return.
+            feedback (bool): True to get feedback along with these messages.
+        Returns:
+            A list of dicts with "room_id" and "membership" keys for all rooms
+            the user is currently invited or joined in on. Rooms where the user
+            is joined on, may return a "messages" key with messages, depending
+            on the specified PaginationConfig.
+        """
+        room_list = yield self.store.get_rooms_for_user_where_membership_is(
+            user_id=user_id,
+            membership_list=[Membership.INVITE, Membership.JOIN]
+        )
+
+        user = self.hs.parse_userid(user_id)
+
+        rooms_ret = []
+
+        now_token = yield self.hs.get_event_sources().get_current_token()
+
+        presence_stream = self.hs.get_event_sources().sources["presence"]
+        pagination_config = PaginationConfig(from_token=now_token)
+        presence, _ = yield presence_stream.get_pagination_rows(
+            user, pagination_config, None
+        )
+
+        limit = pagin_config.limit
+        if not limit:
+            limit = 10
+
+        for event in room_list:
+            d = {
+                "room_id": event.room_id,
+                "membership": event.membership,
+            }
+
+            if event.membership == Membership.INVITE:
+                d["inviter"] = event.user_id
+
+            rooms_ret.append(d)
+
+            if event.membership != Membership.JOIN:
+                continue
+            try:
+                messages, token = yield self.store.get_recent_events_for_room(
+                    event.room_id,
+                    limit=limit,
+                    end_token=now_token.events_key,
+                )
+
+                start_token = now_token.copy_and_replace("events_key", token[0])
+                end_token = now_token.copy_and_replace("events_key", token[1])
+
+                d["messages"] = {
+                    "chunk": [m.get_dict() for m in messages],
+                    "start": start_token.to_string(),
+                    "end": end_token.to_string(),
+                }
+
+                current_state = yield self.store.get_current_state(
+                    event.room_id
+                )
+                d["state"] = [c.get_dict() for c in current_state]
+            except:
+                logger.exception("Failed to get snapshot")
+
+        ret = {
+            "rooms": rooms_ret,
+            "presence": presence,
+            "end": now_token.to_string()
+        }
+
+        defer.returnValue(ret)
+
+

synapse/handlers/presence.py

@@ -17,7 +17,8 @@ from twisted.internet import defer
 
 from synapse.api.errors import SynapseError, AuthError
 from synapse.api.constants import PresenceState
-from synapse.api.streams import StreamData
+
+from synapse.util.logutils import log_function
 
 from ._base import BaseHandler
 
@@ -142,6 +143,10 @@ class PresenceHandler(BaseHandler):
 
     @defer.inlineCallbacks
     def is_presence_visible(self, observer_user, observed_user):
+        defer.returnValue(True)
+        # return
+        # FIXME (erikj): This code path absolutely kills the database.
+
         assert(observed_user.is_mine)
 
         if observer_user == observed_user:
@@ -155,9 +160,11 @@ class PresenceHandler(BaseHandler):
         if allowed_by_subscription:
             defer.returnValue(True)
 
-        # TODO(paul): Check same channel
+        share_room = yield self.store.do_users_share_a_room(
+            [observer_user, observed_user]
+        )
 
-        defer.returnValue(False)
+        defer.returnValue(share_room)
 
     @defer.inlineCallbacks
     def get_state(self, target_user, auth_user):
@@ -176,12 +183,19 @@ class PresenceHandler(BaseHandler):
             # TODO(paul): Have remote server send us permissions set
             state = self._get_or_offline_usercache(target_user).get_state()
 
-        if "mtime" in state:
-            state["mtime_age"] = self.clock.time_msec() - state.pop("mtime")
+        if "mtime" in state and (state["mtime"] is not None):
+            state["mtime_age"] = int(
+                self.clock.time_msec() - state.pop("mtime")
+            )
         defer.returnValue(state)
 
     @defer.inlineCallbacks
+    @log_function
     def set_state(self, target_user, auth_user, state):
+        # return
+        # TODO (erikj): Turn this back on. Why did we end up sending EDUs
+        # everywhere?
+
         if not target_user.is_mine:
             raise SynapseError(400, "User is not hosted on this Home Server")
 
@@ -226,9 +240,6 @@ class PresenceHandler(BaseHandler):
         #   we don't have to do this all the time
         self.changed_presencelike_data(target_user, state)
 
-        if not now_online:
-            del self._user_cachemap[target_user]
-
     def changed_presencelike_data(self, user, state):
         statuscache = self._get_or_make_usercache(user)
 
@@ -237,33 +248,42 @@ class PresenceHandler(BaseHandler):
 
         self.push_presence(user, statuscache=statuscache)
 
+    @log_function
     def started_user_eventstream(self, user):
         # TODO(paul): Use "last online" state
         self.set_state(user, user, {"state": PresenceState.ONLINE})
 
+    @log_function
     def stopped_user_eventstream(self, user):
         # TODO(paul): Save current state as "last online" state
         self.set_state(user, user, {"state": PresenceState.OFFLINE})
 
     @defer.inlineCallbacks
     def user_joined_room(self, user, room_id):
-        localusers = set()
-        remotedomains = set()
-
-        rm_handler = self.homeserver.get_handlers().room_member_handler
-        yield rm_handler.fetch_room_distributions_into(room_id,
-                localusers=localusers, remotedomains=remotedomains,
-                ignore_user=user)
 
         if user.is_mine:
-            yield self._send_presence_to_distribution(srcuser=user,
-                localusers=localusers, remotedomains=remotedomains,
+            self.push_update_to_local_and_remote(
+                observed_user=user,
+                room_ids=[room_id],
                 statuscache=self._get_or_offline_usercache(user),
             )
 
-        for srcuser in localusers:
-            yield self._send_presence(srcuser=srcuser, destuser=user,
-                statuscache=self._get_or_offline_usercache(srcuser),
+        else:
+            self.push_update_to_clients(
+                observed_user=user,
+                room_ids=[room_id],
+                statuscache=self._get_or_offline_usercache(user),
+            )
+
+        # We also want to tell them about current presence of people.
+        rm_handler = self.homeserver.get_handlers().room_member_handler
+        curr_users = yield rm_handler.get_room_members(room_id)
+
+        for local_user in [c for c in curr_users if c.is_mine]:
+            self.push_update_to_local_and_remote(
+                observed_user=local_user,
+                users_to_push=[user],
+                statuscache=self._get_or_offline_usercache(local_user),
             )
 
     @defer.inlineCallbacks
@@ -367,37 +387,64 @@ class PresenceHandler(BaseHandler):
             p["observed_user"] = observed_user
             p.update(self._get_or_offline_usercache(observed_user).get_state())
             if "mtime" in p:
-                p["mtime_age"] = self.clock.time_msec() - p.pop("mtime")
+                p["mtime_age"] = int(
+                    self.clock.time_msec() - p.pop("mtime")
+                )
 
         defer.returnValue(presence)
 
     @defer.inlineCallbacks
+    @log_function
     def start_polling_presence(self, user, target_user=None, state=None):
         logger.debug("Start polling for presence from %s", user)
 
         if target_user:
-            target_users = [target_user]
+            target_users = set([target_user])
+            room_ids = []
         else:
             presence = yield self.store.get_presence_list(
                 user.localpart, accepted=True
             )
-            target_users = [
+            target_users = set([
                 self.hs.parse_userid(x["observed_user_id"]) for x in presence
-            ]
+            ])
+
+            # Also include people in all my rooms
+
+            rm_handler = self.homeserver.get_handlers().room_member_handler
+            room_ids = yield rm_handler.get_rooms_for_user(user)
 
         if state is None:
             state = yield self.store.get_presence_state(user.localpart)
+        else:
+#            statuscache = self._get_or_make_usercache(user)
+#            self._user_cachemap_latest_serial += 1
+#            statuscache.update(state, self._user_cachemap_latest_serial)
+            pass
 
-        localusers, remoteusers = partitionbool(
-            target_users,
-            lambda u: u.is_mine
+        yield self.push_update_to_local_and_remote(
+            observed_user=user,
+            users_to_push=target_users,
+            room_ids=room_ids,
+            statuscache=self._get_or_make_usercache(user),
         )
 
-        for target_user in localusers:
-            self._start_polling_local(user, target_user)
+        for target_user in target_users:
+            if target_user.is_mine:
+                self._start_polling_local(user, target_user)
+
+                # We want to tell the person that just came online
+                # presence state of people they are interested in?
+                self.push_update_to_clients(
+                    observed_user=target_user,
+                    users_to_push=[user],
+                    statuscache=self._get_or_offline_usercache(target_user),
+                )
 
         deferreds = []
-        remoteusers_by_domain = partition(remoteusers, lambda u: u.domain)
+        remote_users = [u for u in target_users if not u.is_mine]
+        remoteusers_by_domain = partition(remote_users, lambda u: u.domain)
+        # Only poll for people in our get_presence_list
         for domain in remoteusers_by_domain:
             remoteusers = remoteusers_by_domain[domain]
 
@@ -419,25 +466,26 @@ class PresenceHandler(BaseHandler):
 
         self._local_pushmap[target_localpart].add(user)
 
-        self.push_update_to_clients(
-            observer_user=user,
-            observed_user=target_user,
-            statuscache=self._get_or_offline_usercache(target_user),
-        )
-
     def _start_polling_remote(self, user, domain, remoteusers):
+        to_poll = set()
+
         for u in remoteusers:
             if u not in self._remote_recvmap:
                 self._remote_recvmap[u] = set()
+                to_poll.add(u)
 
             self._remote_recvmap[u].add(user)
 
+        if not to_poll:
+            return defer.succeed(None)
+
         return self.federation.send_edu(
             destination=domain,
             edu_type="m.presence",
-            content={"poll": [u.to_string() for u in remoteusers]}
+            content={"poll": [u.to_string() for u in to_poll]}
         )
 
+    @log_function
     def stop_polling_presence(self, user, target_user=None):
         logger.debug("Stop polling for presence from %s", user)
 
@@ -447,9 +495,13 @@ class PresenceHandler(BaseHandler):
         deferreds = []
 
         if target_user:
-            raise NotImplementedError("TODO: remove one user")
+            if target_user not in self._remote_recvmap:
+                return
+            target_users = set([target_user])
+        else:
+            target_users = self._remote_recvmap.keys()
 
-        remoteusers = [u for u in self._remote_recvmap
+        remoteusers = [u for u in target_users
                        if user in self._remote_recvmap[u]]
         remoteusers_by_domain = partition(remoteusers, lambda u: u.domain)
 
@@ -473,20 +525,28 @@ class PresenceHandler(BaseHandler):
             if not self._local_pushmap[localpart]:
                 del self._local_pushmap[localpart]
 
+    @log_function
     def _stop_polling_remote(self, user, domain, remoteusers):
+        to_unpoll = set()
+
         for u in remoteusers:
             self._remote_recvmap[u].remove(user)
 
             if not self._remote_recvmap[u]:
                 del self._remote_recvmap[u]
+                to_unpoll.add(u)
+
+        if not to_unpoll:
+            return defer.succeed(None)
 
         return self.federation.send_edu(
             destination=domain,
             edu_type="m.presence",
-            content={"unpoll": [u.to_string() for u in remoteusers]}
+            content={"unpoll": [u.to_string() for u in to_unpoll]}
         )
 
     @defer.inlineCallbacks
+    @log_function
     def push_presence(self, user, statuscache):
         assert(user.is_mine)
 
@@ -502,53 +562,17 @@ class PresenceHandler(BaseHandler):
         rm_handler = self.homeserver.get_handlers().room_member_handler
         room_ids = yield rm_handler.get_rooms_for_user(user)
 
-        for room_id in room_ids:
-            yield rm_handler.fetch_room_distributions_into(
-                room_id, localusers=localusers, remotedomains=remotedomains,
-                ignore_user=user,
-            )
-
-        if not localusers and not remotedomains:
+        if not localusers and not room_ids:
             defer.returnValue(None)
 
-        yield self._send_presence_to_distribution(user,
-            localusers=localusers, remotedomains=remotedomains,
-            statuscache=statuscache
+        yield self.push_update_to_local_and_remote(
+            observed_user=user,
+            users_to_push=localusers,
+            remote_domains=remotedomains,
+            room_ids=room_ids,
+            statuscache=statuscache,
         )
 
-    def _send_presence(self, srcuser, destuser, statuscache):
-        if destuser.is_mine:
-            self.push_update_to_clients(
-                observer_user=destuser,
-                observed_user=srcuser,
-                statuscache=statuscache)
-            return defer.succeed(None)
-        else:
-            return self._push_presence_remote(srcuser, destuser.domain,
-                state=statuscache.get_state()
-            )
-
-    @defer.inlineCallbacks
-    def _send_presence_to_distribution(self, srcuser, localusers=set(),
-            remotedomains=set(), statuscache=None):
-
-        for u in localusers:
-            logger.debug(" | push to local user %s", u)
-            self.push_update_to_clients(
-                observer_user=u,
-                observed_user=srcuser,
-                statuscache=statuscache,
-            )
-
-        deferreds = []
-        for domain in remotedomains:
-            logger.debug(" | push to remote domain %s", domain)
-            deferreds.append(self._push_presence_remote(srcuser, domain,
-                state=statuscache.get_state())
-            )
-
-        yield defer.DeferredList(deferreds)
-
     @defer.inlineCallbacks
     def _push_presence_remote(self, user, destination, state=None):
         if state is None:
@@ -560,14 +584,21 @@ class PresenceHandler(BaseHandler):
 
         if "mtime" in state:
             state = dict(state)
-            state["mtime_age"] = self.clock.time_msec() - state.pop("mtime")
+            state["mtime_age"] = int(
+                self.clock.time_msec() - state.pop("mtime")
+            )
+
+        user_state = {
+            "user_id": user.to_string(),
+        }
+        user_state.update(**state)
 
         yield self.federation.send_edu(
             destination=destination,
             edu_type="m.presence",
             content={
                 "push": [
-                    dict(user_id=user.to_string(), **state),
+                    user_state,
                 ],
             }
         )
@@ -586,31 +617,28 @@ class PresenceHandler(BaseHandler):
             rm_handler = self.homeserver.get_handlers().room_member_handler
             room_ids = yield rm_handler.get_rooms_for_user(user)
 
-            for room_id in room_ids:
-                yield rm_handler.fetch_room_distributions_into(
-                    room_id, localusers=observers, ignore_user=user
-                )
-
-            if not observers:
+            if not observers and not room_ids:
                 break
 
             state = dict(push)
             del state["user_id"]
 
             if "mtime_age" in state:
-                state["mtime"] = self.clock.time_msec() - state.pop("mtime_age")
+                state["mtime"] = int(
+                    self.clock.time_msec() - state.pop("mtime_age")
+                )
 
             statuscache = self._get_or_make_usercache(user)
 
             self._user_cachemap_latest_serial += 1
             statuscache.update(state, serial=self._user_cachemap_latest_serial)
 
-            for observer_user in observers:
-                self.push_update_to_clients(
-                    observer_user=observer_user,
-                    observed_user=user,
-                    statuscache=statuscache,
-                )
+            self.push_update_to_clients(
+                observed_user=user,
+                users_to_push=observers,
+                room_ids=room_ids,
+                statuscache=statuscache,
+            )
 
             if state["state"] == PresenceState.OFFLINE:
                 del self._user_cachemap[user]
@@ -644,46 +672,54 @@ class PresenceHandler(BaseHandler):
 
         yield defer.DeferredList(deferreds)
 
-    def push_update_to_clients(self, observer_user, observed_user,
-                               statuscache):
-        state = statuscache.make_event(user=observed_user, clock=self.clock)
+    @defer.inlineCallbacks
+    def push_update_to_local_and_remote(self, observed_user,
+                                        users_to_push=[], room_ids=[],
+                                        remote_domains=[],
+                                        statuscache=None):
 
-        self.notifier.on_new_user_event(
-            observer_user.to_string(),
-            event_data=statuscache.make_event(
-                user=observed_user,
-                clock=self.clock
-            ),
-            stream_type=PresenceStreamData,
-            store_id=statuscache.serial
+        localusers, remoteusers = partitionbool(
+            users_to_push,
+            lambda u: u.is_mine
         )
 
+        localusers = set(localusers)
 
-class PresenceStreamData(StreamData):
-    def __init__(self, hs):
-        super(PresenceStreamData, self).__init__(hs)
-        self.presence = hs.get_handlers().presence_handler
+        self.push_update_to_clients(
+            observed_user=observed_user,
+            users_to_push=localusers,
+            room_ids=room_ids,
+            statuscache=statuscache,
+        )
 
-    def get_rows(self, user_id, from_key, to_key, limit):
-        cachemap = self.presence._user_cachemap
+        remote_domains = set(remote_domains)
+        remote_domains |= set([r.domain for r in remoteusers])
+        for room_id in room_ids:
+            remote_domains.update(
+                (yield self.store.get_joined_hosts_for_room(room_id))
+            )
 
-        # TODO(paul): limit, and filter by visibility
-        updates = [(k, cachemap[k]) for k in cachemap
-                   if from_key < cachemap[k].serial <= to_key]
+        remote_domains.discard(self.hs.hostname)
 
-        if updates:
-            clock = self.presence.clock
+        deferreds = []
+        for domain in remote_domains:
+            logger.debug(" | push to remote domain %s", domain)
+            deferreds.append(
+                self._push_presence_remote(
+                    observed_user, domain, state=statuscache.get_state()
+                )
+            )
 
-            latest_serial = max([x[1].serial for x in updates])
-            data = [x[1].make_event(user=x[0], clock=clock) for x in updates]
-            return ((data, latest_serial))
-        else:
-            return (([], self.presence._user_cachemap_latest_serial))
+        yield defer.DeferredList(deferreds)
 
-    def max_token(self):
-        return self.presence._user_cachemap_latest_serial
+        defer.returnValue((localusers, remote_domains))
 
-PresenceStreamData.EVENT_TYPE = PresenceStreamData
+    def push_update_to_clients(self, observed_user, users_to_push=[],
+                                 room_ids=[], statuscache=None):
+        self.notifier.on_new_user_event(
+            users_to_push,
+            room_ids,
+        )
 
 
 class UserPresenceCache(object):
@@ -720,6 +756,8 @@ class UserPresenceCache(object):
         content["user_id"] = user.to_string()
 
         if "mtime" in content:
-            content["mtime_age"] = clock.time_msec() - content.pop("mtime")
+            content["mtime_age"] = int(
+                clock.time_msec() - content.pop("mtime")
+            )
 
         return {"type": "m.presence", "content": content}

synapse/handlers/room.py

@@ -18,265 +18,19 @@ from twisted.internet import defer
 
 from synapse.types import UserID, RoomAlias, RoomID
 from synapse.api.constants import Membership
-from synapse.api.errors import RoomError, StoreError, SynapseError
+from synapse.api.errors import StoreError, SynapseError
 from synapse.api.events.room import (
-    RoomTopicEvent, MessageEvent, InviteJoinEvent, RoomMemberEvent,
-    RoomConfigEvent
+    RoomMemberEvent, RoomConfigEvent
 )
-from synapse.api.streams.event import EventStream, MessagesStreamData
 from synapse.util import stringutils
-from ._base import BaseHandler
+from ._base import BaseRoomHandler
 
 import logging
-import json
 
 logger = logging.getLogger(__name__)
 
 
-class MessageHandler(BaseHandler):
-
-    def __init__(self, hs):
-        super(MessageHandler, self).__init__(hs)
-        self.hs = hs
-        self.clock = hs.get_clock()
-        self.event_factory = hs.get_event_factory()
-
-    @defer.inlineCallbacks
-    def get_message(self, msg_id=None, room_id=None, sender_id=None,
-                    user_id=None):
-        """ Retrieve a message.
-
-        Args:
-            msg_id (str): The message ID to obtain.
-            room_id (str): The room where the message resides.
-            sender_id (str): The user ID of the user who sent the message.
-            user_id (str): The user ID of the user making this request.
-        Returns:
-            The message, or None if no message exists.
-        Raises:
-            SynapseError if something went wrong.
-        """
-        yield self.auth.check_joined_room(room_id, user_id)
-
-        # Pull out the message from the db
-        msg = yield self.store.get_message(room_id=room_id,
-                                           msg_id=msg_id,
-                                           user_id=sender_id)
-
-        if msg:
-            defer.returnValue(msg)
-        defer.returnValue(None)
-
-    @defer.inlineCallbacks
-    def send_message(self, event=None, suppress_auth=False, stamp_event=True):
-        """ Send a message.
-
-        Args:
-            event : The message event to store.
-            suppress_auth (bool) : True to suppress auth for this message. This
-            is primarily so the home server can inject messages into rooms at
-            will.
-            stamp_event (bool) : True to stamp event content with server keys.
-        Raises:
-            SynapseError if something went wrong.
-        """
-        if stamp_event:
-            event.content["hsob_ts"] = int(self.clock.time_msec())
-
-        with (yield self.room_lock.lock(event.room_id)):
-            if not suppress_auth:
-                yield self.auth.check(event, raises=True)
-
-            # store message in db
-            store_id = yield self.store.persist_event(event)
-
-            event.destinations = yield self.store.get_joined_hosts_for_room(
-                event.room_id
-            )
-
-            yield self.hs.get_federation().handle_new_event(event)
-
-            self.notifier.on_new_room_event(event, store_id)
-
-    @defer.inlineCallbacks
-    def get_messages(self, user_id=None, room_id=None, pagin_config=None,
-                     feedback=False):
-        """Get messages in a room.
-
-        Args:
-            user_id (str): The user requesting messages.
-            room_id (str): The room they want messages from.
-            pagin_config (synapse.api.streams.PaginationConfig): The pagination
-            config rules to apply, if any.
-            feedback (bool): True to get compressed feedback with the messages
-        Returns:
-            dict: Pagination API results
-        """
-        yield self.auth.check_joined_room(room_id, user_id)
-
-        data_source = [MessagesStreamData(self.hs, room_id=room_id,
-                                          feedback=feedback)]
-        event_stream = EventStream(user_id, data_source)
-        pagin_config = yield event_stream.fix_tokens(pagin_config)
-        data_chunk = yield event_stream.get_chunk(config=pagin_config)
-        defer.returnValue(data_chunk)
-
-    @defer.inlineCallbacks
-    def store_room_data(self, event=None, stamp_event=True):
-        """ Stores data for a room.
-
-        Args:
-            event : The room path event
-            stamp_event (bool) : True to stamp event content with server keys.
-        Raises:
-            SynapseError if something went wrong.
-        """
-
-        with (yield self.room_lock.lock(event.room_id)):
-            yield self.auth.check(event, raises=True)
-
-            if stamp_event:
-                event.content["hsob_ts"] = int(self.clock.time_msec())
-
-            yield self.state_handler.handle_new_event(event)
-
-            # store in db
-            store_id = yield self.store.store_room_data(
-                room_id=event.room_id,
-                etype=event.type,
-                state_key=event.state_key,
-                content=json.dumps(event.content)
-            )
-
-            event.destinations = yield self.store.get_joined_hosts_for_room(
-                event.room_id
-            )
-            self.notifier.on_new_room_event(event, store_id)
-
-        yield self.hs.get_federation().handle_new_event(event)
-
-    @defer.inlineCallbacks
-    def get_room_data(self, user_id=None, room_id=None,
-                      event_type=None, state_key="",
-                      public_room_rules=[],
-                      private_room_rules=["join"]):
-        """ Get data from a room.
-
-        Args:
-            event : The room path event
-            public_room_rules : A list of membership states the user can be in,
-            in order to read this data IN A PUBLIC ROOM. An empty list means
-            'any state'.
-            private_room_rules : A list of membership states the user can be
-            in, in order to read this data IN A PRIVATE ROOM. An empty list
-            means 'any state'.
-        Returns:
-            The path data content.
-        Raises:
-            SynapseError if something went wrong.
-        """
-        if event_type == RoomTopicEvent.TYPE:
-            # anyone invited/joined can read the topic
-            private_room_rules = ["invite", "join"]
-
-        # does this room exist
-        room = yield self.store.get_room(room_id)
-        if not room:
-            raise RoomError(403, "Room does not exist.")
-
-        # does this user exist in this room
-        member = yield self.store.get_room_member(
-            room_id=room_id,
-            user_id="" if not user_id else user_id)
-
-        member_state = member.membership if member else None
-
-        if room.is_public and public_room_rules:
-            # make sure the user meets public room rules
-            if member_state not in public_room_rules:
-                raise RoomError(403, "Member does not meet public room rules.")
-        elif not room.is_public and private_room_rules:
-            # make sure the user meets private room rules
-            if member_state not in private_room_rules:
-                raise RoomError(
-                    403, "Member does not meet private room rules.")
-
-        data = yield self.store.get_room_data(room_id, event_type, state_key)
-        defer.returnValue(data)
-
-    @defer.inlineCallbacks
-    def get_feedback(self, room_id=None, msg_sender_id=None, msg_id=None,
-                     user_id=None, fb_sender_id=None, fb_type=None):
-        yield self.auth.check_joined_room(room_id, user_id)
-
-        # Pull out the feedback from the db
-        fb = yield self.store.get_feedback(
-            room_id=room_id, msg_id=msg_id, msg_sender_id=msg_sender_id,
-            fb_sender_id=fb_sender_id, fb_type=fb_type
-        )
-
-        if fb:
-            defer.returnValue(fb)
-        defer.returnValue(None)
-
-    @defer.inlineCallbacks
-    def send_feedback(self, event, stamp_event=True):
-        if stamp_event:
-            event.content["hsob_ts"] = int(self.clock.time_msec())
-
-        with (yield self.room_lock.lock(event.room_id)):
-            yield self.auth.check(event, raises=True)
-
-            # store message in db
-            store_id = yield self.store.persist_event(event)
-
-            event.destinations = yield self.store.get_joined_hosts_for_room(
-                event.room_id
-            )
-        yield self.hs.get_federation().handle_new_event(event)
-
-        self.notifier.on_new_room_event(event, store_id)
-
-    @defer.inlineCallbacks
-    def snapshot_all_rooms(self, user_id=None, pagin_config=None,
-                           feedback=False):
-        """Retrieve a snapshot of all rooms the user is invited or has joined.
-
-        This snapshot may include messages for all rooms where the user is
-        joined, depending on the pagination config.
-
-        Args:
-            user_id (str): The ID of the user making the request.
-            pagin_config (synapse.api.streams.PaginationConfig): The pagination
-            config used to determine how many messages *PER ROOM* to return.
-            feedback (bool): True to get feedback along with these messages.
-        Returns:
-            A list of dicts with "room_id" and "membership" keys for all rooms
-            the user is currently invited or joined in on. Rooms where the user
-            is joined on, may return a "messages" key with messages, depending
-            on the specified PaginationConfig.
-        """
-        room_list = yield self.store.get_rooms_for_user_where_membership_is(
-            user_id=user_id,
-            membership_list=[Membership.INVITE, Membership.JOIN]
-        )
-        for room_info in room_list:
-            if room_info["membership"] != Membership.JOIN:
-                continue
-            try:
-                event_chunk = yield self.get_messages(
-                    user_id=user_id,
-                    pagin_config=pagin_config,
-                    feedback=feedback,
-                    room_id=room_info["room_id"]
-                )
-                room_info["messages"] = event_chunk
-            except:
-                pass
-        defer.returnValue(room_list)
-
-
-class RoomCreationHandler(BaseHandler):
+class RoomCreationHandler(BaseRoomHandler):
 
     @defer.inlineCallbacks
     def create_room(self, user_id, room_id, config):
@@ -347,6 +101,13 @@ class RoomCreationHandler(BaseHandler):
             content=config,
         )
 
+        snapshot = yield self.store.snapshot_room(
+            room_id=room_id,
+            user_id=user_id,
+            state_type=RoomConfigEvent.TYPE,
+            state_key="",
+        )
+
         if room_alias:
             yield self.store.create_room_alias_association(
                 room_id=room_id,
@@ -354,16 +115,16 @@ class RoomCreationHandler(BaseHandler):
                 servers=[self.hs.hostname],
             )
 
-        yield self.state_handler.handle_new_event(config_event)
+        yield self.state_handler.handle_new_event(config_event, snapshot)
         # store_id = persist...
 
-        yield self.hs.get_federation().handle_new_event(config_event)
-        # self.notifier.on_new_room_event(event, store_id)
+        federation_handler = self.hs.get_handlers().federation_handler
+        yield federation_handler.handle_new_event(config_event, snapshot)
 
         content = {"membership": Membership.JOIN}
         join_event = self.event_factory.create_event(
             etype=RoomMemberEvent.TYPE,
-            target_user_id=user_id,
+            state_key=user_id,
             room_id=room_id,
             user_id=user_id,
             membership=Membership.JOIN,
@@ -372,7 +133,6 @@ class RoomCreationHandler(BaseHandler):
 
         yield self.hs.get_handlers().room_member_handler.change_membership(
             join_event,
-            broadcast_msg=True,
             do_auth=False
         )
 
@@ -383,7 +143,7 @@ class RoomCreationHandler(BaseHandler):
         defer.returnValue(result)
 
 
-class RoomMemberHandler(BaseHandler):
+class RoomMemberHandler(BaseRoomHandler):
     # TODO(paul): This handler currently contains a messy conflation of
     #   low-level API that works on UserID objects and so on, and REST-level
     #   API that takes ID strings and returns pagination chunks. These concerns
@@ -451,11 +211,11 @@ class RoomMemberHandler(BaseHandler):
 
         member_list = yield self.store.get_room_members(room_id=room_id)
         event_list = [
-            entry.as_event(self.event_factory).get_dict()
+            entry.get_dict()
             for entry in member_list
         ]
         chunk_data = {
-            "start": "START",
+            "start": "START",  # FIXME (erikj): START is no longer valid
             "end": "END",
             "chunk": event_list
         }
@@ -484,29 +244,34 @@ class RoomMemberHandler(BaseHandler):
         defer.returnValue(member)
 
     @defer.inlineCallbacks
-    def change_membership(self, event=None, broadcast_msg=False, do_auth=True):
+    def change_membership(self, event=None, do_auth=True):
         """ Change the membership status of a user in a room.
 
         Args:
             event (SynapseEvent): The membership event
-            broadcast_msg (bool): True to inject a membership message into this
-                room on success.
         Raises:
             SynapseError if there was a problem changing the membership.
         """
+        target_user_id = event.state_key
 
-        #broadcast_msg = False
-
+        snapshot = yield self.store.snapshot_room(
+            event.room_id, event.user_id,
+            RoomMemberEvent.TYPE, target_user_id
+        )
+        ## TODO(markjh): get prev state from snapshot.
         prev_state = yield self.store.get_room_member(
-            event.target_user_id, event.room_id
+            target_user_id, event.room_id
         )
 
-        if prev_state and prev_state.membership == event.membership:
-            # treat this event as a NOOP.
-            if do_auth:  # This is mainly to fix a unit test.
-                yield self.auth.check(event, raises=True)
-            defer.returnValue({})
-            return
+        if prev_state:
+            event.content["prev"] = prev_state.membership
+
+#        if prev_state and prev_state.membership == event.membership:
+#            # treat this event as a NOOP.
+#            if do_auth:  # This is mainly to fix a unit test.
+#                yield self.auth.check(event, raises=True)
+#            defer.returnValue({})
+#            return
 
         room_id = event.room_id
 
@@ -514,27 +279,22 @@ class RoomMemberHandler(BaseHandler):
         # if this HS is not currently in the room, i.e. we have to do the
         # invite/join dance.
         if event.membership == Membership.JOIN:
-            yield self._do_join(
-                event, do_auth=do_auth, broadcast_msg=broadcast_msg
-            )
+            yield self._do_join(event, snapshot, do_auth=do_auth)
         else:
             # This is not a JOIN, so we can handle it normally.
             if do_auth:
-                yield self.auth.check(event, raises=True)
+                yield self.auth.check(event, snapshot, raises=True)
 
-            prev_state = yield self.store.get_room_member(
-                event.target_user_id, event.room_id
-            )
             if prev_state and prev_state.membership == event.membership:
                 # double same action, treat this event as a NOOP.
                 defer.returnValue({})
                 return
 
-            yield self.state_handler.handle_new_event(event)
+            yield self.state_handler.handle_new_event(event, snapshot)
             yield self._do_local_membership_update(
                 event,
                 membership=event.content["membership"],
-                broadcast_msg=broadcast_msg,
+                snapshot=snapshot,
             )
 
         defer.returnValue({"room_id": room_id})
@@ -557,26 +317,31 @@ class RoomMemberHandler(BaseHandler):
         content.update({"membership": Membership.JOIN})
         new_event = self.event_factory.create_event(
             etype=RoomMemberEvent.TYPE,
-            target_user_id=joinee.to_string(),
+            state_key=joinee.to_string(),
             room_id=room_id,
             user_id=joinee.to_string(),
             membership=Membership.JOIN,
             content=content,
         )
 
-        yield self._do_join(new_event, room_host=host, do_auth=True)
+        snapshot = yield self.store.snapshot_room(
+            room_id, joinee.to_string(), RoomMemberEvent.TYPE,
+            joinee.to_string()
+        )
+
+        yield self._do_join(new_event, snapshot, room_host=host, do_auth=True)
 
         defer.returnValue({"room_id": room_id})
 
     @defer.inlineCallbacks
-    def _do_join(self, event, room_host=None, do_auth=True, broadcast_msg=True):
-        joinee = self.hs.parse_userid(event.target_user_id)
+    def _do_join(self, event, snapshot, room_host=None, do_auth=True):
+        joinee = self.hs.parse_userid(event.state_key)
         # room_id = RoomID.from_string(event.room_id, self.hs)
         room_id = event.room_id
 
         # If event doesn't include a display name, add one.
-        yield self._fill_out_join_content(
-            joinee, event.content
+        yield self.distributor.fire(
+            "collect_presencelike_data", joinee, event.content
         )
 
         # XXX: We don't do an auth check if we are doing an invite
@@ -584,13 +349,14 @@ class RoomMemberHandler(BaseHandler):
         # that we are allowed to join when we decide whether or not we
         # need to do the invite/join dance.
 
-        room = yield self.store.get_room(room_id)
+        hosts = yield self.store.get_joined_hosts_for_room(room_id)
 
-        if room:
+        if self.hs.hostname in hosts:
             should_do_dance = False
         elif room_host:
             should_do_dance = True
         else:
+            # TODO(markjh): get prev_state from snapshot
             prev_state = yield self.store.get_room_member(
                 joinee.to_string(), room_id
             )
@@ -598,7 +364,7 @@ class RoomMemberHandler(BaseHandler):
             if prev_state and prev_state.membership == Membership.INVITE:
                 room = yield self.store.get_room(room_id)
                 inviter = UserID.from_string(
-                    prev_state.sender, self.hs
+                    prev_state.user_id, self.hs
                 )
 
                 should_do_dance = not inviter.is_mine and not room
@@ -606,27 +372,25 @@ class RoomMemberHandler(BaseHandler):
             else:
                 should_do_dance = False
 
+        have_joined = False
+        if should_do_dance:
+            handler = self.hs.get_handlers().federation_handler
+            have_joined = yield handler.do_invite_join(
+                room_host, room_id, event.user_id, event.content, snapshot
+            )
+
         # We want to do the _do_update inside the room lock.
-        if not should_do_dance:
+        if not have_joined:
             logger.debug("Doing normal join")
 
             if do_auth:
-                yield self.auth.check(event, raises=True)
+                yield self.auth.check(event, snapshot, raises=True)
 
-            yield self.state_handler.handle_new_event(event)
+            yield self.state_handler.handle_new_event(event, snapshot)
             yield self._do_local_membership_update(
                 event,
                 membership=event.content["membership"],
-                broadcast_msg=broadcast_msg,
-            )
-
-
-        if should_do_dance:
-            yield self._do_invite_join_dance(
-                room_id=room_id,
-                joinee=event.user_id,
-                target_host=room_host,
-                content=event.content,
+                snapshot=snapshot,
             )
 
         user = self.hs.parse_userid(event.user_id)
@@ -634,32 +398,6 @@ class RoomMemberHandler(BaseHandler):
             "user_joined_room", user=user, room_id=room_id
         )
 
-    @defer.inlineCallbacks
-    def _fill_out_join_content(self, user_id, content):
-        # If event doesn't include a display name, add one.
-        profile_handler = self.hs.get_handlers().profile_handler
-        if "displayname" not in content:
-            try:
-                display_name = yield profile_handler.get_displayname(
-                    user_id
-                )
-
-                if display_name:
-                    content["displayname"] = display_name
-            except:
-                logger.exception("Failed to set display_name")
-
-        if "avatar_url" not in content:
-            try:
-                avatar_url = yield profile_handler.get_avatar_url(
-                    user_id
-                )
-
-                if avatar_url:
-                    content["avatar_url"] = avatar_url
-            except:
-                logger.exception("Failed to set display_name")
-
     @defer.inlineCallbacks
     def _should_invite_join(self, room_id, prev_state, do_auth):
         logger.debug("_should_invite_join: room_id: %s", room_id)
@@ -694,116 +432,33 @@ class RoomMemberHandler(BaseHandler):
             user_id=user.to_string(), membership_list=membership_list
         )
 
-        defer.returnValue([r["room_id"] for r in rooms])
+        defer.returnValue([r.room_id for r in rooms])
 
-    @defer.inlineCallbacks
-    def _do_local_membership_update(self, event, membership, broadcast_msg):
-        # store membership
-        store_id = yield self.store.store_room_member(
-            user_id=event.target_user_id,
-            sender=event.user_id,
-            room_id=event.room_id,
-            content=event.content,
-            membership=membership
-        )
-
-        # Send a PDU to all hosts who have joined the room.
-        destinations = yield self.store.get_joined_hosts_for_room(
-            event.room_id
-        )
+    def _do_local_membership_update(self, event, membership, snapshot):
+        destinations = []
 
         # If we're inviting someone, then we should also send it to that
         # HS.
+        target_user_id = event.state_key
+        target_user = self.hs.parse_userid(target_user_id)
         if membership == Membership.INVITE:
-            host = UserID.from_string(
-                event.target_user_id, self.hs
-            ).domain
+            host = target_user.domain
             destinations.append(host)
 
         # If we are joining a remote HS, include that.
         if membership == Membership.JOIN:
-            host = UserID.from_string(
-                event.target_user_id, self.hs
-            ).domain
+            host = target_user.domain
             destinations.append(host)
 
-        event.destinations = list(set(destinations))
-
-        yield self.hs.get_federation().handle_new_event(event)
-        self.notifier.on_new_room_event(event, store_id)
-
-        if broadcast_msg:
-            yield self._inject_membership_msg(
-                source=event.user_id,
-                target=event.target_user_id,
-                room_id=event.room_id,
-                membership=event.content["membership"]
-            )
-
-    @defer.inlineCallbacks
-    def _do_invite_join_dance(self, room_id, joinee, target_host, content):
-        logger.debug("Doing remote join dance")
-
-        # do invite join dance
-        federation = self.hs.get_federation()
-        new_event = self.event_factory.create_event(
-            etype=InviteJoinEvent.TYPE,
-            target_host=target_host,
-            room_id=room_id,
-            user_id=joinee,
-            content=content
+        return self._on_new_room_event(
+            event, snapshot, extra_destinations=destinations,
+            extra_users=[target_user]
         )
 
-        new_event.destinations = [target_host]
-
-        yield self.store.store_room(
-            room_id, "", is_public=False
-        )
-
-        #yield self.state_handler.handle_new_event(event)
-        yield federation.handle_new_event(new_event)
-        yield federation.get_state_for_room(
-            target_host, room_id
-        )
-
-    @defer.inlineCallbacks
-    def _inject_membership_msg(self, room_id=None, source=None, target=None,
-                               membership=None):
-        # TODO this should be a different type of message, not m.text
-        if membership == Membership.INVITE:
-            body = "%s invited %s to the room." % (source, target)
-        elif membership == Membership.JOIN:
-            body = "%s joined the room." % (target)
-        elif membership == Membership.LEAVE:
-            body = "%s left the room." % (target)
-        else:
-            raise RoomError(500, "Unknown membership value %s" % membership)
-
-        membership_json = {
-            "msgtype": u"m.text",
-            "body": body,
-            "membership_source": source,
-            "membership_target": target,
-            "membership": membership,
-        }
-
-        msg_id = "m%s" % int(self.clock.time_msec())
-
-        event = self.event_factory.create_event(
-            etype=MessageEvent.TYPE,
-            room_id=room_id,
-            user_id="_homeserver_",
-            msg_id=msg_id,
-            content=membership_json
-        )
-
-        handler = self.hs.get_handlers().message_handler
-        yield handler.send_message(event, suppress_auth=True)
-
-
-class RoomListHandler(BaseHandler):
+class RoomListHandler(BaseRoomHandler):
 
     @defer.inlineCallbacks
     def get_public_room_list(self):
-        chunk = yield self.store.get_rooms(is_public=True, with_topics=True)
+        chunk = yield self.store.get_rooms(is_public=True)
+        # FIXME (erikj): START is no longer a valid value
         defer.returnValue({"start": "START", "end": "END", "chunk": chunk})

synapse/handlers/typing.py

@@ -0,0 +1,147 @@
+# -*- coding: utf-8 -*-
+# Copyright 2014 matrix.org
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from twisted.internet import defer
+
+from ._base import BaseHandler
+
+from synapse.api.errors import SynapseError, AuthError
+
+import logging
+
+from collections import namedtuple
+
+logger = logging.getLogger(__name__)
+
+
+# A tiny object useful for storing a user's membership in a room, as a mapping
+# key
+RoomMember = namedtuple("RoomMember", ("room_id", "user"))
+
+
+class TypingNotificationHandler(BaseHandler):
+    def __init__(self, hs):
+        super(TypingNotificationHandler, self).__init__(hs)
+
+        self.homeserver = hs
+
+        self.clock = hs.get_clock()
+
+        self.federation = hs.get_replication_layer()
+
+        self.federation.register_edu_handler("m.typing", self._recv_edu)
+
+        self._member_typing_until = {}
+
+    @defer.inlineCallbacks
+    def started_typing(self, target_user, auth_user, room_id, timeout):
+        if not target_user.is_mine:
+            raise SynapseError(400, "User is not hosted on this Home Server")
+
+        if target_user != auth_user:
+            raise AuthError(400, "Cannot set another user's typing state")
+
+        until = self.clock.time_msec() + timeout
+        member = RoomMember(room_id=room_id, user=target_user)
+
+        was_present = member in self._member_typing_until
+
+        self._member_typing_until[member] = until
+
+        if was_present:
+            # No point sending another notification
+            defer.returnValue(None)
+
+        yield self._push_update(
+            room_id=room_id,
+            user=target_user,
+            typing=True,
+        )
+
+    @defer.inlineCallbacks
+    def stopped_typing(self, target_user, auth_user, room_id):
+        if not target_user.is_mine:
+            raise SynapseError(400, "User is not hosted on this Home Server")
+
+        if target_user != auth_user:
+            raise AuthError(400, "Cannot set another user's typing state")
+
+        member = RoomMember(room_id=room_id, user=target_user)
+
+        if member not in self._member_typing_until:
+            # No point
+            defer.returnValue(None)
+
+        yield self._push_update(
+            room_id=room_id,
+            user=target_user,
+            typing=False,
+        )
+
+    @defer.inlineCallbacks
+    def _push_update(self, room_id, user, typing):
+        localusers = set()
+        remotedomains = set()
+
+        rm_handler = self.homeserver.get_handlers().room_member_handler
+        yield rm_handler.fetch_room_distributions_into(room_id,
+                localusers=localusers, remotedomains=remotedomains,
+                ignore_user=user)
+
+        for u in localusers:
+            self.push_update_to_clients(
+                room_id=room_id,
+                observer_user=u,
+                observed_user=user,
+                typing=typing,
+            )
+
+        deferreds = []
+        for domain in remotedomains:
+            deferreds.append(self.federation.send_edu(
+                destination=domain,
+                edu_type="m.typing",
+                content={
+                    "room_id": room_id,
+                    "user_id": user.to_string(),
+                    "typing": typing,
+                },
+            ))
+
+        yield defer.DeferredList(deferreds, consumeErrors=False)
+
+    @defer.inlineCallbacks
+    def _recv_edu(self, origin, content):
+        room_id = content["room_id"]
+        user = self.homeserver.parse_userid(content["user_id"])
+
+        localusers = set()
+
+        rm_handler = self.homeserver.get_handlers().room_member_handler
+        yield rm_handler.fetch_room_distributions_into(room_id,
+                localusers=localusers)
+
+        for u in localusers:
+            self.push_update_to_clients(
+                room_id=room_id,
+                observer_user=u,
+                observed_user=user,
+                typing=content["typing"]
+            )
+
+    def push_update_to_clients(self, room_id, observer_user, observed_user,
+            typing):
+        # TODO(paul) steal this from presence.py
+        pass

synapse/http/server.py

@@ -17,15 +17,23 @@
 from syutil.jsonutil import (
     encode_canonical_json, encode_pretty_printed_json
 )
-from synapse.api.errors import cs_exception, CodeMessageException
+from synapse.api.errors import (
+    cs_exception, SynapseError, CodeMessageException, Codes, cs_error
+)
+from synapse.util.stringutils import random_string
 
 from twisted.internet import defer, reactor
+from twisted.protocols.basic import FileSender
 from twisted.web import server, resource
 from twisted.web.server import NOT_DONE_YET
+from twisted.web.util import redirectTo
 
+import base64
 import collections
+import json
 import logging
-
+import os
+import re
 
 logger = logging.getLogger(__name__)
 
@@ -52,10 +60,9 @@ class HttpServer(object):
         pass
 
 
-# The actual HTTP server impl, using twisted http server
-class TwistedHttpServer(HttpServer, resource.Resource):
-    """ This wraps the twisted HTTP server, and triggers the correct callbacks
-    on the transport_layer.
+class JsonResource(HttpServer, resource.Resource):
+    """ This implements the HttpServer interface and provides JSON support for
+    Resources.
 
     Register callbacks via register_path()
     """
@@ -125,7 +132,11 @@ class TwistedHttpServer(HttpServer, resource.Resource):
                 {"error": "Unrecognized request"}
             )
         except CodeMessageException as e:
-            logger.exception(e)
+            if isinstance(e, SynapseError):
+                logger.error("%s SynapseError: %s - %s", request, e.code,
+                             e.msg)
+            else:
+                logger.exception(e)
             self._send_response(
                 request,
                 e.code,
@@ -140,6 +151,14 @@ class TwistedHttpServer(HttpServer, resource.Resource):
             )
 
     def _send_response(self, request, code, response_json_object):
+        # could alternatively use request.notifyFinish() and flip a flag when
+        # the Deferred fires, but since the flag is RIGHT THERE it seems like
+        # a waste.
+        if request._disconnected:
+            logger.warn(
+                "Not sending response to request %s, already disconnected.",
+                request)
+            return
 
         if not self._request_user_agent_is_curl(request):
             json_bytes = encode_canonical_json(response_json_object)
@@ -160,6 +179,173 @@ class TwistedHttpServer(HttpServer, resource.Resource):
         return False
 
 
+class RootRedirect(resource.Resource):
+    """Redirects the root '/' path to another path."""
+
+    def __init__(self, path):
+        resource.Resource.__init__(self)
+        self.url = path
+
+    def render_GET(self, request):
+        return redirectTo(self.url, request)
+
+    def getChild(self, name, request):
+        if len(name) == 0:
+            return self  # select ourselves as the child to render
+        return resource.Resource.getChild(self, name, request)
+
+
+class ContentRepoResource(resource.Resource):
+    """Provides file uploading and downloading.
+
+    Uploads are POSTed to wherever this Resource is linked to. This resource
+    returns a "content token" which can be used to GET this content again. The
+    token is typically a path, but it may not be. Tokens can expire, be one-time
+    uses, etc.
+
+    In this case, the token is a path to the file and contains 3 interesting
+    sections:
+        - User ID base64d (for namespacing content to each user)
+        - random 24 char string
+        - Content type base64d (so we can return it when clients GET it)
+
+    """
+    isLeaf = True
+
+    def __init__(self, hs, directory, auth):
+        resource.Resource.__init__(self)
+        self.hs = hs
+        self.directory = directory
+        self.auth = auth
+
+        if not os.path.isdir(self.directory):
+            os.mkdir(self.directory)
+            logger.info("ContentRepoResource : Created %s directory.",
+                        self.directory)
+
+    @defer.inlineCallbacks
+    def map_request_to_name(self, request):
+        # auth the user
+        auth_user = yield self.auth.get_user_by_req(request)
+
+        # namespace all file uploads on the user
+        prefix = base64.urlsafe_b64encode(
+            auth_user.to_string()
+        ).replace('=', '')
+
+        # use a random string for the main portion
+        main_part = random_string(24)
+
+        # suffix with a file extension if we can make one. This is nice to
+        # provide a hint to clients on the file information. We will also reuse
+        # this info to spit back the content type to the client.
+        suffix = ""
+        if request.requestHeaders.hasHeader("Content-Type"):
+            content_type = request.requestHeaders.getRawHeaders(
+                "Content-Type")[0]
+            suffix = "." + base64.urlsafe_b64encode(content_type)
+            if (content_type.split("/")[0].lower() in
+                    ["image", "video", "audio"]):
+                file_ext = content_type.split("/")[-1]
+                # be a little paranoid and only allow a-z
+                file_ext = re.sub("[^a-z]", "", file_ext)
+                suffix += "." + file_ext
+
+        file_name = prefix + main_part + suffix
+        file_path = os.path.join(self.directory, file_name)
+        logger.info("User %s is uploading a file to path %s",
+                    auth_user.to_string(),
+                    file_path)
+
+        # keep trying to make a non-clashing file, with a sensible max attempts
+        attempts = 0
+        while os.path.exists(file_path):
+            main_part = random_string(24)
+            file_name = prefix + main_part + suffix
+            file_path = os.path.join(self.directory, file_name)
+            attempts += 1
+            if attempts > 25:  # really? Really?
+                raise SynapseError(500, "Unable to create file.")
+
+        defer.returnValue(file_path)
+
+    def render_GET(self, request):
+        # no auth here on purpose, to allow anyone to view, even across home
+        # servers.
+
+        # TODO: A little crude here, we could do this better.
+        filename = request.path.split('/')[-1]
+        # be paranoid
+        filename = re.sub("[^0-9A-z.-_]", "", filename)
+
+        file_path = self.directory + "/" + filename
+
+        logger.debug("Searching for %s", file_path)
+
+        if os.path.isfile(file_path):
+            # filename has the content type
+            base64_contentype = filename.split(".")[1]
+            content_type = base64.urlsafe_b64decode(base64_contentype)
+            logger.info("Sending file %s", file_path)
+            f = open(file_path, 'rb')
+            request.setHeader('Content-Type', content_type)
+            d = FileSender().beginFileTransfer(f, request)
+
+            # after the file has been sent, clean up and finish the request
+            def cbFinished(ignored):
+                f.close()
+                request.finish()
+            d.addCallback(cbFinished)
+        else:
+            respond_with_json_bytes(
+                request,
+                404,
+                json.dumps(cs_error("Not found", code=Codes.NOT_FOUND)),
+                send_cors=True)
+
+        return server.NOT_DONE_YET
+
+    def render_POST(self, request):
+        self._async_render(request)
+        return server.NOT_DONE_YET
+
+    def render_OPTIONS(self, request):
+        respond_with_json_bytes(request, 200, {}, send_cors=True)
+        return server.NOT_DONE_YET
+
+    @defer.inlineCallbacks
+    def _async_render(self, request):
+        try:
+            fname = yield self.map_request_to_name(request)
+
+            # TODO I have a suspcious feeling this is just going to block
+            with open(fname, "wb") as f:
+                f.write(request.content.read())
+
+
+            # FIXME (erikj): These should use constants.
+            file_name = os.path.basename(fname)
+            url = "http://%s/matrix/content/%s" % (
+                self.hs.domain_with_port, file_name
+            )
+
+            respond_with_json_bytes(request, 200,
+                                    json.dumps({"content_token": url}),
+                                    send_cors=True)
+
+        except CodeMessageException as e:
+            logger.exception(e)
+            respond_with_json_bytes(request, e.code,
+                                    json.dumps(cs_exception(e)))
+        except Exception as e:
+            logger.error("Failed to store file: %s" % e)
+            respond_with_json_bytes(
+                request,
+                500,
+                json.dumps({"error": "Internal server error"}),
+                send_cors=True)
+
+
 def respond_with_json_bytes(request, code, json_bytes, send_cors=False):
     """Sends encoded JSON in response to the given request.
 

synapse/notifier.py

@@ -0,0 +1,258 @@
+# -*- coding: utf-8 -*-
+# Copyright 2014 matrix.org
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from twisted.internet import defer, reactor
+
+from synapse.util.logutils import log_function
+
+import logging
+
+
+logger = logging.getLogger(__name__)
+
+
+class _NotificationListener(object):
+    """ This represents a single client connection to the events stream.
+
+    The events stream handler will have yielded to the deferred, so to
+    notify the handler it is sufficient to resolve the deferred.
+
+    This listener will also keep track of which rooms it is listening in
+    so that it can remove itself from the indexes in the Notifier class.
+    """
+
+    def __init__(self, user, rooms, from_token, limit, timeout, deferred):
+        self.user = user
+        self.from_token = from_token
+        self.limit = limit
+        self.timeout = timeout
+        self.deferred = deferred
+
+        self.rooms = rooms
+
+        self.pending_notifications = []
+
+    def notify(self, notifier, events, start_token, end_token):
+        """ Inform whoever is listening about the new events. This will
+        also remove this listener from all the indexes in the Notifier
+        it knows about.
+        """
+
+        result = (events, (start_token, end_token))
+
+        try:
+            self.deferred.callback(result)
+        except defer.AlreadyCalledError:
+            pass
+
+        for room in self.rooms:
+            lst = notifier.rooms_to_listeners.get(room, set())
+            lst.discard(self)
+
+        notifier.user_to_listeners.get(self.user, set()).discard(self)
+
+
+class Notifier(object):
+    """ This class is responsible for notifying any listeners when there are
+    new events available for it.
+
+    Primarily used from the /events stream.
+    """
+
+    def __init__(self, hs):
+        self.hs = hs
+
+        self.rooms_to_listeners = {}
+        self.user_to_listeners = {}
+
+        self.event_sources = hs.get_event_sources()
+
+        hs.get_distributor().observe(
+            "user_joined_room", self._user_joined_room
+        )
+
+    @log_function
+    @defer.inlineCallbacks
+    def on_new_room_event(self, event, extra_users=[]):
+        """ Used by handlers to inform the notifier something has happened
+        in the room, room event wise.
+
+        This triggers the notifier to wake up any listeners that are
+        listening to the room, and any listeners for the users in the
+        `extra_users` param.
+        """
+        room_id = event.room_id
+
+        source = self.event_sources.sources["room"]
+
+        listeners = self.rooms_to_listeners.get(room_id, set()).copy()
+
+        for user in extra_users:
+            listeners |= self.user_to_listeners.get(user, set()).copy()
+
+        logger.debug("on_new_room_event listeners %s", listeners)
+
+        # TODO (erikj): Can we make this more efficient by hitting the
+        # db once?
+
+        @defer.inlineCallbacks
+        def notify(listener):
+            events, end_token = yield source.get_new_events_for_user(
+                listener.user,
+                listener.from_token,
+                listener.limit,
+            )
+
+            if events:
+                listener.notify(
+                    self, events, listener.from_token, end_token
+                )
+
+        def eb(failure):
+            logger.exception("Failed to notify listener", failure)
+
+        yield defer.DeferredList(
+            [notify(l).addErrback(eb) for l in listeners]
+        )
+
+    @defer.inlineCallbacks
+    @log_function
+    def on_new_user_event(self, users=[], rooms=[]):
+        """ Used to inform listeners that something has happend
+        presence/user event wise.
+
+        Will wake up all listeners for the given users and rooms.
+        """
+        source = self.event_sources.sources["presence"]
+
+        listeners = set()
+
+        for user in users:
+            listeners |= self.user_to_listeners.get(user, set()).copy()
+
+        for room in rooms:
+            listeners |= self.rooms_to_listeners.get(room, set()).copy()
+
+        @defer.inlineCallbacks
+        def notify(listener):
+            events, end_token = yield source.get_new_events_for_user(
+                listener.user,
+                listener.from_token,
+                listener.limit,
+            )
+
+            if events:
+                listener.notify(
+                    self, events, listener.from_token, end_token
+                )
+
+        def eb(failure):
+            logger.exception("Failed to notify listener", failure)
+
+        yield defer.DeferredList(
+            [notify(l).addErrback(eb) for l in listeners]
+        )
+
+    def get_events_for(self, user, rooms, pagination_config, timeout):
+        """ For the given user and rooms, return any new events for them. If
+        there are no new events wait for up to `timeout` milliseconds for any
+        new events to happen before returning.
+        """
+        deferred = defer.Deferred()
+
+        self._get_events(
+            deferred, user, rooms, pagination_config.from_token,
+            pagination_config.limit, timeout
+        ).addErrback(deferred.errback)
+
+        return deferred
+
+    @defer.inlineCallbacks
+    def _get_events(self, deferred, user, rooms, from_token, limit, timeout):
+        if not from_token:
+            from_token = yield self.event_sources.get_current_token()
+
+        listener = _NotificationListener(
+            user,
+            rooms,
+            from_token,
+            limit,
+            timeout,
+            deferred,
+        )
+
+        if timeout:
+            reactor.callLater(timeout/1000, self._timeout_listener, listener)
+
+            self._register_with_keys(listener)
+
+        yield self._check_for_updates(listener)
+
+        if not timeout:
+            self._timeout_listener(listener)
+
+        return
+
+    def _timeout_listener(self, listener):
+        # TODO (erikj): We should probably set to_token to the current max
+        # rather than reusing from_token.
+        listener.notify(
+            self,
+            [],
+            listener.from_token,
+            listener.from_token,
+        )
+
+    @log_function
+    def _register_with_keys(self, listener):
+        for room in listener.rooms:
+            s = self.rooms_to_listeners.setdefault(room, set())
+            s.add(listener)
+
+        self.user_to_listeners.setdefault(listener.user, set()).add(listener)
+
+    @defer.inlineCallbacks
+    @log_function
+    def _check_for_updates(self, listener):
+        # TODO (erikj): We need to think about limits across multiple sources
+        events = []
+
+        from_token = listener.from_token
+        limit = listener.limit
+
+        # TODO (erikj): DeferredList?
+        for source in self.event_sources.sources.values():
+            stuff, new_token = yield source.get_new_events_for_user(
+                listener.user,
+                from_token,
+                limit,
+            )
+
+            events.extend(stuff)
+
+            from_token = new_token
+
+        end_token = from_token
+
+        if events:
+            listener.notify(self, events, listener.from_token, end_token)
+
+        defer.returnValue(listener)
+
+    def _user_joined_room(self, user, room_id):
+        new_listeners = self.user_to_listeners.get(user, set())
+
+        listeners = self.rooms_to_listeners.setdefault(room_id, set())
+        listeners |= new_listeners

synapse/rest/__init__.py

@@ -15,8 +15,7 @@
 
 
 from . import (
-    room, events, register, login, profile, public, presence, im, directory,
-    webclient
+    room, events, register, login, profile, presence, initial_sync, directory
 )
 
 
@@ -32,19 +31,14 @@ class RestServletFactory(object):
     """
 
     def __init__(self, hs):
-        http_server = hs.get_http_server()
+        client_resource = hs.get_resource_for_client()
 
         # TODO(erikj): There *must* be a better way of doing this.
-        room.register_servlets(hs, http_server)
-        events.register_servlets(hs, http_server)
-        register.register_servlets(hs, http_server)
-        login.register_servlets(hs, http_server)
-        profile.register_servlets(hs, http_server)
-        public.register_servlets(hs, http_server)
-        presence.register_servlets(hs, http_server)
-        im.register_servlets(hs, http_server)
-        directory.register_servlets(hs, http_server)
-
-    def register_web_client(self, hs):
-        http_server = hs.get_http_server()
-        webclient.register_servlets(hs, http_server)
+        room.register_servlets(hs, client_resource)
+        events.register_servlets(hs, client_resource)
+        register.register_servlets(hs, client_resource)
+        login.register_servlets(hs, client_resource)
+        profile.register_servlets(hs, client_resource)
+        presence.register_servlets(hs, client_resource)
+        initial_sync.register_servlets(hs, client_resource)
+        directory.register_servlets(hs, client_resource)

synapse/rest/base.py

@@ -14,6 +14,8 @@
 # limitations under the License.
 
 """ This module contains base REST classes for constructing REST servlets. """
+from synapse.api.urls import CLIENT_PREFIX
+from synapse.rest.transactions import HttpTransactionStore
 import re
 
 
@@ -27,7 +29,7 @@ def client_path_pattern(path_regex):
     Returns:
         SRE_Pattern
     """
-    return re.compile("^/matrix/client/api/v1" + path_regex)
+    return re.compile("^" + CLIENT_PREFIX + path_regex)
 
 
 class RestServlet(object):
@@ -58,6 +60,7 @@ class RestServlet(object):
         self.handlers = hs.get_handlers()
         self.event_factory = hs.get_event_factory()
         self.auth = hs.get_auth()
+        self.txns = HttpTransactionStore()
 
     def register(self, http_server):
         """ Register this servlet with the given HTTP server. """

synapse/rest/directory.py

@@ -31,7 +31,7 @@ def register_servlets(hs, http_server):
 
 
 class ClientDirectoryServer(RestServlet):
-    PATTERN = client_path_pattern("/ds/room/(?P<room_alias>[^/]*)$")
+    PATTERN = client_path_pattern("/directory/room/(?P<room_alias>[^/]*)$")
 
     @defer.inlineCallbacks
     def on_GET(self, request, room_alias):

synapse/rest/events.py

@@ -17,14 +17,14 @@
 from twisted.internet import defer
 
 from synapse.api.errors import SynapseError
-from synapse.api.streams import PaginationConfig
+from synapse.streams.config import PaginationConfig
 from synapse.rest.base import RestServlet, client_path_pattern
 
 
 class EventStreamRestServlet(RestServlet):
     PATTERN = client_path_pattern("/events$")
 
-    DEFAULT_LONGPOLL_TIME_MS = 5000
+    DEFAULT_LONGPOLL_TIME_MS = 30000
 
     @defer.inlineCallbacks
     def on_GET(self, request):
@@ -41,11 +41,29 @@ class EventStreamRestServlet(RestServlet):
 
         chunk = yield handler.get_stream(auth_user.to_string(), pagin_config,
                                          timeout=timeout)
+
         defer.returnValue((200, chunk))
 
     def on_OPTIONS(self, request):
         return (200, {})
 
 
+# TODO: Unit test gets, with and without auth, with different kinds of events.
+class EventRestServlet(RestServlet):
+    PATTERN = client_path_pattern("/events/(?P<event_id>[^/]*)$")
+
+    @defer.inlineCallbacks
+    def on_GET(self, request, event_id):
+        auth_user = yield self.auth.get_user_by_req(request)
+        handler = self.handlers.event_handler
+        event = yield handler.get_event(auth_user, event_id)
+
+        if event:
+            defer.returnValue((200, event.get_dict()))
+        else:
+            defer.returnValue((404, "Event not found."))
+
+
 def register_servlets(hs, http_server):
     EventStreamRestServlet(hs).register(http_server)
+    EventRestServlet(hs).register(http_server)

synapse/rest/initial_sync.py

@@ -15,12 +15,13 @@
 
 from twisted.internet import defer
 
-from synapse.api.streams import PaginationConfig
+from synapse.streams.config import PaginationConfig
 from base import RestServlet, client_path_pattern
 
 
-class ImSyncRestServlet(RestServlet):
-    PATTERN = client_path_pattern("/im/sync$")
+# TODO: Needs unit testing
+class InitialSyncRestServlet(RestServlet):
+    PATTERN = client_path_pattern("/initialSync$")
 
     @defer.inlineCallbacks
     def on_GET(self, request):
@@ -37,4 +38,4 @@ class ImSyncRestServlet(RestServlet):
 
 
 def register_servlets(hs, http_server):
-    ImSyncRestServlet(hs).register(http_server)
+    InitialSyncRestServlet(hs).register(http_server)

synapse/rest/login.py

@@ -16,6 +16,7 @@
 from twisted.internet import defer
 
 from synapse.api.errors import SynapseError
+from synapse.types import UserID
 from base import RestServlet, client_path_pattern
 
 import json
@@ -26,7 +27,7 @@ class LoginRestServlet(RestServlet):
     PASS_TYPE = "m.login.password"
 
     def on_GET(self, request):
-        return (200, {"type": LoginRestServlet.PASS_TYPE})
+        return (200, {"flows": [{"type": LoginRestServlet.PASS_TYPE}]})
 
     def on_OPTIONS(self, request):
         return (200, {})
@@ -45,12 +46,17 @@ class LoginRestServlet(RestServlet):
 
     @defer.inlineCallbacks
     def do_password_login(self, login_submission):
+        if not login_submission["user"].startswith('@'):
+            login_submission["user"] = UserID.create_local(
+                login_submission["user"], self.hs).to_string()
+
         handler = self.handlers.login_handler
         token = yield handler.login(
             user=login_submission["user"],
             password=login_submission["password"])
 
         result = {
+            "user_id": login_submission["user"],  # may have changed
             "access_token": token,
             "home_server": self.hs.hostname,
         }

synapse/rest/presence.py

@@ -68,7 +68,7 @@ class PresenceStatusRestServlet(RestServlet):
 
 
 class PresenceListRestServlet(RestServlet):
-    PATTERN = client_path_pattern("/presence_list/(?P<user_id>[^/]*)")
+    PATTERN = client_path_pattern("/presence/list/(?P<user_id>[^/]*)")
 
     @defer.inlineCallbacks
     def on_GET(self, request, user_id):

synapse/rest/register.py

@@ -33,10 +33,10 @@ class RegisterRestServlet(RestServlet):
         try:
             register_json = json.loads(request.content.read())
             if "password" in register_json:
-                password = register_json["password"]
+                password = register_json["password"].encode("utf-8")
 
             if type(register_json["user_id"]) == unicode:
-                desired_user_id = register_json["user_id"]
+                desired_user_id = register_json["user_id"].encode("utf-8")
                 if urllib.quote(desired_user_id) != desired_user_id:
                     raise SynapseError(
                         400,

synapse/rest/room.py

@@ -18,10 +18,9 @@ from twisted.internet import defer
 
 from base import RestServlet, client_path_pattern
 from synapse.api.errors import SynapseError, Codes
-from synapse.api.events.room import (RoomTopicEvent, MessageEvent,
-                                     RoomMemberEvent, FeedbackEvent)
-from synapse.api.constants import Feedback, Membership
-from synapse.api.streams import PaginationConfig
+from synapse.streams.config import PaginationConfig
+from synapse.api.events.room import RoomMemberEvent
+from synapse.api.constants import Membership
 
 import json
 import logging
@@ -35,31 +34,28 @@ class RoomCreateRestServlet(RestServlet):
     # No PATTERN; we have custom dispatch rules here
 
     def register(self, http_server):
-        # /rooms OR /rooms/<roomid>
-        http_server.register_path("POST",
-                                  client_path_pattern("/rooms$"),
-                                  self.on_POST)
-        http_server.register_path("PUT",
-                                  client_path_pattern(
-                                      "/rooms/(?P<room_id>[^/]*)$"),
-                                  self.on_PUT)
+        PATTERN = "/createRoom"
+        register_txn_path(self, PATTERN, http_server)
         # define CORS for all of /rooms in RoomCreateRestServlet for simplicity
         http_server.register_path("OPTIONS",
                                   client_path_pattern("/rooms(?:/.*)?$"),
                                   self.on_OPTIONS)
+        # define CORS for /createRoom[/txnid]
+        http_server.register_path("OPTIONS",
+                                  client_path_pattern("/createRoom(?:/.*)?$"),
+                                  self.on_OPTIONS)
 
     @defer.inlineCallbacks
-    def on_PUT(self, request, room_id):
-        room_id = urllib.unquote(room_id)
-        auth_user = yield self.auth.get_user_by_req(request)
+    def on_PUT(self, request, txn_id):
+        try:
+            defer.returnValue(self.txns.get_client_transaction(request, txn_id))
+        except KeyError:
+            pass
 
-        if not room_id:
-            raise SynapseError(400, "PUT must specify a room ID")
+        response = yield self.on_POST(request)
 
-        room_config = self.get_room_config(request)
-        info = yield self.make_room(room_config, auth_user, room_id)
-        room_config.update(info)
-        defer.returnValue((200, info))
+        self.txns.store_client_transaction(request, txn_id, response)
+        defer.returnValue(response)
 
     @defer.inlineCallbacks
     def on_POST(self, request):
@@ -95,249 +91,193 @@ class RoomCreateRestServlet(RestServlet):
         return (200, {})
 
 
-class RoomTopicRestServlet(RestServlet):
-    PATTERN = client_path_pattern("/rooms/(?P<room_id>[^/]*)/topic$")
+# TODO: Needs unit testing for generic events
+class RoomStateEventRestServlet(RestServlet):
+    def register(self, http_server):
+        # /room/$roomid/state/$eventtype
+        no_state_key = "/rooms/(?P<room_id>[^/]*)/state/(?P<event_type>[^/]*)$"
 
-    def get_event_type(self):
-        return RoomTopicEvent.TYPE
+        # /room/$roomid/state/$eventtype/$statekey
+        state_key = ("/rooms/(?P<room_id>[^/]*)/state/" +
+            "(?P<event_type>[^/]*)/(?P<state_key>[^/]*)$")
+
+        http_server.register_path("GET",
+                                  client_path_pattern(state_key),
+                                  self.on_GET)
+        http_server.register_path("PUT",
+                                  client_path_pattern(state_key),
+                                  self.on_PUT)
+        http_server.register_path("GET",
+                                  client_path_pattern(no_state_key),
+                                  self.on_GET_no_state_key)
+        http_server.register_path("PUT",
+                                  client_path_pattern(no_state_key),
+                                  self.on_PUT_no_state_key)
+
+    def on_GET_no_state_key(self, request, room_id, event_type):
+        return self.on_GET(request, room_id, event_type, "")
+
+    def on_PUT_no_state_key(self, request, room_id, event_type):
+        return self.on_PUT(request, room_id, event_type, "")
 
     @defer.inlineCallbacks
-    def on_GET(self, request, room_id):
+    def on_GET(self, request, room_id, event_type, state_key):
         user = yield self.auth.get_user_by_req(request)
 
         msg_handler = self.handlers.message_handler
         data = yield msg_handler.get_room_data(
             user_id=user.to_string(),
             room_id=urllib.unquote(room_id),
-            event_type=RoomTopicEvent.TYPE,
-            state_key="",
+            event_type=urllib.unquote(event_type),
+            state_key=urllib.unquote(state_key),
         )
 
         if not data:
-            raise SynapseError(404, "Topic not found.", errcode=Codes.NOT_FOUND)
-        defer.returnValue((200, json.loads(data.content)))
+            raise SynapseError(404, "Event not found.", errcode=Codes.NOT_FOUND)
+        defer.returnValue((200, data[0].get_dict()["content"]))
 
     @defer.inlineCallbacks
-    def on_PUT(self, request, room_id):
+    def on_PUT(self, request, room_id, event_type, state_key):
         user = yield self.auth.get_user_by_req(request)
+        event_type = urllib.unquote(event_type)
 
         content = _parse_json(request)
 
         event = self.event_factory.create_event(
-            etype=self.get_event_type(),
+            etype=event_type,
             content=content,
             room_id=urllib.unquote(room_id),
             user_id=user.to_string(),
+            state_key=urllib.unquote(state_key)
             )
-
-        msg_handler = self.handlers.message_handler
-        yield msg_handler.store_room_data(
-            event=event
-        )
-        defer.returnValue((200, ""))
-
-
-class JoinRoomAliasServlet(RestServlet):
-    PATTERN = client_path_pattern("/join/(?P<room_alias>[^/]+)$")
-
-    @defer.inlineCallbacks
-    def on_PUT(self, request, room_alias):
-        user = yield self.auth.get_user_by_req(request)
-
-        if not user:
-            defer.returnValue((403, "Unrecognized user"))
-
-        logger.debug("room_alias: %s", room_alias)
-
-        room_alias = self.hs.parse_roomalias(urllib.unquote(room_alias))
-
-        handler = self.handlers.room_member_handler
-        ret_dict = yield handler.join_room_alias(user, room_alias)
-
-        defer.returnValue((200, ret_dict))
-
-
-class RoomMemberRestServlet(RestServlet):
-    PATTERN = client_path_pattern("/rooms/(?P<room_id>[^/]*)/members/"
-                                  + "(?P<target_user_id>[^/]*)/state$")
-
-    def get_event_type(self):
-        return RoomMemberEvent.TYPE
-
-    @defer.inlineCallbacks
-    def on_GET(self, request, room_id, target_user_id):
-        room_id = urllib.unquote(room_id)
-        user = yield self.auth.get_user_by_req(request)
-
-        handler = self.handlers.room_member_handler
-        member = yield handler.get_room_member(room_id, target_user_id,
-                                               user.to_string())
-        if not member:
-            raise SynapseError(404, "Member not found.",
-                               errcode=Codes.NOT_FOUND)
-        defer.returnValue((200, json.loads(member.content)))
-
-    @defer.inlineCallbacks
-    def on_DELETE(self, request, roomid, target_user_id):
-        user = yield self.auth.get_user_by_req(request)
-
-        event = self.event_factory.create_event(
-            etype=self.get_event_type(),
-            target_user_id=target_user_id,
-            room_id=urllib.unquote(roomid),
-            user_id=user.to_string(),
-            membership=Membership.LEAVE,
-            content={"membership": Membership.LEAVE}
+        if event_type == RoomMemberEvent.TYPE:
+            # membership events are special
+            handler = self.handlers.room_member_handler
+            yield handler.change_membership(event)
+            defer.returnValue((200, ""))
+        else:
+            # store random bits of state
+            msg_handler = self.handlers.message_handler
+            yield msg_handler.store_room_data(
+                event=event
             )
+            defer.returnValue((200, ""))
 
-        handler = self.handlers.room_member_handler
-        yield handler.change_membership(event, broadcast_msg=True)
-        defer.returnValue((200, ""))
+
+# TODO: Needs unit testing for generic events + feedback
+class RoomSendEventRestServlet(RestServlet):
+
+    def register(self, http_server):
+        # /rooms/$roomid/send/$event_type[/$txn_id]
+        PATTERN = ("/rooms/(?P<room_id>[^/]*)/send/(?P<event_type>[^/]*)")
+        register_txn_path(self, PATTERN, http_server, with_get=True)
 
     @defer.inlineCallbacks
-    def on_PUT(self, request, roomid, target_user_id):
+    def on_POST(self, request, room_id, event_type):
         user = yield self.auth.get_user_by_req(request)
-
-        content = _parse_json(request)
-        if "membership" not in content:
-            raise SynapseError(400, "No membership key.",
-                               errcode=Codes.BAD_JSON)
-
-        valid_membership_values = [Membership.JOIN, Membership.INVITE]
-        if (content["membership"] not in valid_membership_values):
-            raise SynapseError(400, "Membership value must be %s." % (
-                    valid_membership_values,), errcode=Codes.BAD_JSON)
-
-        event = self.event_factory.create_event(
-            etype=self.get_event_type(),
-            target_user_id=target_user_id,
-            room_id=urllib.unquote(roomid),
-            user_id=user.to_string(),
-            membership=content["membership"],
-            content=content
-            )
-
-        handler = self.handlers.room_member_handler
-        result = yield handler.change_membership(event, broadcast_msg=True)
-        defer.returnValue((200, result))
-
-
-class MessageRestServlet(RestServlet):
-    PATTERN = client_path_pattern("/rooms/(?P<room_id>[^/]*)/messages/"
-                                  + "(?P<sender_id>[^/]*)/(?P<msg_id>[^/]*)$")
-
-    def get_event_type(self):
-        return MessageEvent.TYPE
-
-    @defer.inlineCallbacks
-    def on_GET(self, request, room_id, sender_id, msg_id):
-        user = yield self.auth.get_user_by_req(request)
-
-        msg_handler = self.handlers.message_handler
-        msg = yield msg_handler.get_message(room_id=urllib.unquote(room_id),
-                                            sender_id=sender_id,
-                                            msg_id=msg_id,
-                                            user_id=user.to_string(),
-                                            )
-
-        if not msg:
-            raise SynapseError(404, "Message not found.",
-                               errcode=Codes.NOT_FOUND)
-
-        defer.returnValue((200, json.loads(msg.content)))
-
-    @defer.inlineCallbacks
-    def on_PUT(self, request, room_id, sender_id, msg_id):
-        user = yield self.auth.get_user_by_req(request)
-
-        if user.to_string() != sender_id:
-            raise SynapseError(403, "Must send messages as yourself.",
-                               errcode=Codes.FORBIDDEN)
-
         content = _parse_json(request)
 
         event = self.event_factory.create_event(
-            etype=self.get_event_type(),
+            etype=event_type,
             room_id=urllib.unquote(room_id),
             user_id=user.to_string(),
-            msg_id=msg_id,
             content=content
-            )
+        )
 
         msg_handler = self.handlers.message_handler
         yield msg_handler.send_message(event)
 
-        defer.returnValue((200, ""))
+        defer.returnValue((200, {"event_id": event.event_id}))
 
-
-class FeedbackRestServlet(RestServlet):
-    PATTERN = client_path_pattern(
-        "/rooms/(?P<room_id>[^/]*)/messages/" +
-        "(?P<msg_sender_id>[^/]*)/(?P<msg_id>[^/]*)/feedback/" +
-        "(?P<sender_id>[^/]*)/(?P<feedback_type>[^/]*)$"
-    )
-
-    def get_event_type(self):
-        return FeedbackEvent.TYPE
+    def on_GET(self, request, room_id, event_type, txn_id):
+        return (200, "Not implemented")
 
     @defer.inlineCallbacks
-    def on_GET(self, request, room_id, msg_sender_id, msg_id, fb_sender_id,
-               feedback_type):
-        user = yield (self.auth.get_user_by_req(request))
+    def on_PUT(self, request, room_id, event_type, txn_id):
+        try:
+            defer.returnValue(self.txns.get_client_transaction(request, txn_id))
+        except KeyError:
+            pass
 
-        if feedback_type not in Feedback.LIST:
-            raise SynapseError(400, "Bad feedback type.",
-                               errcode=Codes.BAD_JSON)
+        response = yield self.on_POST(request, room_id, event_type)
 
-        msg_handler = self.handlers.message_handler
-        feedback = yield msg_handler.get_feedback(
-            room_id=urllib.unquote(room_id),
-            msg_sender_id=msg_sender_id,
-            msg_id=msg_id,
-            user_id=user.to_string(),
-            fb_sender_id=fb_sender_id,
-            fb_type=feedback_type
-        )
+        self.txns.store_client_transaction(request, txn_id, response)
+        defer.returnValue(response)
 
-        if not feedback:
-            raise SynapseError(404, "Feedback not found.",
-                               errcode=Codes.NOT_FOUND)
 
-        defer.returnValue((200, json.loads(feedback.content)))
+# TODO: Needs unit testing for room ID + alias joins
+class JoinRoomAliasServlet(RestServlet):
+
+    def register(self, http_server):
+        # /join/$room_identifier[/$txn_id]
+        PATTERN = ("/join/(?P<room_identifier>[^/]*)")
+        register_txn_path(self, PATTERN, http_server)
 
     @defer.inlineCallbacks
-    def on_PUT(self, request, room_id, sender_id, msg_id, fb_sender_id,
-               feedback_type):
-        user = yield (self.auth.get_user_by_req(request))
+    def on_POST(self, request, room_identifier):
+        user = yield self.auth.get_user_by_req(request)
 
-        if user.to_string() != fb_sender_id:
-            raise SynapseError(403, "Must send feedback as yourself.",
-                               errcode=Codes.FORBIDDEN)
+        # the identifier could be a room alias or a room id. Try one then the
+        # other if it fails to parse, without swallowing other valid
+        # SynapseErrors.
 
-        if feedback_type not in Feedback.LIST:
-            raise SynapseError(400, "Bad feedback type.",
-                               errcode=Codes.BAD_JSON)
-
-        content = _parse_json(request)
-
-        event = self.event_factory.create_event(
-            etype=self.get_event_type(),
-            room_id=urllib.unquote(room_id),
-            msg_sender_id=sender_id,
-            msg_id=msg_id,
-            user_id=user.to_string(),  # user sending the feedback
-            feedback_type=feedback_type,
-            content=content
+        identifier = None
+        is_room_alias = False
+        try:
+            identifier = self.hs.parse_roomalias(
+                urllib.unquote(room_identifier)
+            )
+            is_room_alias = True
+        except SynapseError:
+            identifier = self.hs.parse_roomid(
+                urllib.unquote(room_identifier)
             )
 
-        msg_handler = self.handlers.message_handler
-        yield msg_handler.send_feedback(event)
+        # TODO: Support for specifying the home server to join with?
 
-        defer.returnValue((200, ""))
+        if is_room_alias:
+            handler = self.handlers.room_member_handler
+            ret_dict = yield handler.join_room_alias(user, identifier)
+            defer.returnValue((200, ret_dict))
+        else:  # room id
+            event = self.event_factory.create_event(
+                etype=RoomMemberEvent.TYPE,
+                content={"membership": Membership.JOIN},
+                room_id=urllib.unquote(identifier.to_string()),
+                user_id=user.to_string(),
+                state_key=user.to_string()
+            )
+            handler = self.handlers.room_member_handler
+            yield handler.change_membership(event)
+            defer.returnValue((200, ""))
+
+    @defer.inlineCallbacks
+    def on_PUT(self, request, room_identifier, txn_id):
+        try:
+            defer.returnValue(self.txns.get_client_transaction(request, txn_id))
+        except KeyError:
+            pass
+
+        response = yield self.on_POST(request, room_identifier)
+
+        self.txns.store_client_transaction(request, txn_id, response)
+        defer.returnValue(response)
 
 
+# TODO: Needs unit testing
+class PublicRoomListRestServlet(RestServlet):
+    PATTERN = client_path_pattern("/publicRooms$")
+
+    @defer.inlineCallbacks
+    def on_GET(self, request):
+        handler = self.handlers.room_list_handler
+        data = yield handler.get_public_room_list()
+        defer.returnValue((200, data))
+
+
+# TODO: Needs unit testing
 class RoomMemberListRestServlet(RestServlet):
-    PATTERN = client_path_pattern("/rooms/(?P<room_id>[^/]*)/members/list$")
+    PATTERN = client_path_pattern("/rooms/(?P<room_id>[^/]*)/members$")
 
     @defer.inlineCallbacks
     def on_GET(self, request, room_id):
@@ -348,11 +288,24 @@ class RoomMemberListRestServlet(RestServlet):
             room_id=urllib.unquote(room_id),
             user_id=user.to_string())
 
+        for event in members["chunk"]:
+            # FIXME: should probably be state_key here, not user_id
+            target_user = self.hs.parse_userid(event["user_id"])
+            # Presence is an optional cache; don't fail if we can't fetch it
+            try:
+                presence_state = yield self.handlers.presence_handler.get_state(
+                    target_user=target_user, auth_user=user
+                )
+                event["content"].update(presence_state)
+            except:
+                pass
+
         defer.returnValue((200, members))
 
 
+# TODO: Needs unit testing
 class RoomMessageListRestServlet(RestServlet):
-    PATTERN = client_path_pattern("/rooms/(?P<room_id>[^/]*)/messages/list$")
+    PATTERN = client_path_pattern("/rooms/(?P<room_id>[^/]*)/messages$")
 
     @defer.inlineCallbacks
     def on_GET(self, request, room_id):
@@ -369,6 +322,112 @@ class RoomMessageListRestServlet(RestServlet):
         defer.returnValue((200, msgs))
 
 
+# TODO: Needs unit testing
+class RoomStateRestServlet(RestServlet):
+    PATTERN = client_path_pattern("/rooms/(?P<room_id>[^/]*)/state$")
+
+    @defer.inlineCallbacks
+    def on_GET(self, request, room_id):
+        user = yield self.auth.get_user_by_req(request)
+        # TODO: Get all the current state for this room and return in the same
+        # format as initial sync, that is:
+        # [
+        #   { state event }, { state event }
+        # ]
+        defer.returnValue((200, []))
+
+
+# TODO: Needs unit testing
+class RoomInitialSyncRestServlet(RestServlet):
+    PATTERN = client_path_pattern("/rooms/(?P<room_id>[^/]*)/initialSync$")
+
+    @defer.inlineCallbacks
+    def on_GET(self, request, room_id):
+        user = yield self.auth.get_user_by_req(request)
+        # TODO: Get all the initial sync data for this room and return in the
+        # same format as initial sync, that is:
+        # {
+        #   membership: join,
+        #   messages: [
+        #       chunk: [ msg events ],
+        #       start: s_tok,
+        #       end: e_tok
+        #   ],
+        #   room_id: foo,
+        #   state: [
+        #       { state event } , { state event }
+        #   ]
+        # }
+        # Probably worth keeping the keys room_id and membership for parity with
+        # /initialSync even though they must be joined to sync this and know the
+        # room ID, so clients can reuse the same code (room_id and membership
+        # are MANDATORY for /initialSync, so the code will expect it to be
+        # there)
+        defer.returnValue((200, {}))
+
+
+class RoomTriggerBackfill(RestServlet):
+    PATTERN = client_path_pattern("/rooms/(?P<room_id>[^/]*)/backfill$")
+
+    @defer.inlineCallbacks
+    def on_GET(self, request, room_id):
+        remote_server = urllib.unquote(request.args["remote"][0])
+        room_id = urllib.unquote(room_id)
+        limit = int(request.args["limit"][0])
+
+        handler = self.handlers.federation_handler
+        events = yield handler.backfill(remote_server, room_id, limit)
+
+        res = [event.get_dict() for event in events]
+        defer.returnValue((200, res))
+
+
+# TODO: Needs unit testing
+class RoomMembershipRestServlet(RestServlet):
+
+    def register(self, http_server):
+        # /rooms/$roomid/[invite|join|leave]
+        PATTERN = ("/rooms/(?P<room_id>[^/]*)/" +
+            "(?P<membership_action>join|invite|leave)")
+        register_txn_path(self, PATTERN, http_server)
+
+    @defer.inlineCallbacks
+    def on_POST(self, request, room_id, membership_action):
+        user = yield self.auth.get_user_by_req(request)
+
+        content = _parse_json(request)
+
+        # target user is you unless it is an invite
+        state_key = user.to_string()
+        if membership_action == "invite":
+            if "user_id" not in content:
+                raise SynapseError(400, "Missing user_id key.")
+            state_key = content["user_id"]
+
+        event = self.event_factory.create_event(
+            etype=RoomMemberEvent.TYPE,
+            content={"membership": unicode(membership_action)},
+            room_id=urllib.unquote(room_id),
+            user_id=user.to_string(),
+            state_key=state_key
+        )
+        handler = self.handlers.room_member_handler
+        yield handler.change_membership(event)
+        defer.returnValue((200, ""))
+
+    @defer.inlineCallbacks
+    def on_PUT(self, request, room_id, membership_action, txn_id):
+        try:
+            defer.returnValue(self.txns.get_client_transaction(request, txn_id))
+        except KeyError:
+            pass
+
+        response = yield self.on_POST(request, room_id, membership_action)
+
+        self.txns.store_client_transaction(request, txn_id, response)
+        defer.returnValue(response)
+
+
 def _parse_json(request):
     try:
         content = json.loads(request.content.read())
@@ -380,12 +439,46 @@ def _parse_json(request):
         raise SynapseError(400, "Content not JSON.", errcode=Codes.NOT_JSON)
 
 
+def register_txn_path(servlet, regex_string, http_server, with_get=False):
+    """Registers a transaction-based path.
+
+    This registers two paths:
+        PUT regex_string/$txnid
+        POST regex_string
+
+    Args:
+        regex_string (str): The regex string to register. Must NOT have a
+        trailing $ as this string will be appended to.
+        http_server : The http_server to register paths with.
+        with_get: True to also register respective GET paths for the PUTs.
+    """
+    http_server.register_path(
+        "POST",
+        client_path_pattern(regex_string + "$"),
+        servlet.on_POST
+    )
+    http_server.register_path(
+        "PUT",
+        client_path_pattern(regex_string + "/(?P<txn_id>[^/]*)$"),
+        servlet.on_PUT
+    )
+    if with_get:
+        http_server.register_path(
+        "GET",
+        client_path_pattern(regex_string + "/(?P<txn_id>[^/]*)$"),
+        servlet.on_GET
+    )
+
+
 def register_servlets(hs, http_server):
-    RoomTopicRestServlet(hs).register(http_server)
-    RoomMemberRestServlet(hs).register(http_server)
-    MessageRestServlet(hs).register(http_server)
-    FeedbackRestServlet(hs).register(http_server)
+    RoomStateEventRestServlet(hs).register(http_server)
     RoomCreateRestServlet(hs).register(http_server)
     RoomMemberListRestServlet(hs).register(http_server)
     RoomMessageListRestServlet(hs).register(http_server)
     JoinRoomAliasServlet(hs).register(http_server)
+    RoomTriggerBackfill(hs).register(http_server)
+    RoomMembershipRestServlet(hs).register(http_server)
+    RoomSendEventRestServlet(hs).register(http_server)
+    PublicRoomListRestServlet(hs).register(http_server)
+    RoomStateRestServlet(hs).register(http_server)
+    RoomInitialSyncRestServlet(hs).register(http_server)

synapse/rest/transactions.py

@@ -0,0 +1,96 @@
+# -*- coding: utf-8 -*-
+# Copyright 2014 matrix.org
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""This module contains logic for storing HTTP PUT transactions. This is used
+to ensure idempotency when performing PUTs using the REST API."""
+import logging
+
+logger = logging.getLogger(__name__)
+
+
+class HttpTransactionStore(object):
+
+    def __init__(self):
+        # { key : (txn_id, response) }
+        self.transactions = {}
+
+    def get_response(self, key, txn_id):
+        """Retrieve a response for this request.
+
+        Args:
+            key (str): A transaction-independent key for this request. Typically
+            this is a combination of the path (without the transaction id) and
+            the user's access token.
+            txn_id (str): The transaction ID for this request
+        Returns:
+            A tuple of (HTTP response code, response content) or None.
+        """
+        try:
+            logger.debug("get_response Key: %s TxnId: %s", key, txn_id)
+            (last_txn_id, response) = self.transactions[key]
+            if txn_id == last_txn_id:
+                logger.info("get_response: Returning a response for %s", key)
+                return response
+        except KeyError:
+            pass
+        return None
+
+    def store_response(self, key, txn_id, response):
+        """Stores an HTTP response tuple.
+
+        Args:
+            key (str): A transaction-independent key for this request. Typically
+            this is a combination of the path (without the transaction id) and
+            the user's access token.
+            txn_id (str): The transaction ID for this request.
+            response (tuple): A tuple of (HTTP response code, response content)
+        """
+        logger.debug("store_response Key: %s TxnId: %s", key, txn_id)
+        self.transactions[key] = (txn_id, response)
+
+    def store_client_transaction(self, request, txn_id, response):
+        """Stores the request/response pair of an HTTP transaction.
+
+        Args:
+            request (twisted.web.http.Request): The twisted HTTP request. This
+            request must have the transaction ID as the last path segment.
+            response (tuple): A tuple of (response code, response dict)
+            txn_id (str): The transaction ID for this request.
+        """
+        self.store_response(self._get_key(request), txn_id, response)
+
+    def get_client_transaction(self, request, txn_id):
+        """Retrieves a stored response if there was one.
+
+        Args:
+            request (twisted.web.http.Request): The twisted HTTP request. This
+            request must have the transaction ID as the last path segment.
+            txn_id (str): The transaction ID for this request.
+        Returns:
+            The response tuple.
+        Raises:
+            KeyError if the transaction was not found.
+        """
+        response = self.get_response(self._get_key(request), txn_id)
+        if response is None:
+            raise KeyError("Transaction not found.")
+        return response
+
+    def _get_key(self, request):
+        token = request.args["access_token"][0]
+        path_without_txn_id = request.path.rsplit("/", 1)[0]
+        return path_without_txn_id + "/" + token
+
+

synapse/rest/webclient.py

@@ -1,45 +0,0 @@
-# -*- coding: utf-8 -*-
-# Copyright 2014 matrix.org
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-from synapse.rest.base import RestServlet
-
-import logging
-import re
-
-logger = logging.getLogger(__name__)
-
-
-class WebClientRestServlet(RestServlet):
-    # No PATTERN; we have custom dispatch rules here
-
-    def register(self, http_server):
-        http_server.register_path("GET",
-                                  re.compile("^/$"),
-                                  self.on_GET_redirect)
-        http_server.register_path("GET",
-                                  re.compile("^/matrix/client$"),
-                                  self.on_GET)
-
-    def on_GET(self, request):
-        return (200, "not implemented")
-
-    def on_GET_redirect(self, request):
-        request.setHeader("Location", request.uri + "matrix/client")
-        return (302, None)
-
-
-def register_servlets(hs, http_server):
-    logger.info("Registering web client.")
-    WebClientRestServlet(hs).register(http_server)

synapse/server.py

@@ -20,18 +20,18 @@
 
 # Imports required for the default HomeServer() implementation
 from synapse.federation import initialize_http_replication
-from synapse.federation.handler import FederationEventHandler
 from synapse.api.events.factory import EventFactory
-from synapse.api.notifier import Notifier
+from synapse.notifier import Notifier
 from synapse.api.auth import Auth
 from synapse.handlers import Handlers
 from synapse.rest import RestServletFactory
 from synapse.state import StateHandler
 from synapse.storage import DataStore
-from synapse.types import UserID, RoomAlias
+from synapse.types import UserID, RoomAlias, RoomID
 from synapse.util import Clock
 from synapse.util.distributor import Distributor
 from synapse.util.lockutils import LockManager
+from synapse.streams.events import EventSources
 
 
 class BaseHomeServer(object):
@@ -55,11 +55,9 @@ class BaseHomeServer(object):
 
     DEPENDENCIES = [
         'clock',
-        'http_server',
         'http_client',
         'db_pool',
         'persistence_service',
-        'federation',
         'replication_layer',
         'datastore',
         'event_factory',
@@ -70,6 +68,11 @@ class BaseHomeServer(object):
         'room_lock_manager',
         'notifier',
         'distributor',
+        'resource_for_client',
+        'resource_for_federation',
+        'resource_for_web_client',
+        'resource_for_content_repo',
+        'event_sources',
     ]
 
     def __init__(self, hostname, **kwargs):
@@ -114,6 +117,9 @@ class BaseHomeServer(object):
 
         setattr(BaseHomeServer, "get_%s" % (depname), _get)
 
+    # TODO: Why are these parse_ methods so high up along with other globals?
+    # Surely these should be in a util package or in the api package?
+
     # Other utility methods
     def parse_userid(self, s):
         """Parse the string given by 's' as a User ID and return a UserID
@@ -125,6 +131,11 @@ class BaseHomeServer(object):
         object."""
         return RoomAlias.from_string(s, hs=self)
 
+    def parse_roomid(self, s):
+        """Parse the string given by 's' as a Room ID and return a RoomID
+        object."""
+        return RoomID.from_string(s, hs=self)
+
 # Build magic accessors for every dependency
 for depname in BaseHomeServer.DEPENDENCIES:
     BaseHomeServer._make_dependency_method(depname)
@@ -135,7 +146,10 @@ class HomeServer(BaseHomeServer):
     required.
 
     It still requires the following to be specified by the caller:
-        http_server
+        resource_for_client
+        resource_for_web_client
+        resource_for_federation
+        resource_for_content_repo
         http_client
         db_pool
     """
@@ -146,14 +160,11 @@ class HomeServer(BaseHomeServer):
     def build_replication_layer(self):
         return initialize_http_replication(self)
 
-    def build_federation(self):
-        return FederationEventHandler(self)
-
     def build_datastore(self):
         return DataStore(self)
 
     def build_event_factory(self):
-        return EventFactory()
+        return EventFactory(self)
 
     def build_handlers(self):
         return Handlers(self)
@@ -176,11 +187,11 @@ class HomeServer(BaseHomeServer):
     def build_distributor(self):
         return Distributor()
 
+    def build_event_sources(self):
+        return EventSources(self)
+
     def register_servlets(self):
         """ Register all servlets associated with this HomeServer.
-
-        Args:
-            host_web_client (bool): True to host the web client as well.
         """
         # Simply building the ServletFactory is sufficient to have it register
-        factory = self.get_rest_servlet_factory()
+        self.get_rest_servlet_factory()

synapse/state.py

@@ -45,7 +45,7 @@ class StateHandler(object):
 
     @defer.inlineCallbacks
     @log_function
-    def handle_new_event(self, event):
+    def handle_new_event(self, event, snapshot):
         """ Given an event this works out if a) we have sufficient power level
         to update the state and b) works out what the prev_state should be.
 
@@ -70,25 +70,13 @@ class StateHandler(object):
         # Now I need to fill out the prev state and work out if it has auth
         # (w.r.t. to power levels)
 
-        results = yield self.store.get_latest_pdus_in_context(
-            event.room_id
-        )
+        snapshot.fill_out_prev_events(event)
 
-        event.prev_events = [
-            encode_event_id(p_id, origin) for p_id, origin, _ in results
-        ]
         event.prev_events = [
             e for e in event.prev_events if e != event.event_id
         ]
 
-        if results:
-            event.depth = max([int(v) for _, _, v in results]) + 1
-        else:
-            event.depth = 0
-
-        current_state = yield self.store.get_current_state(
-            key.context, key.type, key.state_key
-        )
+        current_state = snapshot.prev_state_pdu
 
         if current_state:
             event.prev_state = encode_event_id(
@@ -157,7 +145,10 @@ class StateHandler(object):
             defer.returnValue(True)
             return
 
-        if new_branch[-1] == current_branch[-1]:
+        n = new_branch[-1]
+        c = current_branch[-1]
+
+        if n.pdu_id == c.pdu_id and n.origin == c.origin:
             # We have all the PDUs we need, so we can just do the conflict
             # resolution.
 
@@ -188,10 +179,18 @@ class StateHandler(object):
                 key=lambda x: x.depth
             )
 
+            pdu_id = missing_prev.prev_state_id
+            origin = missing_prev.prev_state_origin
+
+            is_missing = yield self.store.get_pdu(pdu_id, origin) is None
+
+            if not is_missing:
+                raise Exception("Conflict resolution failed.")
+
             yield self._replication.get_pdu(
                 destination=missing_prev.origin,
-                pdu_origin=missing_prev.prev_state_origin,
-                pdu_id=missing_prev.prev_state_id,
+                pdu_origin=origin,
+                pdu_id=pdu_id,
                 outlier=True
             )
 

synapse/storage/__init__.py

@@ -13,30 +13,36 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
+from twisted.internet import defer
 
 from synapse.api.events.room import (
-    RoomMemberEvent, MessageEvent, RoomTopicEvent, FeedbackEvent,
-    RoomConfigEvent
+    RoomMemberEvent, RoomTopicEvent, FeedbackEvent,
+#   RoomConfigEvent,
+    RoomNameEvent,
 )
 
+from synapse.util.logutils import log_function
+
 from .directory import DirectoryStore
 from .feedback import FeedbackStore
-from .message import MessageStore
 from .presence import PresenceStore
 from .profile import ProfileStore
 from .registration import RegistrationStore
 from .room import RoomStore
 from .roommember import RoomMemberStore
-from .roomdata import RoomDataStore
 from .stream import StreamStore
 from .pdu import StatePduStore, PduStore
 from .transactions import TransactionStore
 
 import json
+import logging
 import os
 
 
-class DataStore(RoomDataStore, RoomMemberStore, MessageStore, RoomStore,
+logger = logging.getLogger(__name__)
+
+
+class DataStore(RoomMemberStore, RoomStore,
                 RegistrationStore, StreamStore, ProfileStore, FeedbackStore,
                 PresenceStore, PduStore, StatePduStore, TransactionStore,
                 DirectoryStore):
@@ -44,51 +50,256 @@ class DataStore(RoomDataStore, RoomMemberStore, MessageStore, RoomStore,
     def __init__(self, hs):
         super(DataStore, self).__init__(hs)
         self.event_factory = hs.get_event_factory()
+        self.hs = hs
 
-    def persist_event(self, event):
-        if event.type == MessageEvent.TYPE:
-            return self.store_message(
-                user_id=event.user_id,
-                room_id=event.room_id,
-                msg_id=event.msg_id,
-                content=json.dumps(event.content)
-            )
-        elif event.type == RoomMemberEvent.TYPE:
-            return self.store_room_member(
-                user_id=event.target_user_id,
-                sender=event.user_id,
-                room_id=event.room_id,
-                content=event.content,
-                membership=event.content["membership"]
-            )
-        elif event.type == FeedbackEvent.TYPE:
-            return self.store_feedback(
-                room_id=event.room_id,
-                msg_id=event.msg_id,
-                msg_sender_id=event.msg_sender_id,
-                fb_sender_id=event.user_id,
-                fb_type=event.feedback_type,
-                content=json.dumps(event.content)
-            )
-        elif event.type == RoomTopicEvent.TYPE:
-            return self.store_room_data(
-                room_id=event.room_id,
-                etype=event.type,
-                state_key=event.state_key,
-                content=json.dumps(event.content)
-            )
-        elif event.type == RoomConfigEvent.TYPE:
-            if "visibility" in event.content:
-                visibility = event.content["visibility"]
-                return self.store_room_config(
-                    room_id=event.room_id,
-                    visibility=visibility
-                )
+        self.min_token_deferred = self._get_min_token()
+        self.min_token = None
 
+    @defer.inlineCallbacks
+    @log_function
+    def persist_event(self, event=None, backfilled=False, pdu=None):
+        stream_ordering = None
+        if backfilled:
+            if not self.min_token_deferred.called:
+                yield self.min_token_deferred
+            self.min_token -= 1
+            stream_ordering = self.min_token
+
+        latest = yield self._db_pool.runInteraction(
+            self._persist_pdu_event_txn,
+            pdu=pdu,
+            event=event,
+            backfilled=backfilled,
+            stream_ordering=stream_ordering,
+        )
+        defer.returnValue(latest)
+
+    @defer.inlineCallbacks
+    def get_event(self, event_id):
+        events_dict = yield self._simple_select_one(
+            "events",
+            {"event_id": event_id},
+            [
+                "event_id",
+                "type",
+                "room_id",
+                "content",
+                "unrecognized_keys"
+            ],
+        )
+
+        event = self._parse_event_from_row(events_dict)
+        defer.returnValue(event)
+
+    def _persist_pdu_event_txn(self, txn, pdu=None, event=None,
+                               backfilled=False, stream_ordering=None):
+        if pdu is not None:
+            self._persist_event_pdu_txn(txn, pdu)
+        if event is not None:
+            return self._persist_event_txn(
+                txn, event, backfilled, stream_ordering
+            )
+
+    def _persist_event_pdu_txn(self, txn, pdu):
+        cols = dict(pdu.__dict__)
+        unrec_keys = dict(pdu.unrecognized_keys)
+        del cols["content"]
+        del cols["prev_pdus"]
+        cols["content_json"] = json.dumps(pdu.content)
+        cols["unrecognized_keys"] = json.dumps(unrec_keys)
+
+        logger.debug("Persisting: %s", repr(cols))
+
+        if pdu.is_state:
+            self._persist_state_txn(txn, pdu.prev_pdus, cols)
         else:
-            raise NotImplementedError(
-                "Don't know how to persist type=%s" % event.type
+            self._persist_pdu_txn(txn, pdu.prev_pdus, cols)
+
+        self._update_min_depth_for_context_txn(txn, pdu.context, pdu.depth)
+
+    @log_function
+    def _persist_event_txn(self, txn, event, backfilled, stream_ordering=None):
+        if event.type == RoomMemberEvent.TYPE:
+            self._store_room_member_txn(txn, event)
+        elif event.type == FeedbackEvent.TYPE:
+            self._store_feedback_txn(txn,event)
+#        elif event.type == RoomConfigEvent.TYPE:
+#            self._store_room_config_txn(txn, event)
+        elif event.type == RoomNameEvent.TYPE:
+            self._store_room_name_txn(txn, event)
+        elif event.type == RoomTopicEvent.TYPE:
+            self._store_room_topic_txn(txn, event)
+
+        vals = {
+            "topological_ordering": event.depth,
+            "event_id": event.event_id,
+            "type": event.type,
+            "room_id": event.room_id,
+            "content": json.dumps(event.content),
+            "processed": True,
+        }
+
+        if stream_ordering is not None:
+            vals["stream_ordering"] = stream_ordering
+
+        if hasattr(event, "outlier"):
+            vals["outlier"] = event.outlier
+        else:
+            vals["outlier"] = False
+
+        unrec = {
+            k: v
+            for k, v in event.get_full_dict().items()
+            if k not in vals.keys()
+        }
+        vals["unrecognized_keys"] = json.dumps(unrec)
+
+        try:
+            self._simple_insert_txn(txn, "events", vals)
+        except:
+            logger.exception(
+                "Failed to persist, probably duplicate: %s",
+                event.event_id
             )
+            return
+
+        if not backfilled and hasattr(event, "state_key"):
+            vals = {
+                "event_id": event.event_id,
+                "room_id": event.room_id,
+                "type": event.type,
+                "state_key": event.state_key,
+            }
+
+            if hasattr(event, "prev_state"):
+                vals["prev_state"] = event.prev_state
+
+            self._simple_insert_txn(txn, "state_events", vals)
+
+            self._simple_insert_txn(
+                txn,
+                "current_state_events",
+                {
+                    "event_id": event.event_id,
+                    "room_id": event.room_id,
+                    "type": event.type,
+                    "state_key": event.state_key,
+                }
+            )
+
+        return self._get_room_events_max_id_txn(txn)
+
+    @defer.inlineCallbacks
+    def get_current_state(self, room_id, event_type=None, state_key=""):
+        sql = (
+            "SELECT e.* FROM events as e "
+            "INNER JOIN current_state_events as c ON e.event_id = c.event_id "
+            "INNER JOIN state_events as s ON e.event_id = s.event_id "
+            "WHERE c.room_id = ? "
+        )
+
+        if event_type:
+            sql += " AND s.type = ? AND s.state_key = ? "
+            args = (room_id, event_type, state_key)
+        else:
+            args = (room_id, )
+
+        results = yield self._execute_and_decode(sql, *args)
+
+        defer.returnValue([self._parse_event_from_row(r) for r in results])
+
+    @defer.inlineCallbacks
+    def _get_min_token(self):
+        row = yield self._execute(
+            None,
+            "SELECT MIN(stream_ordering) FROM events"
+        )
+
+        self.min_token = row[0][0] if row and row[0] and row[0][0] else -1
+        self.min_token = min(self.min_token, -1)
+
+        logger.debug("min_token is: %s", self.min_token)
+
+        defer.returnValue(self.min_token)
+
+
+    def snapshot_room(self, room_id, user_id, state_type=None, state_key=None):
+        """Snapshot the room for an update by a user
+        Args:
+            room_id (synapse.types.RoomId): The room to snapshot.
+            user_id (synapse.types.UserId): The user to snapshot the room for.
+            state_type (str): Optional state type to snapshot.
+            state_key (str): Optional state key to snapshot.
+        Returns:
+            synapse.storage.Snapshot: A snapshot of the state of the room.
+        """
+        def _snapshot(txn):
+            membership_state = self._get_room_member(txn, user_id, room_id)
+            prev_pdus = self._get_latest_pdus_in_context(
+                txn, room_id
+            )
+            if state_type is not None and state_key is not None:
+                prev_state_pdu = self._get_current_state_pdu(
+                    txn, room_id, state_type, state_key
+                )
+            else:
+                prev_state_pdu = None
+
+            return Snapshot(
+                store=self,
+                room_id=room_id,
+                user_id=user_id,
+                prev_pdus=prev_pdus,
+                membership_state=membership_state,
+                state_type=state_type,
+                state_key=state_key,
+                prev_state_pdu=prev_state_pdu,
+            )
+
+        return self._db_pool.runInteraction(_snapshot)
+
+
+class Snapshot(object):
+    """Snapshot of the state of a room
+    Args:
+        store (DataStore): The datastore.
+        room_id (RoomId): The room of the snapshot.
+        user_id (UserId): The user this snapshot is for.
+        prev_pdus (list): The list of PDU ids this snapshot is after.
+        membership_state (RoomMemberEvent): The current state of the user in
+            the room.
+        state_type (str, optional): State type captured by the snapshot
+        state_key (str, optional): State key captured by the snapshot
+        prev_state_pdu (PduEntry, optional): pdu id of
+            the previous value of the state type and key in the room.
+    """
+
+    def __init__(self, store, room_id, user_id, prev_pdus,
+                 membership_state, state_type=None, state_key=None,
+                 prev_state_pdu=None):
+        self.store = store
+        self.room_id = room_id
+        self.user_id = user_id
+        self.prev_pdus = prev_pdus
+        self.membership_state = membership_state
+        self.state_type = state_type
+        self.state_key = state_key
+        self.prev_state_pdu = prev_state_pdu
+
+    def fill_out_prev_events(self, event):
+        if hasattr(event, "prev_events"):
+            return
+
+        es = [
+            "%s@%s" % (p_id, origin) for p_id, origin, _ in self.prev_pdus
+        ]
+
+        event.prev_events = [e for e in es if e != event.event_id]
+
+        if self.prev_pdus:
+            event.depth = max([int(v) for _, _, v in self.prev_pdus]) + 1
+        else:
+            event.depth = 0
 
 
 def schema_path(schema):

synapse/storage/_base.py

@@ -12,7 +12,6 @@
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
-
 import logging
 
 from twisted.internet import defer
@@ -20,6 +19,9 @@ from twisted.internet import defer
 from synapse.api.errors import StoreError
 
 import collections
+import copy
+import json
+
 
 logger = logging.getLogger(__name__)
 
@@ -29,6 +31,7 @@ class SQLBaseStore(object):
     def __init__(self, hs):
         self.hs = hs
         self._db_pool = hs.get_db_pool()
+        self.event_factory = hs.get_event_factory()
         self._clock = hs.get_clock()
 
     def cursor_to_dict(self, cursor):
@@ -57,14 +60,22 @@ class SQLBaseStore(object):
             The result of decoder(results)
         """
         logger.debug(
-            "[SQL] %s  Args=%s Func=%s", query, args, decoder.__name__
+            "[SQL] %s  Args=%s Func=%s",
+            query, args, decoder.__name__ if decoder else None
         )
 
         def interaction(txn):
             cursor = txn.execute(query, args)
-            return decoder(cursor)
+            if decoder:
+                return decoder(cursor)
+            else:
+                return cursor.fetchall()
+
         return self._db_pool.runInteraction(interaction)
 
+    def _execute_and_decode(self, query, *args):
+        return self._execute(self.cursor_to_dict, query, *args)
+
     # "Simple" SQL API methods that operate on a single table with no JOINs,
     # no complex WHERE clauses, just a dict of values for columns.
 
@@ -75,16 +86,18 @@ class SQLBaseStore(object):
             table : string giving the table name
             values : dict of new column names and values for them
         """
+        return self._db_pool.runInteraction(
+            self._simple_insert_txn, table, values,
+        )
+
+    def _simple_insert_txn(self, txn, table, values):
         sql = "INSERT INTO %s (%s) VALUES(%s)" % (
             table,
             ", ".join(k for k in values),
             ", ".join("?" for k in values)
         )
-
-        def func(txn):
-            txn.execute(sql, values.values())
-            return txn.lastrowid
-        return self._db_pool.runInteraction(func)
+        txn.execute(sql, values.values())
+        return txn.lastrowid
 
     def _simple_select_one(self, table, keyvalues, retcols,
                            allow_none=False):
@@ -281,6 +294,22 @@ class SQLBaseStore(object):
 
         return self._db_pool.runInteraction(func)
 
+    def _parse_event_from_row(self, row_dict):
+        d = copy.deepcopy({k: v for k, v in row_dict.items() if v})
+
+        d.pop("stream_ordering", None)
+        d.pop("topological_ordering", None)
+        d.pop("processed", None)
+
+        d.update(json.loads(row_dict["unrecognized_keys"]))
+        d["content"] = json.loads(d["content"])
+        del d["unrecognized_keys"]
+
+        return self.event_factory.create_event(
+            etype=d["type"],
+            **d
+        )
+
 
 class Table(object):
     """ A base class used to store information about a particular table.

synapse/storage/feedback.py

@@ -13,63 +13,35 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-from ._base import SQLBaseStore, Table
-from synapse.api.events.room import FeedbackEvent
+from twisted.internet import defer
 
-import collections
-import json
+from ._base import SQLBaseStore
 
 
 class FeedbackStore(SQLBaseStore):
 
-    def store_feedback(self, room_id, msg_id, msg_sender_id,
-                       fb_sender_id, fb_type, content):
-        return self._simple_insert(FeedbackTable.table_name, dict(
-            room_id=room_id,
-            msg_id=msg_id,
-            msg_sender_id=msg_sender_id,
-            fb_sender_id=fb_sender_id,
-            fb_type=fb_type,
-            content=content,
-        ))
+    def _store_feedback_txn(self, txn, event):
+        self._simple_insert_txn(txn, "feedback", {
+            "event_id": event.event_id,
+            "feedback_type": event.content["type"],
+            "room_id": event.room_id,
+            "target_event_id": event.content["target_event_id"],
+            "sender": event.user_id,
+        })
 
-    def get_feedback(self, room_id=None, msg_id=None, msg_sender_id=None,
-                     fb_sender_id=None, fb_type=None):
-        query = FeedbackTable.select_statement(
-            "msg_sender_id = ? AND room_id = ? AND msg_id = ? " +
-            "AND fb_sender_id = ? AND feedback_type = ? " +
-            "ORDER BY id DESC LIMIT 1")
-        return self._execute(
-            FeedbackTable.decode_single_result,
-            query, msg_sender_id, room_id, msg_id, fb_sender_id, fb_type,
+    @defer.inlineCallbacks
+    def get_feedback_for_event(self, event_id):
+        sql = (
+            "SELECT events.* FROM events INNER JOIN feedback "
+            "ON events.event_id = feedback.event_id "
+            "WHERE feedback.target_event_id = ? "
         )
 
-    def get_max_feedback_id(self):
-        return self._simple_max_id(FeedbackTable.table_name)
+        rows = yield self._execute_and_decode(sql, event_id)
 
-
-class FeedbackTable(Table):
-    table_name = "feedback"
-
-    fields = [
-        "id",
-        "content",
-        "feedback_type",
-        "fb_sender_id",
-        "msg_id",
-        "room_id",
-        "msg_sender_id"
-    ]
-
-    class EntryType(collections.namedtuple("FeedbackEntry", fields)):
-
-        def as_event(self, event_factory):
-            return event_factory.create_event(
-                etype=FeedbackEvent.TYPE,
-                room_id=self.room_id,
-                msg_id=self.msg_id,
-                msg_sender_id=self.msg_sender_id,
-                user_id=self.fb_sender_id,
-                feedback_type=self.feedback_type,
-                content=json.loads(self.content),
-            )
+        defer.returnValue(
+            [
+                self._parse_event_from_row(r)
+                for r in rows
+            ]
+        )

synapse/storage/message.py

@@ -1,81 +0,0 @@
-# -*- coding: utf-8 -*-
-# Copyright 2014 matrix.org
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-from ._base import SQLBaseStore, Table
-from synapse.api.events.room import MessageEvent
-
-import collections
-import json
-
-
-class MessageStore(SQLBaseStore):
-
-    def get_message(self, user_id, room_id, msg_id):
-        """Get a message from the store.
-
-        Args:
-            user_id (str): The ID of the user who sent the message.
-            room_id (str): The room the message was sent in.
-            msg_id (str): The unique ID for this user/room combo.
-        """
-        query = MessagesTable.select_statement(
-            "user_id = ? AND room_id = ? AND msg_id = ? " +
-            "ORDER BY id DESC LIMIT 1")
-        return self._execute(
-            MessagesTable.decode_single_result,
-            query, user_id, room_id, msg_id,
-        )
-
-    def store_message(self, user_id, room_id, msg_id, content):
-        """Store a message in the store.
-
-        Args:
-            user_id (str): The ID of the user who sent the message.
-            room_id (str): The room the message was sent in.
-            msg_id (str): The unique ID for this user/room combo.
-            content (str): The content of the message (JSON)
-        """
-        return self._simple_insert(MessagesTable.table_name, dict(
-            user_id=user_id,
-            room_id=room_id,
-            msg_id=msg_id,
-            content=content,
-        ))
-
-    def get_max_message_id(self):
-        return self._simple_max_id(MessagesTable.table_name)
-
-
-class MessagesTable(Table):
-    table_name = "messages"
-
-    fields = [
-        "id",
-        "user_id",
-        "room_id",
-        "msg_id",
-        "content"
-    ]
-
-    class EntryType(collections.namedtuple("MessageEntry", fields)):
-
-        def as_event(self, event_factory):
-            return event_factory.create_event(
-                etype=MessageEvent.TYPE,
-                room_id=self.room_id,
-                user_id=self.user_id,
-                msg_id=self.msg_id,
-                content=json.loads(self.content),
-            )

synapse/storage/pdu.py

@@ -13,6 +13,8 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
+from twisted.internet import defer
+
 from ._base import SQLBaseStore, Table, JoinHelper
 
 from synapse.util.logutils import log_function
@@ -112,7 +114,7 @@ class PduStore(SQLBaseStore):
 
         return self._get_pdu_tuples(txn, res)
 
-    def persist_pdu(self, prev_pdus, **cols):
+    def _persist_pdu_txn(self, txn, prev_pdus, cols):
         """Inserts a (non-state) PDU into the database.
 
         Args:
@@ -120,11 +122,6 @@ class PduStore(SQLBaseStore):
             prev_pdus (list)
             **cols: The columns to insert into the PdusTable.
         """
-        return self._db_pool.runInteraction(
-            self._persist_pdu, prev_pdus, cols
-        )
-
-    def _persist_pdu(self, txn, prev_pdus, cols):
         entry = PdusTable.EntryType(
             **{k: cols.get(k, None) for k in PdusTable.fields}
         )
@@ -260,7 +257,7 @@ class PduStore(SQLBaseStore):
 
         return row[0] if row else None
 
-    def update_min_depth_for_context(self, context, depth):
+    def _update_min_depth_for_context_txn(self, txn, context, depth):
         """Update the minimum `depth` of the given context, which is the line
         on which we stop backfilling backwards.
 
@@ -268,11 +265,6 @@ class PduStore(SQLBaseStore):
             context (str)
             depth (int)
         """
-        return self._db_pool.runInteraction(
-            self._update_min_depth_for_context, context, depth
-        )
-
-    def _update_min_depth_for_context(self, txn, context, depth):
         min_depth = self._get_min_depth_interaction(txn, context)
 
         do_insert = depth < min_depth if min_depth else True
@@ -284,7 +276,7 @@ class PduStore(SQLBaseStore):
                 (context, depth)
             )
 
-    def get_latest_pdus_in_context(self, context):
+    def _get_latest_pdus_in_context(self, txn, context):
         """Get's a list of the most current pdus for a given context. This is
         used when we are sending a Pdu and need to fill out the `prev_pdus`
         key
@@ -293,11 +285,6 @@ class PduStore(SQLBaseStore):
             txn
             context
         """
-        return self._db_pool.runInteraction(
-            self._get_latest_pdus_in_context, context
-        )
-
-    def _get_latest_pdus_in_context(self, txn, context):
         query = (
             "SELECT p.pdu_id, p.origin, p.depth FROM %(pdus)s as p "
             "INNER JOIN %(forward)s as f ON p.pdu_id = f.pdu_id "
@@ -319,6 +306,7 @@ class PduStore(SQLBaseStore):
 
         return [(row[0], row[1], row[2]) for row in results]
 
+    @defer.inlineCallbacks
     def get_oldest_pdus_in_context(self, context):
         """Get a list of Pdus that we haven't backfilled beyond yet (and haven't    
         seen). This list is used when we want to backfill backwards and is the 
@@ -331,17 +319,14 @@ class PduStore(SQLBaseStore):
         Returns:
             list: A list of PduIdTuple.
         """
-        return self._db_pool.runInteraction(
-            self._get_oldest_pdus_in_context, context
-        )
-
-    def _get_oldest_pdus_in_context(self, txn, context):
-        txn.execute(
+        results = yield self._execute(
+            None,
             "SELECT pdu_id, origin FROM %(back)s WHERE context = ?"
             % {"back": PduBackwardExtremitiesTable.table_name, },
-            (context,)
+            context
         )
-        return [PduIdTuple(i, o) for i, o in txn.fetchall()]
+
+        defer.returnValue([PduIdTuple(i, o) for i, o in results])
 
     def is_pdu_new(self, pdu_id, origin, context, depth):
         """For a given Pdu, try and figure out if it's 'new', i.e., if it's
@@ -485,7 +470,7 @@ class StatePduStore(SQLBaseStore):
     """A collection of queries for handling state PDUs.
     """
 
-    def persist_state(self, prev_pdus, **cols):
+    def _persist_state_txn(self, txn, prev_pdus, cols):
         """Inserts a state PDU into the database
 
         Args:
@@ -493,12 +478,6 @@ class StatePduStore(SQLBaseStore):
             prev_pdus (list)
             **cols: The columns to insert into the PdusTable and StatePdusTable
         """
-
-        return self._db_pool.runInteraction(
-            self._persist_state, prev_pdus, cols
-        )
-
-    def _persist_state(self, txn, prev_pdus, cols):
         pdu_entry = PdusTable.EntryType(
             **{k: cols.get(k, None) for k in PdusTable.fields}
         )
@@ -580,7 +559,7 @@ class StatePduStore(SQLBaseStore):
 
         txn.execute(query, query_args)
 
-    def get_current_state(self, context, pdu_type, state_key):
+    def get_current_state_pdu(self, context, pdu_type, state_key):
         """For a given context, pdu_type, state_key 3-tuple, return what is
         currently considered the current state.
 
@@ -595,10 +574,10 @@ class StatePduStore(SQLBaseStore):
         """
 
         return self._db_pool.runInteraction(
-            self._get_current_state, context, pdu_type, state_key
+            self._get_current_state_pdu, context, pdu_type, state_key
         )
 
-    def _get_current_state(self, txn, context, pdu_type, state_key):
+    def _get_current_state_pdu(self, txn, context, pdu_type, state_key):
         return self._get_current_interaction(txn, context, pdu_type, state_key)
 
     def _get_current_interaction(self, txn, context, pdu_type, state_key):

synapse/storage/presence.py

@@ -67,6 +67,7 @@ class PresenceStore(SQLBaseStore):
             table="presence_allow_inbound",
             keyvalues={"observed_user_id": observed_localpart,
                        "observer_user_id": observer_userid},
+            retcols=["observed_user_id"],
             allow_none=True,
         )
 

synapse/storage/registration.py

@@ -17,7 +17,7 @@ from twisted.internet import defer
 
 from sqlite3 import IntegrityError
 
-from synapse.api.errors import StoreError
+from synapse.api.errors import StoreError, Codes
 
 from ._base import SQLBaseStore
 
@@ -73,7 +73,7 @@ class RegistrationStore(SQLBaseStore):
                         "VALUES (?,?,?)",
                         [user_id, password_hash, now])
         except IntegrityError:
-            raise StoreError(400, "User ID already taken.")
+            raise StoreError(400, "User ID already taken.", errcode=Codes.USER_IN_USE)
 
         # it's possible for this to get a conflict, but only for a single user
         # since tokens are namespaced based on their user ID

synapse/storage/room.py

@@ -18,12 +18,10 @@ from twisted.internet import defer
 from sqlite3 import IntegrityError
 
 from synapse.api.errors import StoreError
-from synapse.api.events.room import RoomTopicEvent
 
 from ._base import SQLBaseStore, Table
 
 import collections
-import json
 import logging
 
 logger = logging.getLogger(__name__)
@@ -76,49 +74,82 @@ class RoomStore(SQLBaseStore):
         )
 
     @defer.inlineCallbacks
-    def get_rooms(self, is_public, with_topics):
+    def get_rooms(self, is_public):
         """Retrieve a list of all public rooms.
 
         Args:
             is_public (bool): True if the rooms returned should be public.
-            with_topics (bool): True to include the current topic for the room
-            in the response.
         Returns:
-            A list of room dicts containing at least a "room_id" key, and a
-            "topic" key if one is set and with_topic=True.
+            A list of room dicts containing at least a "room_id" key, a
+            "topic" key if one is set, and a "name" key if one is set
         """
-        room_data_type = RoomTopicEvent.TYPE
-        public = 1 if is_public else 0
 
-        latest_topic = ("SELECT max(room_data.id) FROM room_data WHERE "
-                        + "room_data.type = ? GROUP BY room_id")
-
-        query = ("SELECT rooms.*, room_data.content, room_alias FROM rooms "
-                 + "LEFT JOIN "
-                 + "room_aliases ON room_aliases.room_id = rooms.room_id "
-                 + "LEFT JOIN "
-                 + "room_data ON rooms.room_id = room_data.room_id WHERE "
-                 + "(room_data.id IN (" + latest_topic + ") "
-                 + "OR room_data.id IS NULL) AND rooms.is_public = ?")
-
-        res = yield self._execute(
-            self.cursor_to_dict, query, room_data_type, public
+        topic_subquery = (
+            "SELECT topics.event_id as event_id, "
+            "topics.room_id as room_id, topic "
+            "FROM topics "
+            "INNER JOIN current_state_events as c "
+            "ON c.event_id = topics.event_id "
         )
 
-        # return only the keys the specification expects
-        ret_keys = ["room_id", "topic", "room_alias"]
+        name_subquery = (
+            "SELECT room_names.event_id as event_id, "
+            "room_names.room_id as room_id, name "
+            "FROM room_names "
+            "INNER JOIN current_state_events as c "
+            "ON c.event_id = room_names.event_id "
+        )
 
-        # extract topic from the json (icky) FIXME
-        for i, room_row in enumerate(res):
-            try:
-                content_json = json.loads(room_row["content"])
-                room_row["topic"] = content_json["topic"]
-            except:
-                pass  # no topic set
-            # filter the dict based on ret_keys
-            res[i] = {k: v for k, v in room_row.iteritems() if k in ret_keys}
+        # We use non printing ascii character US () as a seperator
+        sql = (
+            "SELECT r.room_id, n.name, t.topic, "
+            "group_concat(a.room_alias, '') "
+            "FROM rooms AS r "
+            "LEFT JOIN (%(topic)s) AS t ON t.room_id = r.room_id "
+            "LEFT JOIN (%(name)s) AS n ON n.room_id = r.room_id "
+            "INNER JOIN room_aliases AS a ON a.room_id = r.room_id "
+            "WHERE r.is_public = ? "
+            "GROUP BY r.room_id "
+        ) % {
+            "topic": topic_subquery,
+            "name": name_subquery,
+        }
 
-        defer.returnValue(res)
+        rows = yield self._execute(None, sql, is_public)
+
+        ret = [
+            {
+                "room_id": r[0],
+                "name": r[1],
+                "topic": r[2],
+                "aliases": r[3].split(""),
+            }
+            for r in rows
+        ]
+
+        defer.returnValue(ret)
+
+    def _store_room_topic_txn(self, txn, event):
+        self._simple_insert_txn(
+            txn,
+            "topics",
+            {
+                "event_id": event.event_id,
+                "room_id": event.room_id,
+                "topic": event.topic,
+            }
+        )
+
+    def _store_room_name_txn(self, txn, event):
+        self._simple_insert_txn(
+            txn,
+            "room_names",
+            {
+                "event_id": event.event_id,
+                "room_id": event.room_id,
+                "name": event.name,
+            }
+        )
 
 
 class RoomsTable(Table):

synapse/storage/roomdata.py

@@ -1,85 +0,0 @@
-# -*- coding: utf-8 -*-
-# Copyright 2014 matrix.org
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-from ._base import SQLBaseStore, Table
-
-import collections
-import json
-
-
-class RoomDataStore(SQLBaseStore):
-
-    """Provides various CRUD operations for Room Events. """
-
-    def get_room_data(self, room_id, etype, state_key=""):
-        """Retrieve the data stored under this type and state_key.
-
-        Args:
-            room_id (str)
-            etype (str)
-            state_key (str)
-        Returns:
-            namedtuple: Or None if nothing exists at this path.
-        """
-        query = RoomDataTable.select_statement(
-            "room_id = ? AND type = ? AND state_key = ? "
-            "ORDER BY id DESC LIMIT 1"
-        )
-        return self._execute(
-            RoomDataTable.decode_single_result,
-            query, room_id, etype, state_key,
-        )
-
-    def store_room_data(self, room_id, etype, state_key="", content=None):
-        """Stores room specific data.
-
-        Args:
-            room_id (str)
-            etype (str)
-            state_key (str)
-            data (str)- The data to store for this path in JSON.
-        Returns:
-            The store ID for this data.
-        """
-        return self._simple_insert(RoomDataTable.table_name, dict(
-            etype=etype,
-            state_key=state_key,
-            room_id=room_id,
-            content=content,
-        ))
-
-    def get_max_room_data_id(self):
-        return self._simple_max_id(RoomDataTable.table_name)
-
-
-class RoomDataTable(Table):
-    table_name = "room_data"
-
-    fields = [
-        "id",
-        "room_id",
-        "type",
-        "state_key",
-        "content"
-    ]
-
-    class EntryType(collections.namedtuple("RoomDataEntry", fields)):
-
-        def as_event(self, event_factory):
-            return event_factory.create_event(
-                etype=self.type,
-                room_id=self.room_id,
-                content=json.loads(self.content),
-            )

synapse/storage/roommember.py

@@ -15,15 +15,10 @@
 
 from twisted.internet import defer
 
-from synapse.types import UserID
+from ._base import SQLBaseStore
+
 from synapse.api.constants import Membership
-from synapse.api.events.room import RoomMemberEvent
 
-from ._base import SQLBaseStore, Table
-
-
-import collections
-import json
 import logging
 
 logger = logging.getLogger(__name__)
@@ -31,6 +26,39 @@ logger = logging.getLogger(__name__)
 
 class RoomMemberStore(SQLBaseStore):
 
+    def _store_room_member_txn(self, txn, event):
+        """Store a room member in the database.
+        """
+        target_user_id = event.state_key
+        domain = self.hs.parse_userid(target_user_id).domain
+
+        self._simple_insert_txn(
+            txn,
+            "room_memberships",
+            {
+                "event_id": event.event_id,
+                "user_id": target_user_id,
+                "sender": event.user_id,
+                "room_id": event.room_id,
+                "membership": event.membership,
+            }
+        )
+
+        # Update room hosts table
+        if event.membership == Membership.JOIN:
+            sql = (
+                "INSERT OR IGNORE INTO room_hosts (room_id, host) "
+                "VALUES (?, ?)"
+            )
+            txn.execute(sql, (event.room_id, domain))
+        else:
+            sql = (
+                "DELETE FROM room_hosts WHERE room_id = ? AND host = ?"
+            )
+
+            txn.execute(sql, (event.room_id, domain))
+
+    @defer.inlineCallbacks
     def get_room_member(self, user_id, room_id):
         """Retrieve the current state of a room member.
 
@@ -38,36 +66,33 @@ class RoomMemberStore(SQLBaseStore):
             user_id (str): The member's user ID.
             room_id (str): The room the member is in.
         Returns:
-            namedtuple: The room member from the database, or None if this
-            member does not exist.
+            Deferred: Results in a MembershipEvent or None.
         """
-        query = RoomMemberTable.select_statement(
-            "room_id = ? AND user_id = ? ORDER BY id DESC LIMIT 1")
-        return self._execute(
-            RoomMemberTable.decode_single_result,
-            query, room_id, user_id,
+        rows = yield self._get_members_by_dict({
+            "e.room_id": room_id,
+            "m.user_id": user_id,
+        })
+
+        defer.returnValue(rows[0] if rows else None)
+
+    def _get_room_member(self, txn, user_id, room_id):
+        sql = (
+            "SELECT e.* FROM events as e"
+            " INNER JOIN room_memberships as m"
+            " ON e.event_id = m.event_id"
+            " INNER JOIN current_state_events as c"
+            " ON m.event_id = c.event_id"
+            " WHERE m.user_id = ? and e.room_id = ?"
+            " LIMIT 1"
         )
+        txn.execute(sql, (user_id, room_id))
+        rows = self.cursor_to_dict(txn)
+        if rows:
+            return self._parse_event_from_row(rows[0])
+        else:
+            return None
 
-    def store_room_member(self, user_id, sender, room_id, membership, content):
-        """Store a room member in the database.
 
-        Args:
-            user_id (str): The member's user ID.
-            room_id (str): The room in relation to the member.
-            membership (synapse.api.constants.Membership): The new membership
-            state.
-            content (dict): The content of the membership (JSON).
-        """
-        content_json = json.dumps(content)
-        return self._simple_insert(RoomMemberTable.table_name, dict(
-            user_id=user_id,
-            sender=sender,
-            room_id=room_id,
-            membership=membership,
-            content=content_json,
-        ))
-
-    @defer.inlineCallbacks
     def get_room_members(self, room_id, membership=None):
         """Retrieve the current room member list for a room.
 
@@ -79,17 +104,12 @@ class RoomMemberStore(SQLBaseStore):
         Returns:
             list of namedtuples representing the members in this room.
         """
-        query = RoomMemberTable.select_statement(
-            "id IN (SELECT MAX(id) FROM " + RoomMemberTable.table_name
-            + " WHERE room_id = ? GROUP BY user_id)"
-        )
-        res = yield self._execute(
-            RoomMemberTable.decode_results, query, room_id,
-        )
-        # strip memberships which don't match
+
+        where = {"m.room_id": room_id}
         if membership:
-            res = [entry for entry in res if entry.membership == membership]
-        defer.returnValue(res)
+            where["m.membership"] = membership
+
+        return self._get_members_by_dict(where)
 
     def get_rooms_for_user_where_membership_is(self, user_id, membership_list):
         """ Get all the rooms for this user where the membership for this user
@@ -106,67 +126,61 @@ class RoomMemberStore(SQLBaseStore):
             return defer.succeed(None)
 
         args = [user_id]
-        membership_placeholder = ["membership=?"] * len(membership_list)
-        where_membership = "(" + " OR ".join(membership_placeholder) + ")"
-        for membership in membership_list:
-            args.append(membership)
+        args.extend(membership_list)
 
-        query = ("SELECT room_id, membership FROM room_memberships"
-                 + " WHERE user_id=? AND " + where_membership
-                 + " GROUP BY room_id ORDER BY id DESC")
-        return self._execute(
-            self.cursor_to_dict, query, *args
+        where_clause = "user_id = ? AND (%s)" % (
+            " OR ".join(["membership = ?" for _ in membership_list]),
         )
 
+        return self._get_members_query(where_clause, args)
+
+    def get_joined_hosts_for_room(self, room_id):
+        return self._simple_select_onecol(
+            "room_hosts",
+            {"room_id": room_id},
+            "host"
+        )
+
+    def _get_members_by_dict(self, where_dict):
+        clause = " AND ".join("%s = ?" % k for k in where_dict.keys())
+        vals = where_dict.values()
+        return self._get_members_query(clause, vals)
+
     @defer.inlineCallbacks
-    def get_joined_hosts_for_room(self, room_id):
-        query = RoomMemberTable.select_statement(
-            "id IN (SELECT MAX(id) FROM " + RoomMemberTable.table_name
-            + " WHERE room_id = ? GROUP BY user_id)"
-        )
+    def _get_members_query(self, where_clause, where_values):
+        sql = (
+            "SELECT e.* FROM events as e "
+            "INNER JOIN room_memberships as m "
+            "ON e.event_id = m.event_id "
+            "INNER JOIN current_state_events as c "
+            "ON m.event_id = c.event_id "
+            "WHERE %s "
+        ) % (where_clause,)
 
-        res = yield self._execute(
-            RoomMemberTable.decode_results, query, room_id,
-        )
+        rows = yield self._execute_and_decode(sql, *where_values)
 
-        def host_from_user_id_string(user_id):
-            domain = UserID.from_string(entry.user_id, self.hs).domain
-            return domain
+        # logger.debug("_get_members_query Got rows %s", rows)
 
-        # strip memberships which don't match
-        hosts = [
-            host_from_user_id_string(entry.user_id)
-            for entry in res
-            if entry.membership == Membership.JOIN
-        ]
+        results = [self._parse_event_from_row(r) for r in rows]
+        defer.returnValue(results)
 
-        logger.debug("Returning hosts: %s from results: %s", hosts, res)
+    @defer.inlineCallbacks
+    def do_users_share_a_room(self, user_list):
+        """ Checks whether a list of users share a room.
+        """
+        user_list_clause = " OR ".join(["m.user_id = ?"] * len(user_list))
+        sql = (
+            "SELECT m.room_id FROM room_memberships as m "
+            "INNER JOIN current_state_events as c "
+            "ON m.event_id = c.event_id "
+            "WHERE m.membership = 'join' "
+            "AND (%(clause)s) "
+            "GROUP BY m.room_id HAVING COUNT(m.room_id) = ?"
+        ) % {"clause": user_list_clause}
 
-        defer.returnValue(hosts)
+        args = user_list
+        args.append(len(user_list))
 
-    def get_max_room_member_id(self):
-        return self._simple_max_id(RoomMemberTable.table_name)
+        rows = yield self._execute(None, sql, *args)
 
-
-class RoomMemberTable(Table):
-    table_name = "room_memberships"
-
-    fields = [
-        "id",
-        "user_id",
-        "sender",
-        "room_id",
-        "membership",
-        "content"
-    ]
-
-    class EntryType(collections.namedtuple("RoomMemberEntry", fields)):
-
-        def as_event(self, event_factory):
-            return event_factory.create_event(
-                etype=RoomMemberEvent.TYPE,
-                room_id=self.room_id,
-                target_user_id=self.user_id,
-                user_id=self.sender,
-                content=json.loads(self.content),
-            )
+        defer.returnValue(len(rows) > 0)

synapse/storage/schema/im.sql

@@ -12,43 +12,92 @@
  * See the License for the specific language governing permissions and
  * limitations under the License.
  */
+
+CREATE TABLE IF NOT EXISTS events(
+    stream_ordering INTEGER PRIMARY KEY AUTOINCREMENT,
+    topological_ordering INTEGER NOT NULL,
+    event_id TEXT NOT NULL,
+    type TEXT NOT NULL,
+    room_id TEXT NOT NULL,
+    content TEXT NOT NULL,
+    unrecognized_keys TEXT,
+    processed BOOL NOT NULL,
+    outlier BOOL NOT NULL,
+    CONSTRAINT ev_uniq UNIQUE (event_id)
+);
+
+CREATE INDEX IF NOT EXISTS events_event_id ON events (event_id);
+CREATE INDEX IF NOT EXISTS events_stream_ordering ON events (stream_ordering);
+CREATE INDEX IF NOT EXISTS events_topological_ordering ON events (topological_ordering);
+CREATE INDEX IF NOT EXISTS events_room_id ON events (room_id);
+
+CREATE TABLE IF NOT EXISTS state_events(
+    event_id TEXT NOT NULL,
+    room_id TEXT NOT NULL,
+    type TEXT NOT NULL,
+    state_key TEXT NOT NULL,
+    prev_state TEXT
+);
+
+CREATE UNIQUE INDEX IF NOT EXISTS state_events_event_id ON state_events (event_id);
+CREATE INDEX IF NOT EXISTS state_events_room_id ON state_events (room_id);
+CREATE INDEX IF NOT EXISTS state_events_type ON state_events (type);
+CREATE INDEX IF NOT EXISTS state_events_state_key ON state_events (state_key);
+
+
+CREATE TABLE IF NOT EXISTS current_state_events(
+    event_id TEXT NOT NULL,
+    room_id TEXT NOT NULL,
+    type TEXT NOT NULL,
+    state_key TEXT NOT NULL,
+    CONSTRAINT curr_uniq UNIQUE (room_id, type, state_key) ON CONFLICT REPLACE
+);
+
+CREATE INDEX IF NOT EXISTS curr_events_event_id ON current_state_events (event_id);
+CREATE INDEX IF NOT EXISTS current_state_events_room_id ON current_state_events (room_id);
+CREATE INDEX IF NOT EXISTS current_state_events_type ON current_state_events (type);
+CREATE INDEX IF NOT EXISTS current_state_events_state_key ON current_state_events (state_key);
+
+CREATE TABLE IF NOT EXISTS room_memberships(
+    event_id TEXT NOT NULL,
+    user_id TEXT NOT NULL,
+    sender TEXT NOT NULL,
+    room_id TEXT NOT NULL,
+    membership TEXT NOT NULL
+);
+
+CREATE INDEX IF NOT EXISTS room_memberships_event_id ON room_memberships (event_id);
+CREATE INDEX IF NOT EXISTS room_memberships_room_id ON room_memberships (room_id);
+CREATE INDEX IF NOT EXISTS room_memberships_user_id ON room_memberships (user_id);
+
+CREATE TABLE IF NOT EXISTS feedback(
+    event_id TEXT NOT NULL,
+    feedback_type TEXT,
+    target_event_id TEXT,
+    sender TEXT,
+    room_id TEXT
+);
+
+CREATE TABLE IF NOT EXISTS topics(
+    event_id TEXT NOT NULL,
+    room_id TEXT NOT NULL,
+    topic TEXT NOT NULL
+);
+
+CREATE TABLE IF NOT EXISTS room_names(
+    event_id TEXT NOT NULL,
+    room_id TEXT NOT NULL,
+    name TEXT NOT NULL
+);
+
 CREATE TABLE IF NOT EXISTS rooms(
     room_id TEXT PRIMARY KEY NOT NULL,
     is_public INTEGER,
     creator TEXT
 );
 
-CREATE TABLE IF NOT EXISTS room_memberships(
-    id INTEGER PRIMARY KEY AUTOINCREMENT,
-    user_id TEXT NOT NULL, -- no foreign key to users table, it could be an id belonging to another home server
-    sender TEXT NOT NULL,
+CREATE TABLE IF NOT EXISTS room_hosts(
     room_id TEXT NOT NULL,
-    membership TEXT NOT NULL,
-    content TEXT NOT NULL
-);
-
-CREATE TABLE IF NOT EXISTS messages(
-    id INTEGER PRIMARY KEY AUTOINCREMENT,
-    user_id TEXT, 
-    room_id TEXT,
-    msg_id TEXT,
-    content TEXT
-);
-
-CREATE TABLE IF NOT EXISTS feedback(
-    id INTEGER PRIMARY KEY AUTOINCREMENT,
-    content TEXT,
-    feedback_type TEXT,
-    fb_sender_id TEXT,
-    msg_id TEXT,
-    room_id TEXT,
-    msg_sender_id TEXT
-);
-
-CREATE TABLE IF NOT EXISTS room_data(
-    id INTEGER PRIMARY KEY AUTOINCREMENT,
-    room_id TEXT NOT NULL,
-    type TEXT NOT NULL,
-    state_key TEXT NOT NULL,
-    content TEXT
+    host TEXT NOT NULL,
+    CONSTRAINT room_hosts_uniq UNIQUE (room_id, host) ON CONFLICT IGNORE
 );