Merge branch 'release-v0.3.0' of github.com:matrix-org/synapse

InitialSync: load the 30 last messages of each room so that a full page of messages can be displayed without additionnal request

.gitignore

@@ -18,9 +18,12 @@ htmlcov
 demo/*.db
 demo/*.log
 demo/*.pid
+demo/etc
 
 graph/*.svg
 graph/*.png
 graph/*.dot
 
+webclient/config.js
+
 uploads

CHANGES.rst

@@ -1,3 +1,101 @@
+Changes in synapse 0.3.0 (2014-09-18)
+=====================================
+See UPGRADE for information about changes to the client server API, including
+breaking backwards compatibility with VoIP calls and registration API.
+
+Homeserver:
+ * When a user changes their displayname or avatar the server will now update 
+   all their join states to reflect this.
+ * The server now adds "age" key to events to indicate how old they are. This
+   is clock independent, so at no point does any server or webclient have to
+   assume their clock is in sync with everyone else.
+ * Fix bug where we didn't correctly pull in missing PDUs.
+ * Fix bug where prev_content key wasn't always returned.
+ * Add support for password resets.
+
+Webclient:
+ * Improve page content loading.
+ * Join/parts now trigger desktop notifications.
+ * Always show room aliases in the UI if one is present.
+ * No longer show user-count in the recents side panel.
+ * Add up & down arrow support to the text box for message sending to step
+   through your sent history.
+ * Don't display notifications for our own messages.
+ * Emotes are now formatted correctly in desktop notifications.
+ * The recents list now differentiates between public & private rooms.
+ * Fix bug where when switching between rooms the pagination flickered before
+   the view jumped to the bottom of the screen.
+ * Add support for password resets.
+ * Add bing word support.
+
+Registration API:
+ * The registration API has been overhauled to function like the login API. In
+   practice, this means registration requests must now include the following:
+   'type':'m.login.password'. See UPGRADE for more information on this.
+ * The 'user_id' key has been renamed to 'user' to better match the login API.
+ * There is an additional login type: 'm.login.email.identity'.
+ * The command client and web client have been updated to reflect these changes.
+
+Changes in synapse 0.2.3 (2014-09-12)
+=====================================
+
+Homeserver:
+ * Fix bug where we stopped sending events to remote home servers if a
+   user from that home server left, even if there were some still in the
+   room.
+ * Fix bugs in the state conflict resolution where it was incorrectly
+   rejecting events.
+
+Webclient:
+ * Display room names and topics.
+ * Allow setting/editing of room names and topics.
+ * Display information about rooms on the main page.
+ * Handle ban and kick events in real time.
+ * VoIP UI and reliability improvements.
+ * Add glare support for VoIP.
+ * Improvements to initial startup speed.
+ * Don't display duplicate join events.
+ * Local echo of messages.
+ * Differentiate sending and sent of local echo.
+ * Various minor bug fixes.
+
+Changes in synapse 0.2.2 (2014-09-06)
+=====================================
+
+Homeserver:
+ * When the server returns state events it now also includes the previous 
+   content.
+ * Add support for inviting people when creating a new room.
+ * Make the homeserver inform the room via `m.room.aliases` when a new alias
+   is added for a room.
+ * Validate `m.room.power_level` events.
+
+Webclient:
+ * Add support for captchas on registration.
+ * Handle `m.room.aliases` events.
+ * Asynchronously send messages and show a local echo.
+ * Inform the UI when a message failed to send.
+ * Only autoscroll on receiving a new message if the user was already at the 
+   bottom of the screen.
+ * Add support for ban/kick reasons.
+
+Changes in synapse 0.2.1 (2014-09-03)
+=====================================
+
+Homeserver:
+ * Added support for signing up with a third party id.
+ * Add synctl scripts.
+ * Added rate limiting.
+ * Add option to change the external address the content repo uses.
+ * Presence bug fixes.
+
+Webclient:
+ * Added support for signing up with a third party id.
+ * Added support for banning and kicking users.
+ * Added support for displaying and setting ops.
+ * Added support for room names.
+ * Fix bugs with room membership event display.
+
 Changes in synapse 0.2.0 (2014-09-02)
 =====================================
 This update changes many configuration options, updates the

README.rst

@@ -2,11 +2,11 @@ Introduction
 ============
 
 Matrix is an ambitious new ecosystem for open federated Instant Messaging and
-VoIP[1].  The basics you need to know to get up and running are:
+VoIP.  The basics you need to know to get up and running are:
 
     - Chatrooms are distributed and do not exist on any single server.  Rooms 
-      can be found using names like ``#matrix:matrix.org`` or 
-      ``#test:localhost:8080`` or they can be ephemeral.
+      can be found using aliases like ``#matrix:matrix.org`` or 
+      ``#test:localhost:8008`` or they can be ephemeral.
     
     - Matrix user IDs look like ``@matthew:matrix.org`` (although in the future
       you will normally refer to yourself and others using a 3PID: email
@@ -14,33 +14,52 @@ VoIP[1].  The basics you need to know to get up and running are:
 
 The overall architecture is::
 
-      client <----> homeserver <=================> homeserver <-----> client
-                e.g. matrix.org:8080        e.g. mydomain.net:8080
+      client <----> homeserver <=====================> homeserver <----> client
+             https://somewhere.org/_matrix      https://elsewhere.net/_matrix
+
+WARNING
+=======
+
+**Synapse is currently in a state of rapid development, and not all features are yet functional.
+Critically, some security features are still in development, which means Synapse can *not*
+be considered secure or reliable at this point.**  For instance:
+
+- **SSL Certificates used by server-server federation are not yet validated.**
+- **Room permissions are not yet enforced on traffic received via federation.**
+- **Homeservers do not yet cryptographically sign their events to avoid tampering**
+- Default configuration provides open signup to the service from the internet
+
+Despite this, we believe Synapse is more than useful as a way for experimenting and
+exploring Synapse, and the missing features will land shortly.  **Until then, please do *NOT*
+use Synapse for any remotely important or secure communication.**
+
 
 Quick Start
 ===========
 
+System requirements:
+    - POSIX-compliant system (tested on Linux & OSX)
+    - Python 2.7
+
 To get up and running:
       
     - To simply play with an **existing** homeserver you can
       just go straight to http://matrix.org/alpha.
     
-    - To run your own **private** homeserver on localhost:8080, install synapse 
-      with ``python setup.py develop --user`` and then run one with
-      ``python synapse/app/homeserver.py`` - you will find a webclient running
-      at http://localhost:8080 (use a recent Chrome, Safari or Firefox for now,
-      please...)
+    - To run your own **private** homeserver on localhost:8008, install synapse with
+      ``python setup.py develop --user`` and then run ``./synctl start`` twice (once to
+      generate a config; once to actually run) - you will find a webclient running at
+      http://localhost:8008. Please use a recent Chrome, Safari or Firefox for now...
              
-    - To make the homeserver **public** and let it exchange messages with 
-      other homeservers and participate in the overall Matrix federation, open 
-      up port 8080 and run ``python synapse/app/homeserver.py --host 
-      machine.my.domain.name``.  Then come join ``#matrix:matrix.org`` and
-      say hi! :)
+    - To run a **public** homeserver and let it exchange messages with other homeservers
+      and participate in the global Matrix federation, you must expose port 8448 to the
+      internet and edit homeserver.yaml to specify server_name (the public DNS entry for
+      this server) and then run ``synctl start``. If you changed the server_name, you may
+      need to move the old database (homeserver.db) out of the way first. Then come join
+      ``#matrix:matrix.org`` and say hi! :)
 
 For more detailed setup instructions, please see further down this document.
 
-[1] VoIP currently in development
-
    
 About Matrix
 ============
@@ -50,15 +69,15 @@ which handle:
 
     - Creating and managing fully distributed chat rooms with no
       single points of control or failure
-    - Eventually-consistent cryptographically secure[2] synchronisation of room 
+    - Eventually-consistent cryptographically secure[1] synchronisation of room 
       state across a global open network of federated servers and services
     - Sending and receiving extensible messages in a room with (optional)
-      end-to-end encryption[3]
+      end-to-end encryption[2]
     - Inviting, joining, leaving, kicking, banning room members
     - Managing user accounts (registration, login, logout)
     - Using 3rd Party IDs (3PIDs) such as email addresses, phone numbers,
       Facebook accounts to authenticate, identify and discover users on Matrix.
-    - Placing 1:1 VoIP and Video calls (in development)
+    - Placing 1:1 VoIP and Video calls
 
 These APIs are intended to be implemented on a wide range of servers, services
 and clients, letting developers build messaging and VoIP functionality on top of
@@ -92,9 +111,9 @@ https://github.com/matrix-org/synapse/issues or at matrix@matrix.org.
 
 Thanks for trying Matrix!
 
-[2] Cryptographic signing of messages isn't turned on yet
+[1] Cryptographic signing of messages isn't turned on yet
 
-[3] End-to-end encryption is currently in development
+[2] End-to-end encryption is currently in development
 
 
 Homeserver Installation
@@ -102,7 +121,6 @@ Homeserver Installation
 
 First, the dependencies need to be installed.  Start by installing 
 'python2.7-dev' and the various tools of the compiler toolchain.
-N.B. synapse requires python 2.x where x >= 7
 
   Installing prerequisites on ubuntu::
 
@@ -133,6 +151,9 @@ you can check PyNaCl out of git directly (https://github.com/pyca/pynacl) and
 installing it. Installing PyNaCl using pip may also work (remember to remove any
 other versions installed by setuputils in, for example, ~/.local/lib).
 
+On OSX, if you encounter ``clang: error: unknown argument: '-mno-fused-madd'`` you will
+need to ``export CFLAGS=-Qunused-arguments``.
+
 This will run a process of downloading and installing into your
 user's .local/lib directory all of the required dependencies that are
 missing.
@@ -181,6 +202,10 @@ For the first form, simply pass the required hostname (of the machine) as the
         --config-path homeserver.config \
         --generate-config
     $ python synapse/app/homeserver.py --config-path homeserver.config
+    
+Alternatively, you can run synapse via synctl - running ``synctl start`` to generate a
+homeserver.yaml config file, where you can then edit server-name to specify
+machine.my.domain.name, and then set the actual server running again with synctl start.
 
 For the second form, first create your SRV record and publish it in DNS. This
 needs to be named _matrix._tcp.YOURDOMAIN, and point at at least one hostname
@@ -268,7 +293,7 @@ track 3PID logins and publish end-user public keys.
 
 It's currently early days for identity servers as Matrix is not yet using 3PIDs
 as the primary means of identity and E2E encryption is not complete. As such,
-we're not yet running an identity server in public.
+we are running a single identity server (http://matrix.org:8090) at the current time.
 
 
 Where's the spec?!

UPGRADE.rst

@@ -1,10 +1,36 @@
-Upgrading to v0.2.0
+Upgrading to v0.3.0
 ===================
 
-To upgrade the database schema, run::
+This registration API now closely matches the login API. This introduces a bit
+more backwards and forwards between the HS and the client, but this improves
+the overall flexibility of the API. You can now GET on /register to retrieve a list
+of valid registration flows. Upon choosing one, they are submitted in the same
+way as login, e.g::
 
-    ./database-prepare-for-0.2.0.sh "<database>.db"
+  {
+    type: m.login.password,
+    user: foo,
+    password: bar
+  }
 
+The default HS supports 2 flows, with and without Identity Server email
+authentication. Enabling captcha on the HS will add in an extra step to all
+flows: ``m.login.recaptcha`` which must be completed before you can transition
+to the next stage. There is a new login type: ``m.login.email.identity`` which
+contains the ``threepidCreds`` key which were previously sent in the original
+register request. For more information on this, see the specification.
+
+Web Client
+----------
+
+The VoIP specification has changed between v0.2.0 and v0.3.0. Users should
+refresh any browser tabs to get the latest web client code. Users on
+v0.2.0 of the web client will not be able to call those on v0.3.0 and
+vice versa.
+
+
+Upgrading to v0.2.0
+===================
 
 The home server now requires setting up of SSL config before it can run. To
 automatically generate default config use::

VERSION

@@ -1 +1 @@
-0.2.0
+0.3.0

WISHLIST.rst

@@ -5,3 +5,5 @@ Broad-sweeping stuff which would be nice to have
  - homeserver implementation in go
  - homeserver implementation in node.js
  - client SDKs
+ - libpurple library
+ - irssi plugin?

cmdclient/console.py

@@ -1,6 +1,6 @@
 #!/usr/bin/env python
 
-# Copyright 2014 matrix.org
+# Copyright 2014 OpenMarket Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -88,6 +88,8 @@ class SynapseCmd(cmd.Cmd):
         return False
 
     def _domain(self):
+        if "user" not in self.config or not self.config["user"]:
+            return None
         return self.config["user"].split(":")[1]
 
     def do_config(self, line):
@@ -143,35 +145,50 @@ class SynapseCmd(cmd.Cmd):
         <noupdate> : Do not automatically clobber config values.
         """
         args = self._parse(line, ["userid", "noupdate"])
-        path = "/register"
 
         password = None
         pwd = None
         pwd2 = "_"
         while pwd != pwd2:
-            pwd = getpass.getpass("(Optional) Type a password for this user: ")
-            if len(pwd) == 0:
-                print "Not using a password for this user."
-                break
+            pwd = getpass.getpass("Type a password for this user: ")
             pwd2 = getpass.getpass("Retype the password: ")
-            if pwd != pwd2:
+            if pwd != pwd2 or len(pwd) == 0:
                 print "Password mismatch."
+                pwd = None
             else:
                 password = pwd
 
-        body = {}
+        body = {
+            "type": "m.login.password"
+        }
         if "userid" in args:
-            body["user_id"] = args["userid"]
+            body["user"] = args["userid"]
         if password:
             body["password"] = password
 
-        reactor.callFromThread(self._do_register, "POST", path, body,
+        reactor.callFromThread(self._do_register, body,
                                "noupdate" not in args)
 
     @defer.inlineCallbacks
-    def _do_register(self, method, path, data, update_config):
-        url = self._url() + path
-        json_res = yield self.http_client.do_request(method, url, data=data)
+    def _do_register(self, data, update_config):
+        # check the registration flows
+        url = self._url() + "/register"
+        json_res = yield self.http_client.do_request("GET", url)
+        print json.dumps(json_res, indent=4)
+
+        passwordFlow = None
+        for flow in json_res["flows"]:
+            if flow["type"] == "m.login.recaptcha" or ("stages" in flow and "m.login.recaptcha" in flow["stages"]):
+                print "Unable to register: Home server requires captcha."
+                return
+            if flow["type"] == "m.login.password" and "stages" not in flow:
+                passwordFlow = flow
+                break
+
+        if not passwordFlow:
+            return
+
+        json_res = yield self.http_client.do_request("POST", url, data=data)
         print json.dumps(json_res, indent=4)
         if update_config and "user_id" in json_res:
             self.config["user"] = json_res["user_id"]
@@ -191,10 +208,12 @@ class SynapseCmd(cmd.Cmd):
                 p = getpass.getpass("Enter your password: ")
                 user = args["user_id"]
                 if self._is_on("complete_usernames") and not user.startswith("@"):
-                    user = "@" + user + ":" + self._domain()
-
+                    domain = self._domain()
+                    if domain:
+                        user = "@" + user + ":" + domain
+                
                 reactor.callFromThread(self._do_login, user, p)
-                print " got %s " % p
+                #print " got %s " % p
         except Exception as e:
             print e
 
@@ -312,7 +331,7 @@ class SynapseCmd(cmd.Cmd):
         try:
             args = self._parse(line, ["roomname"], force_keys=True)
             path = "/join/%s" % urllib.quote(args["roomname"])
-            reactor.callFromThread(self._run_and_pprint, "PUT", path, {})
+            reactor.callFromThread(self._run_and_pprint, "POST", path, {})
         except Exception as e:
             print e
 
@@ -700,7 +719,7 @@ def main(server_url, identity_server_url, username, token, config_path):
 if __name__ == '__main__':
     parser = argparse.ArgumentParser("Starts a synapse client.")
     parser.add_argument(
-        "-s", "--server", dest="server", default="http://localhost:8080",
+        "-s", "--server", dest="server", default="http://localhost:8008",
         help="The URL of the home server to talk to.")
     parser.add_argument(
         "-i", "--identity-server", dest="identityserver", default="http://localhost:8090",

cmdclient/http.py

@@ -1,5 +1,5 @@
 # -*- coding: utf-8 -*-
-# Copyright 2014 matrix.org
+# Copyright 2014 OpenMarket Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.

database-prepare-for-0.2.0.sh

@@ -1,10 +0,0 @@
-#!/bin/bash
-
-# This is will prepare a synapse database for running with v0.2.0 of synapse. 
-
-set -e
-
-cp "$1" "$1.bak"
-
-sqlite3 "$1" < "synapse/storage/schema/im.sql" 
-sqlite3 "$1" <<< "PRAGMA user_version = 2;"

docs/client-server/OLD_specification.rst

@@ -2,6 +2,20 @@
 Matrix Client-Server API
 ========================
 
+
+.. WARNING::
+  This specification is old. Please see /docs/specification.rst instead.
+
+
+
+
+
+
+
+
+
+
+
 The following specification outlines how a client can send and receive data from 
 a home server.
 

docs/client-server/howto.rst

@@ -1,9 +1,8 @@
-TODO(kegan): Tweak joinalias API keys/path? Event stream historical > live needs
-a token (currently doesn't). im/sync responses include outdated event formats
-(room membership change messages). Room config (specifically: message history,
-public rooms). /register seems super simplistic compared to /login, maybe it
-would be better if /register used the same technique as /login? /register should
-be "user" not "user_id".
+.. TODO kegan
+  Room config (specifically: message history,
+  public rooms). /register seems super simplistic compared to /login, maybe it
+  would be better if /register used the same technique as /login? /register should
+  be "user" not "user_id".
 
 
 How to use the client-server API
@@ -15,7 +14,7 @@ implementation, there may be variations in relation to registering/logging in
 which are not covered in extensive detail in this guide.
 
 If you haven't already, get a home server up and running on 
-``http://localhost:8080``.
+``http://localhost:8008``.
 
 
 Accounts
@@ -23,14 +22,16 @@ Accounts
 Before you can send and receive messages, you must **register** for an account. 
 If you already have an account, you must **login** into it.
 
-**Try out the fiddle: http://jsfiddle.net/jrf1h02d/**
+`Try out the fiddle`__
+
+.. __: http://jsfiddle.net/gh/get/jquery/1.8.3/matrix-org/synapse/tree/master/jsfiddles/register_login
 
 Registration
 ------------
 The aim of registration is to get a user ID and access token which you will need
 when accessing other APIs::
 
-    curl -XPOST -d '{"user_id":"example", "password":"wordpass"}' "http://localhost:8080/_matrix/client/api/v1/register"
+    curl -XPOST -d '{"user_id":"example", "password":"wordpass"}' "http://localhost:8008/_matrix/client/api/v1/register"
 
     {
         "access_token": "QGV4YW1wbGU6bG9jYWxob3N0.AqdSzFmFYrLrTmteXc", 
@@ -51,13 +52,17 @@ Login
 -----
 The aim when logging in is to get an access token for your existing user ID::
 
-    curl -XGET "http://localhost:8080/_matrix/client/api/v1/login"
+    curl -XGET "http://localhost:8008/_matrix/client/api/v1/login"
 
     {
-        "type": "m.login.password"
+        "flows": [
+            {
+                "type": "m.login.password"
+            }
+        ]
     }
 
-    curl -XPOST -d '{"type":"m.login.password", "user":"example", "password":"wordpass"}' "http://localhost:8080/_matrix/client/api/v1/login"
+    curl -XPOST -d '{"type":"m.login.password", "user":"example", "password":"wordpass"}' "http://localhost:8008/_matrix/client/api/v1/login"
 
     {
         "access_token": "QGV4YW1wbGU6bG9jYWxob3N0.vRDLTgxefmKWQEtgGd", 
@@ -80,14 +85,16 @@ Communicating
 In order to communicate with another user, you must **create a room** with that 
 user and **send a message** to that room. 
 
-**Try out the fiddle: http://jsfiddle.net/jnwqcshc/**
+`Try out the fiddle`__
+
+.. __: http://jsfiddle.net/gh/get/jquery/1.8.3/matrix-org/synapse/tree/master/jsfiddles/create_room_send_msg
 
 Creating a room
 ---------------
 If you want to send a message to someone, you have to be in a room with them. To
 create a room::
 
-    curl -XPOST -d '{"room_alias_name":"tutorial"}' "http://localhost:8080/_matrix/client/api/v1/rooms?access_token=QGV4YW1wbGU6bG9jYWxob3N0.vRDLTgxefmKWQEtgGd"
+    curl -XPOST -d '{"room_alias_name":"tutorial"}' "http://localhost:8008/_matrix/client/api/v1/createRoom?access_token=YOUR_ACCESS_TOKEN"
 
     {
         "room_alias": "#tutorial:localhost", 
@@ -98,20 +105,27 @@ The "room alias" is a human-readable string which can be shared with other users
 so they can join a room, rather than the room ID which is a randomly generated
 string. You can have multiple room aliases per room.
 
-TODO(kegan): How to add/remove aliases from an existing room.
+.. TODO(kegan)
+  How to add/remove aliases from an existing room.
     
 
 Sending messages
 ----------------
 You can now send messages to this room::
 
-    curl -XPUT -d '{"msgtype":"m.text", "body":"hello"}' "http://localhost:8080/_matrix/client/api/v1/rooms/%21CvcvRuDYDzTOzfKKgh:localhost/messages/%40example%3Alocalhost/msgid1?access_token=QGV4YW1wbGU6bG9jYWxob3N0.vRDLTgxefmKWQEtgGd"
+    curl -XPOST -d '{"msgtype":"m.text", "body":"hello"}' "http://localhost:8008/_matrix/client/api/v1/rooms/%21CvcvRuDYDzTOzfKKgh%3Alocalhost/send/m.room.message?access_token=YOUR_ACCESS_TOKEN"
+    
+    {
+        "event_id": "YUwRidLecu"
+    }
+    
+The event ID returned is a unique ID which identifies this message.
     
 NB: There are no limitations to the types of messages which can be exchanged.
-The only requirement is that ``"msgtype"`` is specified.
-
-NB: Depending on the room config, users who join the room may be able to see
-message history from before they joined.
+The only requirement is that ``"msgtype"`` is specified. The Matrix 
+specification outlines the following standard types: ``m.text``, ``m.image``,
+``m.audio``, ``m.video``, ``m.location``, ``m.emote``. See the specification for
+more information on these types.
 
 Users and rooms
 ===============
@@ -121,33 +135,34 @@ these rules may specify if you require an **invitation** from someone already in
 the room in order to **join the room**. In addition, you may also be able to 
 join a room **via a room alias** if one was set up.
 
-**Try out the fiddle: http://jsfiddle.net/og1xokcr/**
+`Try out the fiddle`__
+
+.. __: http://jsfiddle.net/gh/get/jquery/1.8.3/matrix-org/synapse/tree/master/jsfiddles/room_memberships
 
 Inviting a user to a room
 -------------------------
 You can directly invite a user to a room like so::
 
-    curl -XPUT -d '{"membership":"invite"}' "http://localhost:8080/_matrix/client/api/v1/rooms/%21CvcvRuDYDzTOzfKKgh:localhost/members/%40myfriend%3Alocalhost/state?access_token=QGV4YW1wbGU6bG9jYWxob3N0.vRDLTgxefmKWQEtgGd"
+    curl -XPOST -d '{"user_id":"@myfriend:localhost"}' "http://localhost:8008/_matrix/client/api/v1/rooms/%21CvcvRuDYDzTOzfKKgh%3Alocalhost/invite?access_token=YOUR_ACCESS_TOKEN"
     
 This informs ``@myfriend:localhost`` of the room ID 
 ``!CvcvRuDYDzTOzfKKgh:localhost`` and allows them to join the room.
 
 Joining a room via an invite
 ----------------------------
-If you receive an invite, you can join the room by changing the membership to
-join::
+If you receive an invite, you can join the room::
 
-    curl -XPUT -d '{"membership":"join"}' "http://localhost:8080/_matrix/client/api/v1/rooms/%21CvcvRuDYDzTOzfKKgh:localhost/members/%40myfriend%3Alocalhost/state?access_token=QG15ZnJpZW5kOmxvY2FsaG9zdA...XKuGdVsovHmwMyDDvK"
+    curl -XPOST -d '{}' "http://localhost:8008/_matrix/client/api/v1/rooms/%21CvcvRuDYDzTOzfKKgh%3Alocalhost/join?access_token=YOUR_ACCESS_TOKEN"
     
 NB: Only the person invited (``@myfriend:localhost``) can change the membership
-state to ``"join"``.
+state to ``"join"``. Repeatedly joining a room does nothing.
 
 Joining a room via an alias
 ---------------------------
 Alternatively, if you know the room alias for this room and the room config 
 allows it, you can directly join a room via the alias::
 
-    curl -XPUT -d '{}' "http://localhost:8080/_matrix/client/api/v1/join/%23tutorial%3Alocalhost?access_token=QG15ZnJpZW5kOmxvY2FsaG9zdA...XKuGdVsovHmwMyDDvK"
+    curl -XPOST -d '{}' "http://localhost:8008/_matrix/client/api/v1/join/%23tutorial%3Alocalhost?access_token=YOUR_ACCESS_TOKEN"
     
     {
         "room_id": "!CvcvRuDYDzTOzfKKgh:localhost"
@@ -166,128 +181,444 @@ An event is some interesting piece of data that a client may be interested in.
 It can be a message in a room, a room invite, etc. There are many different ways
 of getting events, depending on what the client already knows.
 
-**Try out the fiddle: http://jsfiddle.net/5uk4dqe2/**
+`Try out the fiddle`__
+
+.. __: http://jsfiddle.net/gh/get/jquery/1.8.3/matrix-org/synapse/tree/master/jsfiddles/event_stream
 
 Getting all state
 -----------------
 If the client doesn't know any information on the rooms the user is 
 invited/joined on, they can get all the user's state for all rooms::
 
-    curl -XGET "http://localhost:8080/_matrix/client/api/v1/im/sync?access_token=QG15ZnJpZW5kOmxvY2FsaG9zdA...XKuGdVsovHmwMyDDvK"
+    curl -XGET "http://localhost:8008/_matrix/client/api/v1/initialSync?access_token=YOUR_ACCESS_TOKEN"
     
-    [
-        {
-            "membership": "join", 
-            "messages": {
-                "chunk": [
-                    {
-                        "content": {
-                            "body": "@example:localhost joined the room.", 
-                            "hsob_ts": 1408444664249, 
+    {
+        "end": "s39_18_0", 
+        "presence": [
+            {
+                "content": {
+                    "last_active_ago": 1061436, 
+                    "user_id": "@example:localhost"
+                }, 
+                "type": "m.presence"
+            }
+        ], 
+        "rooms": [
+            {
+                "membership": "join", 
+                "messages": {
+                    "chunk": [
+                        {
+                            "content": {
+                                "@example:localhost": 10, 
+                                "default": 0
+                            }, 
+                            "event_id": "wAumPSTsWF", 
+                            "required_power_level": 10, 
+                            "room_id": "!MkDbyRqnvTYnoxjLYx:localhost", 
+                            "state_key": "", 
+                            "ts": 1409665585188, 
+                            "type": "m.room.power_levels", 
+                            "user_id": "@example:localhost"
+                        }, 
+                        {
+                            "content": {
+                                "join_rule": "public"
+                            }, 
+                            "event_id": "jrLVqKHKiI", 
+                            "required_power_level": 10, 
+                            "room_id": "!MkDbyRqnvTYnoxjLYx:localhost", 
+                            "state_key": "", 
+                            "ts": 1409665585188, 
+                            "type": "m.room.join_rules", 
+                            "user_id": "@example:localhost"
+                        }, 
+                        {
+                            "content": {
+                                "level": 10
+                            }, 
+                            "event_id": "WpmTgsNWUZ", 
+                            "required_power_level": 10, 
+                            "room_id": "!MkDbyRqnvTYnoxjLYx:localhost", 
+                            "state_key": "", 
+                            "ts": 1409665585188, 
+                            "type": "m.room.add_state_level", 
+                            "user_id": "@example:localhost"
+                        }, 
+                        {
+                            "content": {
+                                "level": 0
+                            }, 
+                            "event_id": "qUMBJyKsTQ", 
+                            "required_power_level": 10, 
+                            "room_id": "!MkDbyRqnvTYnoxjLYx:localhost", 
+                            "state_key": "", 
+                            "ts": 1409665585188, 
+                            "type": "m.room.send_event_level", 
+                            "user_id": "@example:localhost"
+                        }, 
+                        {
+                            "content": {
+                                "ban_level": 5, 
+                                "kick_level": 5
+                            }, 
+                            "event_id": "YAaDmKvoUW", 
+                            "required_power_level": 10, 
+                            "room_id": "!MkDbyRqnvTYnoxjLYx:localhost", 
+                            "state_key": "", 
+                            "ts": 1409665585188, 
+                            "type": "m.room.ops_levels", 
+                            "user_id": "@example:localhost"
+                        }, 
+                        {
+                            "content": {
+                                "avatar_url": null, 
+                                "displayname": null, 
+                                "membership": "join"
+                            }, 
+                            "event_id": "RJbPMtCutf", 
                             "membership": "join", 
-                            "membership_source": "@example:localhost", 
-                            "membership_target": "@example:localhost", 
-                            "msgtype": "m.text"
+                            "room_id": "!MkDbyRqnvTYnoxjLYx:localhost", 
+                            "state_key": "@example:localhost", 
+                            "ts": 1409665586730, 
+                            "type": "m.room.member", 
+                            "user_id": "@example:localhost"
                         }, 
-                        "event_id": "lZjmmlrEvo", 
-                        "msg_id": "m1408444664249", 
-                        "room_id": "!CvcvRuDYDzTOzfKKgh:localhost", 
-                        "type": "m.room.message", 
-                        "user_id": "_homeserver_"
-                    }, 
+                        {
+                            "content": {
+                                "body": "hello", 
+                                "hsob_ts": 1409665660439, 
+                                "msgtype": "m.text"
+                            }, 
+                            "event_id": "YUwRidLecu", 
+                            "room_id": "!MkDbyRqnvTYnoxjLYx:localhost", 
+                            "ts": 1409665660439, 
+                            "type": "m.room.message", 
+                            "user_id": "@example:localhost"
+                        }, 
+                        {
+                            "content": {
+                                "membership": "invite"
+                            }, 
+                            "event_id": "YjNuBKnPsb", 
+                            "membership": "invite", 
+                            "room_id": "!MkDbyRqnvTYnoxjLYx:localhost", 
+                            "state_key": "@myfriend:localhost", 
+                            "ts": 1409666426819, 
+                            "type": "m.room.member", 
+                            "user_id": "@example:localhost"
+                        }, 
+                        {
+                            "content": {
+                                "avatar_url": null, 
+                                "displayname": null, 
+                                "membership": "join", 
+                                "prev": "join"
+                            }, 
+                            "event_id": "KWwdDjNZnm", 
+                            "membership": "join", 
+                            "room_id": "!MkDbyRqnvTYnoxjLYx:localhost", 
+                            "state_key": "@example:localhost", 
+                            "ts": 1409666551582, 
+                            "type": "m.room.member", 
+                            "user_id": "@example:localhost"
+                        }, 
+                        {
+                            "content": {
+                                "avatar_url": null, 
+                                "displayname": null, 
+                                "membership": "join"
+                            }, 
+                            "event_id": "JFLVteSvQc", 
+                            "membership": "join", 
+                            "room_id": "!MkDbyRqnvTYnoxjLYx:localhost", 
+                            "state_key": "@example:localhost", 
+                            "ts": 1409666587265, 
+                            "type": "m.room.member", 
+                            "user_id": "@example:localhost"
+                        }
+                    ], 
+                    "end": "s39_18_0", 
+                    "start": "t1-11_18_0"
+                }, 
+                "room_id": "!MkDbyRqnvTYnoxjLYx:localhost", 
+                "state": [
                     {
                         "content": {
-                            "body": "hello", 
-                            "hsob_ts": 1408445405672, 
-                            "msgtype": "m.text"
+                            "creator": "@example:localhost"
                         }, 
-                        "event_id": "BiBJqamISg", 
-                        "msg_id": "msgid1", 
-                        "room_id": "!CvcvRuDYDzTOzfKKgh:localhost", 
-                        "type": "m.room.message", 
+                        "event_id": "dMUoqVTZca", 
+                        "required_power_level": 10, 
+                        "room_id": "!MkDbyRqnvTYnoxjLYx:localhost", 
+                        "state_key": "", 
+                        "ts": 1409665585188, 
+                        "type": "m.room.create", 
                         "user_id": "@example:localhost"
                     }, 
-                    [...]
                     {
                         "content": {
-                            "body": "@myfriend:localhost joined the room.", 
-                            "hsob_ts": 1408446501661, 
-                            "membership": "join", 
-                            "membership_source": "@myfriend:localhost", 
-                            "membership_target": "@myfriend:localhost", 
-                            "msgtype": "m.text"
+                            "@example:localhost": 10, 
+                            "default": 0
                         }, 
-                        "event_id": "IMmXbOzFAa", 
-                        "msg_id": "m1408446501661", 
-                        "room_id": "!CvcvRuDYDzTOzfKKgh:localhost", 
-                        "type": "m.room.message", 
-                        "user_id": "_homeserver_"
+                        "event_id": "wAumPSTsWF", 
+                        "required_power_level": 10, 
+                        "room_id": "!MkDbyRqnvTYnoxjLYx:localhost", 
+                        "state_key": "", 
+                        "ts": 1409665585188, 
+                        "type": "m.room.power_levels", 
+                        "user_id": "@example:localhost"
+                    }, 
+                    {
+                        "content": {
+                            "join_rule": "public"
+                        }, 
+                        "event_id": "jrLVqKHKiI", 
+                        "required_power_level": 10, 
+                        "room_id": "!MkDbyRqnvTYnoxjLYx:localhost", 
+                        "state_key": "", 
+                        "ts": 1409665585188, 
+                        "type": "m.room.join_rules", 
+                        "user_id": "@example:localhost"
+                    }, 
+                    {
+                        "content": {
+                            "level": 10
+                        }, 
+                        "event_id": "WpmTgsNWUZ", 
+                        "required_power_level": 10, 
+                        "room_id": "!MkDbyRqnvTYnoxjLYx:localhost", 
+                        "state_key": "", 
+                        "ts": 1409665585188, 
+                        "type": "m.room.add_state_level", 
+                        "user_id": "@example:localhost"
+                    }, 
+                    {
+                        "content": {
+                            "level": 0
+                        }, 
+                        "event_id": "qUMBJyKsTQ", 
+                        "required_power_level": 10, 
+                        "room_id": "!MkDbyRqnvTYnoxjLYx:localhost", 
+                        "state_key": "", 
+                        "ts": 1409665585188, 
+                        "type": "m.room.send_event_level", 
+                        "user_id": "@example:localhost"
+                    }, 
+                    {
+                        "content": {
+                            "ban_level": 5, 
+                            "kick_level": 5
+                        }, 
+                        "event_id": "YAaDmKvoUW", 
+                        "required_power_level": 10, 
+                        "room_id": "!MkDbyRqnvTYnoxjLYx:localhost", 
+                        "state_key": "", 
+                        "ts": 1409665585188, 
+                        "type": "m.room.ops_levels", 
+                        "user_id": "@example:localhost"
+                    }, 
+                    {
+                        "content": {
+                            "membership": "invite"
+                        }, 
+                        "event_id": "YjNuBKnPsb", 
+                        "membership": "invite", 
+                        "room_id": "!MkDbyRqnvTYnoxjLYx:localhost", 
+                        "state_key": "@myfriend:localhost", 
+                        "ts": 1409666426819, 
+                        "type": "m.room.member", 
+                        "user_id": "@example:localhost"
+                    }, 
+                    {
+                        "content": {
+                            "avatar_url": null, 
+                            "displayname": null, 
+                            "membership": "join"
+                        }, 
+                        "event_id": "JFLVteSvQc", 
+                        "membership": "join", 
+                        "room_id": "!MkDbyRqnvTYnoxjLYx:localhost", 
+                        "state_key": "@example:localhost", 
+                        "ts": 1409666587265, 
+                        "type": "m.room.member", 
+                        "user_id": "@example:localhost"
                     }
-                ], 
-                "end": "20", 
-                "start": "0"
-            }, 
-            "room_id": "!CvcvRuDYDzTOzfKKgh:localhost"
-        }
-    ]
+                ]
+            }
+        ]
+    }
     
-This returns all the room IDs of rooms the user is invited/joined on, as well as
-all of the messages and feedback for these rooms. This can be a LOT of data. You
-may just want the most recent message for each room. This can be achieved by 
-applying pagination stream parameters to this request::
+This returns all the room information the user is invited/joined on, as well as
+all of the presences relevant for these rooms. This can be a LOT of data. You
+may just want the most recent event for each room. This can be achieved by 
+applying query parameters to ``limit`` this request::
 
-    curl -XGET "http://localhost:8080/_matrix/client/api/v1/im/sync?access_token=QG15ZnJpZW5kOmxvY2FsaG9zdA...XKuGdVsovHmwMyDDvK&from=END&to=START&limit=1"
+    curl -XGET "http://localhost:8008/_matrix/client/api/v1/initialSync?limit=1&access_token=YOUR_ACCESS_TOKEN"
     
-    [
-        {
-            "membership": "join", 
-            "messages": {
-                "chunk": [
+    {
+        "end": "s39_18_0", 
+        "presence": [
+            {
+                "content": {
+                    "last_active_ago": 1279484, 
+                    "user_id": "@example:localhost"
+                }, 
+                "type": "m.presence"
+            }
+        ], 
+        "rooms": [
+            {
+                "membership": "join", 
+                "messages": {
+                    "chunk": [
+                        {
+                            "content": {
+                                "avatar_url": null, 
+                                "displayname": null, 
+                                "membership": "join"
+                            }, 
+                            "event_id": "JFLVteSvQc", 
+                            "membership": "join", 
+                            "room_id": "!MkDbyRqnvTYnoxjLYx:localhost", 
+                            "state_key": "@example:localhost", 
+                            "ts": 1409666587265, 
+                            "type": "m.room.member", 
+                            "user_id": "@example:localhost"
+                        }
+                    ], 
+                    "end": "s39_18_0", 
+                    "start": "t10-30_18_0"
+                }, 
+                "room_id": "!MkDbyRqnvTYnoxjLYx:localhost", 
+                "state": [
                     {
                         "content": {
-                            "body": "@myfriend:localhost joined the room.", 
-                            "hsob_ts": 1408446501661, 
-                            "membership": "join", 
-                            "membership_source": "@myfriend:localhost", 
-                            "membership_target": "@myfriend:localhost", 
-                            "msgtype": "m.text"
+                            "creator": "@example:localhost"
                         }, 
-                        "event_id": "IMmXbOzFAa", 
-                        "msg_id": "m1408446501661", 
-                        "room_id": "!CvcvRuDYDzTOzfKKgh:localhost", 
-                        "type": "m.room.message", 
-                        "user_id": "_homeserver_"
+                        "event_id": "dMUoqVTZca", 
+                        "required_power_level": 10, 
+                        "room_id": "!MkDbyRqnvTYnoxjLYx:localhost", 
+                        "state_key": "", 
+                        "ts": 1409665585188, 
+                        "type": "m.room.create", 
+                        "user_id": "@example:localhost"
+                    }, 
+                    {
+                        "content": {
+                            "@example:localhost": 10, 
+                            "default": 0
+                        }, 
+                        "event_id": "wAumPSTsWF", 
+                        "required_power_level": 10, 
+                        "room_id": "!MkDbyRqnvTYnoxjLYx:localhost", 
+                        "state_key": "", 
+                        "ts": 1409665585188, 
+                        "type": "m.room.power_levels", 
+                        "user_id": "@example:localhost"
+                    }, 
+                    {
+                        "content": {
+                            "join_rule": "public"
+                        }, 
+                        "event_id": "jrLVqKHKiI", 
+                        "required_power_level": 10, 
+                        "room_id": "!MkDbyRqnvTYnoxjLYx:localhost", 
+                        "state_key": "", 
+                        "ts": 1409665585188, 
+                        "type": "m.room.join_rules", 
+                        "user_id": "@example:localhost"
+                    }, 
+                    {
+                        "content": {
+                            "level": 10
+                        }, 
+                        "event_id": "WpmTgsNWUZ", 
+                        "required_power_level": 10, 
+                        "room_id": "!MkDbyRqnvTYnoxjLYx:localhost", 
+                        "state_key": "", 
+                        "ts": 1409665585188, 
+                        "type": "m.room.add_state_level", 
+                        "user_id": "@example:localhost"
+                    }, 
+                    {
+                        "content": {
+                            "level": 0
+                        }, 
+                        "event_id": "qUMBJyKsTQ", 
+                        "required_power_level": 10, 
+                        "room_id": "!MkDbyRqnvTYnoxjLYx:localhost", 
+                        "state_key": "", 
+                        "ts": 1409665585188, 
+                        "type": "m.room.send_event_level", 
+                        "user_id": "@example:localhost"
+                    }, 
+                    {
+                        "content": {
+                            "ban_level": 5, 
+                            "kick_level": 5
+                        }, 
+                        "event_id": "YAaDmKvoUW", 
+                        "required_power_level": 10, 
+                        "room_id": "!MkDbyRqnvTYnoxjLYx:localhost", 
+                        "state_key": "", 
+                        "ts": 1409665585188, 
+                        "type": "m.room.ops_levels", 
+                        "user_id": "@example:localhost"
+                    }, 
+                    {
+                        "content": {
+                            "membership": "invite"
+                        }, 
+                        "event_id": "YjNuBKnPsb", 
+                        "membership": "invite", 
+                        "room_id": "!MkDbyRqnvTYnoxjLYx:localhost", 
+                        "state_key": "@myfriend:localhost", 
+                        "ts": 1409666426819, 
+                        "type": "m.room.member", 
+                        "user_id": "@example:localhost"
+                    }, 
+                    {
+                        "content": {
+                            "avatar_url": null, 
+                            "displayname": null, 
+                            "membership": "join"
+                        }, 
+                        "event_id": "JFLVteSvQc", 
+                        "membership": "join", 
+                        "room_id": "!MkDbyRqnvTYnoxjLYx:localhost", 
+                        "state_key": "@example:localhost", 
+                        "ts": 1409666587265, 
+                        "type": "m.room.member", 
+                        "user_id": "@example:localhost"
                     }
-                ], 
-                "end": "20", 
-                "start": "21"
-            }, 
-            "room_id": "!CvcvRuDYDzTOzfKKgh:localhost"
-        }
-    ]
+                ]
+            }
+        ]
+    }
 
 Getting live state
 ------------------
 Once you know which rooms the client has previously interacted with, you need to
 listen for incoming events. This can be done like so::
 
-    curl -XGET "http://localhost:8080/_matrix/client/api/v1/events?access_token=QG15ZnJpZW5kOmxvY2FsaG9zdA...XKuGdVsovHmwMyDDvK&from=END"
+    curl -XGET "http://localhost:8008/_matrix/client/api/v1/events?access_token=YOUR_ACCESS_TOKEN"
     
     {
         "chunk": [], 
-        "end": "215", 
-        "start": "215"
+        "end": "s39_18_0", 
+        "start": "s39_18_0"
     }
     
 This will block waiting for an incoming event, timing out after several seconds.
 Even if there are no new events (as in the example above), there will be some
 pagination stream response keys. The client should make subsequent requests 
-using the value of the ``"end"`` key (in this case ``215``) as the ``from`` 
-query parameter. This value should be stored so when the client reopens your app
-after a period of inactivity, you can resume from where you got up to in the 
-event stream. If it has been a long period of inactivity, there may be LOTS of 
-events waiting for the user. In this case, you may wish to get all state instead
-and then resume getting live state from a newer end token.
+using the value of the ``"end"`` key (in this case ``s39_18_0``) as the ``from`` 
+query parameter e.g. ``http://localhost:8008/_matrix/client/api/v1/events?access
+_token=YOUR_ACCESS_TOKEN&from=s39_18_0``. This value should be stored so when the 
+client reopens your app after a period of inactivity, you can resume from where 
+you got up to in the event stream. If it has been a long period of inactivity, 
+there may be LOTS of events waiting for the user. In this case, you may wish to 
+get all state instead and then resume getting live state from a newer end token.
 
 NB: The timeout can be changed by adding a ``timeout`` query parameter, which is
 in milliseconds. A timeout of 0 will not block.
@@ -300,4 +631,6 @@ creating and joining rooms, sending messages, getting member lists and getting
 historical messages for a room. This covers most functionality of a messaging
 application.
 
-**Try out the fiddle: http://jsfiddle.net/L8r3o1wr/**
+`Try out the fiddle`__
+
+.. __: http://jsfiddle.net/gh/get/jquery/1.8.3/matrix-org/synapse/tree/master/jsfiddles/example_app

docs/client-server/swagger_matrix/api-docs-directory

@@ -1,7 +1,7 @@
 {
   "apiVersion": "1.0.0",
   "swaggerVersion": "1.2",
-  "basePath": "http://localhost:8080/_matrix/client/api/v1",
+  "basePath": "http://localhost:8008/_matrix/client/api/v1",
   "resourcePath": "/directory",
   "produces": [
     "application/json"
@@ -13,6 +13,7 @@
         {
           "method": "GET",
           "summary": "Get the room ID corresponding to this room alias.",
+          "notes": "Volatile: This API is likely to change.",
           "type": "DirectoryResponse",
           "nickname": "get_room_id_for_alias",
           "parameters": [
@@ -28,6 +29,7 @@
         {
           "method": "PUT",
           "summary": "Create a new mapping from room alias to room ID.",
+          "notes": "Volatile: This API is likely to change.",
           "type": "void",
           "nickname": "add_room_alias",
           "parameters": [

docs/client-server/swagger_matrix/api-docs-events

@@ -1,7 +1,7 @@
 {
   "apiVersion": "1.0.0",
   "swaggerVersion": "1.2",
-  "basePath": "http://localhost:8080/_matrix/client/api/v1",
+  "basePath": "http://localhost:8008/_matrix/client/api/v1",
   "resourcePath": "/events",
   "produces": [
     "application/json"

docs/client-server/swagger_matrix/api-docs-login

@@ -8,7 +8,7 @@
           "nickname": "get_login_info", 
           "notes": "All login stages MUST be mentioned if there is >1 login type.", 
           "summary": "Get the login mechanism to use when logging in.", 
-          "type": "LoginInfo"
+          "type": "LoginFlows"
         }, 
         {
           "method": "POST", 
@@ -40,17 +40,31 @@
       "path": "/login"
     }
   ], 
-  "basePath": "http://localhost:8080/_matrix/client/api/v1", 
+  "basePath": "http://localhost:8008/_matrix/client/api/v1", 
   "consumes": [
     "application/json"
   ], 
   "models": {
+    "LoginFlows": {
+      "id": "LoginFlows",
+      "properties": {
+        "flows": {
+          "description": "A list of valid login flows.",
+          "type": "array",
+          "items": {
+            "$ref": "LoginInfo"
+          }
+        }
+      }
+    },
     "LoginInfo": {
       "id": "LoginInfo", 
       "properties": {
         "stages": {
           "description": "Multi-stage login only: An array of all the login types required to login.", 
-          "format": "string", 
+          "items": {
+            "$ref": "string"
+          }, 
           "type": "array"
         }, 
         "type": {
@@ -65,6 +79,10 @@
         "access_token": {
           "description": "The access token for this user's login if this is the final stage of the login process.", 
           "type": "string"
+        },
+        "user_id": {
+          "description": "The user's fully-qualified user ID.",
+          "type": "string"
         }, 
         "next": {
           "description": "Multi-stage login only: The next login type to submit.", 

docs/client-server/swagger_matrix/api-docs-presence

@@ -1,7 +1,7 @@
 {
   "apiVersion": "1.0.0",
   "swaggerVersion": "1.2",
-  "basePath": "http://localhost:8080/_matrix/client/api/v1",
+  "basePath": "http://localhost:8008/_matrix/client/api/v1",
   "resourcePath": "/presence",
   "produces": [
     "application/json"
@@ -106,7 +106,7 @@
     "PresenceUpdate": {
       "id": "PresenceUpdate",
       "properties": {
-        "state": {
+        "presence": {
           "type": "string",
           "description": "Enum: The presence state.",
           "enum": [
@@ -128,10 +128,10 @@
     "Presence": {
       "id": "Presence",
       "properties": {
-        "mtime_age": {
+        "last_active_ago": {
           "type": "integer",
           "format": "int64",
-          "description": "The last time this user's presence state changed, in milliseconds."
+          "description": "The last time this user performed an action on their home server."
         },
         "user_id": {
           "type": "string",

docs/client-server/swagger_matrix/api-docs-profile

@@ -1,7 +1,7 @@
 {
   "apiVersion": "1.0.0",
   "swaggerVersion": "1.2",
-  "basePath": "http://localhost:8080/_matrix/client/api/v1",
+  "basePath": "http://localhost:8008/_matrix/client/api/v1",
   "resourcePath": "/profile",
   "produces": [
     "application/json"

docs/client-server/swagger_matrix/api-docs-registration

@@ -3,65 +3,110 @@
   "apis": [
     {
       "operations": [
+        {
+          "method": "GET", 
+          "nickname": "get_registration_info", 
+          "notes": "All login stages MUST be mentioned if there is >1 login type.", 
+          "summary": "Get the login mechanism to use when registering.", 
+          "type": "RegistrationFlows"
+        }, 
         {
           "method": "POST", 
-          "nickname": "register", 
-          "notes": "Volatile: This API is likely to change.", 
+          "nickname": "submit_registration", 
+          "notes": "If this is part of a multi-stage registration, there MUST be a 'session' key.", 
           "parameters": [
             {
-              "description": "A registration request", 
+              "description": "A registration submission", 
               "name": "body", 
               "paramType": "body", 
               "required": true, 
-              "type": "RegistrationRequest"
+              "type": "RegistrationSubmission"
             }
           ], 
           "responseMessages": [
             {
               "code": 400, 
-              "message": "No JSON object."
+              "message": "Bad login type"
             }, 
             {
               "code": 400, 
-              "message": "User ID must only contain characters which do not require url encoding."
-            },
-            {
-              "code": 400, 
-              "message": "User ID already taken."
+              "message": "Missing JSON keys"
             }
           ], 
-          "summary": "Register with the home server.", 
-          "type": "RegistrationResponse"
+          "summary": "Submit a registration action.", 
+          "type": "RegistrationResult"
         }
       ], 
       "path": "/register"
     }
   ], 
-  "basePath": "http://localhost:8080/_matrix/client/api/v1", 
+  "basePath": "http://localhost:8008/_matrix/client/api/v1", 
   "consumes": [
     "application/json"
   ], 
   "models": {
-    "RegistrationResponse": {
-      "id": "RegistrationResponse", 
+    "RegistrationFlows": {
+      "id": "RegistrationFlows",
       "properties": {
-        "access_token": {
-          "description": "The access token for this user.", 
-          "type": "string"
+        "flows": {
+          "description": "A list of valid registration flows.",
+          "type": "array",
+          "items": {
+            "$ref": "RegistrationInfo"
+          }
+        }
+      }
+    },
+    "RegistrationInfo": {
+      "id": "RegistrationInfo", 
+      "properties": {
+        "stages": {
+          "description": "Multi-stage registration only: An array of all the login types required to registration.", 
+          "items": {
+            "$ref": "string"
+          }, 
+          "type": "array"
         }, 
-        "user_id": {
-          "description": "The fully-qualified user ID.", 
+        "type": {
+          "description": "The first login type that must be used when logging in.", 
           "type": "string"
         }
       }
     }, 
-    "RegistrationRequest": {
-      "id": "RegistrationRequest", 
+    "RegistrationResult": {
+      "id": "RegistrationResult", 
       "properties": {
+        "access_token": {
+          "description": "The access token for this user's registration if this is the final stage of the registration process.", 
+          "type": "string"
+        },
         "user_id": {
-          "description": "The desired user ID. If not specified, a random user ID will be allocated.", 
-          "type": "string",
-          "required": false
+          "description": "The user's fully-qualified user ID.",
+          "type": "string"
+        }, 
+        "next": {
+          "description": "Multi-stage registration only: The next registration type to submit.", 
+          "type": "string"
+        },
+        "session": {
+          "description": "Multi-stage registration only: The session token to send when submitting the next registration type.",
+          "type": "string"
+        }
+      }
+    }, 
+    "RegistrationSubmission": {
+      "id": "RegistrationSubmission", 
+      "properties": {
+        "type": {
+          "description": "The type of registration being submitted.", 
+          "type": "string"
+        },
+        "session": {
+          "description": "Multi-stage registration only: The session token from an earlier registration stage.",
+          "type": "string"
+        },
+        "_registration_type_defined_keys_": {
+          "description": "Keys as defined by the specified registration type, e.g. \"user\", \"password\""
         }
       }
     }

docs/client-server/swagger_matrix/api-docs-rooms

@@ -1,7 +1,7 @@
 {
   "apiVersion": "1.0.0",
   "swaggerVersion": "1.2",
-  "basePath": "http://localhost:8080/_matrix/client/api/v1", 
+  "basePath": "http://localhost:8008/_matrix/client/api/v1", 
   "resourcePath": "/rooms",
   "produces": [
     "application/json"
@@ -180,6 +180,59 @@
         }
       ]
     },
+    {
+      "path": "/rooms/{roomId}/state/m.room.name",
+      "operations": [
+        {
+          "method": "PUT",
+          "summary": "Set the name of this room.",
+          "notes": "Set the name of this room.",
+          "type": "void",
+          "nickname": "set_room_name",
+          "consumes": [
+            "application/json"
+          ],
+          "parameters": [
+            {
+              "name": "body",
+              "description": "The name contents",
+              "required": true,
+              "type": "RoomName",
+              "paramType": "body"
+            },
+            {
+              "name": "roomId",
+              "description": "The room to set the name of.",
+              "required": true,
+              "type": "string",
+              "paramType": "path"
+            }
+          ]
+        },
+        {
+          "method": "GET",
+          "summary": "Get the room's name.",
+          "notes": "",
+          "type": "RoomName",
+          "nickname": "get_room_name",
+          "parameters": [
+            {
+              "name": "roomId",
+              "description": "The room to get the name of.",
+              "required": true,
+              "type": "string",
+              "paramType": "path"
+            }
+          ],
+          "responseMessages": [
+            {
+              "code": 404,
+              "message": "Name not found."
+            }
+          ]
+        }
+      ]
+    },
     {
       "path": "/rooms/{roomId}/send/m.room.message.feedback",
       "operations": [
@@ -267,6 +320,12 @@
               "required": true,
               "type": "string",
               "paramType": "path"
+            },
+            {
+              "name": "body",
+              "required": true,
+              "type": "JoinRequest",
+              "paramType": "body"
             }
           ]
         }  
@@ -291,6 +350,12 @@
               "required": true,
               "type": "string",
               "paramType": "path"
+            },
+            {
+              "name": "body",
+              "required": true,
+              "type": "LeaveRequest",
+              "paramType": "body"
             }
           ]
         }  
@@ -424,10 +489,10 @@
       "path": "/join/{roomAliasOrId}",
       "operations": [
         {
-          "method": "PUT",
+          "method": "POST",
           "summary": "Join a room via a room alias or room ID.",
           "notes": "Join a room via a room alias or room ID.",
-          "type": "RoomInfo",
+          "type": "JoinRoomInfo",
           "nickname": "join",
           "consumes": [
             "application/json"
@@ -574,7 +639,7 @@
         {
           "method": "GET",
           "summary": "Get a list of all the current state events for this room.",
-          "notes": "Get a list of all the current state events for this room.",
+          "notes": "NOT YET IMPLEMENTED.",
           "type": "array",
           "items": {
             "$ref": "Event"
@@ -598,7 +663,7 @@
         {
           "method": "GET",
           "summary": "Get all the current information for this room, including messages and state events.",
-          "notes": "Get all the current information for this room, including messages and state events.",
+          "notes": "NOT YET IMPLEMENTED.",
           "type": "InitialSyncRoomData",
           "nickname": "get_room_sync_data",
           "parameters": [
@@ -624,6 +689,15 @@
         }
       }
     },
+    "RoomName": {
+      "id": "RoomName",
+      "properties": {
+        "name": {
+          "type": "string",
+          "description": "The human-readable name for the room. Can contain spaces."
+        }
+      }
+    },
     "Message": {
       "id": "Message",
       "properties": {
@@ -640,6 +714,16 @@
     "Feedback": {
       "id": "Feedback",
       "properties": {
+        "target_event_id": {
+          "type": "string",
+          "description": "The event ID being acknowledged.",
+          "required": true
+        },
+        "type": {
+          "type": "string",
+          "description": "The type of feedback. Either 'delivered' or 'read'.",
+          "required": true
+        }
       }
     },
     "Member": {
@@ -652,7 +736,7 @@
             "invite",
             "join",
             "leave",
-            "knock"
+            "ban"
           ]
         }
       }
@@ -672,6 +756,16 @@
         }
       }
     },
+    "JoinRoomInfo": {
+      "id": "JoinRoomInfo",
+      "properties": {
+        "room_id": {
+          "type": "string",
+          "description": "The room ID joined, if joined via a room alias only.",
+          "required": true
+        }
+      }
+    },
     "RoomConfig": {
       "id": "RoomConfig",
       "properties": {
@@ -830,6 +924,14 @@
         }
       }
     },
+    "JoinRequest": {
+      "id": "JoinRequest",
+      "properties": {}
+    },
+    "LeaveRequest": {
+      "id": "LeaveRequest",
+      "properties": {}
+    },
     "BanRequest": {
       "id": "BanRequest",
       "properties": {

docs/freenode.txt

@@ -0,0 +1 @@
+NCjcRSEG

docs/human-id-rules.rst

@@ -0,0 +1,79 @@
+This document outlines the format for human-readable IDs within matrix.
+
+Overview
+--------
+UTF-8 is quickly becoming the standard character encoding set on the web. As
+such, Matrix requires that all strings MUST be encoded as UTF-8. However,
+using Unicode as the character set for human-readable IDs is troublesome. There
+are many different characters which appear identical to each other, but would
+identify different users. In addition, there are non-printable characters which
+cannot be rendered by the end-user. This opens up a security vulnerability with
+phishing/spoofing of IDs, commonly known as a homograph attack.
+
+Web browers encountered this problem when International Domain Names were
+introduced. A variety of checks were put in place in order to protect users. If
+an address failed the check, the raw punycode would be displayed to disambiguate
+the address. Similar checks are performed by home servers in Matrix. However, 
+Matrix does not use punycode representations, and so does not show raw punycode 
+on a failed check. Instead, home servers must outright reject these misleading 
+IDs.
+
+Types of human-readable IDs
+---------------------------
+There are two main human-readable IDs in question:
+
+- Room aliases
+- User IDs
+ 
+Room aliases look like ``#localpart:domain``. These aliases point to opaque
+non human-readable room IDs. These pointers can change, so there is already an
+issue present with the same ID pointing to a different destination at a later
+date.
+
+User IDs look like ``@localpart:domain``. These represent actual end-users, and
+unlike room aliases, there is no layer of indirection. This presents a much
+greater concern with homograph attacks. 
+
+Checks
+------
+- Similar to web browsers.
+- blacklisted chars (e.g. non-printable characters)
+- mix of language sets from 'preferred' language not allowed. 
+- Language sets from CLDR dataset.
+- Treated in segments (localpart, domain)
+- Additional restrictions for ease of processing IDs.
+   - Room alias localparts MUST NOT have ``#`` or ``:``.
+   - User ID localparts MUST NOT have ``@`` or ``:``.
+
+Rejecting
+---------
+- Home servers MUST reject room aliases which do not pass the check, both on 
+  GETs and PUTs.
+- Home servers MUST reject user ID localparts which do not pass the check, both
+  on creation and on events.
+- Any home server whose domain does not pass this check, MUST use their punycode
+  domain name instead of the IDN, to prevent other home servers rejecting you.
+- Error code is ``M_FAILED_HUMAN_ID_CHECK``. (generic enough for both failing 
+  due to homograph attacks, and failing due to including ``:`` s, etc)
+- Error message MAY go into further information about which characters were
+  rejected and why.
+- Error message SHOULD contain a ``failed_keys`` key which contains an array
+  of strings which represent the keys which failed the check e.g::
+  
+    failed_keys: [ user_id, room_alias ]
+  
+Other considerations
+--------------------
+- Basic security: Informational key on the event attached by HS to say "unsafe 
+  ID". Problem: clients can just ignore it, and since it will appear only very
+  rarely, easy to forget when implementing clients.
+- Moderate security: Requires client handshake. Forces clients to implement
+  a check, else they cannot communicate with the misleading ID. However, this is
+  extra overhead in both client implementations and round-trips.
+- High security: Outright rejection of the ID at the point of creation / 
+  receiving event. Point of creation rejection is preferable to avoid the ID
+  entering the system in the first place. However, malicious HSes can just allow
+  the ID. Hence, other home servers must reject them if they see them in events.
+  Client never sees the problem ID, provided the HS is correctly implemented.
+- High security decided; client doesn't need to worry about it, no additional
+  protocol complexity aside from rejection of an event.

docs/server-server/security-threat-model.rst

@@ -1,141 +0,0 @@
-Overview
-========
-
-Scope
------
-
-This document considers threats specific to the server to server federation 
-synapse protocol.
-
-
-Attacker
---------
-
-It is assumed that the attacker can see and manipulate all network traffic 
-between any of the servers and may be in control of one or more homeservers 
-participating in the federation protocol.
-
-Threat Model
-============
-
-Denial of Service
------------------
-
-The attacker could attempt to prevent delivery of messages to or from the 
-victim in order to:
-
-    * Disrupt service or marketing campaign of a commercial competitor.
-    * Censor a discussion or censor a participant in a discussion.
-    * Perform general vandalism.
-
-Threat: Resource Exhaustion
-~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-An attacker could cause the victims server to exhaust a particular resource 
-(e.g. open TCP connections, CPU, memory, disk storage)
-
-Threat: Unrecoverable Consistency Violations
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-An attacker could send messages which created an unrecoverable "split-brain"
-state in the cluster such that the victim's servers could no longer dervive a
-consistent view of the chatroom state.
-
-Threat: Bad History
-~~~~~~~~~~~~~~~~~~~
-
-An attacker could convince the victim to accept invalid messages which the 
-victim would then include in their view of the chatroom history. Other servers
-in the chatroom would reject the invalid messages and potentially reject the
-victims messages as well since they depended on the invalid messages.
-
-Threat: Block Network Traffic
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-An attacker could try to firewall traffic between the victim's server and some
-or all of the other servers in the chatroom.
-
-Threat: High Volume of Messages
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-An attacker could send large volumes of messages to a chatroom with the victim
-making the chatroom unusable.
-
-Threat: Banning users without necessary authorisation
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-An attacker could attempt to ban a user from a chatroom with the necessary
-authorisation.
-
-Spoofing
---------
-
-An attacker could try to send a message claiming to be from the victim without 
-the victim having sent the message in order to:
-
-    * Impersonate the victim while performing illict activity.
-    * Obtain privileges of the victim.
-
-Threat: Altering Message Contents
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-An attacker could try to alter the contents of an existing message from the 
-victim.
-
-Threat: Fake Message "origin" Field
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-An attacker could try to send a new message purporting to be from the victim
-with a phony "origin" field.
-
-Spamming
---------
-
-The attacker could try to send a high volume of solicicted or unsolicted 
-messages to the victim in order to:
-    
-    * Find victims for scams.
-    * Market unwanted products.
-
-Threat: Unsoliticted Messages
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-An attacker could try to send messages to victims who do not wish to receive 
-them.
-
-Threat: Abusive Messages
-~~~~~~~~~~~~~~~~~~~~~~~~
-
-An attacker could send abusive or threatening messages to the victim
-
-Spying
-------
-
-The attacker could try to access message contents or metadata for messages sent
-by the victim or to the victim that were not intended to reach the attacker in
-order to:
-
-    * Gain sensitive personal or commercial information.
-    * Impersonate the victim using credentials contained in the messages.
-      (e.g. password reset messages)
-    * Discover who the victim was talking to and when.
-
-Threat: Disclosure during Transmission
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-An attacker could try to expose the message contents or metadata during 
-transmission between the servers.
-
-Threat: Disclosure to Servers Outside Chatroom
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-An attacker could try to convince servers within a chatroom to send messages to
-a server it controls that was not authorised to be within the chatroom.
-
-Threat: Disclosure to Servers Within Chatroom
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-An attacker could take control of a server within a chatroom to expose message
-contents or metadata for messages in that room.
-
-

experiments/cursesio.py

@@ -1,4 +1,4 @@
-# Copyright 2014 matrix.org
+# Copyright 2014 OpenMarket Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.

experiments/test_messaging.py

@@ -1,5 +1,5 @@
 # -*- coding: utf-8 -*-
-# Copyright 2014 matrix.org
+# Copyright 2014 OpenMarket Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.

graph/graph.py

@@ -1,4 +1,4 @@
-# Copyright 2014 matrix.org
+# Copyright 2014 OpenMarket Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.

jsfiddles/create_room_send_msg/demo.html

@@ -1,5 +1,5 @@
 <div>
-    <p>This room creation / message sending demo requires a home server to be running on http://localhost:8080</p>
+    <p>This room creation / message sending demo requires a home server to be running on http://localhost:8008</p>
 </div>
 <form class="loginForm">
     <input type="text" id="userLogin" placeholder="Username"></input>

jsfiddles/create_room_send_msg/demo.js

@@ -10,7 +10,7 @@ $('.login').live('click', function() {
     var user = $("#userLogin").val();
     var password = $("#passwordLogin").val();
     $.ajax({
-        url: "http://localhost:8080/_matrix/client/api/v1/login",
+        url: "http://localhost:8008/_matrix/client/api/v1/login",
         type: "POST",
         contentType: "application/json; charset=utf-8",
         data: JSON.stringify({ user: user, password: password, type: "m.login.password" }),
@@ -19,13 +19,18 @@ $('.login').live('click', function() {
             showLoggedIn(data);
         },
         error: function(err) {
-            alert(JSON.stringify($.parseJSON(err.responseText)));  
+            var errMsg = "To try this, you need a home server running!";
+            var errJson = $.parseJSON(err.responseText);
+            if (errJson) {
+                errMsg = JSON.stringify(errJson);   
+            }
+            alert(errMsg);  
         }
     }); 
 });
 
 var getCurrentRoomList = function() {
-    var url = "http://localhost:8080/_matrix/client/api/v1/initialSync?access_token=" + accountInfo.access_token + "&limit=1";
+    var url = "http://localhost:8008/_matrix/client/api/v1/initialSync?access_token=" + accountInfo.access_token + "&limit=1";
     $.getJSON(url, function(data) {
         var rooms = data.rooms;
         for (var i=0; i<rooms.length; ++i) {
@@ -44,7 +49,7 @@ $('.createRoom').live('click', function() {
         data.room_alias_name = roomAlias;   
     }
     $.ajax({
-        url: "http://localhost:8080/_matrix/client/api/v1/createRoom?access_token="+accountInfo.access_token,
+        url: "http://localhost:8008/_matrix/client/api/v1/createRoom?access_token="+accountInfo.access_token,
         type: "POST",
         contentType: "application/json; charset=utf-8",
         data: JSON.stringify(data),
@@ -79,7 +84,7 @@ $('.sendMessage').live('click', function() {
         return;
     }
     
-    var url = "http://localhost:8080/_matrix/client/api/v1/rooms/$roomid/send/m.room.message?access_token=$token";
+    var url = "http://localhost:8008/_matrix/client/api/v1/rooms/$roomid/send/m.room.message?access_token=$token";
     url = url.replace("$token", accountInfo.access_token);
     url = url.replace("$roomid", encodeURIComponent(roomId));
     

jsfiddles/event_stream/demo.html

@@ -1,5 +1,5 @@
 <div>
-    <p>This event stream demo requires a home server to be running on http://localhost:8080</p>
+    <p>This event stream demo requires a home server to be running on http://localhost:8008</p>
 </div>
 <form class="loginForm">
     <input type="text" id="userLogin" placeholder="Username"></input>

jsfiddles/event_stream/demo.js

@@ -7,7 +7,7 @@ var eventStreamInfo = {
 var roomInfo = [];
 
 var longpollEventStream = function() {
-    var url = "http://localhost:8080/_matrix/client/api/v1/events?access_token=$token&from=$from";
+    var url = "http://localhost:8008/_matrix/client/api/v1/events?access_token=$token&from=$from";
     url = url.replace("$token", accountInfo.access_token);
     url = url.replace("$from", eventStreamInfo.from);
     
@@ -48,7 +48,7 @@ $('.login').live('click', function() {
     var user = $("#userLogin").val();
     var password = $("#passwordLogin").val();
     $.ajax({
-        url: "http://localhost:8080/_matrix/client/api/v1/login",
+        url: "http://localhost:8008/_matrix/client/api/v1/login",
         type: "POST",
         contentType: "application/json; charset=utf-8",
         data: JSON.stringify({ user: user, password: password, type: "m.login.password" }),
@@ -58,14 +58,19 @@ $('.login').live('click', function() {
             showLoggedIn(data);
         },
         error: function(err) {
-            alert(JSON.stringify($.parseJSON(err.responseText)));  
+            var errMsg = "To try this, you need a home server running!";
+            var errJson = $.parseJSON(err.responseText);
+            if (errJson) {
+                errMsg = JSON.stringify(errJson);   
+            }
+            alert(errMsg);  
         }
     }); 
 });
 
 var getCurrentRoomList = function() {
     $("#roomId").val("");
-    var url = "http://localhost:8080/_matrix/client/api/v1/initialSync?access_token=" + accountInfo.access_token + "&limit=1";
+    var url = "http://localhost:8008/_matrix/client/api/v1/initialSync?access_token=" + accountInfo.access_token + "&limit=1";
     $.getJSON(url, function(data) {
         var rooms = data.rooms;
         for (var i=0; i<rooms.length; ++i) {
@@ -98,7 +103,7 @@ var sendMessage = function(roomId) {
         return;
     }
     
-    var url = "http://localhost:8080/_matrix/client/api/v1/rooms/$roomid/send/m.room.message?access_token=$token";
+    var url = "http://localhost:8008/_matrix/client/api/v1/rooms/$roomid/send/m.room.message?access_token=$token";
     url = url.replace("$token", accountInfo.access_token);
     url = url.replace("$roomid", encodeURIComponent(roomId));
     

jsfiddles/example_app/demo.details

@@ -0,0 +1,7 @@
+ name: Example Matrix Client
+ description: Includes login, live event streaming, creating rooms, sending messages and viewing member lists.
+ authors:
+   - matrix.org
+ resources:
+   - http://matrix.org
+ normalize_css: no

jsfiddles/example_app/demo.html

@@ -1,5 +1,5 @@
 <div class="signUp">
-    <p>Matrix example application: Requires a local home server running at http://localhost:8080</p>
+    <p>Matrix example application: Requires a local home server running at http://localhost:8008</p>
     <form class="registrationForm">
         <p>No account? Register:</p>
         <input type="text" id="userReg" placeholder="Username"></input>

jsfiddles/example_app/demo.js

@@ -10,7 +10,7 @@ var viewingRoomId;
 
 // ************** Event Streaming **************
 var longpollEventStream = function() {
-    var url = "http://localhost:8080/_matrix/client/api/v1/events?access_token=$token&from=$from";
+    var url = "http://localhost:8008/_matrix/client/api/v1/events?access_token=$token&from=$from";
     url = url.replace("$token", accountInfo.access_token);
     url = url.replace("$from", eventStreamInfo.from);
 
@@ -89,7 +89,7 @@ $('.login').live('click', function() {
     var user = $("#userLogin").val();
     var password = $("#passwordLogin").val();
     $.ajax({
-        url: "http://localhost:8080/_matrix/client/api/v1/login",
+        url: "http://localhost:8008/_matrix/client/api/v1/login",
         type: "POST",
         contentType: "application/json; charset=utf-8",
         data: JSON.stringify({ user: user, password: password, type: "m.login.password" }),
@@ -107,7 +107,7 @@ $('.register').live('click', function() {
     var user = $("#userReg").val();
     var password = $("#passwordReg").val();
     $.ajax({
-        url: "http://localhost:8080/_matrix/client/api/v1/register",
+        url: "http://localhost:8008/_matrix/client/api/v1/register",
         type: "POST",
         contentType: "application/json; charset=utf-8",
         data: JSON.stringify({ user_id: user, password: password }),
@@ -134,7 +134,7 @@ $('.createRoom').live('click', function() {
         data.room_alias_name = roomAlias;   
     }
     $.ajax({
-        url: "http://localhost:8080/_matrix/client/api/v1/createRoom?access_token="+accountInfo.access_token,
+        url: "http://localhost:8008/_matrix/client/api/v1/createRoom?access_token="+accountInfo.access_token,
         type: "POST",
         contentType: "application/json; charset=utf-8",
         data: JSON.stringify(data),
@@ -155,7 +155,7 @@ $('.createRoom').live('click', function() {
 
 // ************** Getting current state **************
 var getCurrentRoomList = function() {
-    var url = "http://localhost:8080/_matrix/client/api/v1/initialSync?access_token=" + accountInfo.access_token + "&limit=1";
+    var url = "http://localhost:8008/_matrix/client/api/v1/initialSync?access_token=" + accountInfo.access_token + "&limit=1";
     $.getJSON(url, function(data) {
         var rooms = data.rooms;
         for (var i=0; i<rooms.length; ++i) {
@@ -181,7 +181,7 @@ var loadRoomContent = function(roomId) {
 
 var getMessages = function(roomId) {
     $("#messages").empty();
-    var url = "http://localhost:8080/_matrix/client/api/v1/rooms/" + 
+    var url = "http://localhost:8008/_matrix/client/api/v1/rooms/" + 
               encodeURIComponent(roomId) + "/messages?access_token=" + accountInfo.access_token + "&from=END&dir=b&limit=10";
     $.getJSON(url, function(data) {
         for (var i=data.chunk.length-1; i>=0; --i) {
@@ -193,7 +193,7 @@ var getMessages = function(roomId) {
 var getMemberList = function(roomId) {
     $("#members").empty();
     memberInfo = [];
-    var url = "http://localhost:8080/_matrix/client/api/v1/rooms/" + 
+    var url = "http://localhost:8008/_matrix/client/api/v1/rooms/" + 
               encodeURIComponent(roomId) + "/members?access_token=" + accountInfo.access_token;
     $.getJSON(url, function(data) {
         for (var i=0; i<data.chunk.length; ++i) {
@@ -216,7 +216,7 @@ $('.sendMessage').live('click', function() {
 var sendMessage = function(roomId, body) {
     var msgId = $.now();
     
-    var url = "http://localhost:8080/_matrix/client/api/v1/rooms/$roomid/send/m.room.message?access_token=$token";
+    var url = "http://localhost:8008/_matrix/client/api/v1/rooms/$roomid/send/m.room.message?access_token=$token";
     url = url.replace("$token", accountInfo.access_token);
     url = url.replace("$roomid", encodeURIComponent(roomId));
     
@@ -262,7 +262,7 @@ var setRooms = function(roomList) {
         var membership = $(this).find('td:eq(1)').text();
         if (membership !== "join") {
             console.log("Joining room " + roomId); 
-            var url = "http://localhost:8080/_matrix/client/api/v1/rooms/$roomid/join?access_token=$token";
+            var url = "http://localhost:8008/_matrix/client/api/v1/rooms/$roomid/join?access_token=$token";
             url = url.replace("$token", accountInfo.access_token);
             url = url.replace("$roomid", encodeURIComponent(roomId));
             $.ajax({
@@ -290,6 +290,9 @@ var addMessage = function(data) {
 
     var msg = data.content.body;
     if (data.type === "m.room.member") {
+        if (data.content.membership === undefined) {
+            return;
+        }
         if (data.content.membership === "invite") {
             msg = "<em>invited " + data.state_key + " to the room</em>";
         }
@@ -299,10 +302,13 @@ var addMessage = function(data) {
         else if (data.content.membership === "leave") {
             msg = "<em>left the room</em>";
         }
-        else {
-            msg = "<em>" + data.content.membership + "</em>";
+        else if (data.content.membership === "ban") {
+            msg = "<em>was banned from the room</em>";
         }
     }
+    if (msg === undefined) {
+        return;
+    }
 
     var row = "<tr>" +
               "<td>"+data.user_id+"</td>" +

jsfiddles/register_login/demo.html

@@ -1,5 +1,5 @@
 <div>
-    <p>This registration/login demo requires a home server to be running on http://localhost:8080</p>
+    <p>This registration/login demo requires a home server to be running on http://localhost:8008</p>
 </div>
 <form class="registrationForm">
     <input type="text" id="user" placeholder="Username"></input>

jsfiddles/register_login/demo.js

@@ -11,7 +11,7 @@ $('.register').live('click', function() {
     var user = $("#user").val();
     var password = $("#password").val();
     $.ajax({
-        url: "http://localhost:8080/_matrix/client/api/v1/register",
+        url: "http://localhost:8008/_matrix/client/api/v1/register",
         type: "POST",
         contentType: "application/json; charset=utf-8",
         data: JSON.stringify({ user_id: user, password: password }),
@@ -20,14 +20,19 @@ $('.register').live('click', function() {
             showLoggedIn(data);
         },
         error: function(err) {
-            alert(JSON.stringify($.parseJSON(err.responseText)));  
+            var errMsg = "To try this, you need a home server running!";
+            var errJson = $.parseJSON(err.responseText);
+            if (errJson) {
+                errMsg = JSON.stringify(errJson);   
+            }
+            alert(errMsg);   
         }
     });
 });
 
 var login = function(user, password) {
     $.ajax({
-        url: "http://localhost:8080/_matrix/client/api/v1/login",
+        url: "http://localhost:8008/_matrix/client/api/v1/login",
         type: "POST",
         contentType: "application/json; charset=utf-8",
         data: JSON.stringify({ user: user, password: password, type: "m.login.password" }),
@@ -36,7 +41,12 @@ var login = function(user, password) {
             showLoggedIn(data);
         },
         error: function(err) {
-            alert(JSON.stringify($.parseJSON(err.responseText)));  
+            var errMsg = "To try this, you need a home server running!";
+            var errJson = $.parseJSON(err.responseText);
+            if (errJson) {
+                errMsg = JSON.stringify(errJson);   
+            }
+            alert(errMsg);  
         }
     });  
 };
@@ -44,7 +54,7 @@ var login = function(user, password) {
 $('.login').live('click', function() {
     var user = $("#userLogin").val();
     var password = $("#passwordLogin").val();
-    $.getJSON("http://localhost:8080/_matrix/client/api/v1/login", function(data) {
+    $.getJSON("http://localhost:8008/_matrix/client/api/v1/login", function(data) {
         if (data.flows[0].type !== "m.login.password") {
             alert("I don't know how to login with this type: " + data.type);
             return;
@@ -60,7 +70,7 @@ $('.logout').live('click', function() {
 });
 
 $('.testToken').live('click', function() {
-    var url = "http://localhost:8080/_matrix/client/api/v1/initialSync?access_token=" + accountInfo.access_token + "&limit=1";
+    var url = "http://localhost:8008/_matrix/client/api/v1/initialSync?access_token=" + accountInfo.access_token + "&limit=1";
     $.getJSON(url, function(data) {
          $("#imSyncText").text(JSON.stringify(data, undefined, 2));
     }).fail(function(err) {

jsfiddles/room_memberships/demo.html

@@ -1,5 +1,5 @@
 <div>
-    <p>This room membership demo requires a home server to be running on http://localhost:8080</p>
+    <p>This room membership demo requires a home server to be running on http://localhost:8008</p>
 </div>
 <form class="loginForm">
     <input type="text" id="userLogin" placeholder="Username"></input>

jsfiddles/room_memberships/demo.js

@@ -18,7 +18,7 @@ $('.login').live('click', function() {
     var user = $("#userLogin").val();
     var password = $("#passwordLogin").val();
     $.ajax({
-        url: "http://localhost:8080/_matrix/client/api/v1/login",
+        url: "http://localhost:8008/_matrix/client/api/v1/login",
         type: "POST",
         contentType: "application/json; charset=utf-8",
         data: JSON.stringify({ user: user, password: password, type: "m.login.password" }),
@@ -28,7 +28,12 @@ $('.login').live('click', function() {
             showLoggedIn(data);
         },
         error: function(err) {
-            alert(JSON.stringify($.parseJSON(err.responseText)));  
+            var errMsg = "To try this, you need a home server running!";
+            var errJson = $.parseJSON(err.responseText);
+            if (errJson) {
+                errMsg = JSON.stringify(errJson);   
+            }
+            alert(errMsg); 
         }
     }); 
 });
@@ -39,7 +44,7 @@ var getCurrentRoomList = function() {
     // solution but that is out of scope of this fiddle.
     $("#rooms").find("tr:gt(0)").remove();
     
-    var url = "http://localhost:8080/_matrix/client/api/v1/initialSync?access_token=" + accountInfo.access_token + "&limit=1";
+    var url = "http://localhost:8008/_matrix/client/api/v1/initialSync?access_token=" + accountInfo.access_token + "&limit=1";
     $.getJSON(url, function(data) {
         var rooms = data.rooms;
         for (var i=0; i<rooms.length; ++i) {
@@ -53,7 +58,7 @@ var getCurrentRoomList = function() {
 $('.createRoom').live('click', function() {
     var data = {};
     $.ajax({
-        url: "http://localhost:8080/_matrix/client/api/v1/createRoom?access_token="+accountInfo.access_token,
+        url: "http://localhost:8008/_matrix/client/api/v1/createRoom?access_token="+accountInfo.access_token,
         type: "POST",
         contentType: "application/json; charset=utf-8",
         data: JSON.stringify(data),
@@ -87,7 +92,7 @@ $('.changeMembership').live('click', function() {
         return;
     }
     
-    var url = "http://localhost:8080/_matrix/client/api/v1/rooms/$roomid/$membership?access_token=$token";
+    var url = "http://localhost:8008/_matrix/client/api/v1/rooms/$roomid/$membership?access_token=$token";
     url = url.replace("$token", accountInfo.access_token);
     url = url.replace("$roomid", encodeURIComponent(roomId));
     url = url.replace("$membership", membership);
@@ -117,7 +122,7 @@ $('.changeMembership').live('click', function() {
 
 $('.joinAlias').live('click', function() {
     var roomAlias = $("#roomAlias").val();
-    var url = "http://localhost:8080/_matrix/client/api/v1/join/$roomalias?access_token=$token";
+    var url = "http://localhost:8008/_matrix/client/api/v1/join/$roomalias?access_token=$token";
     url = url.replace("$token", accountInfo.access_token);
     url = url.replace("$roomalias", encodeURIComponent(roomAlias));
     $.ajax({

scripts/basic.css

@@ -0,0 +1,510 @@
+/*
+ * basic.css
+ * ~~~~~~~~~
+ *
+ * Sphinx stylesheet -- basic theme.
+ *
+ * :copyright: Copyright 2007-2010 by the Sphinx team, see AUTHORS.
+ * :license: BSD, see LICENSE for details.
+ *
+ */
+
+/* -- main layout ----------------------------------------------------------- */
+
+div.clearer {
+    clear: both;
+}
+
+/* -- relbar ---------------------------------------------------------------- */
+
+div.related {
+    width: 100%;
+    font-size: 90%;
+}
+
+div.related h3 {
+    display: none;
+}
+
+div.related ul {
+    margin: 0;
+    padding: 0 0 0 10px;
+    list-style: none;
+}
+
+div.related li {
+    display: inline;
+}
+
+div.related li.right {
+    float: right;
+    margin-right: 5px;
+}
+
+/* -- sidebar --------------------------------------------------------------- */
+
+div.sphinxsidebarwrapper {
+    padding: 10px 5px 0 10px;
+}
+
+div.sphinxsidebar {
+    float: left;
+    width: 230px;
+    margin-left: -100%;
+    font-size: 90%;
+}
+
+div.sphinxsidebar ul {
+    list-style: none;
+}
+
+div.sphinxsidebar ul ul,
+div.sphinxsidebar ul.want-points {
+    margin-left: 20px;
+    list-style: square;
+}
+
+div.sphinxsidebar ul ul {
+    margin-top: 0;
+    margin-bottom: 0;
+}
+
+div.sphinxsidebar form {
+    margin-top: 10px;
+}
+
+div.sphinxsidebar input {
+    border: 1px solid #98dbcc;
+    font-family: sans-serif;
+    font-size: 1em;
+}
+
+img {
+    border: 0;
+}
+
+/* -- search page ----------------------------------------------------------- */
+
+ul.search {
+    margin: 10px 0 0 20px;
+    padding: 0;
+}
+
+ul.search li {
+    padding: 5px 0 5px 20px;
+    background-image: url(file.png);
+    background-repeat: no-repeat;
+    background-position: 0 7px;
+}
+
+ul.search li a {
+    font-weight: bold;
+}
+
+ul.search li div.context {
+    color: #888;
+    margin: 2px 0 0 30px;
+    text-align: left;
+}
+
+ul.keywordmatches li.goodmatch a {
+    font-weight: bold;
+}
+
+/* -- index page ------------------------------------------------------------ */
+
+table.contentstable {
+    width: 90%;
+}
+
+table.contentstable p.biglink {
+    line-height: 150%;
+}
+
+a.biglink {
+    font-size: 1.3em;
+}
+
+span.linkdescr {
+    font-style: italic;
+    padding-top: 5px;
+    font-size: 90%;
+}
+
+/* -- general index --------------------------------------------------------- */
+
+table.indextable {
+    width: 100%;
+}
+
+table.indextable td {
+    text-align: left;
+    vertical-align: top;
+}
+
+table.indextable dl, table.indextable dd {
+    margin-top: 0;
+    margin-bottom: 0;
+}
+
+table.indextable tr.pcap {
+    height: 10px;
+}
+
+table.indextable tr.cap {
+    margin-top: 10px;
+    background-color: #f2f2f2;
+}
+
+img.toggler {
+    margin-right: 3px;
+    margin-top: 3px;
+    cursor: pointer;
+}
+
+div.modindex-jumpbox {
+    border-top: 1px solid #ddd;
+    border-bottom: 1px solid #ddd;
+    margin: 1em 0 1em 0;
+    padding: 0.4em;
+}
+
+div.genindex-jumpbox {
+    border-top: 1px solid #ddd;
+    border-bottom: 1px solid #ddd;
+    margin: 1em 0 1em 0;
+    padding: 0.4em;
+}
+
+/* -- general body styles --------------------------------------------------- */
+
+a.headerlink {
+    visibility: hidden;
+}
+
+h1:hover > a.headerlink,
+h2:hover > a.headerlink,
+h3:hover > a.headerlink,
+h4:hover > a.headerlink,
+h5:hover > a.headerlink,
+h6:hover > a.headerlink,
+dt:hover > a.headerlink {
+    visibility: visible;
+}
+
+div.document p.caption {
+    text-align: inherit;
+}
+
+div.document td {
+    text-align: left;
+}
+
+.field-list ul {
+    padding-left: 1em;
+}
+
+.first {
+    margin-top: 0 !important;
+}
+
+p.rubric {
+    margin-top: 30px;
+    font-weight: bold;
+}
+
+.align-left {
+    text-align: left;
+}
+
+.align-center {
+    clear: both;
+    text-align: center;
+}
+
+.align-right {
+    text-align: right;
+}
+
+/* -- sidebars -------------------------------------------------------------- */
+
+div.sidebar {
+    margin: 0 0 0.5em 1em;
+    border: 1px solid #ddb;
+    padding: 7px 7px 0 7px;
+    background-color: #ffe;
+    width: 40%;
+    float: right;
+}
+
+p.sidebar-title {
+    font-weight: bold;
+}
+
+/* -- topics ---------------------------------------------------------------- */
+
+div.topic {
+    border: 1px solid #ccc;
+    padding: 7px 7px 0 7px;
+    margin: 10px 0 10px 0;
+}
+
+p.topic-title {
+    font-size: 1.1em;
+    font-weight: bold;
+    margin-top: 10px;
+}
+
+/* -- admonitions ----------------------------------------------------------- */
+
+div.admonition {
+    margin-top: 10px;
+    margin-bottom: 10px;
+    padding: 7px;
+}
+
+div.admonition dt {
+    font-weight: bold;
+}
+
+div.admonition dl {
+    margin-bottom: 0;
+}
+
+p.admonition-title {
+    margin: 0px 10px 5px 0px;
+    font-weight: bold;
+}
+
+div.document p.centered {
+    text-align: center;
+    margin-top: 25px;
+}
+
+/* -- tables ---------------------------------------------------------------- */
+
+table.docutils {
+    border: 0;
+    border-collapse: collapse;
+}
+
+table.docutils td, table.docutils th {
+    padding: 1px 8px 1px 5px;
+    border-top: 0;
+    border-left: 0;
+    border-right: 0;
+    border-bottom: 1px solid #aaa;
+}
+
+table.field-list td, table.field-list th {
+    border: 0 !important;
+}
+
+table.footnote td, table.footnote th {
+    border: 0 !important;
+}
+
+th {
+    text-align: left;
+    padding-right: 5px;
+}
+
+table.citation {
+    border-left: solid 1px gray;
+    margin-left: 1px;
+}
+
+table.citation td {
+    border-bottom: none;
+}
+
+/* -- other body styles ----------------------------------------------------- */
+
+ol.arabic {
+    list-style: decimal;
+}
+
+ol.loweralpha {
+    list-style: lower-alpha;
+}
+
+ol.upperalpha {
+    list-style: upper-alpha;
+}
+
+ol.lowerroman {
+    list-style: lower-roman;
+}
+
+ol.upperroman {
+    list-style: upper-roman;
+}
+
+dl {
+    margin-bottom: 15px;
+}
+
+dd p {
+    margin-top: 0px;
+}
+
+dd ul, dd table {
+    margin-bottom: 10px;
+}
+
+dd {
+    margin-top: 3px;
+    margin-bottom: 10px;
+    margin-left: 30px;
+}
+
+dt:target, .highlighted {
+    background-color: #fbe54e;
+}
+
+dl.glossary dt {
+    font-weight: bold;
+    font-size: 1.1em;
+}
+
+.field-list ul {
+    margin: 0;
+    padding-left: 1em;
+}
+
+.field-list p {
+    margin: 0;
+}
+
+.refcount {
+    color: #060;
+}
+
+.optional {
+    font-size: 1.3em;
+}
+
+.versionmodified {
+    font-style: italic;
+}
+
+.system-message {
+    background-color: #fda;
+    padding: 5px;
+    border: 3px solid red;
+}
+
+.footnote:target  {
+    background-color: #ffa
+}
+
+.line-block {
+    display: block;
+    margin-top: 1em;
+    margin-bottom: 1em;
+}
+
+.line-block .line-block {
+    margin-top: 0;
+    margin-bottom: 0;
+    margin-left: 1.5em;
+}
+
+.guilabel, .menuselection {
+    font-family: sans-serif;
+}
+
+.accelerator {
+    text-decoration: underline;
+}
+
+.classifier {
+    font-style: oblique;
+}
+
+/* -- code displays --------------------------------------------------------- */
+
+pre {
+    overflow: auto;
+}
+
+td.linenos pre {
+    padding: 5px 0px;
+    border: 0;
+    background-color: transparent;
+    color: #aaa;
+}
+
+table.highlighttable {
+    margin-left: 0.5em;
+}
+
+table.highlighttable td {
+    padding: 0 0.5em 0 0.5em;
+}
+
+tt.descname {
+    background-color: transparent;
+    font-weight: bold;
+    font-size: 1.2em;
+}
+
+tt.descclassname {
+    background-color: transparent;
+}
+
+tt.xref, a tt {
+    background-color: transparent;
+    font-weight: bold;
+}
+
+h1 tt, h2 tt, h3 tt, h4 tt, h5 tt, h6 tt {
+    background-color: transparent;
+}
+
+.viewcode-link {
+    float: right;
+}
+
+.viewcode-back {
+    float: right;
+    font-family: sans-serif;
+}
+
+div.viewcode-block:target {
+    margin: -1px -10px;
+    padding: 0 10px;
+}
+
+/* -- math display ---------------------------------------------------------- */
+
+img.math {
+    vertical-align: middle;
+}
+
+div.document div.math p {
+    text-align: center;
+}
+
+span.eqno {
+    float: right;
+}
+
+/* -- printout stylesheet --------------------------------------------------- */
+
+@media print {
+    div.document,
+    div.documentwrapper,
+    div.bodywrapper {
+        margin: 0 !important;
+        width: 100%;
+    }
+
+    div.sphinxsidebar,
+    div.related,
+    div.footer,
+    #top-link {
+        display: none;
+    }
+}
+

scripts/copyrighter.pl

@@ -1,5 +1,5 @@
 #!/usr/bin/perl -pi
-# Copyright 2014 matrix.org
+# Copyright 2014 OpenMarket Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -14,7 +14,7 @@
 # limitations under the License.
 
 $copyright = <<EOT;
-# Copyright 2014 matrix.org
+# Copyright 2014 OpenMarket Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.

scripts/gendoc.sh

@@ -0,0 +1,14 @@
+#!/bin/bash
+
+MATRIXDOTORG=$HOME/workspace/matrix.org
+
+rst2html-2.7.py --stylesheet=basic.css,nature.css ../docs/specification.rst > $MATRIXDOTORG/docs/spec/index.html
+rst2html-2.7.py --stylesheet=basic.css,nature.css ../docs/client-server/howto.rst > $MATRIXDOTORG/docs/howtos/client-server.html
+
+perl -pi -e 's#<head>#<head><link rel="stylesheet" href="/site.css">#' $MATRIXDOTORG/docs/spec/index.html $MATRIXDOTORG/docs/howtos/client-server.html
+
+perl -pi -e 's#<body>#<body><div id="header"><div id="headerContent">&nbsp;</div></div><div id="page"><div id="wrapper"><div style="text-align: center; padding: 40px;"><a href="/"><img src="/matrix.png" width="305" height="130" alt="[matrix]"/></a></div>#' $MATRIXDOTORG/docs/spec/index.html $MATRIXDOTORG/docs/howtos/client-server.html
+
+perl -pi -e 's#</body>#</div></div><div id="footer"><div id="footerContent">&copy 2014 Matrix.org</div></div></body>#' $MATRIXDOTORG/docs/spec/index.html $MATRIXDOTORG/docs/howtos/client-server.html
+
+scp -r $MATRIXDOTORG/docs matrix@ldc-prd-matrix-001:/sites/matrix

scripts/nature.css

@@ -0,0 +1,270 @@
+/*
+ * nature.css_t
+ * ~~~~~~~~~~~~
+ *
+ * Sphinx stylesheet -- nature theme.
+ *
+ * :copyright: Copyright 2007-2010 by the Sphinx team, see AUTHORS.
+ * :license: BSD, see LICENSE for details.
+ *
+ */
+ 
+/* -- page layout ----------------------------------------------------------- */
+ 
+body {
+    font-family: Arial, sans-serif;
+    font-size: 100%;
+    /*background-color: #111;*/
+    color: #555;
+    margin: 0;
+    padding: 0;
+}
+
+div.documentwrapper {
+    float: left;
+    width: 100%;
+}
+
+div.bodywrapper {
+    margin: 0 0 0 230px;
+}
+
+hr {
+    border: 1px solid #B1B4B6;
+}
+ 
+/*
+div.document {
+    background-color: #eee;
+}
+*/
+ 
+div.document {
+    background-color: #ffffff;
+    color: #3E4349;
+    padding: 0 30px 30px 30px;
+    font-size: 0.9em;
+}
+ 
+div.footer {
+    color: #555;
+    width: 100%;
+    padding: 13px 0;
+    text-align: center;
+    font-size: 75%;
+}
+ 
+div.footer a {
+    color: #444;
+    text-decoration: underline;
+}
+ 
+div.related {
+    background-color: #6BA81E;
+    line-height: 32px;
+    color: #fff;
+    text-shadow: 0px 1px 0 #444;
+    font-size: 0.9em;
+}
+ 
+div.related a {
+    color: #E2F3CC;
+}
+ 
+div.sphinxsidebar {
+    font-size: 0.75em;
+    line-height: 1.5em;
+}
+
+div.sphinxsidebarwrapper{
+    padding: 20px 0;
+}
+ 
+div.sphinxsidebar h3,
+div.sphinxsidebar h4 {
+    font-family: Arial, sans-serif;
+    color: #222;
+    font-size: 1.2em;
+    font-weight: normal;
+    margin: 0;
+    padding: 5px 10px;
+    background-color: #ddd;
+    text-shadow: 1px 1px 0 white
+}
+
+div.sphinxsidebar h4{
+    font-size: 1.1em;
+}
+ 
+div.sphinxsidebar h3 a {
+    color: #444;
+}
+ 
+ 
+div.sphinxsidebar p {
+    color: #888;
+    padding: 5px 20px;
+}
+ 
+div.sphinxsidebar p.topless {
+}
+ 
+div.sphinxsidebar ul {
+    margin: 10px 20px;
+    padding: 0;
+    color: #000;
+}
+ 
+div.sphinxsidebar a {
+    color: #444;
+}
+ 
+div.sphinxsidebar input {
+    border: 1px solid #ccc;
+    font-family: sans-serif;
+    font-size: 1em;
+}
+
+div.sphinxsidebar input[type=text]{
+    margin-left: 20px;
+}
+ 
+/* -- body styles ----------------------------------------------------------- */
+ 
+a {
+    color: #005B81;
+    text-decoration: none;
+}
+ 
+a:hover {
+    color: #E32E00;
+    text-decoration: underline;
+}
+ 
+div.document h1,
+div.document h2,
+div.document h3,
+div.document h4,
+div.document h5,
+div.document h6 {
+    font-family: Arial, sans-serif;
+    background-color: #BED4EB;
+    font-weight: normal;
+    color: #212224;
+    margin: 30px 0px 10px 0px;
+    padding: 5px 0 5px 10px;
+    text-shadow: 0px 1px 0 white
+}
+ 
+div.document h1 { border-top: 20px solid white; margin-top: 0; font-size: 200%; }
+div.document h2 { font-size: 150%; background-color: #C8D5E3; }
+div.document h3 { font-size: 120%; background-color: #D8DEE3; }
+div.document h4 { font-size: 110%; background-color: #D8DEE3; }
+div.document h5 { font-size: 100%; background-color: #D8DEE3; }
+div.document h6 { font-size: 100%; background-color: #D8DEE3; }
+ 
+a.headerlink {
+    color: #c60f0f;
+    font-size: 0.8em;
+    padding: 0 4px 0 4px;
+    text-decoration: none;
+}
+ 
+a.headerlink:hover {
+    background-color: #c60f0f;
+    color: white;
+}
+ 
+div.document p, div.document dd, div.document li {
+    line-height: 1.5em;
+}
+ 
+div.admonition p.admonition-title + p {
+    display: inline;
+}
+
+div.highlight{
+    background-color: white;
+}
+
+div.note {
+    background-color: #eee;
+    border: 1px solid #ccc;
+}
+ 
+div.seealso {
+    background-color: #ffc;
+    border: 1px solid #ff6;
+}
+ 
+div.topic {
+    background-color: #eee;
+}
+ 
+div.warning {
+    background-color: #ffe4e4;
+    border: 1px solid #f66;
+}
+ 
+p.admonition-title {
+    display: inline;
+}
+ 
+p.admonition-title:after {
+    content: ":";
+}
+ 
+pre {
+    padding: 10px;
+    background-color: White;
+    color: #222;
+    line-height: 1.2em;
+    border: 1px solid #C6C9CB;
+    font-size: 1.1em;
+    margin: 1.5em 0 1.5em 0;
+    -webkit-box-shadow: 1px 1px 1px #d8d8d8;
+    -moz-box-shadow: 1px 1px 1px #d8d8d8;
+}
+ 
+tt {
+    background-color: #ecf0f3;
+    color: #222;
+    /* padding: 1px 2px; */
+    font-size: 1.1em;
+    font-family: monospace;
+}
+
+.viewcode-back {
+    font-family: Arial, sans-serif;
+}
+
+div.viewcode-block:target {
+    background-color: #f4debf;
+    border-top: 1px solid #ac9;
+    border-bottom: 1px solid #ac9;
+}
+
+p {
+	margin: 0;
+}
+
+ul li dd {
+	margin-top: 0;
+}
+
+ul li dl {
+	margin-bottom: 0;
+}
+
+li dl dd {
+	margin-bottom: 0;
+}
+
+dd ul {
+	padding-left: 0;
+}
+
+li dd ul {
+	margin-bottom: 0;
+}
+

setup.py

@@ -1,6 +1,6 @@
 #!/usr/bin/env python
 
-# Copyright 2014 matrix.org
+# Copyright 2014 OpenMarket Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.

synapse/__init__.py

@@ -1,5 +1,5 @@
 # -*- coding: utf-8 -*-
-# Copyright 2014 matrix.org
+# Copyright 2014 OpenMarket Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -16,4 +16,4 @@
 """ This is a reference implementation of a synapse home server.
 """
 
-__version__ = "0.2.0"
+__version__ = "0.3.0"

synapse/api/__init__.py

@@ -1,5 +1,5 @@
 # -*- coding: utf-8 -*-
-# Copyright 2014 matrix.org
+# Copyright 2014 OpenMarket Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.

synapse/api/auth.py

@@ -1,5 +1,5 @@
 # -*- coding: utf-8 -*-
-# Copyright 2014 matrix.org
+# Copyright 2014 OpenMarket Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -18,8 +18,8 @@
 from twisted.internet import defer
 
 from synapse.api.constants import Membership, JoinRules
-from synapse.api.errors import AuthError, StoreError, Codes
-from synapse.api.events.room import RoomMemberEvent
+from synapse.api.errors import AuthError, StoreError, Codes, SynapseError
+from synapse.api.events.room import RoomMemberEvent, RoomPowerLevelsEvent
 from synapse.util.logutils import log_function
 
 import logging
@@ -67,6 +67,9 @@ class Auth(object):
                 else:
                     yield self._can_send_event(event)
 
+                if event.type == RoomPowerLevelsEvent.TYPE:
+                    yield self._check_power_levels(event)
+
                 defer.returnValue(True)
             else:
                 raise AuthError(500, "Unknown event: %s" % event)
@@ -172,7 +175,7 @@ class Auth(object):
                 if kick_level:
                     kick_level = int(kick_level)
                 else:
-                    kick_level = 5
+                    kick_level = 50
 
                 if user_level < kick_level:
                     raise AuthError(
@@ -189,7 +192,7 @@ class Auth(object):
             if ban_level:
                 ban_level = int(ban_level)
             else:
-                ban_level = 5  # FIXME (erikj): What should we do here?
+                ban_level = 50  # FIXME (erikj): What should we do here?
 
             if user_level < ban_level:
                 raise AuthError(403, "You don't have permission to ban")
@@ -305,7 +308,9 @@ class Auth(object):
         else:
             user_level = 0
 
-        logger.debug("Checking power level for %s, %s", event.user_id, user_level)
+        logger.debug(
+            "Checking power level for %s, %s", event.user_id, user_level
+        )
         if current_state and hasattr(current_state, "required_power_level"):
             req = current_state.required_power_level
 
@@ -315,3 +320,101 @@ class Auth(object):
                     403,
                     "You don't have permission to change that state"
                 )
+
+    @defer.inlineCallbacks
+    def _check_power_levels(self, event):
+        for k, v in event.content.items():
+            if k == "default":
+                continue
+
+            # FIXME (erikj): We don't want hsob_Ts in content.
+            if k == "hsob_ts":
+                continue
+
+            try:
+                self.hs.parse_userid(k)
+            except:
+                raise SynapseError(400, "Not a valid user_id: %s" % (k,))
+
+            try:
+                int(v)
+            except:
+                raise SynapseError(400, "Not a valid power level: %s" % (v,))
+
+        current_state = yield self.store.get_current_state(
+            event.room_id,
+            event.type,
+            event.state_key,
+        )
+
+        if not current_state:
+            return
+        else:
+            current_state = current_state[0]
+
+        user_level = yield self.store.get_power_level(
+            event.room_id,
+            event.user_id,
+        )
+
+        if user_level:
+            user_level = int(user_level)
+        else:
+            user_level = 0
+
+        old_list = current_state.content
+
+        # FIXME (erikj)
+        old_people = {k: v for k, v in old_list.items() if k.startswith("@")}
+        new_people = {
+            k: v for k, v in event.content.items()
+            if k.startswith("@")
+        }
+
+        removed = set(old_people.keys()) - set(new_people.keys())
+        added = set(old_people.keys()) - set(new_people.keys())
+        same = set(old_people.keys()) & set(new_people.keys())
+
+        for r in removed:
+            if int(old_list.content[r]) > user_level:
+                raise AuthError(
+                    403,
+                    "You don't have permission to remove user: %s" % (r, )
+                )
+
+        for n in added:
+            if int(event.content[n]) > user_level:
+                raise AuthError(
+                    403,
+                    "You don't have permission to add ops level greater "
+                    "than your own"
+                )
+
+        for s in same:
+            if int(event.content[s]) != int(old_list[s]):
+                if int(event.content[s]) > user_level:
+                    raise AuthError(
+                        403,
+                        "You don't have permission to add ops level greater "
+                        "than your own"
+                    )
+
+        if "default" in old_list:
+            old_default = int(old_list["default"])
+
+            if old_default > user_level:
+                raise AuthError(
+                    403,
+                    "You don't have permission to add ops level greater than "
+                    "your own"
+                )
+
+            if "default" in event.content:
+                new_default = int(event.content["default"])
+
+                if new_default > user_level:
+                    raise AuthError(
+                        403,
+                        "You don't have permission to add ops level greater "
+                        "than your own"
+                    )

synapse/api/constants.py

@@ -1,5 +1,5 @@
 # -*- coding: utf-8 -*-
-# Copyright 2014 matrix.org
+# Copyright 2014 OpenMarket Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -50,3 +50,12 @@ class JoinRules(object):
     KNOCK = u"knock"
     INVITE = u"invite"
     PRIVATE = u"private"
+
+
+class LoginType(object):
+    PASSWORD = u"m.login.password"
+    OAUTH = u"m.login.oauth2"
+    EMAIL_CODE = u"m.login.email.code"
+    EMAIL_URL = u"m.login.email.url"
+    EMAIL_IDENTITY = u"m.login.email.identity"
+    RECAPTCHA = u"m.login.recaptcha"

synapse/api/errors.py

@@ -1,5 +1,5 @@
 # -*- coding: utf-8 -*-
-# Copyright 2014 matrix.org
+# Copyright 2014 OpenMarket Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -28,6 +28,9 @@ class Codes(object):
     UNKNOWN = "M_UNKNOWN"
     NOT_FOUND = "M_NOT_FOUND"
     UNKNOWN_TOKEN = "M_UNKNOWN_TOKEN"
+    LIMIT_EXCEEDED = "M_LIMIT_EXCEEDED"
+    CAPTCHA_NEEDED = "M_CAPTCHA_NEEDED"
+    CAPTCHA_INVALID = "M_CAPTCHA_INVALID"
 
 
 class CodeMessageException(Exception):
@@ -38,11 +41,15 @@ class CodeMessageException(Exception):
         super(CodeMessageException, self).__init__("%d: %s" % (code, msg))
         self.code = code
         self.msg = msg
+        self.response_code_message = None
+
+    def error_dict(self):
+        return cs_error(self.msg)
 
 
 class SynapseError(CodeMessageException):
     """A base error which can be caught for all synapse events."""
-    def __init__(self, code, msg, errcode=""):
+    def __init__(self, code, msg, errcode=Codes.UNKNOWN):
         """Constructs a synapse error.
 
         Args:
@@ -53,6 +60,11 @@ class SynapseError(CodeMessageException):
         super(SynapseError, self).__init__(code, msg)
         self.errcode = errcode
 
+    def error_dict(self):
+        return cs_error(
+            self.msg,
+            self.errcode,
+        )
 
 class RoomError(SynapseError):
     """An error raised when a room event fails."""
@@ -91,13 +103,39 @@ class StoreError(SynapseError):
     pass
 
 
-def cs_exception(exception):
-    if isinstance(exception, SynapseError):
+class InvalidCaptchaError(SynapseError):
+    def __init__(self, code=400, msg="Invalid captcha.", error_url=None,
+                 errcode=Codes.CAPTCHA_INVALID):
+        super(InvalidCaptchaError, self).__init__(code, msg, errcode)
+        self.error_url = error_url
+
+    def error_dict(self):
         return cs_error(
-            exception.msg,
-            Codes.UNKNOWN if not exception.errcode else exception.errcode)
-    elif isinstance(exception, CodeMessageException):
-        return cs_error(exception.msg)
+            self.msg,
+            self.errcode,
+            error_url=self.error_url,
+        )
+
+class LimitExceededError(SynapseError):
+    """A client has sent too many requests and is being throttled.
+    """
+    def __init__(self, code=429, msg="Too Many Requests", retry_after_ms=None,
+                 errcode=Codes.LIMIT_EXCEEDED):
+        super(LimitExceededError, self).__init__(code, msg, errcode)
+        self.retry_after_ms = retry_after_ms
+        self.response_code_message = "Too Many Requests"
+
+    def error_dict(self):
+        return cs_error(
+            self.msg,
+            self.errcode,
+            retry_after_ms=self.retry_after_ms,
+        )
+
+
+def cs_exception(exception):
+    if isinstance(exception, CodeMessageException):
+        return exception.error_dict()
     else:
         logging.error("Unknown exception type: %s", type(exception))
 

synapse/api/events/__init__.py

@@ -1,5 +1,5 @@
 # -*- coding: utf-8 -*-
-# Copyright 2014 matrix.org
+# Copyright 2014 OpenMarket Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -17,6 +17,19 @@ from synapse.api.errors import SynapseError, Codes
 from synapse.util.jsonobject import JsonEncodedObject
 
 
+def serialize_event(hs, e):
+    # FIXME(erikj): To handle the case of presence events and the like
+    if not isinstance(e, SynapseEvent):
+        return e
+
+    d = e.get_dict()
+    if "age_ts" in d:
+        d["age"] = int(hs.get_clock().time_msec()) - d["age_ts"]
+        del d["age_ts"]
+
+    return d
+
+
 class SynapseEvent(JsonEncodedObject):
 
     """Base class for Synapse events. These are JSON objects which must abide
@@ -43,6 +56,8 @@ class SynapseEvent(JsonEncodedObject):
         "content",  # HTTP body, JSON
         "state_key",
         "required_power_level",
+        "age_ts",
+        "prev_content",
     ]
 
     internal_keys = [
@@ -157,7 +172,8 @@ class SynapseEvent(JsonEncodedObject):
 
 
 class SynapseStateEvent(SynapseEvent):
-     def __init__(self, **kwargs):
+
+    def __init__(self, **kwargs):
         if "state_key" not in kwargs:
             kwargs["state_key"] = ""
         super(SynapseStateEvent, self).__init__(**kwargs)

synapse/api/events/factory.py

@@ -1,5 +1,5 @@
 # -*- coding: utf-8 -*-
-# Copyright 2014 matrix.org
+# Copyright 2014 OpenMarket Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -47,15 +47,26 @@ class EventFactory(object):
             self._event_list[event_class.TYPE] = event_class
 
         self.clock = hs.get_clock()
+        self.hs = hs
 
     def create_event(self, etype=None, **kwargs):
         kwargs["type"] = etype
         if "event_id" not in kwargs:
-            kwargs["event_id"] = random_string(10)
+            kwargs["event_id"] = "%s@%s" % (
+                random_string(10), self.hs.hostname
+            )
 
         if "ts" not in kwargs:
             kwargs["ts"] = int(self.clock.time_msec())
 
+        # The "age" key is a delta timestamp that should be converted into an
+        # absolute timestamp the minute we see it.
+        if "age" in kwargs:
+            kwargs["age_ts"] = int(self.clock.time_msec()) - int(kwargs["age"])
+            del kwargs["age"]
+        elif "age_ts" not in kwargs:
+            kwargs["age_ts"] = int(self.clock.time_msec())
+
         if etype in self._event_list:
             handler = self._event_list[etype]
         else:

synapse/api/events/room.py

@@ -1,5 +1,5 @@
 # -*- coding: utf-8 -*-
-# Copyright 2014 matrix.org
+# Copyright 2014 OpenMarket Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -103,8 +103,7 @@ class FeedbackEvent(SynapseEvent):
     def get_content_template(self):
         return {
             "type": u"string",
-            "target_event_id": u"string",
-            "msg_sender_id": u"string"
+            "target_event_id": u"string"
         }
 
 
@@ -174,3 +173,10 @@ class RoomOpsPowerLevelsEvent(SynapseStateEvent):
 
     def get_content_template(self):
         return {}
+
+
+class RoomAliasesEvent(SynapseStateEvent):
+    TYPE = "m.room.aliases"
+
+    def get_content_template(self):
+        return {}

synapse/api/ratelimiting.py

@@ -0,0 +1,79 @@
+# Copyright 2014 OpenMarket Ltd
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import collections
+
+
+class Ratelimiter(object):
+    """
+    Ratelimit message sending by user.
+    """
+
+    def __init__(self):
+        self.message_counts = collections.OrderedDict()
+
+    def send_message(self, user_id, time_now_s, msg_rate_hz, burst_count):
+        """Can the user send a message?
+        Args:
+            user_id: The user sending a message.
+            time_now_s: The time now.
+            msg_rate_hz: The long term number of messages a user can send in a
+                second.
+            burst_count: How many messages the user can send before being
+                limited.
+        Returns:
+            A pair of a bool indicating if they can send a message now and a
+                time in seconds of when they can next send a message.
+        """
+        self.prune_message_counts(time_now_s)
+        message_count, time_start, _ignored = self.message_counts.pop(
+            user_id, (0., time_now_s, None),
+        )
+        time_delta = time_now_s - time_start
+        sent_count = message_count - time_delta * msg_rate_hz
+        if sent_count < 0:
+            allowed = True
+            time_start = time_now_s
+            message_count = 1.
+        elif sent_count > burst_count - 1.:
+            allowed = False
+        else:
+            allowed = True
+            message_count += 1
+
+        self.message_counts[user_id] = (
+            message_count, time_start, msg_rate_hz
+        )
+
+        if msg_rate_hz > 0:
+            time_allowed = (
+                time_start + (message_count - burst_count + 1) / msg_rate_hz
+            )
+            if time_allowed < time_now_s:
+                time_allowed = time_now_s
+        else:
+            time_allowed = -1
+
+        return allowed, time_allowed
+
+    def prune_message_counts(self, time_now_s):
+        for user_id in self.message_counts.keys():
+            message_count, time_start, msg_rate_hz = (
+                self.message_counts[user_id]
+            )
+            time_delta = time_now_s - time_start
+            if message_count - time_delta * msg_rate_hz > 0:
+                break
+            else:
+                del self.message_counts[user_id]

synapse/api/urls.py

@@ -1,5 +1,5 @@
 # -*- coding: utf-8 -*-
-# Copyright 2014 matrix.org
+# Copyright 2014 OpenMarket Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.

synapse/app/__init__.py

@@ -1,5 +1,5 @@
 # -*- coding: utf-8 -*-
-# Copyright 2014 matrix.org
+# Copyright 2014 OpenMarket Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.

synapse/app/homeserver.py

@@ -1,6 +1,6 @@
 #!/usr/bin/env python
 # -*- coding: utf-8 -*-
-# Copyright 2014 matrix.org
+# Copyright 2014 OpenMarket Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -23,7 +23,8 @@ from twisted.enterprise import adbapi
 from twisted.web.resource import Resource
 from twisted.web.static import File
 from twisted.web.server import Site
-from synapse.http.server import JsonResource, RootRedirect, ContentRepoResource
+from synapse.http.server import JsonResource, RootRedirect
+from synapse.http.content_repository import ContentRepoResource
 from synapse.http.client import TwistedHttpClient
 from synapse.api.urls import (
     CLIENT_PREFIX, FEDERATION_PREFIX, WEB_CLIENT_PREFIX, CONTENT_REPO_PREFIX
@@ -56,7 +57,7 @@ SCHEMAS = [
 
 # Remember to update this number every time an incompatible change is made to
 # database schema files, so the users will be informed on server restarts.
-SCHEMA_VERSION = 2
+SCHEMA_VERSION = 3
 
 
 class SynapseHomeServer(HomeServer):
@@ -74,7 +75,9 @@ class SynapseHomeServer(HomeServer):
         return File("webclient")  # TODO configurable?
 
     def build_resource_for_content_repo(self):
-        return ContentRepoResource(self, self.upload_dir, self.auth)
+        return ContentRepoResource(
+            self, self.upload_dir, self.auth, self.content_addr
+        )
 
     def build_db_pool(self):
         """ Set up all the dbs. Since all the *.sql have IF NOT EXISTS, so we
@@ -90,20 +93,28 @@ class SynapseHomeServer(HomeServer):
             if row and row[0]:
                 user_version = row[0]
 
-                if user_version < SCHEMA_VERSION:
-                    # TODO(paul): add some kind of intelligent fixup here
-                    raise ValueError("Cannot use this database as the " +
-                        "schema version (%d) does not match (%d)" %
-                        (user_version, SCHEMA_VERSION)
+                if user_version > SCHEMA_VERSION:
+                    raise ValueError("Cannot use this database as it is too " +
+                        "new for the server to understand"
                     )
+                elif user_version < SCHEMA_VERSION:
+                    logging.info("Upgrading database from version %d",
+                        user_version
+                    )
+
+                    # Run every version since after the current version.
+                    for v in range(user_version + 1, SCHEMA_VERSION + 1):
+                        sql_script = read_schema("delta/v%d" % (v))
+                        c.executescript(sql_script)
+
+                    db_conn.commit()
 
             else:
                 for sql_loc in SCHEMAS:
                     sql_script = read_schema(sql_loc)
 
                     c.executescript(sql_script)
-                    db_conn.commit()
-
+                db_conn.commit()
                 c.execute("PRAGMA user_version = %d" % SCHEMA_VERSION)
 
             c.close()
@@ -247,6 +258,8 @@ def setup():
         upload_dir=os.path.abspath("uploads"),
         db_name=config.database_path,
         tls_context_factory=tls_context_factory,
+        config=config,
+        content_addr=config.content_addr,
     )
 
     hs.register_servlets()

synapse/config/__init__.py

@@ -1,5 +1,5 @@
 # -*- coding: utf-8 -*-
-# Copyright 2014 matrix.org
+# Copyright 2014 OpenMarket Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.

synapse/config/_base.py

@@ -1,5 +1,5 @@
 # -*- coding: utf-8 -*-
-# Copyright 2014 matrix.org
+# Copyright 2014 OpenMarket Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -13,8 +13,6 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-
-import ConfigParser as configparser
 import argparse
 import sys
 import os
@@ -121,6 +119,9 @@ class Config(object):
                     and value is not None):
                     config[key] = value
             with open(config_args.config_path, "w") as config_file:
+                # TODO(paul) it would be lovely if we wrote out vim- and emacs-
+                #   style mode markers into the file, to hint to people that
+                #   this is a YAML file.
                 yaml.dump(config, config_file, default_flow_style=False)
             sys.exit(0)
 

synapse/config/captcha.py

@@ -0,0 +1,42 @@
+# Copyright 2014 OpenMarket Ltd
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from ._base import Config
+
+class CaptchaConfig(Config):
+
+    def __init__(self, args):
+        super(CaptchaConfig, self).__init__(args)
+        self.recaptcha_private_key = args.recaptcha_private_key
+        self.enable_registration_captcha = args.enable_registration_captcha
+        self.captcha_ip_origin_is_x_forwarded = args.captcha_ip_origin_is_x_forwarded
+
+    @classmethod
+    def add_arguments(cls, parser):
+        super(CaptchaConfig, cls).add_arguments(parser)
+        group = parser.add_argument_group("recaptcha")
+        group.add_argument(
+            "--recaptcha-private-key", type=str, default="YOUR_PRIVATE_KEY",
+            help="The matching private key for the web client's public key."
+        )
+        group.add_argument(
+            "--enable-registration-captcha", type=bool, default=False,
+            help="Enables ReCaptcha checks when registering, preventing signup "+
+            "unless a captcha is answered. Requires a valid ReCaptcha public/private key."
+        )
+        group.add_argument(
+            "--captcha_ip_origin_is_x_forwarded", type=bool, default=False,
+            help="When checking captchas, use the X-Forwarded-For (XFF) header as the client IP "+
+            "and not the actual client IP."
+        )

synapse/config/database.py

@@ -1,5 +1,5 @@
 # -*- coding: utf-8 -*-
-# Copyright 2014 matrix.org
+# Copyright 2014 OpenMarket Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.

synapse/config/email.py

@@ -0,0 +1,39 @@
+# -*- coding: utf-8 -*-
+# Copyright 2014 OpenMarket Ltd
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from ._base import Config
+
+
+class EmailConfig(Config):
+
+    def __init__(self, args):
+        super(EmailConfig, self).__init__(args)
+        self.email_from_address = args.email_from_address
+        self.email_smtp_server = args.email_smtp_server
+
+    @classmethod
+    def add_arguments(cls, parser):
+        super(EmailConfig, cls).add_arguments(parser)
+        email_group = parser.add_argument_group("email")
+        email_group.add_argument(
+            "--email-from-address",
+            default="FROM@EXAMPLE.COM",
+            help="The address to send emails from (e.g. for password resets)."
+        )
+        email_group.add_argument(
+            "--email-smtp-server",
+            default="",
+            help="The SMTP server to send emails from (e.g. for password resets)."
+        )

synapse/config/homeserver.py

@@ -1,5 +1,5 @@
 # -*- coding: utf-8 -*-
-# Copyright 2014 matrix.org
+# Copyright 2014 OpenMarket Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -17,10 +17,18 @@ from .tls import TlsConfig
 from .server import ServerConfig
 from .logger import LoggingConfig
 from .database import DatabaseConfig
+from .ratelimiting import RatelimitConfig
+from .repository import ContentRepositoryConfig
+from .captcha import CaptchaConfig
+from .email import EmailConfig
 
-class HomeServerConfig(TlsConfig, ServerConfig, DatabaseConfig, LoggingConfig):
+
+class HomeServerConfig(TlsConfig, ServerConfig, DatabaseConfig, LoggingConfig,
+                       RatelimitConfig, ContentRepositoryConfig, CaptchaConfig,
+                       EmailConfig):
     pass
 
-if __name__=='__main__':
+
+if __name__ == '__main__':
     import sys
     HomeServerConfig.load_config("Generate config", sys.argv[1:], "HomeServer")

synapse/config/logger.py

@@ -1,5 +1,5 @@
 # -*- coding: utf-8 -*-
-# Copyright 2014 matrix.org
+# Copyright 2014 OpenMarket Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.

synapse/config/ratelimiting.py

@@ -0,0 +1,35 @@
+# Copyright 2014 OpenMarket Ltd
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from ._base import Config
+
+class RatelimitConfig(Config):
+
+    def __init__(self, args):
+        super(RatelimitConfig, self).__init__(args)
+        self.rc_messages_per_second = args.rc_messages_per_second
+        self.rc_message_burst_count = args.rc_message_burst_count
+
+    @classmethod
+    def add_arguments(cls, parser):
+        super(RatelimitConfig, cls).add_arguments(parser)
+        rc_group = parser.add_argument_group("ratelimiting")
+        rc_group.add_argument(
+            "--rc-messages-per-second", type=float, default=0.2,
+            help="number of messages a client can send per second"
+        )
+        rc_group.add_argument(
+            "--rc-message-burst-count", type=float, default=10,
+            help="number of message a client can send before being throttled"
+        )

synapse/config/repository.py

@@ -0,0 +1,39 @@
+# -*- coding: utf-8 -*-
+# Copyright 2014 matrix.org
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from ._base import Config
+import os
+
+class ContentRepositoryConfig(Config):
+    def __init__(self, args):
+        super(ContentRepositoryConfig, self).__init__(args)
+        self.max_upload_size = self.parse_size(args.max_upload_size)
+
+    def parse_size(self, string):
+        sizes = {"K": 1024, "M": 1024 * 1024}
+        size = 1
+        suffix = string[-1]
+        if suffix in sizes:
+            string = string[:-1]
+            size = sizes[suffix]
+        return int(string) * size
+
+    @classmethod
+    def add_arguments(cls, parser):
+        super(ContentRepositoryConfig, cls).add_arguments(parser)
+        db_group = parser.add_argument_group("content_repository")
+        db_group.add_argument(
+            "--max-upload-size", default="1M"
+        )

synapse/config/server.py

@@ -1,5 +1,5 @@
 # -*- coding: utf-8 -*-
-# Copyright 2014 matrix.org
+# Copyright 2014 OpenMarket Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -32,6 +32,14 @@ class ServerConfig(Config):
         self.webclient = True
         self.manhole = args.manhole
 
+        if not args.content_addr:
+            host = args.server_name
+            if ':' not in host:
+                host  = "%s:%d" % (host, args.bind_port)
+            args.content_addr = "https://%s" % (host,)
+
+        self.content_addr = args.content_addr
+
     @classmethod
     def add_arguments(cls, parser):
         super(ServerConfig, cls).add_arguments(parser)
@@ -50,13 +58,16 @@ class ServerConfig(Config):
                                   help="Local interface to listen on")
         server_group.add_argument("-D", "--daemonize", action='store_true',
                                   help="Daemonize the home server")
-        server_group.add_argument('--pid-file', default="hs.pid",
+        server_group.add_argument('--pid-file', default="homeserver.pid",
                                   help="When running as a daemon, the file to"
                                   " store the pid in")
         server_group.add_argument("--manhole", metavar="PORT", dest="manhole",
                                   type=int,
                                   help="Turn on the twisted telnet manhole"
                                   " service on the given port.")
+        server_group.add_argument("--content-addr", default=None,
+                                  help="The host and scheme to use for the "
+                                  "content repository")
 
     def read_signing_key(self, signing_key_path):
         signing_key_base64 = self.read_file(signing_key_path, "signing_key")
@@ -77,3 +88,4 @@ class ServerConfig(Config):
             with open(args.signing_key_path, "w") as signing_key_file:
                 key = nacl.signing.SigningKey.generate()
                 signing_key_file.write(encode_base64(key.encode()))
+

synapse/config/tls.py

@@ -1,5 +1,5 @@
 # -*- coding: utf-8 -*-
-# Copyright 2014 matrix.org
+# Copyright 2014 OpenMarket Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.

synapse/crypto/__init__.py

@@ -1,5 +1,5 @@
 # -*- coding: utf-8 -*-
-# Copyright 2014 matrix.org
+# Copyright 2014 OpenMarket Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.

synapse/crypto/context_factory.py

@@ -1,4 +1,18 @@
-from twisted.internet import reactor, ssl
+# Copyright 2014 OpenMarket Ltd
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from twisted.internet import ssl
 from OpenSSL import SSL
 from twisted.internet._sslverify import _OpenSSLECCurve, _defaultCurveName
 

synapse/crypto/keyclient.py

@@ -1,5 +1,5 @@
 # -*- coding: utf-8 -*-
-# Copyright 2014 matrix.org
+# Copyright 2014 OpenMarket Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.

synapse/crypto/keyserver.py

@@ -1,5 +1,5 @@
 # -*- coding: utf-8 -*-
-# Copyright 2014 matrix.org
+# Copyright 2014 OpenMarket Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.

synapse/crypto/resource/__init__.py

@@ -1,5 +1,5 @@
 # -*- coding: utf-8 -*-
-# Copyright 2014 matrix.org
+# Copyright 2014 OpenMarket Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.

synapse/crypto/resource/key.py

@@ -1,5 +1,5 @@
 # -*- coding: utf-8 -*-
-# Copyright 2014 matrix.org
+# Copyright 2014 OpenMarket Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.

synapse/federation/__init__.py

@@ -1,5 +1,5 @@
 # -*- coding: utf-8 -*-
-# Copyright 2014 matrix.org
+# Copyright 2014 OpenMarket Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.

synapse/federation/pdu_codec.py

@@ -1,5 +1,5 @@
 # -*- coding: utf-8 -*-
-# Copyright 2014 matrix.org
+# Copyright 2014 OpenMarket Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.

synapse/federation/persistence.py

@@ -1,5 +1,5 @@
 # -*- coding: utf-8 -*-
-# Copyright 2014 matrix.org
+# Copyright 2014 OpenMarket Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.

synapse/federation/replication.py

@@ -1,5 +1,5 @@
 # -*- coding: utf-8 -*-
-# Copyright 2014 matrix.org
+# Copyright 2014 OpenMarket Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -291,6 +291,13 @@ class ReplicationLayer(object):
     def on_incoming_transaction(self, transaction_data):
         transaction = Transaction(**transaction_data)
 
+        for p in transaction.pdus:
+            if "age" in p:
+                p["age_ts"] = int(self._clock.time_msec()) - int(p["age"])
+                del p["age"]
+
+        pdu_list = [Pdu(**p) for p in transaction.pdus]
+
         logger.debug("[%s] Got transaction", transaction.transaction_id)
 
         response = yield self.transaction_actions.have_responded(transaction)
@@ -303,8 +310,6 @@ class ReplicationLayer(object):
 
         logger.debug("[%s] Transacition is new", transaction.transaction_id)
 
-        pdu_list = [Pdu(**p) for p in transaction.pdus]
-
         dl = []
         for pdu in pdu_list:
             dl.append(self._handle_new_pdu(pdu))
@@ -405,9 +410,14 @@ class ReplicationLayer(object):
         """Returns a new Transaction containing the given PDUs suitable for
         transmission.
         """
+        pdus = [p.get_dict() for p in pdu_list]
+        for p in pdus:
+            if "age_ts" in pdus:
+                p["age"] = int(self.clock.time_msec()) - p["age_ts"]
+
         return Transaction(
-            pdus=[p.get_dict() for p in pdu_list],
             origin=self.server_name,
+            pdus=pdus,
             ts=int(self._clock.time_msec()),
             destination=None,
         )
@@ -593,8 +603,21 @@ class _TransactionQueue(object):
             logger.debug("TX [%s] Sending transaction...", destination)
 
             # Actually send the transaction
+
+            # FIXME (erikj): This is a bit of a hack to make the Pdu age
+            # keys work
+            def cb(transaction):
+                now = int(self._clock.time_msec())
+                if "pdus" in transaction:
+                    for p in transaction["pdus"]:
+                        if "age_ts" in p:
+                            p["age"] = now - int(p["age_ts"])
+
+                return transaction
+
             code, response = yield self.transport_layer.send_transaction(
-                transaction
+                transaction,
+                on_send_callback=cb,
             )
 
             logger.debug("TX [%s] Sent transaction", destination)
@@ -628,7 +651,6 @@ class _TransactionQueue(object):
 
             for deferred in deferreds:
                 deferred.errback(e)
-                yield deferred
 
         finally:
             # We want to be *very* sure we delete this after we stop processing

synapse/federation/transport.py

@@ -1,5 +1,5 @@
 # -*- coding: utf-8 -*-
-# Copyright 2014 matrix.org
+# Copyright 2014 OpenMarket Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -144,7 +144,7 @@ class TransportLayer(object):
 
     @defer.inlineCallbacks
     @log_function
-    def send_transaction(self, transaction):
+    def send_transaction(self, transaction, on_send_callback=None):
         """ Sends the given Transaction to it's destination
 
         Args:
@@ -165,10 +165,23 @@ class TransportLayer(object):
 
         data = transaction.get_dict()
 
+        # FIXME (erikj): This is a bit of a hack to make the Pdu age
+        # keys work
+        def cb(destination, method, path_bytes, producer):
+            if not on_send_callback:
+                return
+
+            transaction = json.loads(producer.body)
+
+            new_transaction = on_send_callback(transaction)
+
+            producer.reset(new_transaction)
+
         code, response = yield self.client.put_json(
             transaction.destination,
             path=PREFIX + "/send/%s/" % transaction.transaction_id,
-            data=data
+            data=data,
+            on_send_callback=cb,
         )
 
         logger.debug(

synapse/federation/units.py

@@ -1,5 +1,5 @@
 # -*- coding: utf-8 -*-
-# Copyright 2014 matrix.org
+# Copyright 2014 OpenMarket Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -69,6 +69,7 @@ class Pdu(JsonEncodedObject):
         "prev_state_id",
         "prev_state_origin",
         "required_power_level",
+        "user_id",
     ]
 
     internal_keys = [

synapse/handlers/__init__.py

@@ -1,5 +1,5 @@
 # -*- coding: utf-8 -*-
-# Copyright 2014 matrix.org
+# Copyright 2014 OpenMarket Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.

synapse/handlers/_base.py

@@ -1,5 +1,5 @@
 # -*- coding: utf-8 -*-
-# Copyright 2014 matrix.org
+# Copyright 2014 OpenMarket Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -14,6 +14,7 @@
 # limitations under the License.
 
 from twisted.internet import defer
+from synapse.api.errors import LimitExceededError
 
 class BaseHandler(object):
 
@@ -25,10 +26,21 @@ class BaseHandler(object):
         self.room_lock = hs.get_room_lock_manager()
         self.state_handler = hs.get_state_handler()
         self.distributor = hs.get_distributor()
+        self.ratelimiter = hs.get_ratelimiter()
+        self.clock = hs.get_clock()
         self.hs = hs
 
-
-class BaseRoomHandler(BaseHandler):
+    def ratelimit(self, user_id):
+        time_now = self.clock.time()
+        allowed, time_allowed = self.ratelimiter.send_message(
+            user_id, time_now,
+            msg_rate_hz=self.hs.config.rc_messages_per_second,
+            burst_count=self.hs.config.rc_message_burst_count,
+        )
+        if not allowed:
+            raise LimitExceededError(
+                retry_after_ms=int(1000*(time_allowed - time_now)),
+            )
 
     @defer.inlineCallbacks
     def _on_new_room_event(self, event, snapshot, extra_destinations=[],

synapse/handlers/directory.py

@@ -1,5 +1,5 @@
 # -*- coding: utf-8 -*-
-# Copyright 2014 matrix.org
+# Copyright 2014 OpenMarket Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -18,8 +18,11 @@ from twisted.internet import defer
 from ._base import BaseHandler
 
 from synapse.api.errors import SynapseError
+from synapse.http.client import HttpClient
+from synapse.api.events.room import RoomAliasesEvent
 
 import logging
+import sqlite3
 
 
 logger = logging.getLogger(__name__)
@@ -36,7 +39,8 @@ class DirectoryHandler(BaseHandler):
         )
 
     @defer.inlineCallbacks
-    def create_association(self, room_alias, room_id, servers):
+    def create_association(self, user_id, room_alias, room_id, servers=None):
+
         # TODO(erikj): Do auth.
 
         if not room_alias.is_mine:
@@ -47,12 +51,43 @@ class DirectoryHandler(BaseHandler):
 
         # TODO(erikj): Check if there is a current association.
 
-        yield self.store.create_room_alias_association(
-            room_alias,
-            room_id,
-            servers
+        if not servers:
+            servers = yield self.store.get_joined_hosts_for_room(room_id)
+
+        if not servers:
+            raise SynapseError(400, "Failed to get server list")
+
+
+        try:
+            yield self.store.create_room_alias_association(
+                room_alias,
+                room_id,
+                servers
+            )
+        except sqlite3.IntegrityError:
+            defer.returnValue("Already exists")
+
+        # TODO: Send the room event.
+
+        aliases = yield self.store.get_aliases_for_room(room_id)
+
+        event = self.event_factory.create_event(
+            etype=RoomAliasesEvent.TYPE,
+            state_key=self.hs.hostname,
+            room_id=room_id,
+            user_id=user_id,
+            content={"aliases": aliases},
         )
 
+        snapshot = yield self.store.snapshot_room(
+            room_id=room_id,
+            user_id=user_id,
+        )
+
+        yield self.state_handler.handle_new_event(event, snapshot)
+        yield self._on_new_room_event(event, snapshot, extra_users=[user_id])
+
+
     @defer.inlineCallbacks
     def get_association(self, room_alias):
         room_id = None
@@ -68,7 +103,10 @@ class DirectoryHandler(BaseHandler):
             result = yield self.federation.make_query(
                 destination=room_alias.domain,
                 query_type="directory",
-                args={"room_alias": room_alias.to_string()},
+                args={
+                    "room_alias": room_alias.to_string(),
+                    HttpClient.RETRY_DNS_LOOKUP_FAILURES: False
+                }
             )
 
             if result and "room_id" in result and "servers" in result:
@@ -79,6 +117,9 @@ class DirectoryHandler(BaseHandler):
             defer.returnValue({})
             return
 
+        extra_servers = yield self.store.get_joined_hosts_for_room(room_id)
+        servers = list(set(extra_servers) | set(servers))
+
         defer.returnValue({
             "room_id": room_id,
             "servers": servers,

synapse/handlers/events.py

@@ -1,5 +1,5 @@
 # -*- coding: utf-8 -*-
-# Copyright 2014 matrix.org
+# Copyright 2014 OpenMarket Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -15,7 +15,6 @@
 
 from twisted.internet import defer
 
-from synapse.api.events import SynapseEvent
 from synapse.util.logutils import log_function
 
 from ._base import BaseHandler
@@ -71,10 +70,7 @@ class EventStreamHandler(BaseHandler):
                 auth_user, room_ids, pagin_config, timeout
             )
 
-            chunks = [
-                e.get_dict() if isinstance(e, SynapseEvent) else e
-                for e in events
-            ]
+            chunks = [self.hs.serialize_event(e) for e in events]
 
             chunk = {
                 "chunk": chunks,
@@ -92,7 +88,9 @@ class EventStreamHandler(BaseHandler):
                 # 10 seconds of grace to allow the client to reconnect again
                 #   before we think they're gone
                 def _later():
-                    logger.debug("_later stopped_user_eventstream %s", auth_user)
+                    logger.debug(
+                        "_later stopped_user_eventstream %s", auth_user
+                    )
                     self.distributor.fire(
                         "stopped_user_eventstream", auth_user
                     )
@@ -126,5 +124,7 @@ class EventHandler(BaseHandler):
             defer.returnValue(None)
             return
 
-        yield self.auth.check(event, raises=True)
+        if hasattr(event, "room_id"):
+            yield self.auth.check_joined_room(event.room_id, user.to_string())
+
         defer.returnValue(event)

synapse/handlers/federation.py

@@ -1,5 +1,5 @@
 # -*- coding: utf-8 -*-
-# Copyright 2014 matrix.org
+# Copyright 2014 OpenMarket Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -21,8 +21,9 @@ from synapse.api.events.room import InviteJoinEvent, RoomMemberEvent
 from synapse.api.constants import Membership
 from synapse.util.logutils import log_function
 from synapse.federation.pdu_codec import PduCodec
+from synapse.api.errors import SynapseError
 
-from twisted.internet import defer
+from twisted.internet import defer, reactor
 
 import logging
 
@@ -92,22 +93,18 @@ class FederationHandler(BaseHandler):
         """
         event = self.pdu_codec.event_from_pdu(pdu)
 
+        logger.debug("Got event: %s", event.event_id)
+
         with (yield self.lock_manager.lock(pdu.context)):
             if event.is_state and not backfilled:
                 is_new_state = yield self.state_handler.handle_new_state(
                     pdu
                 )
-                if not is_new_state:
-                    return
             else:
                 is_new_state = False
         # TODO: Implement something in federation that allows us to
         # respond to PDU.
 
-        if hasattr(event, "state_key") and not is_new_state:
-            logger.debug("Ignoring old state.")
-            return
-
         target_is_mine = False
         if hasattr(event, "target_host"):
             target_is_mine = event.target_host == self.hs.hostname
@@ -133,12 +130,16 @@ class FederationHandler(BaseHandler):
 
             yield self.hs.get_handlers().room_member_handler.change_membership(
                 new_event,
-                do_auth=True
+                do_auth=False,
             )
 
         else:
             with (yield self.room_lock.lock(event.room_id)):
-                yield self.store.persist_event(event, backfilled)
+                yield self.store.persist_event(
+                    event,
+                    backfilled,
+                    is_new_state=is_new_state
+                )
 
             room = yield self.store.get_room(event.room_id)
 
@@ -231,7 +232,12 @@ class FederationHandler(BaseHandler):
         # TODO (erikj): Time out here.
         d = defer.Deferred()
         self.waiting_for_join_list.setdefault((joinee, room_id), []).append(d)
-        yield d
+        reactor.callLater(10, d.cancel)
+
+        try:
+            yield d
+        except defer.CancelledError:
+            raise SynapseError(500, "Unable to join remote room")
 
         try:
             yield self.store.store_room(

synapse/handlers/login.py

@@ -1,5 +1,5 @@
 # -*- coding: utf-8 -*-
-# Copyright 2014 matrix.org
+# Copyright 2014 OpenMarket Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -17,9 +17,13 @@ from twisted.internet import defer
 
 from ._base import BaseHandler
 from synapse.api.errors import LoginError, Codes
+from synapse.http.client import PlainHttpClient
+from synapse.util.emailutils import EmailException
+import synapse.util.emailutils as emailutils
 
 import bcrypt
 import logging
+import urllib
 
 logger = logging.getLogger(__name__)
 
@@ -62,4 +66,41 @@ class LoginHandler(BaseHandler):
             defer.returnValue(token)
         else:
             logger.warn("Failed password login for user %s", user)
-            raise LoginError(403, "", errcode=Codes.FORBIDDEN)
+            raise LoginError(403, "", errcode=Codes.FORBIDDEN)
+
+    @defer.inlineCallbacks
+    def reset_password(self, user_id, email):
+        is_valid = yield self._check_valid_association(user_id, email)
+        logger.info("reset_password user=%s email=%s valid=%s", user_id, email,
+                    is_valid)
+        if is_valid:
+            try:
+                # send an email out
+                emailutils.send_email(
+                    smtp_server=self.hs.config.email_smtp_server,
+                    from_addr=self.hs.config.email_from_address,
+                    to_addr=email,
+                    subject="Password Reset",
+                    body="TODO."
+                )
+            except EmailException as e:
+                logger.exception(e)
+
+    @defer.inlineCallbacks
+    def _check_valid_association(self, user_id, email):
+        identity = yield self._query_email(email)
+        if identity and "mxid" in identity:
+            if identity["mxid"] == user_id:
+                defer.returnValue(True)
+                return
+        defer.returnValue(False)
+
+    @defer.inlineCallbacks
+    def _query_email(self, email):
+        httpCli = PlainHttpClient(self.hs)
+        data = yield httpCli.get_json(
+            'matrix.org:8090',  # TODO FIXME This should be configurable.
+            "/_matrix/identity/api/v1/lookup?medium=email&address=" +
+            "%s" % urllib.quote(email)
+        )
+        defer.returnValue(data)

synapse/handlers/message.py

@@ -1,5 +1,5 @@
 # -*- coding: utf-8 -*-
-# Copyright 2014 matrix.org
+# Copyright 2014 OpenMarket Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -19,7 +19,7 @@ from synapse.api.constants import Membership
 from synapse.api.events.room import RoomTopicEvent
 from synapse.api.errors import RoomError
 from synapse.streams.config import PaginationConfig
-from ._base import BaseRoomHandler
+from ._base import BaseHandler
 
 import logging
 
@@ -27,7 +27,7 @@ logger = logging.getLogger(__name__)
 
 
 
-class MessageHandler(BaseRoomHandler):
+class MessageHandler(BaseHandler):
 
     def __init__(self, hs):
         super(MessageHandler, self).__init__(hs)
@@ -76,6 +76,8 @@ class MessageHandler(BaseRoomHandler):
         Raises:
             SynapseError if something went wrong.
         """
+
+        self.ratelimit(event.user_id)
         # TODO(paul): Why does 'event' not have a 'user' object?
         user = self.hs.parse_userid(event.user_id)
         assert user.is_mine, "User must be our own: %s" % (user,)
@@ -122,7 +124,7 @@ class MessageHandler(BaseRoomHandler):
         )
 
         chunk = {
-            "chunk": [e.get_dict() for e in events],
+            "chunk": [self.hs.serialize_event(e) for e in events],
             "start": pagin_config.from_token.to_string(),
             "end": next_token.to_string(),
         }
@@ -140,7 +142,12 @@ class MessageHandler(BaseRoomHandler):
             SynapseError if something went wrong.
         """
 
-        snapshot = yield self.store.snapshot_room(event.room_id, event.user_id)
+        snapshot = yield self.store.snapshot_room(
+            event.room_id,
+            event.user_id,
+            state_type=event.type,
+            state_key=event.state_key,
+        )
 
         yield self.auth.check(event, snapshot, raises=True)
 
@@ -261,6 +268,9 @@ class MessageHandler(BaseRoomHandler):
             user, pagination_config, None
         )
 
+        public_rooms = yield self.store.get_rooms(is_public=True)
+        public_room_ids = [r["room_id"] for r in public_rooms]
+
         limit = pagin_config.limit
         if not limit:
             limit = 10
@@ -269,6 +279,8 @@ class MessageHandler(BaseRoomHandler):
             d = {
                 "room_id": event.room_id,
                 "membership": event.membership,
+                "visibility": ("public" if event.room_id in
+                              public_room_ids else "private"),
             }
 
             if event.membership == Membership.INVITE:
@@ -289,7 +301,7 @@ class MessageHandler(BaseRoomHandler):
                 end_token = now_token.copy_and_replace("room_key", token[1])
 
                 d["messages"] = {
-                    "chunk": [m.get_dict() for m in messages],
+                    "chunk": [self.hs.serialize_event(m) for m in messages],
                     "start": start_token.to_string(),
                     "end": end_token.to_string(),
                 }
@@ -297,7 +309,7 @@ class MessageHandler(BaseRoomHandler):
                 current_state = yield self.store.get_current_state(
                     event.room_id
                 )
-                d["state"] = [c.get_dict() for c in current_state]
+                d["state"] = [self.hs.serialize_event(c) for c in current_state]
             except:
                 logger.exception("Failed to get snapshot")
 

synapse/handlers/presence.py

@@ -1,5 +1,5 @@
 # -*- coding: utf-8 -*-
-# Copyright 2014 matrix.org
+# Copyright 2014 OpenMarket Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -155,19 +155,18 @@ class PresenceHandler(BaseHandler):
         if observer_user == observed_user:
             defer.returnValue(True)
 
-        allowed_by_subscription = yield self.store.is_presence_visible(
-            observed_localpart=observed_user.localpart,
-            observer_userid=observer_user.to_string(),
-        )
-
-        if allowed_by_subscription:
+        if (yield self.store.user_rooms_intersect(
+            [u.to_string() for u in observer_user, observed_user]
+        )):
             defer.returnValue(True)
 
-        share_room = yield self.store.do_users_share_a_room(
-            [observer_user, observed_user]
-        )
+        if (yield self.store.is_presence_visible(
+            observed_localpart=observed_user.localpart,
+            observer_userid=observer_user.to_string(),
+        )):
+            defer.returnValue(True)
 
-        defer.returnValue(share_room)
+        defer.returnValue(False)
 
     @defer.inlineCallbacks
     def get_state(self, target_user, auth_user):
@@ -181,7 +180,7 @@ class PresenceHandler(BaseHandler):
             state = yield self.store.get_presence_state(target_user.localpart)
             if "mtime" in state:
                 del state["mtime"]
-            state["presence"] = state["state"]
+            state["presence"] = state.pop("state")
 
             if target_user in self._user_cachemap:
                 state["last_active"] = (
@@ -208,21 +207,17 @@ class PresenceHandler(BaseHandler):
             raise SynapseError(400, "User is not hosted on this Home Server")
 
         if target_user != auth_user:
-            raise AuthError(400, "Cannot set another user's displayname")
+            raise AuthError(400, "Cannot set another user's presence")
 
         if "status_msg" not in state:
             state["status_msg"] = None
 
         for k in state.keys():
-            if k not in ("presence", "state", "status_msg"):
+            if k not in ("presence", "status_msg"):
                 raise SynapseError(
                     400, "Unexpected presence state key '%s'" % (k,)
                 )
 
-        # Handle legacy "state" key for now
-        if "state" in state:
-            state["presence"] = state.pop("state")
-
         if state["presence"] not in self.STATE_LEVELS:
             raise SynapseError(400, "'%s' is not a valid presence state" %
                 state["presence"]
@@ -601,7 +596,7 @@ class PresenceHandler(BaseHandler):
         if state is None:
             state = yield self.store.get_presence_state(user.localpart)
             del state["mtime"]
-            state["presence"] = state["state"]
+            state["presence"] = state.pop("state")
 
             if user in self._user_cachemap:
                 state["last_active"] = (
@@ -622,8 +617,6 @@ class PresenceHandler(BaseHandler):
             "user_id": user.to_string(),
         }
         user_state.update(**state)
-        if "state" in user_state and "presence" not in user_state:
-            user_state["presence"] = user_state["state"]
 
         yield self.federation.send_edu(
             destination=destination,
@@ -655,21 +648,12 @@ class PresenceHandler(BaseHandler):
             state = dict(push)
             del state["user_id"]
 
-            if "presence" in state:
-                # all is OK
-                pass
-            elif "state" in state:
-                # Legacy handling
-                state["presence"] = state["state"]
-            else:
+            if "presence" not in state:
                 logger.warning("Received a presence 'push' EDU from %s without"
-                    + " either a 'presence' or 'state' key", origin
+                    + " a 'presence' key", origin
                 )
                 continue
 
-            if "state" in state:
-                del state["state"]
-
             if "last_active_ago" in state:
                 state["last_active"] = int(
                     self.clock.time_msec() - state.pop("last_active_ago")
@@ -773,15 +757,53 @@ class PresenceEventSource(object):
         self.hs = hs
         self.clock = hs.get_clock()
 
+    @defer.inlineCallbacks
+    def is_visible(self, observer_user, observed_user):
+        if observer_user == observed_user:
+            defer.returnValue(True)
+
+        presence = self.hs.get_handlers().presence_handler
+
+        if (yield presence.store.user_rooms_intersect(
+            [u.to_string() for u in observer_user, observed_user]
+        )):
+            defer.returnValue(True)
+
+        if observed_user.is_mine:
+            pushmap = presence._local_pushmap
+
+            defer.returnValue(
+                observed_user.localpart in pushmap and
+                observer_user in pushmap[observed_user.localpart]
+            )
+        else:
+            recvmap = presence._remote_recvmap
+
+            defer.returnValue(
+                observed_user in recvmap and
+                observer_user in recvmap[observed_user]
+            )
+
+    @defer.inlineCallbacks
     def get_new_events_for_user(self, user, from_key, limit):
         from_key = int(from_key)
 
+        observer_user = user
+
         presence = self.hs.get_handlers().presence_handler
         cachemap = presence._user_cachemap
 
-        # TODO(paul): limit, and filter by visibility
-        updates = [(k, cachemap[k]) for k in cachemap
-                   if from_key < cachemap[k].serial]
+        updates = []
+        # TODO(paul): use a DeferredList ? How to limit concurrency.
+        for observed_user in cachemap.keys():
+            cached = cachemap[observed_user]
+            if not (from_key < cached.serial):
+                continue
+
+            if (yield self.is_visible(observer_user, observed_user)):
+                updates.append((observed_user, cached))
+
+        # TODO(paul): limit
 
         if updates:
             clock = self.clock
@@ -789,20 +811,23 @@ class PresenceEventSource(object):
             latest_serial = max([x[1].serial for x in updates])
             data = [x[1].make_event(user=x[0], clock=clock) for x in updates]
 
-            return ((data, latest_serial))
+            defer.returnValue((data, latest_serial))
         else:
-            return (([], presence._user_cachemap_latest_serial))
+            defer.returnValue(([], presence._user_cachemap_latest_serial))
 
     def get_current_key(self):
         presence = self.hs.get_handlers().presence_handler
         return presence._user_cachemap_latest_serial
 
+    @defer.inlineCallbacks
     def get_pagination_rows(self, user, pagination_config, key):
         # TODO (erikj): Does this make sense? Ordering?
 
         from_token = pagination_config.from_token
         to_token = pagination_config.to_token
 
+        observer_user = user
+
         from_key = int(from_token.presence_key)
 
         if to_token:
@@ -813,7 +838,17 @@ class PresenceEventSource(object):
         presence = self.hs.get_handlers().presence_handler
         cachemap = presence._user_cachemap
 
-        # TODO(paul): limit, and filter by visibility
+        updates = []
+        # TODO(paul): use a DeferredList ? How to limit concurrency.
+        for observed_user in cachemap.keys():
+            if not (to_key < cachemap[observed_user].serial < from_key):
+                continue
+
+            if (yield self.is_visible(observer_user, observed_user)):
+                updates.append((observed_user, cachemap[observed_user]))
+
+        # TODO(paul): limit
+
         updates = [(k, cachemap[k]) for k in cachemap
                    if to_key < cachemap[k].serial < from_key]
 
@@ -831,13 +866,13 @@ class PresenceEventSource(object):
             next_token = next_token.copy_and_replace(
                 "presence_key", earliest_serial
             )
-            return ((data, next_token))
+            defer.returnValue((data, next_token))
         else:
             if not to_token:
                 to_token = from_token.copy_and_replace(
                     "presence_key", 0
                 )
-            return (([], to_token))
+            defer.returnValue(([], to_token))
 
 
 class UserPresenceCache(object):
@@ -851,7 +886,6 @@ class UserPresenceCache(object):
 
     def update(self, state, serial):
         assert("mtime_age" not in state)
-        assert("state" not in state)
 
         self.state.update(state)
         # Delete keys that are now 'None'
@@ -869,11 +903,6 @@ class UserPresenceCache(object):
     def get_state(self):
         # clone it so caller can't break our cache
         state = dict(self.state)
-
-        # Legacy handling
-        if "presence" in state:
-            state["state"] = state["presence"]
-
         return state
 
     def make_event(self, user, clock):

synapse/handlers/profile.py

@@ -1,5 +1,5 @@
 # -*- coding: utf-8 -*-
-# Copyright 2014 matrix.org
+# Copyright 2014 OpenMarket Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -15,9 +15,9 @@
 
 from twisted.internet import defer
 
-from synapse.api.errors import SynapseError, AuthError
-
-from synapse.api.errors import CodeMessageException
+from synapse.api.errors import SynapseError, AuthError, CodeMessageException
+from synapse.api.constants import Membership
+from synapse.api.events.room import RoomMemberEvent
 
 from ._base import BaseHandler
 
@@ -97,6 +97,8 @@ class ProfileHandler(BaseHandler):
             }
         )
 
+        yield self._update_join_states(target_user)
+
     @defer.inlineCallbacks
     def get_avatar_url(self, target_user):
         if target_user.is_mine:
@@ -144,6 +146,8 @@ class ProfileHandler(BaseHandler):
             }
         )
 
+        yield self._update_join_states(target_user)
+
     @defer.inlineCallbacks
     def collect_presencelike_data(self, user, state):
         if not user.is_mine:
@@ -180,3 +184,39 @@ class ProfileHandler(BaseHandler):
             )
 
         defer.returnValue(response)
+
+    @defer.inlineCallbacks
+    def _update_join_states(self, user):
+        if not user.is_mine:
+            return
+
+        joins = yield self.store.get_rooms_for_user_where_membership_is(
+            user.to_string(),
+            [Membership.JOIN],
+        )
+
+        for j in joins:
+            snapshot = yield self.store.snapshot_room(
+                j.room_id, j.state_key, RoomMemberEvent.TYPE,
+                j.state_key
+            )
+
+            content = {
+                "membership": j.content["membership"],
+                "prev": j.content["membership"],
+            }
+
+            yield self.distributor.fire(
+                "collect_presencelike_data", user, content
+            )
+
+            new_event = self.event_factory.create_event(
+                etype=j.type,
+                room_id=j.room_id,
+                state_key=j.state_key,
+                content=content,
+                user_id=j.state_key,
+            )
+
+            yield self.state_handler.handle_new_event(new_event, snapshot)
+            yield self._on_new_room_event(new_event, snapshot)

synapse/handlers/register.py

@@ -1,5 +1,5 @@
 # -*- coding: utf-8 -*-
-# Copyright 2014 matrix.org
+# Copyright 2014 OpenMarket Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -17,12 +17,18 @@
 from twisted.internet import defer
 
 from synapse.types import UserID
-from synapse.api.errors import SynapseError, RegistrationError
+from synapse.api.errors import (
+    SynapseError, RegistrationError, InvalidCaptchaError
+)
 from ._base import BaseHandler
 import synapse.util.stringutils as stringutils
+from synapse.http.client import PlainHttpClient
 
 import base64
 import bcrypt
+import logging
+
+logger = logging.getLogger(__name__)
 
 
 class RegistrationHandler(BaseHandler):
@@ -61,7 +67,6 @@ class RegistrationHandler(BaseHandler):
                 password_hash=password_hash)
 
             self.distributor.fire("registered_user", user)
-            defer.returnValue((user_id, token))
         else:
             # autogen a random user ID
             attempts = 0
@@ -80,7 +85,6 @@ class RegistrationHandler(BaseHandler):
                         password_hash=password_hash)
 
                     self.distributor.fire("registered_user", user)
-                    defer.returnValue((user_id, token))
                 except SynapseError:
                     # if user id is taken, just generate another
                     user_id = None
@@ -90,6 +94,54 @@ class RegistrationHandler(BaseHandler):
                         raise RegistrationError(
                             500, "Cannot generate user ID.")
 
+        defer.returnValue((user_id, token))
+
+    @defer.inlineCallbacks
+    def check_recaptcha(self, ip, private_key, challenge, response):
+        """Checks a recaptcha is correct."""
+
+        captcha_response = yield self._validate_captcha(
+            ip,
+            private_key,
+            challenge,
+            response
+        )
+        if not captcha_response["valid"]:
+            logger.info("Invalid captcha entered from %s. Error: %s",
+                        ip, captcha_response["error_url"])
+            raise InvalidCaptchaError(
+                error_url=captcha_response["error_url"]
+            )
+        else:
+            logger.info("Valid captcha entered from %s", ip)
+
+    @defer.inlineCallbacks
+    def register_email(self, threepidCreds):
+        """Registers emails with an identity server."""
+
+        for c in threepidCreds:
+            logger.info("validating theeepidcred sid %s on id server %s",
+                        c['sid'], c['idServer'])
+            try:
+                threepid = yield self._threepid_from_creds(c)
+            except:
+                logger.err()
+                raise RegistrationError(400, "Couldn't validate 3pid")
+
+            if not threepid:
+                raise RegistrationError(400, "Couldn't validate 3pid")
+            logger.info("got threepid medium %s address %s",
+                        threepid['medium'], threepid['address'])
+
+    @defer.inlineCallbacks
+    def bind_emails(self, user_id, threepidCreds):
+        """Links emails with a user ID and informs an identity server."""
+
+        # Now we have a matrix ID, bind it to the threepids we were given
+        for c in threepidCreds:
+            # XXX: This should be a deferred list, shouldn't it?
+            yield self._bind_threepid(c, user_id)
+
     def _generate_token(self, user_id):
         # urlsafe variant uses _ and - so use . as the separator and replace
         # all =s with .s so http clients don't quote =s when it is used as
@@ -99,3 +151,71 @@ class RegistrationHandler(BaseHandler):
 
     def _generate_user_id(self):
         return "-" + stringutils.random_string(18)
+
+    @defer.inlineCallbacks
+    def _threepid_from_creds(self, creds):
+        httpCli = PlainHttpClient(self.hs)
+        # XXX: make this configurable!
+        trustedIdServers = ['matrix.org:8090']
+        if not creds['idServer'] in trustedIdServers:
+            logger.warn('%s is not a trusted ID server: rejecting 3pid ' +
+                        'credentials', creds['idServer'])
+            defer.returnValue(None)
+        data = yield httpCli.get_json(
+            creds['idServer'],
+            "/_matrix/identity/api/v1/3pid/getValidated3pid",
+            {'sid': creds['sid'], 'clientSecret': creds['clientSecret']}
+        )
+
+        if 'medium' in data:
+            defer.returnValue(data)
+        defer.returnValue(None)
+
+    @defer.inlineCallbacks
+    def _bind_threepid(self, creds, mxid):
+        httpCli = PlainHttpClient(self.hs)
+        data = yield httpCli.post_urlencoded_get_json(
+            creds['idServer'],
+            "/_matrix/identity/api/v1/3pid/bind",
+            {'sid': creds['sid'], 'clientSecret': creds['clientSecret'],
+            'mxid': mxid}
+        )
+        defer.returnValue(data)
+
+    @defer.inlineCallbacks
+    def _validate_captcha(self, ip_addr, private_key, challenge, response):
+        """Validates the captcha provided.
+
+        Returns:
+            dict: Containing 'valid'(bool) and 'error_url'(str) if invalid.
+
+        """
+        response = yield self._submit_captcha(ip_addr, private_key, challenge,
+                                              response)
+        # parse Google's response. Lovely format..
+        lines = response.split('\n')
+        json = {
+            "valid": lines[0] == 'true',
+            "error_url": "http://www.google.com/recaptcha/api/challenge?" +
+                         "error=%s" % lines[1]
+        }
+        defer.returnValue(json)
+
+    @defer.inlineCallbacks
+    def _submit_captcha(self, ip_addr, private_key, challenge, response):
+        client = PlainHttpClient(self.hs)
+        data = yield client.post_urlencoded_get_raw(
+            "www.google.com:80",
+            "/recaptcha/api/verify",
+            # twisted dislikes google's response, no content length.
+            accept_partial=True,
+            args={
+                'privatekey': private_key,
+                'remoteip': ip_addr,
+                'challenge': challenge,
+                'response': response
+            }
+        )
+        defer.returnValue(data)
+
+

synapse/handlers/room.py

@@ -1,5 +1,5 @@
 # -*- coding: utf-8 -*-
-# Copyright 2014 matrix.org
+# Copyright 2014 OpenMarket Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -25,14 +25,14 @@ from synapse.api.events.room import (
     RoomSendEventLevelEvent, RoomOpsPowerLevelsEvent, RoomNameEvent,
 )
 from synapse.util import stringutils
-from ._base import BaseRoomHandler
+from ._base import BaseHandler
 
 import logging
 
 logger = logging.getLogger(__name__)
 
 
-class RoomCreationHandler(BaseRoomHandler):
+class RoomCreationHandler(BaseHandler):
 
     @defer.inlineCallbacks
     def create_room(self, user_id, room_id, config):
@@ -49,6 +49,7 @@ class RoomCreationHandler(BaseRoomHandler):
             SynapseError if the room ID was taken, couldn't be stored, or
             something went horribly wrong.
         """
+        self.ratelimit(user_id)
 
         if "room_alias_name" in config:
             room_alias = RoomAlias.create_local(
@@ -64,6 +65,13 @@ class RoomCreationHandler(BaseRoomHandler):
         else:
             room_alias = None
 
+        invite_list = config.get("invite", [])
+        for i in invite_list:
+            try:
+                self.hs.parse_userid(i)
+            except:
+                raise SynapseError(400, "Invalid user_id: %s" % (i,))
+
         is_public = config.get("visibility", None) == "public"
 
         if room_id:
@@ -104,14 +112,14 @@ class RoomCreationHandler(BaseRoomHandler):
         )
 
         if room_alias:
-            yield self.store.create_room_alias_association(
+            directory_handler = self.hs.get_handlers().directory_handler
+            yield directory_handler.create_association(
+                user_id=user_id,
                 room_id=room_id,
                 room_alias=room_alias,
                 servers=[self.hs.hostname],
             )
 
-        federation_handler = self.hs.get_handlers().federation_handler
-
         @defer.inlineCallbacks
         def handle_event(event):
             snapshot = yield self.store.snapshot_room(
@@ -133,7 +141,18 @@ class RoomCreationHandler(BaseRoomHandler):
                 etype=RoomNameEvent.TYPE,
                 room_id=room_id,
                 user_id=user_id,
-                required_power_level=5,
+                required_power_level=50,
+                content={"name": name},
+            )
+
+            yield handle_event(name_event)
+        elif room_alias:
+            name = room_alias.to_string()
+            name_event = self.event_factory.create_event(
+                etype=RoomNameEvent.TYPE,
+                room_id=room_id,
+                user_id=user_id,
+                required_power_level=50,
                 content={"name": name},
             )
 
@@ -145,7 +164,7 @@ class RoomCreationHandler(BaseRoomHandler):
                 etype=RoomTopicEvent.TYPE,
                 room_id=room_id,
                 user_id=user_id,
-                required_power_level=5,
+                required_power_level=50,
                 content={"topic": topic},
             )
 
@@ -166,6 +185,25 @@ class RoomCreationHandler(BaseRoomHandler):
             do_auth=False
         )
 
+        content = {"membership": Membership.INVITE}
+        for invitee in invite_list:
+            invite_event = self.event_factory.create_event(
+                etype=RoomMemberEvent.TYPE,
+                state_key=invitee,
+                room_id=room_id,
+                user_id=user_id,
+                content=content
+            )
+
+            yield self.hs.get_handlers().room_member_handler.change_membership(
+                invite_event,
+                do_auth=False
+            )
+
+        yield self.hs.get_handlers().room_member_handler.change_membership(
+            join_event,
+            do_auth=False
+        )
         result = {"room_id": room_id}
         if room_alias:
             result["room_alias"] = room_alias.to_string()
@@ -176,7 +214,7 @@ class RoomCreationHandler(BaseRoomHandler):
         event_keys = {
             "room_id": room_id,
             "user_id": creator.to_string(),
-            "required_power_level": 10,
+            "required_power_level": 100,
         }
 
         def create(etype, **content):
@@ -193,7 +231,7 @@ class RoomCreationHandler(BaseRoomHandler):
 
         power_levels_event = self.event_factory.create_event(
             etype=RoomPowerLevelsEvent.TYPE,
-            content={creator.to_string(): 10, "default": 0},
+            content={creator.to_string(): 100, "default": 0},
             **event_keys
         )
 
@@ -205,7 +243,7 @@ class RoomCreationHandler(BaseRoomHandler):
 
         add_state_event = create(
             etype=RoomAddStateLevelEvent.TYPE,
-            level=10,
+            level=100,
         )
 
         send_event = create(
@@ -215,8 +253,8 @@ class RoomCreationHandler(BaseRoomHandler):
 
         ops = create(
             etype=RoomOpsPowerLevelsEvent.TYPE,
-            ban_level=5,
-            kick_level=5,
+            ban_level=50,
+            kick_level=50,
         )
 
         return [
@@ -229,7 +267,7 @@ class RoomCreationHandler(BaseRoomHandler):
         ]
 
 
-class RoomMemberHandler(BaseRoomHandler):
+class RoomMemberHandler(BaseHandler):
     # TODO(paul): This handler currently contains a messy conflation of
     #   low-level API that works on UserID objects and so on, and REST-level
     #   API that takes ID strings and returns pagination chunks. These concerns
@@ -297,7 +335,7 @@ class RoomMemberHandler(BaseRoomHandler):
 
         member_list = yield self.store.get_room_members(room_id=room_id)
         event_list = [
-            entry.get_dict()
+            self.hs.serialize_event(entry)
             for entry in member_list
         ]
         chunk_data = {
@@ -550,11 +588,17 @@ class RoomMemberHandler(BaseRoomHandler):
             extra_users=[target_user]
         )
 
-class RoomListHandler(BaseRoomHandler):
+class RoomListHandler(BaseHandler):
 
     @defer.inlineCallbacks
     def get_public_room_list(self):
         chunk = yield self.store.get_rooms(is_public=True)
+        for room in chunk:
+            joined_members = yield self.store.get_room_members(
+                room_id=room["room_id"],
+                membership=Membership.JOIN
+            )
+            room["num_joined_members"] = len(joined_members)
         # FIXME (erikj): START is no longer a valid value
         defer.returnValue({"start": "START", "end": "END", "chunk": chunk})
 

synapse/handlers/typing.py

@@ -1,5 +1,5 @@
 # -*- coding: utf-8 -*-
-# Copyright 2014 matrix.org
+# Copyright 2014 OpenMarket Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.

synapse/http/__init__.py

@@ -1,5 +1,5 @@
 # -*- coding: utf-8 -*-
-# Copyright 2014 matrix.org
+# Copyright 2014 OpenMarket Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.

synapse/http/client.py

@@ -1,5 +1,5 @@
 # -*- coding: utf-8 -*-
-# Copyright 2014 matrix.org
+# Copyright 2014 OpenMarket Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -15,7 +15,8 @@
 
 
 from twisted.internet import defer, reactor
-from twisted.web.client import _AgentBase, _URI, readBody
+from twisted.internet.error import DNSLookupError
+from twisted.web.client import _AgentBase, _URI, readBody, FileBodyProducer, PartialDownloadError
 from twisted.web.http_headers import Headers
 
 from synapse.http.endpoint import matrix_endpoint
@@ -23,7 +24,9 @@ from synapse.util.async import sleep
 
 from syutil.jsonutil import encode_canonical_json
 
-from synapse.api.errors import CodeMessageException
+from synapse.api.errors import CodeMessageException, SynapseError
+
+from StringIO import StringIO
 
 import json
 import logging
@@ -43,6 +46,7 @@ _destination_mappings = {
 class HttpClient(object):
     """ Interface for talking json over http
     """
+    RETRY_DNS_LOOKUP_FAILURES = "__retry_dns"
 
     def put_json(self, destination, path, data):
         """ Sends the specifed json data using PUT
@@ -118,7 +122,7 @@ class TwistedHttpClient(HttpClient):
         self.hs = hs
 
     @defer.inlineCallbacks
-    def put_json(self, destination, path, data):
+    def put_json(self, destination, path, data, on_send_callback=None):
         if destination in _destination_mappings:
             destination = _destination_mappings[destination]
 
@@ -127,7 +131,8 @@ class TwistedHttpClient(HttpClient):
             "PUT",
             path.encode("ascii"),
             producer=_JsonProducer(data),
-            headers_dict={"Content-Type": ["application/json"]}
+            headers_dict={"Content-Type": ["application/json"]},
+            on_send_callback=on_send_callback,
         )
 
         logger.debug("Getting resp body")
@@ -142,22 +147,79 @@ class TwistedHttpClient(HttpClient):
             destination = _destination_mappings[destination]
 
         logger.debug("get_json args: %s", args)
+
+        retry_on_dns_fail = True
+        if HttpClient.RETRY_DNS_LOOKUP_FAILURES in args:
+            # FIXME: This isn't ideal, but the interface exposed in get_json
+            # isn't comprehensive enough to give caller's any control over
+            # their connection mechanics.
+            retry_on_dns_fail = args.pop(HttpClient.RETRY_DNS_LOOKUP_FAILURES)
+
         query_bytes = urllib.urlencode(args, True)
+        logger.debug("Query bytes: %s Retry DNS: %s", args, retry_on_dns_fail)
 
         response = yield self._create_request(
             destination.encode("ascii"),
             "GET",
             path.encode("ascii"),
-            query_bytes=query_bytes
+            query_bytes=query_bytes,
+            retry_on_dns_fail=retry_on_dns_fail
         )
 
         body = yield readBody(response)
 
         defer.returnValue(json.loads(body))
 
+    @defer.inlineCallbacks
+    def post_urlencoded_get_json(self, destination, path, args={}):
+        if destination in _destination_mappings:
+            destination = _destination_mappings[destination]
+
+        logger.debug("post_urlencoded_get_json args: %s", args)
+        query_bytes = urllib.urlencode(args, True)
+
+        response = yield self._create_request(
+            destination.encode("ascii"),
+            "POST",
+            path.encode("ascii"),
+            producer=FileBodyProducer(StringIO(urllib.urlencode(args))),
+            headers_dict={"Content-Type": ["application/x-www-form-urlencoded"]}
+        )
+
+        body = yield readBody(response)
+
+        defer.returnValue(json.loads(body))
+        
+    # XXX FIXME : I'm so sorry.
+    @defer.inlineCallbacks
+    def post_urlencoded_get_raw(self, destination, path, accept_partial=False, args={}):
+        if destination in _destination_mappings:
+            destination = _destination_mappings[destination]
+
+        query_bytes = urllib.urlencode(args, True)
+
+        response = yield self._create_request(
+            destination.encode("ascii"),
+            "POST",
+            path.encode("ascii"),
+            producer=FileBodyProducer(StringIO(urllib.urlencode(args))),
+            headers_dict={"Content-Type": ["application/x-www-form-urlencoded"]}
+        )
+
+        try:
+            body = yield readBody(response)
+            defer.returnValue(body)
+        except PartialDownloadError as e:
+            if accept_partial:
+                defer.returnValue(e.response)
+            else:
+                raise e
+        
+
     @defer.inlineCallbacks
     def _create_request(self, destination, method, path_bytes, param_bytes=b"",
-                        query_bytes=b"", producer=None, headers_dict={}):
+                        query_bytes=b"", producer=None, headers_dict={},
+                        retry_on_dns_fail=True, on_send_callback=None):
         """ Creates and sends a request to the given url
         """
         headers_dict[b"User-Agent"] = [b"Synapse"]
@@ -178,12 +240,12 @@ class TwistedHttpClient(HttpClient):
         retries_left = 5
 
         # TODO: setup and pass in an ssl_context to enable TLS
-        endpoint = matrix_endpoint(
-            reactor, destination, timeout=10,
-            ssl_context_factory=self.hs.tls_context_factory
-        )
+        endpoint = self._getEndpoint(reactor, destination);
 
         while True:
+            if on_send_callback:
+                on_send_callback(destination, method, path_bytes, producer)
+
             try:
                 response = yield self.agent.request(
                     destination,
@@ -199,6 +261,11 @@ class TwistedHttpClient(HttpClient):
                 logger.debug("Got response to %s", method)
                 break
             except Exception as e:
+                if not retry_on_dns_fail and isinstance(e, DNSLookupError):
+                    logger.warn("DNS Lookup failed to %s with %s", destination,
+                                e)
+                    raise SynapseError(400, "Domain specified not found.")
+
                 logger.exception("Got error in _create_request")
                 _print_ex(e)
 
@@ -223,6 +290,17 @@ class TwistedHttpClient(HttpClient):
 
         defer.returnValue(response)
 
+    def _getEndpoint(self, reactor, destination):
+        return matrix_endpoint(
+            reactor, destination, timeout=10,
+            ssl_context_factory=self.hs.tls_context_factory
+        )
+
+
+class PlainHttpClient(TwistedHttpClient):
+    def _getEndpoint(self, reactor, destination):
+        return matrix_endpoint(reactor, destination, timeout=10)
+    
 
 def _print_ex(e):
     if hasattr(e, "reasons") and e.reasons:
@@ -236,6 +314,9 @@ class _JsonProducer(object):
     """ Used by the twisted http client to create the HTTP body from json
     """
     def __init__(self, jsn):
+        self.reset(jsn)
+
+    def reset(self, jsn):
         self.body = encode_canonical_json(jsn)
         self.length = len(self.body)
 

synapse/http/content_repository.py

@@ -0,0 +1,206 @@
+# -*- coding: utf-8 -*-
+# Copyright 2014 OpenMarket Ltd
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from .server import respond_with_json_bytes
+
+from synapse.util.stringutils import random_string
+from synapse.api.errors import (
+    cs_exception, SynapseError, CodeMessageException, Codes, cs_error
+)
+
+from twisted.protocols.basic import FileSender
+from twisted.web import server, resource
+from twisted.internet import defer
+
+import base64
+import json
+import logging
+import os
+import re
+
+logger = logging.getLogger(__name__)
+
+
+class ContentRepoResource(resource.Resource):
+    """Provides file uploading and downloading.
+
+    Uploads are POSTed to wherever this Resource is linked to. This resource
+    returns a "content token" which can be used to GET this content again. The
+    token is typically a path, but it may not be. Tokens can expire, be one-time
+    uses, etc.
+
+    In this case, the token is a path to the file and contains 3 interesting
+    sections:
+        - User ID base64d (for namespacing content to each user)
+        - random 24 char string
+        - Content type base64d (so we can return it when clients GET it)
+
+    """
+    isLeaf = True
+
+    def __init__(self, hs, directory, auth, external_addr):
+        resource.Resource.__init__(self)
+        self.hs = hs
+        self.directory = directory
+        self.auth = auth
+        self.external_addr = external_addr.rstrip('/')
+        self.max_upload_size = hs.config.max_upload_size
+
+        if not os.path.isdir(self.directory):
+            os.mkdir(self.directory)
+            logger.info("ContentRepoResource : Created %s directory.",
+                        self.directory)
+
+    @defer.inlineCallbacks
+    def map_request_to_name(self, request):
+        # auth the user
+        auth_user = yield self.auth.get_user_by_req(request)
+
+        # namespace all file uploads on the user
+        prefix = base64.urlsafe_b64encode(
+            auth_user.to_string()
+        ).replace('=', '')
+
+        # use a random string for the main portion
+        main_part = random_string(24)
+
+        # suffix with a file extension if we can make one. This is nice to
+        # provide a hint to clients on the file information. We will also reuse
+        # this info to spit back the content type to the client.
+        suffix = ""
+        if request.requestHeaders.hasHeader("Content-Type"):
+            content_type = request.requestHeaders.getRawHeaders(
+                "Content-Type")[0]
+            suffix = "." + base64.urlsafe_b64encode(content_type)
+            if (content_type.split("/")[0].lower() in
+                    ["image", "video", "audio"]):
+                file_ext = content_type.split("/")[-1]
+                # be a little paranoid and only allow a-z
+                file_ext = re.sub("[^a-z]", "", file_ext)
+                suffix += "." + file_ext
+
+        file_name = prefix + main_part + suffix
+        file_path = os.path.join(self.directory, file_name)
+        logger.info("User %s is uploading a file to path %s",
+                    auth_user.to_string(),
+                    file_path)
+
+        # keep trying to make a non-clashing file, with a sensible max attempts
+        attempts = 0
+        while os.path.exists(file_path):
+            main_part = random_string(24)
+            file_name = prefix + main_part + suffix
+            file_path = os.path.join(self.directory, file_name)
+            attempts += 1
+            if attempts > 25:  # really? Really?
+                raise SynapseError(500, "Unable to create file.")
+
+        defer.returnValue(file_path)
+
+    def render_GET(self, request):
+        # no auth here on purpose, to allow anyone to view, even across home
+        # servers.
+
+        # TODO: A little crude here, we could do this better.
+        filename = request.path.split('/')[-1]
+        # be paranoid
+        filename = re.sub("[^0-9A-z.-_]", "", filename)
+
+        file_path = self.directory + "/" + filename
+
+        logger.debug("Searching for %s", file_path)
+
+        if os.path.isfile(file_path):
+            # filename has the content type
+            base64_contentype = filename.split(".")[1]
+            content_type = base64.urlsafe_b64decode(base64_contentype)
+            logger.info("Sending file %s", file_path)
+            f = open(file_path, 'rb')
+            request.setHeader('Content-Type', content_type)
+            d = FileSender().beginFileTransfer(f, request)
+
+            # after the file has been sent, clean up and finish the request
+            def cbFinished(ignored):
+                f.close()
+                request.finish()
+            d.addCallback(cbFinished)
+        else:
+            respond_with_json_bytes(
+                request,
+                404,
+                json.dumps(cs_error("Not found", code=Codes.NOT_FOUND)),
+                send_cors=True)
+
+        return server.NOT_DONE_YET
+
+    def render_POST(self, request):
+        self._async_render(request)
+        return server.NOT_DONE_YET
+
+    def render_OPTIONS(self, request):
+        respond_with_json_bytes(request, 200, {}, send_cors=True)
+        return server.NOT_DONE_YET
+
+    @defer.inlineCallbacks
+    def _async_render(self, request):
+        try:
+            # TODO: The checks here are a bit late. The content will have
+            # already been uploaded to a tmp file at this point
+            content_length = request.getHeader("Content-Length")
+            if content_length is None:
+                raise SynapseError(
+                    msg="Request must specify a Content-Length", code=400
+                )
+            if int(content_length) > self.max_upload_size:
+                raise SynapseError(
+                    msg="Upload request body is too large",
+                    code=413,
+                )
+
+            fname = yield self.map_request_to_name(request)
+
+            # TODO I have a suspcious feeling this is just going to block
+            with open(fname, "wb") as f:
+                f.write(request.content.read())
+
+
+            # FIXME (erikj): These should use constants.
+            file_name = os.path.basename(fname)
+            # FIXME: we can't assume what the public mounted path of the repo is
+            # ...plus self-signed SSL won't work to remote clients anyway
+            # ...and we can't assume that it's SSL anyway, as we might want to
+            # server it via the non-SSL listener...
+            url = "%s/_matrix/content/%s" % (
+                self.external_addr, file_name
+            )
+
+            respond_with_json_bytes(request, 200,
+                                    json.dumps({"content_token": url}),
+                                    send_cors=True)
+
+        except CodeMessageException as e:
+            logger.exception(e)
+            respond_with_json_bytes(request, e.code,
+                                    json.dumps(cs_exception(e)))
+        except Exception as e:
+            logger.error("Failed to store file: %s" % e)
+            respond_with_json_bytes(
+                request,
+                500,
+                json.dumps({"error": "Internal server error"}),
+                send_cors=True)
+
+
+

synapse/http/endpoint.py

@@ -1,5 +1,5 @@
 # -*- coding: utf-8 -*-
-# Copyright 2014 matrix.org
+# Copyright 2014 OpenMarket Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.

synapse/http/server.py

@@ -1,5 +1,5 @@
 # -*- coding: utf-8 -*-
-# Copyright 2014 matrix.org
+# Copyright 2014 OpenMarket Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -18,22 +18,16 @@ from syutil.jsonutil import (
     encode_canonical_json, encode_pretty_printed_json
 )
 from synapse.api.errors import (
-    cs_exception, SynapseError, CodeMessageException, Codes, cs_error
+    cs_exception, SynapseError, CodeMessageException
 )
-from synapse.util.stringutils import random_string
 
 from twisted.internet import defer, reactor
-from twisted.protocols.basic import FileSender
 from twisted.web import server, resource
 from twisted.web.server import NOT_DONE_YET
 from twisted.web.util import redirectTo
 
-import base64
 import collections
-import json
 import logging
-import os
-import re
 
 logger = logging.getLogger(__name__)
 
@@ -140,7 +134,8 @@ class JsonResource(HttpServer, resource.Resource):
             self._send_response(
                 request,
                 e.code,
-                cs_exception(e)
+                cs_exception(e),
+                response_code_message=e.response_code_message
             )
         except Exception as e:
             logger.exception(e)
@@ -150,7 +145,8 @@ class JsonResource(HttpServer, resource.Resource):
                 {"error": "Internal server error"}
             )
 
-    def _send_response(self, request, code, response_json_object):
+    def _send_response(self, request, code, response_json_object,
+                       response_code_message=None):
         # could alternatively use request.notifyFinish() and flip a flag when
         # the Deferred fires, but since the flag is RIGHT THERE it seems like
         # a waste.
@@ -166,7 +162,8 @@ class JsonResource(HttpServer, resource.Resource):
             json_bytes = encode_pretty_printed_json(response_json_object)
 
         # TODO: Only enable CORS for the requests that need it.
-        respond_with_json_bytes(request, code, json_bytes, send_cors=True)
+        respond_with_json_bytes(request, code, json_bytes, send_cors=True,
+                                response_code_message=response_code_message)
 
     @staticmethod
     def _request_user_agent_is_curl(request):
@@ -195,162 +192,8 @@ class RootRedirect(resource.Resource):
         return resource.Resource.getChild(self, name, request)
 
 
-class ContentRepoResource(resource.Resource):
-    """Provides file uploading and downloading.
-
-    Uploads are POSTed to wherever this Resource is linked to. This resource
-    returns a "content token" which can be used to GET this content again. The
-    token is typically a path, but it may not be. Tokens can expire, be one-time
-    uses, etc.
-
-    In this case, the token is a path to the file and contains 3 interesting
-    sections:
-        - User ID base64d (for namespacing content to each user)
-        - random 24 char string
-        - Content type base64d (so we can return it when clients GET it)
-
-    """
-    isLeaf = True
-
-    def __init__(self, hs, directory, auth):
-        resource.Resource.__init__(self)
-        self.hs = hs
-        self.directory = directory
-        self.auth = auth
-
-        if not os.path.isdir(self.directory):
-            os.mkdir(self.directory)
-            logger.info("ContentRepoResource : Created %s directory.",
-                        self.directory)
-
-    @defer.inlineCallbacks
-    def map_request_to_name(self, request):
-        # auth the user
-        auth_user = yield self.auth.get_user_by_req(request)
-
-        # namespace all file uploads on the user
-        prefix = base64.urlsafe_b64encode(
-            auth_user.to_string()
-        ).replace('=', '')
-
-        # use a random string for the main portion
-        main_part = random_string(24)
-
-        # suffix with a file extension if we can make one. This is nice to
-        # provide a hint to clients on the file information. We will also reuse
-        # this info to spit back the content type to the client.
-        suffix = ""
-        if request.requestHeaders.hasHeader("Content-Type"):
-            content_type = request.requestHeaders.getRawHeaders(
-                "Content-Type")[0]
-            suffix = "." + base64.urlsafe_b64encode(content_type)
-            if (content_type.split("/")[0].lower() in
-                    ["image", "video", "audio"]):
-                file_ext = content_type.split("/")[-1]
-                # be a little paranoid and only allow a-z
-                file_ext = re.sub("[^a-z]", "", file_ext)
-                suffix += "." + file_ext
-
-        file_name = prefix + main_part + suffix
-        file_path = os.path.join(self.directory, file_name)
-        logger.info("User %s is uploading a file to path %s",
-                    auth_user.to_string(),
-                    file_path)
-
-        # keep trying to make a non-clashing file, with a sensible max attempts
-        attempts = 0
-        while os.path.exists(file_path):
-            main_part = random_string(24)
-            file_name = prefix + main_part + suffix
-            file_path = os.path.join(self.directory, file_name)
-            attempts += 1
-            if attempts > 25:  # really? Really?
-                raise SynapseError(500, "Unable to create file.")
-
-        defer.returnValue(file_path)
-
-    def render_GET(self, request):
-        # no auth here on purpose, to allow anyone to view, even across home
-        # servers.
-
-        # TODO: A little crude here, we could do this better.
-        filename = request.path.split('/')[-1]
-        # be paranoid
-        filename = re.sub("[^0-9A-z.-_]", "", filename)
-
-        file_path = self.directory + "/" + filename
-
-        logger.debug("Searching for %s", file_path)
-
-        if os.path.isfile(file_path):
-            # filename has the content type
-            base64_contentype = filename.split(".")[1]
-            content_type = base64.urlsafe_b64decode(base64_contentype)
-            logger.info("Sending file %s", file_path)
-            f = open(file_path, 'rb')
-            request.setHeader('Content-Type', content_type)
-            d = FileSender().beginFileTransfer(f, request)
-
-            # after the file has been sent, clean up and finish the request
-            def cbFinished(ignored):
-                f.close()
-                request.finish()
-            d.addCallback(cbFinished)
-        else:
-            respond_with_json_bytes(
-                request,
-                404,
-                json.dumps(cs_error("Not found", code=Codes.NOT_FOUND)),
-                send_cors=True)
-
-        return server.NOT_DONE_YET
-
-    def render_POST(self, request):
-        self._async_render(request)
-        return server.NOT_DONE_YET
-
-    def render_OPTIONS(self, request):
-        respond_with_json_bytes(request, 200, {}, send_cors=True)
-        return server.NOT_DONE_YET
-
-    @defer.inlineCallbacks
-    def _async_render(self, request):
-        try:
-            fname = yield self.map_request_to_name(request)
-
-            # TODO I have a suspcious feeling this is just going to block
-            with open(fname, "wb") as f:
-                f.write(request.content.read())
-
-
-            # FIXME (erikj): These should use constants.
-            file_name = os.path.basename(fname)
-            # FIXME: we can't assume what the public mounted path of the repo is
-            # ...plus self-signed SSL won't work to remote clients anyway
-            # ...and we can't assume that it's SSL anyway, as we might want to
-            # server it via the non-SSL listener...
-            url = "https://%s/_matrix/content/%s" % (
-                self.hs.domain_with_port, file_name
-            )
-
-            respond_with_json_bytes(request, 200,
-                                    json.dumps({"content_token": url}),
-                                    send_cors=True)
-
-        except CodeMessageException as e:
-            logger.exception(e)
-            respond_with_json_bytes(request, e.code,
-                                    json.dumps(cs_exception(e)))
-        except Exception as e:
-            logger.error("Failed to store file: %s" % e)
-            respond_with_json_bytes(
-                request,
-                500,
-                json.dumps({"error": "Internal server error"}),
-                send_cors=True)
-
-
-def respond_with_json_bytes(request, code, json_bytes, send_cors=False):
+def respond_with_json_bytes(request, code, json_bytes, send_cors=False,
+                            response_code_message=None):
     """Sends encoded JSON in response to the given request.
 
     Args:
@@ -362,7 +205,7 @@ def respond_with_json_bytes(request, code, json_bytes, send_cors=False):
     Returns:
         twisted.web.server.NOT_DONE_YET"""
 
-    request.setResponseCode(code)
+    request.setResponseCode(code, message=response_code_message)
     request.setHeader(b"Content-Type", b"application/json")
 
     if send_cors: