Update tracer to give more information

Add a Twisted Service to synapse.app.homeserver

.gitignore

@@ -1,6 +1,7 @@
 *.pyc
 .*.swp
 
+.DS_Store
 _trial_temp/
 logs/
 dbs/
@@ -11,16 +12,33 @@ docs/build/
 
 cmdclient_config.json
 homeserver*.db
+homeserver*.log
+homeserver*.pid
+homeserver*.yaml
+
+*.signing.key
+*.tls.crt
+*.tls.dh
+*.tls.key
 
 .coverage
 htmlcov
 
 demo/*.db
 demo/*.log
+demo/*.log.*
 demo/*.pid
-
-graph/*.svg
-graph/*.png
-graph/*.dot
+demo/media_store.*
+demo/etc
 
 uploads
+
+.idea/
+media_store/
+
+*.tac
+
+build/
+
+localhost-800*/
+static/client/register/register_config.js

CHANGES.rst

@@ -1,3 +1,402 @@
+Changes in synapse v0.8.0 (2015-03-06)
+======================================
+
+General:
+
+* Add support for registration fallback. This is a page hosted on the server
+  which allows a user to register for an account, regardless of what client
+  they are using (e.g. mobile devices).
+
+* Added new default push rules and made them configurable by clients:
+
+  * Suppress all notice messages.
+  * Notify when invited to a new room.
+  * Notify for messages that don't match any rule.
+  * Notify on incoming call.
+
+Federation:
+
+* Added per host server side rate-limiting of incoming federation requests.
+* Added a ``/get_missing_events/`` API to federation to reduce number of
+  ``/events/`` requests.
+
+Configuration:
+
+* Added configuration option to disable registration:
+  ``disable_registration``.
+* Added configuration option to change soft limit of number of open file
+  descriptors: ``soft_file_limit``.
+* Make ``tls_private_key_path`` optional when running with ``no_tls``.
+
+Application services:
+
+* Application services can now poll on the CS API ``/events`` for their events,
+  by providing their application service ``access_token``.
+* Added exclusive namespace support to application services API.
+
+
+Changes in synapse v0.7.1 (2015-02-19)
+======================================
+
+* Initial alpha implementation of parts of the Application Services API.
+  Including:
+
+  - AS Registration / Unregistration
+  - User Query API
+  - Room Alias Query API
+  - Push transport for receiving events.
+  - User/Alias namespace admin control
+
+* Add cache when fetching events from remote servers to stop repeatedly
+  fetching events with bad signatures.
+* Respect the per remote server retry scheme when fetching both events and
+  server keys to reduce the number of times we send requests to dead servers.
+* Inform remote servers when the local server fails to handle a received event.
+* Turn off python bytecode generation due to problems experienced when
+  upgrading from previous versions.
+
+Changes in synapse v0.7.0 (2015-02-12)
+======================================
+
+* Add initial implementation of the query auth federation API, allowing
+  servers to agree on whether an event should be allowed or rejected.
+* Persist events we have rejected from federation, fixing the bug where
+  servers would keep requesting the same events.
+* Various federation performance improvements, including:
+
+  - Add in memory caches on queries such as:
+
+     * Computing the state of a room at a point in time, used for
+       authorization on federation requests.
+     * Fetching events from the database.
+     * User's room membership, used for authorizing presence updates.
+
+  - Upgraded JSON library to improve parsing and serialisation speeds.
+
+* Add default avatars to new user accounts using pydenticon library.
+* Correctly time out federation requests.
+* Retry federation requests against different servers.
+* Add support for push and push rules.
+* Add alpha versions of proposed new CSv2 APIs, including ``/sync`` API.
+
+Changes in synapse 0.6.1 (2015-01-07)
+=====================================
+
+* Major optimizations to improve performance of initial sync and event sending
+  in large rooms (by up to 10x)
+* Media repository now includes a Content-Length header on media downloads.
+* Improve quality of thumbnails by changing resizing algorithm.
+
+Changes in synapse 0.6.0 (2014-12-16)
+=====================================
+
+* Add new API for media upload and download that supports thumbnailing.
+* Replicate media uploads over multiple homeservers so media is always served
+  to clients from their local homeserver.  This obsoletes the
+  --content-addr parameter and confusion over accessing content directly
+  from remote homeservers.
+* Implement exponential backoff when retrying federation requests when
+  sending to remote homeservers which are offline.
+* Implement typing notifications.
+* Fix bugs where we sent events with invalid signatures due to bugs where
+  we incorrectly persisted events.
+* Improve performance of database queries involving retrieving events.
+
+Changes in synapse 0.5.4a (2014-12-13)
+======================================
+
+* Fix bug while generating the error message when a file path specified in
+  the config doesn't exist.
+
+Changes in synapse 0.5.4 (2014-12-03)
+=====================================
+
+* Fix presence bug where some rooms did not display presence updates for
+  remote users.
+* Do not log SQL timing log lines when started with "-v"
+* Fix potential memory leak.
+
+Changes in synapse 0.5.3c (2014-12-02)
+======================================
+
+* Change the default value for the `content_addr` option to use the HTTP
+  listener, as by default the HTTPS listener will be using a self-signed
+  certificate.
+
+Changes in synapse 0.5.3 (2014-11-27)
+=====================================
+
+* Fix bug that caused joining a remote room to fail if a single event was not
+  signed correctly.
+* Fix bug which caused servers to continuously try and fetch events from other
+  servers.
+
+Changes in synapse 0.5.2 (2014-11-26)
+=====================================
+
+Fix major bug that caused rooms to disappear from peoples initial sync.
+
+Changes in synapse 0.5.1 (2014-11-26)
+=====================================
+See UPGRADES.rst for specific instructions on how to upgrade.
+
+ * Fix bug where we served up an Event that did not match its signatures.
+ * Fix regression where we no longer correctly handled the case where a
+   homeserver receives an event for a room it doesn't recognise (but is in.)
+
+Changes in synapse 0.5.0 (2014-11-19)
+=====================================
+This release includes changes to the federation protocol and client-server API
+that is not backwards compatible.
+
+This release also changes the internal database schemas and so requires servers to
+drop their current history. See UPGRADES.rst for details.
+
+Homeserver:
+ * Add authentication and authorization to the federation protocol. Events are
+   now signed by their originating homeservers.
+ * Implement the new authorization model for rooms.
+ * Split out web client into a seperate repository: matrix-angular-sdk.
+ * Change the structure of PDUs.
+ * Fix bug where user could not join rooms via an alias containing 4-byte
+   UTF-8 characters.
+ * Merge concept of PDUs and Events internally.
+ * Improve logging by adding request ids to log lines.
+ * Implement a very basic room initial sync API.
+ * Implement the new invite/join federation APIs.
+
+Webclient:
+ * The webclient has been moved to a seperate repository.
+
+Changes in synapse 0.4.2 (2014-10-31)
+=====================================
+
+Homeserver:
+ * Fix bugs where we did not notify users of correct presence updates.
+ * Fix bug where we did not handle sub second event stream timeouts.
+
+Webclient:
+ * Add ability to click on messages to see JSON.
+ * Add ability to redact messages.
+ * Add ability to view and edit all room state JSON.
+ * Handle incoming redactions.
+ * Improve feedback on errors.
+ * Fix bugs in mobile CSS.
+ * Fix bugs with desktop notifications.
+
+Changes in synapse 0.4.1 (2014-10-17)
+=====================================
+Webclient:
+ * Fix bug with display of timestamps.
+
+Changes in synpase 0.4.0 (2014-10-17)
+=====================================
+This release includes changes to the federation protocol and client-server API
+that is not backwards compatible.
+
+The Matrix specification has been moved to a separate git repository:
+http://github.com/matrix-org/matrix-doc
+
+You will also need an updated syutil and config. See UPGRADES.rst.
+
+Homeserver:
+ * Sign federation transactions to assert strong identity over federation.
+ * Rename timestamp keys in PDUs and events from 'ts' and 'hsob_ts' to 'origin_server_ts'.
+
+
+Changes in synapse 0.3.4 (2014-09-25)
+=====================================
+This version adds support for using a TURN server. See docs/turn-howto.rst on
+how to set one up.
+
+Homeserver:
+ * Add support for redaction of messages.
+ * Fix bug where inviting a user on a remote home server could take up to
+   20-30s.
+ * Implement a get current room state API.
+ * Add support specifying and retrieving turn server configuration.
+
+Webclient:
+ * Add button to send messages to users from the home page.
+ * Add support for using TURN for VoIP calls.
+ * Show display name change messages.
+ * Fix bug where the client didn't get the state of a newly joined room
+   until after it has been refreshed.
+ * Fix bugs with tab complete.
+ * Fix bug where holding down the down arrow caused chrome to chew 100% CPU.
+ * Fix bug where desktop notifications occasionally used "Undefined" as the
+   display name.
+ * Fix more places where we sometimes saw room IDs incorrectly.
+ * Fix bug which caused lag when entering text in the text box.
+
+Changes in synapse 0.3.3 (2014-09-22)
+=====================================
+
+Homeserver:
+ * Fix bug where you continued to get events for rooms you had left.
+
+Webclient:
+ * Add support for video calls with basic UI.
+ * Fix bug where one to one chats were named after your display name rather
+   than the other person's.
+ * Fix bug which caused lag when typing in the textarea.
+ * Refuse to run on browsers we know won't work.
+ * Trigger pagination when joining new rooms.
+ * Fix bug where we sometimes didn't display invitations in recents.
+ * Automatically join room when accepting a VoIP call.
+ * Disable outgoing and reject incoming calls on browsers we don't support
+   VoIP in.
+ * Don't display desktop notifications for messages in the room you are
+   non-idle and speaking in.
+
+Changes in synapse 0.3.2 (2014-09-18)
+=====================================
+
+Webclient:
+ * Fix bug where an empty "bing words" list in old accounts didn't send
+   notifications when it should have done.
+
+Changes in synapse 0.3.1 (2014-09-18)
+=====================================
+This is a release to hotfix v0.3.0 to fix two regressions.
+
+Webclient:
+ * Fix a regression where we sometimes displayed duplicate events.
+ * Fix a regression where we didn't immediately remove rooms you were
+   banned in from the recents list.
+
+Changes in synapse 0.3.0 (2014-09-18)
+=====================================
+See UPGRADE for information about changes to the client server API, including
+breaking backwards compatibility with VoIP calls and registration API.
+
+Homeserver:
+ * When a user changes their displayname or avatar the server will now update 
+   all their join states to reflect this.
+ * The server now adds "age" key to events to indicate how old they are. This
+   is clock independent, so at no point does any server or webclient have to
+   assume their clock is in sync with everyone else.
+ * Fix bug where we didn't correctly pull in missing PDUs.
+ * Fix bug where prev_content key wasn't always returned.
+ * Add support for password resets.
+
+Webclient:
+ * Improve page content loading.
+ * Join/parts now trigger desktop notifications.
+ * Always show room aliases in the UI if one is present.
+ * No longer show user-count in the recents side panel.
+ * Add up & down arrow support to the text box for message sending to step
+   through your sent history.
+ * Don't display notifications for our own messages.
+ * Emotes are now formatted correctly in desktop notifications.
+ * The recents list now differentiates between public & private rooms.
+ * Fix bug where when switching between rooms the pagination flickered before
+   the view jumped to the bottom of the screen.
+ * Add bing word support.
+
+Registration API:
+ * The registration API has been overhauled to function like the login API. In
+   practice, this means registration requests must now include the following:
+   'type':'m.login.password'. See UPGRADE for more information on this.
+ * The 'user_id' key has been renamed to 'user' to better match the login API.
+ * There is an additional login type: 'm.login.email.identity'.
+ * The command client and web client have been updated to reflect these changes.
+
+Changes in synapse 0.2.3 (2014-09-12)
+=====================================
+
+Homeserver:
+ * Fix bug where we stopped sending events to remote home servers if a
+   user from that home server left, even if there were some still in the
+   room.
+ * Fix bugs in the state conflict resolution where it was incorrectly
+   rejecting events.
+
+Webclient:
+ * Display room names and topics.
+ * Allow setting/editing of room names and topics.
+ * Display information about rooms on the main page.
+ * Handle ban and kick events in real time.
+ * VoIP UI and reliability improvements.
+ * Add glare support for VoIP.
+ * Improvements to initial startup speed.
+ * Don't display duplicate join events.
+ * Local echo of messages.
+ * Differentiate sending and sent of local echo.
+ * Various minor bug fixes.
+
+Changes in synapse 0.2.2 (2014-09-06)
+=====================================
+
+Homeserver:
+ * When the server returns state events it now also includes the previous 
+   content.
+ * Add support for inviting people when creating a new room.
+ * Make the homeserver inform the room via `m.room.aliases` when a new alias
+   is added for a room.
+ * Validate `m.room.power_level` events.
+
+Webclient:
+ * Add support for captchas on registration.
+ * Handle `m.room.aliases` events.
+ * Asynchronously send messages and show a local echo.
+ * Inform the UI when a message failed to send.
+ * Only autoscroll on receiving a new message if the user was already at the 
+   bottom of the screen.
+ * Add support for ban/kick reasons.
+
+Changes in synapse 0.2.1 (2014-09-03)
+=====================================
+
+Homeserver:
+ * Added support for signing up with a third party id.
+ * Add synctl scripts.
+ * Added rate limiting.
+ * Add option to change the external address the content repo uses.
+ * Presence bug fixes.
+
+Webclient:
+ * Added support for signing up with a third party id.
+ * Added support for banning and kicking users.
+ * Added support for displaying and setting ops.
+ * Added support for room names.
+ * Fix bugs with room membership event display.
+
+Changes in synapse 0.2.0 (2014-09-02)
+=====================================
+This update changes many configuration options, updates the
+database schema and mandates SSL for server-server connections.
+
+Homeserver:
+ * Require SSL for server-server connections.
+ * Add SSL listener for client-server connections.
+ * Add ability to use config files.
+ * Add support for kicking/banning and power levels.
+ * Allow setting of room names and topics on creation.
+ * Change presence to include last seen time of the user.
+ * Change url path prefix to /_matrix/...
+ * Bug fixes to presence.
+
+Webclient:
+ * Reskin the CSS for registration and login.
+ * Various improvements to rooms CSS.
+ * Support changes in client-server API.
+ * Bug fixes to VOIP UI.
+ * Various bug fixes to handling of changes to room member list.
+
+Changes in synapse 0.1.2 (2014-08-29)
+=====================================
+
+Webclient:
+ * Add basic call state UI for VoIP calls.
+
+Changes in synapse 0.1.1 (2014-08-29)
+=====================================
+
+Homeserver:
+    * Fix bug that caused the event stream to not notify some clients about
+      changes.
+
 Changes in synapse 0.1.0 (2014-08-29)
 =====================================
 Presence has been reenabled in this release.

MANIFEST.in

@@ -1,3 +1,14 @@
+include synctl
+include LICENSE
+include VERSION
+include *.rst
+include demo/README
+
+recursive-include synapse/storage/schema *.sql
+
+recursive-include demo *.dh
+recursive-include demo *.py
+recursive-include demo *.sh
 recursive-include docs *
+recursive-include scripts *
 recursive-include tests *.py
-recursive-include synapse/persistence/schema *.sql

README.rst

@@ -1,65 +1,50 @@
-Quick Start
-===========
+Introduction
+============
 
 Matrix is an ambitious new ecosystem for open federated Instant Messaging and
-VoIP[1].  The basics you need to know to get up and running are:
+VoIP.  The basics you need to know to get up and running are:
 
-    - Chatrooms are distributed and do not exist on any single server.  Rooms 
-      can be found using names like ``#matrix:matrix.org`` or 
-      ``#test:localhost:8080`` or they can be ephemeral.
-    
-    - Matrix user IDs look like ``@matthew:matrix.org`` (although in the future
-      you will normally refer to yourself and others using a 3PID: email
-      address, phone number, etc rather than manipulating Matrix user IDs)
+- Everything in Matrix happens in a room.  Rooms are distributed and do not
+  exist on any single server.  Rooms can be located using convenience aliases 
+  like ``#matrix:matrix.org`` or ``#test:localhost:8448``.
+
+- Matrix user IDs look like ``@matthew:matrix.org`` (although in the future
+  you will normally refer to yourself and others using a 3PID: email
+  address, phone number, etc rather than manipulating Matrix user IDs)
 
 The overall architecture is::
 
-      client <----> homeserver <=================> homeserver <-----> client
-                e.g. matrix.org:8080        e.g. mydomain.net:8080
+      client <----> homeserver <=====================> homeserver <----> client
+             https://somewhere.org/_matrix      https://elsewhere.net/_matrix
 
-To get up and running:
-      
-    - To simply play with an **existing** homeserver you can
-      just go straight to http://matrix.org/alpha.
-    
-    - To run your own **private** homeserver on localhost:8080, install synapse 
-      with ``python setup.py develop --user`` and then run one with
-      ``python synapse/app/homeserver.py`` - you will find a webclient running
-      at http://localhost:8080 (use a recent Chrome, Safari or Firefox for now,
-      please...)
-             
-    - To make the homeserver **public** and let it exchange messages with 
-      other homeservers and participate in the overall Matrix federation, open 
-      up port 8080 and run ``python synapse/app/homeserver.py --host 
-      machine.my.domain.name``.  Then come join ``#matrix:matrix.org`` and
-      say hi! :)
+``#matrix:matrix.org`` is the official support room for Matrix, and can be
+accessed by the web client at http://matrix.org/alpha or via an IRC bridge at
+irc://irc.freenode.net/matrix.
 
-For more detailed setup instructions, please see further down this document.
+Synapse is currently in rapid development, but as of version 0.5 we believe it 
+is sufficiently stable to be run as an internet-facing service for real usage!
 
-[1] VoIP currently in development
-
-   
 About Matrix
 ============
 
 Matrix specifies a set of pragmatic RESTful HTTP JSON APIs as an open standard,
 which handle:
 
-    - Creating and managing fully distributed chat rooms with no
-      single points of control or failure
-    - Eventually-consistent cryptographically secure[2] synchronisation of room 
-      state across a global open network of federated servers and services
-    - Sending and receiving extensible messages in a room with (optional)
-      end-to-end encryption[3]
-    - Inviting, joining, leaving, kicking, banning room members
-    - Managing user accounts (registration, login, logout)
-    - Using 3rd Party IDs (3PIDs) such as email addresses, phone numbers,
-      Facebook accounts to authenticate, identify and discover users on Matrix.
-    - Placing 1:1 VoIP and Video calls (in development)
+- Creating and managing fully distributed chat rooms with no
+  single points of control or failure
+- Eventually-consistent cryptographically secure synchronisation of room
+  state across a global open network of federated servers and services
+- Sending and receiving extensible messages in a room with (optional)
+  end-to-end encryption[1]
+- Inviting, joining, leaving, kicking, banning room members
+- Managing user accounts (registration, login, logout)
+- Using 3rd Party IDs (3PIDs) such as email addresses, phone numbers,
+  Facebook accounts to authenticate, identify and discover users on Matrix.
+- Placing 1:1 VoIP and Video calls
 
 These APIs are intended to be implemented on a wide range of servers, services
-and clients, letting developers build messaging and VoIP functionality on top of
-the entirely open Matrix ecosystem rather than using closed or proprietary
+and clients, letting developers build messaging and VoIP functionality on top
+of the entirely open Matrix ecosystem rather than using closed or proprietary
 solutions. The hope is for Matrix to act as the building blocks for a new
 generation of fully open and interoperable messaging and VoIP apps for the
 internet.
@@ -74,65 +59,222 @@ In Matrix, every user runs one or more Matrix clients, which connect through to
 a Matrix homeserver which stores all their personal chat history and user
 account information - much as a mail client connects through to an IMAP/SMTP
 server. Just like email, you can either run your own Matrix homeserver and
-control and own your own communications and history or use one hosted by someone
-else (e.g. matrix.org) - there is no single point of control or mandatory
-service provider in Matrix, unlike WhatsApp, Facebook, Hangouts, etc.
+control and own your own communications and history or use one hosted by
+someone else (e.g. matrix.org) - there is no single point of control or
+mandatory service provider in Matrix, unlike WhatsApp, Facebook, Hangouts, etc.
 
 Synapse ships with two basic demo Matrix clients: webclient (a basic group chat
 web client demo implemented in AngularJS) and cmdclient (a basic Python
-commandline utility which lets you easily see what the JSON APIs are up to).
+command line utility which lets you easily see what the JSON APIs are up to).
 
-We'd like to invite you to take a look at the Matrix spec, try to run a
-homeserver, and join the existing Matrix chatrooms already out there, experiment
-with the APIs and the demo clients, and let us know your thoughts at
-https://github.com/matrix-org/synapse/issues or at matrix@matrix.org.
+Meanwhile, iOS and Android SDKs and clients are currently in development and available from:
 
-Thanks for trying Matrix!
+- https://github.com/matrix-org/matrix-ios-sdk
+- https://github.com/matrix-org/matrix-android-sdk
 
-[2] Cryptographic signing of messages isn't turned on yet
+We'd like to invite you to join #matrix:matrix.org (via http://matrix.org/alpha), run a homeserver, take a look at the Matrix spec at
+http://matrix.org/docs/spec, experiment with the APIs and the demo
+clients, and report any bugs via http://matrix.org/jira.
 
-[3] End-to-end encryption is currently in development
+Thanks for using Matrix!
 
+[1] End-to-end encryption is currently in development
 
 Homeserver Installation
 =======================
 
-First, the dependencies need to be installed.  Start by installing 
-'python2.7-dev' and the various tools of the compiler toolchain.
-N.B. synapse requires python 2.x where x >= 7
+System requirements:
+- POSIX-compliant system (tested on Linux & OSX)
+- Python 2.7
 
-  Installing prerequisites on ubuntu::
+Synapse is written in python but some of the libraries is uses are written in
+C. So before we can install synapse itself we need a working C compiler and the
+header files for python C extensions.
 
-    $ sudo apt-get install build-essential python2.7-dev libffi-dev
+Installing prerequisites on Ubuntu or Debian::
 
-  Installing prerequisites on Mac OS X::
+    $ sudo apt-get install build-essential python2.7-dev libffi-dev \
+                           python-pip python-setuptools sqlite3 \
+                           libssl-dev python-virtualenv libjpeg-dev
+                           
+Installing prerequisites on ArchLinux::
+
+    $ sudo pacman -S base-devel python2 python-pip \
+                     python-setuptools python-virtualenv sqlite3
+
+Installing prerequisites on Mac OS X::
 
     $ xcode-select --install
+    $ sudo pip install virtualenv
+    
+To install the synapse homeserver run::
+
+    $ virtualenv ~/.synapse
+    $ source ~/.synapse/bin/activate
+    $ pip install --process-dependency-links https://github.com/matrix-org/synapse/tarball/master
+
+This installs synapse, along with the libraries it uses, into a virtual
+environment under ``~/.synapse``.
+
+To set up your homeserver, run (in your virtualenv, as before)::
+
+    $ cd ~/.synapse
+    $ python -m synapse.app.homeserver \
+        --server-name machine.my.domain.name \
+        --config-path homeserver.yaml \
+        --generate-config
+
+Substituting your host and domain name as appropriate.
+
+For reliable VoIP calls to be routed via this homeserver, you MUST configure
+a TURN server.  See docs/turn-howto.rst for details.
+
+Troubleshooting Installation
+----------------------------
+
+Synapse requires pip 1.7 or later, so if your OS provides too old a version and 
+you get errors about ``error: no such option: --process-dependency-links`` you 
+may need to manually upgrade it::
+
+    $ sudo pip install --upgrade pip
+
+If pip crashes mid-installation for reason (e.g. lost terminal), pip may
+refuse to run until you remove the temporary installation directory it
+created. To reset the installation::
+
+    $ rm -rf /tmp/pip_install_matrix
+
+pip seems to leak *lots* of memory during installation.  For instance, a Linux 
+host with 512MB of RAM may run out of memory whilst installing Twisted.  If this 
+happens, you will have to individually install the dependencies which are 
+failing, e.g.::
+
+    $ pip install twisted
+
+On OSX, if you encounter clang: error: unknown argument: '-mno-fused-madd' you
+will need to export CFLAGS=-Qunused-arguments.
+
+ArchLinux
+---------
+
+Installation on ArchLinux may encounter a few hiccups as Arch defaults to
+python 3, but synapse currently assumes python 2.7 by default.
+
+pip may be outdated (6.0.7-1 and needs to be upgraded to 6.0.8-1 )::
+
+    $ sudo pip2.7 install --upgrade pip
+    
+You also may need to explicitly specify python 2.7 again during the install
+request::
+
+    $ pip2.7 install --process-dependency-links \
+        https://github.com/matrix-org/synapse/tarball/master
+    
+If you encounter an error with lib bcrypt causing an Wrong ELF Class:
+ELFCLASS32 (x64 Systems), you may need to reinstall py-bcrypt to correctly
+compile it under the right architecture. (This should not be needed if
+installing under virtualenv)::
+
+    $ sudo pip2.7 uninstall py-bcrypt
+    $ sudo pip2.7 install py-bcrypt
+    
+During setup of homeserver you need to call python2.7 directly again::
+
+    $ cd ~/.synapse
+    $ python2.7 -m synapse.app.homeserver \
+      --server-name machine.my.domain.name \
+      --config-path homeserver.yaml \
+      --generate-config
+        
+...substituting your host and domain name as appropriate.
+
+Windows Install
+---------------
+Synapse can be installed on Cygwin. It requires the following Cygwin packages:
+
+ - gcc
+ - git
+ - libffi-devel
+ - openssl (and openssl-devel, python-openssl)
+ - python
+ - python-setuptools
+
+The content repository requires additional packages and will be unable to process
+uploads without them:
+ - libjpeg8
+ - libjpeg8-devel
+ - zlib
+If you choose to install Synapse without these packages, you will need to reinstall
+``pillow`` for changes to be applied, e.g. ``pip uninstall pillow`` ``pip install
+pillow --user``
+
+Troubleshooting:
+
+- You may need to upgrade ``setuptools`` to get this to work correctly:
+  ``pip install setuptools --upgrade``.
+- You may encounter errors indicating that ``ffi.h`` is missing, even with
+  ``libffi-devel`` installed. If you do, copy the ``.h`` files:
+  ``cp /usr/lib/libffi-3.0.13/include/*.h /usr/include``
+- You may need to install libsodium from source in order to install PyNacl. If
+  you do, you may need to create a symlink to ``libsodium.a`` so ``ld`` can find
+  it: ``ln -s /usr/local/lib/libsodium.a /usr/lib/libsodium.a``
+
+Running Your Homeserver
+=======================
+
+To actually run your new homeserver, pick a working directory for Synapse to run 
+(e.g. ``~/.synapse``), and::
+
+    $ cd ~/.synapse
+    $ source ./bin/activate
+    $ synctl start
+
+Troubleshooting Running
+-----------------------
+
+If synapse fails with ``missing "sodium.h"`` crypto errors, you may need 
+to manually upgrade PyNaCL, as synapse uses NaCl (http://nacl.cr.yp.to/) for 
+encryption and digital signatures.
+Unfortunately PyNACL currently has a few issues
+(https://github.com/pyca/pynacl/issues/53) and
+(https://github.com/pyca/pynacl/issues/79) that mean it may not install
+correctly, causing all tests to fail with errors about missing "sodium.h". To
+fix try re-installing from PyPI or directly from
+(https://github.com/pyca/pynacl)::
+
+    $ # Install from PyPI
+    $ pip install --user --upgrade --force pynacl
+    $ # Install from github
+    $ pip install --user https://github.com/pyca/pynacl/tarball/master
+
+ArchLinux
+---------
+
+If running `$ synctl start` fails wit 'returned non-zero exit status 1', you will need to explicitly call Python2.7 - either running as::
+
+    $ python2.7 -m synapse.app.homeserver --daemonize -c homeserver.yaml --pid-file homeserver.pid
+    
+...or by editing synctl with the correct python executable.
+
+Homeserver Development
+======================
+
+To check out a homeserver for development, clone the git repo into a working
+directory of your choice::
+
+    $ git clone https://github.com/matrix-org/synapse.git
+    $ cd synapse
 
 The homeserver has a number of external dependencies, that are easiest
-to install by making setup.py do so, in --user mode::
+to install using pip and a virtualenv::
 
-    $ python setup.py develop --user
-    
-You'll need a version of setuptools new enough to know about git, so you
-may need to also run:
+    $ virtualenv env
+    $ source env/bin/activate
+    $ python synapse/python_dependencies.py | xargs -n1 pip install
+    $ pip install setuptools_trial mock
 
-    $ sudo apt-get install python-pip
-    $ sudo pip install --upgrade setuptools
-    
-If you don't have access to github, then you may need to install ``syutil``
-manually by checking it out and running ``python setup.py develop --user`` on it
-too.
-    
-If you get errors about ``sodium.h`` being missing, you may also need to
-manually install a newer PyNaCl via pip as setuptools installs an old one. Or
-you can check PyNaCl out of git directly (https://github.com/pyca/pynacl) and
-installing it. Installing PyNaCl using pip may also work (remember to remove any
-other versions installed by setuputils in, for example, ~/.local/lib).
-
-This will run a process of downloading and installing into your
-user's .local/lib directory all of the required dependencies that are
-missing.
+This will run a process of downloading and installing all the needed
+dependencies into a virtual env.
 
 Once this is done, you may wish to run the homeserver's unit tests, to
 check that everything is installed as it should be::
@@ -149,9 +291,14 @@ This should end with a 'PASSED' result::
 Upgrading an existing homeserver
 ================================
 
-Before upgrading an existing homeserver to a new version, please refer to
-UPGRADE.rst for any additional instructions.
- 
+IMPORTANT: Before upgrading an existing homeserver to a new version, please
+refer to UPGRADE.rst for any additional instructions.
+
+Otherwise, simply re-install the new codebase over the current one - e.g.
+by ``pip install --process-dependency-links
+https://github.com/matrix-org/synapse/tarball/master``
+if using pip, or by ``git pull`` if running off a git working copy.
+
 
 Setting up Federation
 =====================
@@ -161,19 +308,25 @@ be publicly visible on the internet, and they will need to know its host name.
 You have two choices here, which will influence the form of your Matrix user
 IDs:
 
- 1) Use the machine's own hostname as available on public DNS in the form of its
-    A or AAAA records. This is easier to set up initially, perhaps for testing,
-    but lacks the flexibility of SRV.
+1) Use the machine's own hostname as available on public DNS in the form of
+   its A or AAAA records. This is easier to set up initially, perhaps for
+   testing, but lacks the flexibility of SRV.
 
- 2) Set up a SRV record for your domain name. This requires you create a SRV
-    record in DNS, but gives the flexibility to run the server on your own
-    choice of TCP port, on a machine that might not be the same name as the
-    domain name.
+2) Set up a SRV record for your domain name. This requires you create a SRV
+   record in DNS, but gives the flexibility to run the server on your own
+   choice of TCP port, on a machine that might not be the same name as the
+   domain name.
 
 For the first form, simply pass the required hostname (of the machine) as the
---host parameter::
+--server-name parameter::
 
-    $ python synapse/app/homeserver.py --host machine.my.domain.name
+    $ python -m synapse.app.homeserver \
+        --server-name machine.my.domain.name \
+        --config-path homeserver.yaml \
+        --generate-config
+    $ python -m synapse.app.homeserver --config-path homeserver.yaml
+
+Alternatively, you can run ``synctl start`` to guide you through the process.
 
 For the second form, first create your SRV record and publish it in DNS. This
 needs to be named _matrix._tcp.YOURDOMAIN, and point at at least one hostname
@@ -181,12 +334,20 @@ and port where the server is running.  (At the current time synapse does not
 support clustering multiple servers into a single logical homeserver).  The DNS
 record would then look something like::
 
+    $ dig -t srv _matrix._tcp.machine.my.domaine.name
     _matrix._tcp    IN      SRV     10 0 8448 machine.my.domain.name.
 
+
 At this point, you should then run the homeserver with the hostname of this
 SRV record, as that is the name other machines will expect it to have::
 
-    $ python synapse/app/homeserver.py --host my.domain.name --port 8448
+    $ python -m synapse.app.homeserver \
+        --server-name YOURDOMAIN \
+        --bind-port 8448 \
+        --config-path homeserver.yaml \
+        --generate-config
+    $ python -m synapse.app.homeserver --config-path homeserver.yaml
+
 
 You may additionally want to pass one or more "-v" options, in order to
 increase the verbosity of logging output; at least for initial testing.
@@ -204,11 +365,13 @@ private federation (``localhost:8080``, ``localhost:8081`` and
 http://localhost:8080. Simply run::
 
     $ demo/start.sh
+    
+This is mainly useful just for development purposes.
 
 Running The Demo Web Client
 ===========================
 
-The homeserver runs a web client by default at http://localhost:8080.
+The homeserver runs a web client by default at https://localhost:8448/.
 
 If this is the first time you have used the client from that browser (it uses
 HTML5 local storage to remember its config), you will need to log in to your
@@ -228,8 +391,8 @@ account. Your name will take the form of::
 
 Specify your desired localpart in the topmost box of the "Register for an
 account" form, and click the "Register" button. Hostnames can contain ports if
-required due to lack of SRV records (e.g. @matthew:localhost:8080 on an internal
-synapse sandbox running on localhost)
+required due to lack of SRV records (e.g. @matthew:localhost:8448 on an
+internal synapse sandbox running on localhost)
 
 
 Logging In To An Existing Account
@@ -244,9 +407,9 @@ Identity Servers
 
 The job of authenticating 3PIDs and tracking which 3PIDs are associated with a
 given Matrix user is very security-sensitive, as there is obvious risk of spam
-if it is too easy to sign up for Matrix accounts or harvest 3PID data. Meanwhile
-the job of publishing the end-to-end encryption public keys for Matrix users is
-also very security-sensitive for similar reasons.
+if it is too easy to sign up for Matrix accounts or harvest 3PID data.
+Meanwhile the job of publishing the end-to-end encryption public keys for
+Matrix users is also very security-sensitive for similar reasons.
 
 Therefore the role of managing trusted identity in the Matrix ecosystem is
 farmed out to a cluster of known trusted ecosystem partners, who run 'Matrix
@@ -255,19 +418,21 @@ track 3PID logins and publish end-user public keys.
 
 It's currently early days for identity servers as Matrix is not yet using 3PIDs
 as the primary means of identity and E2E encryption is not complete. As such,
-we're not yet running an identity server in public.
+we are running a single identity server (http://matrix.org:8090) at the current
+time.
 
 
 Where's the spec?!
 ==================
 
-For now, please go spelunking in the ``docs/`` directory to find out.
+The source of the matrix spec lives at https://github.com/matrix-org/matrix-doc.  
+A recent HTML snapshot of this lives at http://matrix.org/docs/spec
 
 
 Building Internal API Documentation
 ===================================
 
-Before building internal API documentation install spinx and
+Before building internal API documentation install sphinx and
 sphinxcontrib-napoleon::
 
     $ pip install sphinx

UPGRADE.rst

@@ -1,3 +1,167 @@
+Upgrading to v0.8.0
+===================
+
+Servers which use captchas will need to add their public key to::
+
+  static/client/register/register_config.js
+
+    window.matrixRegistrationConfig = {
+        recaptcha_public_key: "YOUR_PUBLIC_KEY"
+    };
+
+This is required in order to support registration fallback (typically used on
+mobile devices).
+
+
+Upgrading to v0.7.0
+===================
+
+New dependencies are:
+
+- pydenticon
+- simplejson
+- syutil
+- matrix-angular-sdk
+
+To pull in these dependencies in a virtual env, run::
+
+    python synapse/python_dependencies.py | xargs -n 1 pip install
+
+Upgrading to v0.6.0
+===================
+
+To pull in new dependencies, run::
+
+    python setup.py develop --user
+
+This update includes a change to the database schema. To upgrade you first need
+to upgrade the database by running::
+
+    python scripts/upgrade_db_to_v0.6.0.py <db> <server_name> <signing_key>
+
+Where `<db>` is the location of the database, `<server_name>` is the
+server name as specified in the synapse configuration, and `<signing_key>` is
+the location of the signing key as specified in the synapse configuration.
+
+This may take some time to complete. Failures of signatures and content hashes
+can safely be ignored.
+
+
+Upgrading to v0.5.1
+===================
+
+Depending on precisely when you installed v0.5.0 you may have ended up with
+a stale release of the reference matrix webclient installed as a python module.
+To uninstall it and ensure you are depending on the latest module, please run::
+
+    $ pip uninstall syweb
+
+Upgrading to v0.5.0
+===================
+
+The webclient has been split out into a seperate repository/pacakage in this
+release. Before you restart your homeserver you will need to pull in the
+webclient package by running::
+
+  python setup.py develop --user
+
+This release completely changes the database schema and so requires upgrading
+it before starting the new version of the homeserver.
+
+The script "database-prepare-for-0.5.0.sh" should be used to upgrade the
+database. This will save all user information, such as logins and profiles, 
+but will otherwise purge the database. This includes messages, which
+rooms the home server was a member of and room alias mappings.
+
+If you would like to keep your history, please take a copy of your database
+file and ask for help in #matrix:matrix.org. The upgrade process is,
+unfortunately, non trivial and requires human intervention to resolve any
+resulting conflicts during the upgrade process.
+
+Before running the command the homeserver should be first completely 
+shutdown. To run it, simply specify the location of the database, e.g.:
+
+  ./scripts/database-prepare-for-0.5.0.sh "homeserver.db"
+
+Once this has successfully completed it will be safe to restart the 
+homeserver. You may notice that the homeserver takes a few seconds longer to 
+restart than usual as it reinitializes the database.
+
+On startup of the new version, users can either rejoin remote rooms using room
+aliases or by being reinvited. Alternatively, if any other homeserver sends a
+message to a room that the homeserver was previously in the local HS will 
+automatically rejoin the room.
+
+Upgrading to v0.4.0
+===================
+
+This release needs an updated syutil version. Run::
+
+    python setup.py develop
+
+You will also need to upgrade your configuration as the signing key format has
+changed. Run::
+
+    python -m synapse.app.homeserver --config-path <CONFIG> --generate-config
+
+
+Upgrading to v0.3.0
+===================
+
+This registration API now closely matches the login API. This introduces a bit
+more backwards and forwards between the HS and the client, but this improves
+the overall flexibility of the API. You can now GET on /register to retrieve a list
+of valid registration flows. Upon choosing one, they are submitted in the same
+way as login, e.g::
+
+  {
+    type: m.login.password,
+    user: foo,
+    password: bar
+  }
+
+The default HS supports 2 flows, with and without Identity Server email
+authentication. Enabling captcha on the HS will add in an extra step to all
+flows: ``m.login.recaptcha`` which must be completed before you can transition
+to the next stage. There is a new login type: ``m.login.email.identity`` which
+contains the ``threepidCreds`` key which were previously sent in the original
+register request. For more information on this, see the specification.
+
+Web Client
+----------
+
+The VoIP specification has changed between v0.2.0 and v0.3.0. Users should
+refresh any browser tabs to get the latest web client code. Users on
+v0.2.0 of the web client will not be able to call those on v0.3.0 and
+vice versa.
+
+
+Upgrading to v0.2.0
+===================
+
+The home server now requires setting up of SSL config before it can run. To
+automatically generate default config use::
+
+    $ python synapse/app/homeserver.py \
+        --server-name machine.my.domain.name \
+        --bind-port 8448 \
+        --config-path homeserver.config \
+        --generate-config
+
+This config can be edited if desired, for example to specify a different SSL 
+certificate to use. Once done you can run the home server using::
+
+    $ python synapse/app/homeserver.py --config-path homeserver.config
+
+See the README.rst for more information.
+
+Also note that some config options have been renamed, including:
+
+- "host" to "server-name"
+- "database" to "database-path"
+- "port" to "bind-port" and "unsecure-port"
+
+
 Upgrading to v0.0.1
 ===================
 
@@ -12,7 +176,7 @@ rooms the home server was a member of and room alias mappings.
 Before running the command the homeserver should be first completely 
 shutdown. To run it, simply specify the location of the database, e.g.:
 
-  ./database-prepare-for-0.0.1.sh "homeserver.db"
+  ./scripts/database-prepare-for-0.0.1.sh "homeserver.db"
 
 Once this has successfully completed it will be safe to restart the 
 homeserver. You may notice that the homeserver takes a few seconds longer to 

VERSION

@@ -1 +0,0 @@
-0.1.0

contrib/cmdclient/console.py

@@ -1,6 +1,6 @@
 #!/usr/bin/env python
 
-# Copyright 2014 matrix.org
+# Copyright 2014 OpenMarket Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -60,7 +60,7 @@ class SynapseCmd(cmd.Cmd):
             "complete_usernames": "on",
             "send_delivery_receipts": "on"
         }
-        self.path_prefix = "/matrix/client/api/v1"
+        self.path_prefix = "/_matrix/client/api/v1"
         self.event_stream_token = "END"
         self.prompt = ">>> "
 
@@ -88,6 +88,8 @@ class SynapseCmd(cmd.Cmd):
         return False
 
     def _domain(self):
+        if "user" not in self.config or not self.config["user"]:
+            return None
         return self.config["user"].split(":")[1]
 
     def do_config(self, line):
@@ -143,35 +145,50 @@ class SynapseCmd(cmd.Cmd):
         <noupdate> : Do not automatically clobber config values.
         """
         args = self._parse(line, ["userid", "noupdate"])
-        path = "/register"
 
         password = None
         pwd = None
         pwd2 = "_"
         while pwd != pwd2:
-            pwd = getpass.getpass("(Optional) Type a password for this user: ")
-            if len(pwd) == 0:
-                print "Not using a password for this user."
-                break
+            pwd = getpass.getpass("Type a password for this user: ")
             pwd2 = getpass.getpass("Retype the password: ")
-            if pwd != pwd2:
+            if pwd != pwd2 or len(pwd) == 0:
                 print "Password mismatch."
+                pwd = None
             else:
                 password = pwd
 
-        body = {}
+        body = {
+            "type": "m.login.password"
+        }
         if "userid" in args:
-            body["user_id"] = args["userid"]
+            body["user"] = args["userid"]
         if password:
             body["password"] = password
 
-        reactor.callFromThread(self._do_register, "POST", path, body,
+        reactor.callFromThread(self._do_register, body,
                                "noupdate" not in args)
 
     @defer.inlineCallbacks
-    def _do_register(self, method, path, data, update_config):
-        url = self._url() + path
-        json_res = yield self.http_client.do_request(method, url, data=data)
+    def _do_register(self, data, update_config):
+        # check the registration flows
+        url = self._url() + "/register"
+        json_res = yield self.http_client.do_request("GET", url)
+        print json.dumps(json_res, indent=4)
+
+        passwordFlow = None
+        for flow in json_res["flows"]:
+            if flow["type"] == "m.login.recaptcha" or ("stages" in flow and "m.login.recaptcha" in flow["stages"]):
+                print "Unable to register: Home server requires captcha."
+                return
+            if flow["type"] == "m.login.password" and "stages" not in flow:
+                passwordFlow = flow
+                break
+
+        if not passwordFlow:
+            return
+
+        json_res = yield self.http_client.do_request("POST", url, data=data)
         print json.dumps(json_res, indent=4)
         if update_config and "user_id" in json_res:
             self.config["user"] = json_res["user_id"]
@@ -191,10 +208,12 @@ class SynapseCmd(cmd.Cmd):
                 p = getpass.getpass("Enter your password: ")
                 user = args["user_id"]
                 if self._is_on("complete_usernames") and not user.startswith("@"):
-                    user = "@" + user + ":" + self._domain()
-
+                    domain = self._domain()
+                    if domain:
+                        user = "@" + user + ":" + domain
+                
                 reactor.callFromThread(self._do_login, user, p)
-                print " got %s " % p
+                #print " got %s " % p
         except Exception as e:
             print e
 
@@ -252,7 +271,7 @@ class SynapseCmd(cmd.Cmd):
 
     @defer.inlineCallbacks
     def _do_emailrequest(self, args):
-        url = self._identityServerUrl()+"/matrix/identity/api/v1/validate/email/requestToken"
+        url = self._identityServerUrl()+"/_matrix/identity/api/v1/validate/email/requestToken"
 
         json_res = yield self.http_client.do_request("POST", url, data=urllib.urlencode(args), jsonreq=False,
                                                      headers={'Content-Type': ['application/x-www-form-urlencoded']})
@@ -274,7 +293,7 @@ class SynapseCmd(cmd.Cmd):
 
     @defer.inlineCallbacks
     def _do_emailvalidate(self, args):
-        url = self._identityServerUrl()+"/matrix/identity/api/v1/validate/email/submitToken"
+        url = self._identityServerUrl()+"/_matrix/identity/api/v1/validate/email/submitToken"
 
         json_res = yield self.http_client.do_request("POST", url, data=urllib.urlencode(args), jsonreq=False,
                                                      headers={'Content-Type': ['application/x-www-form-urlencoded']})
@@ -294,7 +313,7 @@ class SynapseCmd(cmd.Cmd):
 
     @defer.inlineCallbacks
     def _do_3pidbind(self, args):
-        url = self._identityServerUrl()+"/matrix/identity/api/v1/3pid/bind"
+        url = self._identityServerUrl()+"/_matrix/identity/api/v1/3pid/bind"
 
         json_res = yield self.http_client.do_request("POST", url, data=urllib.urlencode(args), jsonreq=False,
                                                      headers={'Content-Type': ['application/x-www-form-urlencoded']})
@@ -312,7 +331,7 @@ class SynapseCmd(cmd.Cmd):
         try:
             args = self._parse(line, ["roomname"], force_keys=True)
             path = "/join/%s" % urllib.quote(args["roomname"])
-            reactor.callFromThread(self._run_and_pprint, "PUT", path, {})
+            reactor.callFromThread(self._run_and_pprint, "POST", path, {})
         except Exception as e:
             print e
 
@@ -360,14 +379,14 @@ class SynapseCmd(cmd.Cmd):
     def _do_invite(self, roomid, userstring):
         if (not userstring.startswith('@') and
                     self._is_on("complete_usernames")):
-            url = self._identityServerUrl()+"/matrix/identity/api/v1/lookup"
+            url = self._identityServerUrl()+"/_matrix/identity/api/v1/lookup"
 
             json_res = yield self.http_client.do_request("GET", url, qparams={'medium':'email','address':userstring})
 
             mxid = None
 
             if 'mxid' in json_res and 'signatures' in json_res:
-                url = self._identityServerUrl()+"/matrix/identity/api/v1/pubkey/ed25519"
+                url = self._identityServerUrl()+"/_matrix/identity/api/v1/pubkey/ed25519"
 
                 pubKey = None
                 pubKeyObj = yield self.http_client.do_request("GET", url)
@@ -700,7 +719,7 @@ def main(server_url, identity_server_url, username, token, config_path):
 if __name__ == '__main__':
     parser = argparse.ArgumentParser("Starts a synapse client.")
     parser.add_argument(
-        "-s", "--server", dest="server", default="http://localhost:8080",
+        "-s", "--server", dest="server", default="http://localhost:8008",
         help="The URL of the home server to talk to.")
     parser.add_argument(
         "-i", "--identity-server", dest="identityserver", default="http://localhost:8090",

contrib/cmdclient/http.py

@@ -1,5 +1,5 @@
 # -*- coding: utf-8 -*-
-# Copyright 2014 matrix.org
+# Copyright 2014 OpenMarket Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.

contrib/experiments/cursesio.py

@@ -1,4 +1,4 @@
-# Copyright 2014 matrix.org
+# Copyright 2014 OpenMarket Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.

contrib/experiments/test_messaging.py

@@ -1,5 +1,5 @@
 # -*- coding: utf-8 -*-
-# Copyright 2014 matrix.org
+# Copyright 2014 OpenMarket Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.

contrib/graph/graph.py

@@ -1,4 +1,4 @@
-# Copyright 2014 matrix.org
+# Copyright 2014 OpenMarket Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -120,7 +120,7 @@ def make_graph(pdus, room, filename_prefix):
 def get_pdus(host, room):
     transaction = json.loads(
         urllib2.urlopen(
-            "http://%s/matrix/federation/v1/context/%s/" % (host, room)
+            "http://%s/_matrix/federation/v1/context/%s/" % (host, room)
         ).read()
     )
 

contrib/graph/graph2.py

@@ -0,0 +1,157 @@
+# Copyright 2014 OpenMarket Ltd
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+import sqlite3
+import pydot
+import cgi
+import json
+import datetime
+import argparse
+
+from synapse.events import FrozenEvent
+from synapse.util.frozenutils import unfreeze
+
+
+def make_graph(db_name, room_id, file_prefix, limit):
+    conn = sqlite3.connect(db_name)
+
+    sql = (
+        "SELECT json FROM event_json as j "
+        "INNER JOIN events as e ON e.event_id = j.event_id "
+        "WHERE j.room_id = ?"
+    )
+
+    args = [room_id]
+
+    if limit:
+        sql += (
+            " ORDER BY topological_ordering DESC, stream_ordering DESC "
+            "LIMIT ?"
+        )
+
+        args.append(limit)
+
+    c = conn.execute(sql, args)
+
+    events = [FrozenEvent(json.loads(e[0])) for e in c.fetchall()]
+
+    events.sort(key=lambda e: e.depth)
+
+    node_map = {}
+    state_groups = {}
+
+    graph = pydot.Dot(graph_name="Test")
+
+    for event in events:
+        c = conn.execute(
+            "SELECT state_group FROM event_to_state_groups "
+            "WHERE event_id = ?",
+            (event.event_id,)
+        )
+
+        res = c.fetchone()
+        state_group = res[0] if res else None
+
+        if state_group is not None:
+            state_groups.setdefault(state_group, []).append(event.event_id)
+
+        t = datetime.datetime.fromtimestamp(
+            float(event.origin_server_ts) / 1000
+        ).strftime('%Y-%m-%d %H:%M:%S,%f')
+
+        content = json.dumps(unfreeze(event.get_dict()["content"]))
+
+        label = (
+            "<"
+            "<b>%(name)s </b><br/>"
+            "Type: <b>%(type)s </b><br/>"
+            "State key: <b>%(state_key)s </b><br/>"
+            "Content: <b>%(content)s </b><br/>"
+            "Time: <b>%(time)s </b><br/>"
+            "Depth: <b>%(depth)s </b><br/>"
+            "State group: %(state_group)s<br/>"
+            ">"
+        ) % {
+            "name": event.event_id,
+            "type": event.type,
+            "state_key": event.get("state_key", None),
+            "content": cgi.escape(content, quote=True),
+            "time": t,
+            "depth": event.depth,
+            "state_group": state_group,
+        }
+
+        node = pydot.Node(
+            name=event.event_id,
+            label=label,
+        )
+
+        node_map[event.event_id] = node
+        graph.add_node(node)
+
+    for event in events:
+        for prev_id, _ in event.prev_events:
+            try:
+                end_node = node_map[prev_id]
+            except:
+                end_node = pydot.Node(
+                    name=prev_id,
+                    label="<<b>%s</b>>" % (prev_id,),
+                )
+
+                node_map[prev_id] = end_node
+                graph.add_node(end_node)
+
+            edge = pydot.Edge(node_map[event.event_id], end_node)
+            graph.add_edge(edge)
+
+    for group, event_ids in state_groups.items():
+        if len(event_ids) <= 1:
+            continue
+
+        cluster = pydot.Cluster(
+            str(group),
+            label="<State Group: %s>" % (str(group),)
+        )
+
+        for event_id in event_ids:
+            cluster.add_node(node_map[event_id])
+
+        graph.add_subgraph(cluster)
+
+    graph.write('%s.dot' % file_prefix, format='raw', prog='dot')
+    graph.write_svg("%s.svg" % file_prefix, prog='dot')
+
+if __name__ == "__main__":
+    parser = argparse.ArgumentParser(
+        description="Generate a PDU graph for a given room by talking "
+                    "to the given homeserver to get the list of PDUs. \n"
+                    "Requires pydot."
+    )
+    parser.add_argument(
+        "-p", "--prefix", dest="prefix",
+        help="String to prefix output files with",
+        default="graph_output"
+    )
+    parser.add_argument(
+        "-l", "--limit",
+        help="Only retrieve the last N events.",
+    )
+    parser.add_argument('db')
+    parser.add_argument('room')
+
+    args = parser.parse_args()
+
+    make_graph(args.db, args.room, args.prefix, args.limit)

contrib/jitsimeetbridge/jitsimeetbridge.py

@@ -0,0 +1,260 @@
+#!/usr/bin/env python
+
+"""
+This is an attempt at bridging matrix clients into a Jitis meet room via Matrix
+video call.  It uses hard-coded xml strings overg XMPP BOSH. It can display one
+of the streams from the Jitsi bridge until the second lot of SDP comes down and
+we set the remote SDP at which point the stream ends. Our video never gets to
+the bridge.
+
+Requires:
+npm install jquery jsdom 
+"""
+
+import gevent
+import grequests
+from BeautifulSoup import BeautifulSoup
+import json
+import urllib
+import subprocess
+import time
+
+#ACCESS_TOKEN="" #
+
+MATRIXBASE = 'https://matrix.org/_matrix/client/api/v1/'
+MYUSERNAME = '@davetest:matrix.org'
+
+HTTPBIND = 'https://meet.jit.si/http-bind'
+#HTTPBIND = 'https://jitsi.vuc.me/http-bind'
+#ROOMNAME = "matrix"
+ROOMNAME = "pibble"
+
+HOST="guest.jit.si"
+#HOST="jitsi.vuc.me"
+
+TURNSERVER="turn.guest.jit.si"
+#TURNSERVER="turn.jitsi.vuc.me"
+
+ROOMDOMAIN="meet.jit.si"
+#ROOMDOMAIN="conference.jitsi.vuc.me"
+
+class TrivialMatrixClient:
+    def __init__(self, access_token):
+        self.token = None
+        self.access_token = access_token
+
+    def getEvent(self):
+        while True:
+            url = MATRIXBASE+'events?access_token='+self.access_token+"&timeout=60000"
+            if self.token:
+                url += "&from="+self.token
+            req = grequests.get(url)
+            resps = grequests.map([req])
+            obj = json.loads(resps[0].content)
+            print "incoming from matrix",obj
+            if 'end' not in obj:
+                continue
+            self.token = obj['end']
+            if len(obj['chunk']):
+                return obj['chunk'][0]
+
+    def joinRoom(self, roomId):
+        url = MATRIXBASE+'rooms/'+roomId+'/join?access_token='+self.access_token
+        print url
+        headers={ 'Content-Type': 'application/json' }
+        req = grequests.post(url, headers=headers, data='{}')
+        resps = grequests.map([req])
+        obj = json.loads(resps[0].content)
+        print "response: ",obj
+
+    def sendEvent(self, roomId, evType, event):
+        url = MATRIXBASE+'rooms/'+roomId+'/send/'+evType+'?access_token='+self.access_token
+        print url
+        print json.dumps(event)
+        headers={ 'Content-Type': 'application/json' }
+        req = grequests.post(url, headers=headers, data=json.dumps(event))
+        resps = grequests.map([req])
+        obj = json.loads(resps[0].content)
+        print "response: ",obj
+
+
+
+xmppClients = {}
+
+
+def matrixLoop():
+    while True:
+        ev = matrixCli.getEvent()
+        print ev
+        if ev['type'] == 'm.room.member':
+            print 'membership event'
+            if ev['membership'] == 'invite' and ev['state_key'] == MYUSERNAME:
+                roomId = ev['room_id']
+                print "joining room %s" % (roomId)
+                matrixCli.joinRoom(roomId)
+        elif ev['type'] == 'm.room.message':
+            if ev['room_id'] in xmppClients:
+                print "already have a bridge for that user, ignoring"
+                continue
+            print "got message, connecting"
+            xmppClients[ev['room_id']] = TrivialXmppClient(ev['room_id'], ev['user_id'])
+            gevent.spawn(xmppClients[ev['room_id']].xmppLoop)
+        elif ev['type'] == 'm.call.invite':
+            print "Incoming call"
+            #sdp = ev['content']['offer']['sdp']
+            #print "sdp: %s" % (sdp)
+            #xmppClients[ev['room_id']] = TrivialXmppClient(ev['room_id'], ev['user_id'])
+            #gevent.spawn(xmppClients[ev['room_id']].xmppLoop)
+        elif ev['type'] == 'm.call.answer':
+            print "Call answered"
+            sdp = ev['content']['answer']['sdp']
+            if ev['room_id'] not in xmppClients:
+                print "We didn't have a call for that room"
+                continue
+            # should probably check call ID too
+            xmppCli = xmppClients[ev['room_id']]
+            xmppCli.sendAnswer(sdp)
+        elif ev['type'] == 'm.call.hangup':
+            if ev['room_id'] in xmppClients:
+                xmppClients[ev['room_id']].stop()
+                del xmppClients[ev['room_id']]
+
+class TrivialXmppClient:
+    def __init__(self, matrixRoom, userId):
+        self.rid = 0
+        self.matrixRoom = matrixRoom
+        self.userId = userId
+        self.running = True
+
+    def stop(self):
+        self.running = False
+
+    def nextRid(self):
+        self.rid += 1
+        return '%d' % (self.rid)
+
+    def sendIq(self, xml):
+        fullXml = "<body rid='%s' xmlns='http://jabber.org/protocol/httpbind' sid='%s'>%s</body>" % (self.nextRid(), self.sid, xml)
+        #print "\t>>>%s" % (fullXml)
+        return self.xmppPoke(fullXml)
+
+    def xmppPoke(self, xml):
+        headers = {'Content-Type': 'application/xml'}
+        req = grequests.post(HTTPBIND, verify=False, headers=headers, data=xml)
+        resps = grequests.map([req])
+        obj = BeautifulSoup(resps[0].content)
+        return obj
+
+    def sendAnswer(self, answer):
+        print "sdp from matrix client",answer
+        p = subprocess.Popen(['node', 'unjingle/unjingle.js', '--sdp'], stdin=subprocess.PIPE, stdout=subprocess.PIPE)
+        jingle, out_err = p.communicate(answer)
+        jingle = jingle % {
+            'tojid': self.callfrom,
+            'action': 'session-accept',
+            'initiator': self.callfrom,
+            'responder': self.jid,
+            'sid': self.callsid
+        }
+        print "answer jingle from sdp",jingle
+        res = self.sendIq(jingle)
+        print "reply from answer: ",res
+
+        self.ssrcs = {}
+        jingleSoup = BeautifulSoup(jingle)
+        for cont in jingleSoup.iq.jingle.findAll('content'):
+            if cont.description:
+                self.ssrcs[cont['name']] = cont.description['ssrc']
+        print "my ssrcs:",self.ssrcs
+
+        gevent.joinall([
+                gevent.spawn(self.advertiseSsrcs)
+        ])
+
+    def advertiseSsrcs(self):
+                time.sleep(7)
+        print "SSRC spammer started"
+        while self.running:
+            ssrcMsg = "<presence to='%(tojid)s' xmlns='jabber:client'><x xmlns='http://jabber.org/protocol/muc'/><c xmlns='http://jabber.org/protocol/caps' hash='sha-1' node='http://jitsi.org/jitsimeet' ver='0WkSdhFnAUxrz4ImQQLdB80GFlE='/><nick xmlns='http://jabber.org/protocol/nick'>%(nick)s</nick><stats xmlns='http://jitsi.org/jitmeet/stats'><stat name='bitrate_download' value='175'/><stat name='bitrate_upload' value='176'/><stat name='packetLoss_total' value='0'/><stat name='packetLoss_download' value='0'/><stat name='packetLoss_upload' value='0'/></stats><media xmlns='http://estos.de/ns/mjs'><source type='audio' ssrc='%(assrc)s' direction='sendre'/><source type='video' ssrc='%(vssrc)s' direction='sendre'/></media></presence>" % { 'tojid': "%s@%s/%s" % (ROOMNAME, ROOMDOMAIN, self.shortJid), 'nick': self.userId, 'assrc': self.ssrcs['audio'], 'vssrc': self.ssrcs['video'] }
+            res = self.sendIq(ssrcMsg)
+            print "reply from ssrc announce: ",res
+            time.sleep(10)
+
+
+
+    def xmppLoop(self):
+        self.matrixCallId = time.time()
+        res = self.xmppPoke("<body rid='%s' xmlns='http://jabber.org/protocol/httpbind' to='%s' xml:lang='en' wait='60' hold='1' content='text/xml; charset=utf-8' ver='1.6' xmpp:version='1.0' xmlns:xmpp='urn:xmpp:xbosh'/>" % (self.nextRid(), HOST))
+
+        print res
+        self.sid = res.body['sid']
+        print "sid %s" % (self.sid)
+
+        res = self.sendIq("<auth xmlns='urn:ietf:params:xml:ns:xmpp-sasl' mechanism='ANONYMOUS'/>")
+
+        res = self.xmppPoke("<body rid='%s' xmlns='http://jabber.org/protocol/httpbind' sid='%s' to='%s' xml:lang='en' xmpp:restart='true' xmlns:xmpp='urn:xmpp:xbosh'/>" % (self.nextRid(), self.sid, HOST))
+
+        res = self.sendIq("<iq type='set' id='_bind_auth_2' xmlns='jabber:client'><bind xmlns='urn:ietf:params:xml:ns:xmpp-bind'/></iq>")
+        print res
+
+        self.jid = res.body.iq.bind.jid.string
+        print "jid: %s" % (self.jid)
+        self.shortJid = self.jid.split('-')[0]
+
+        res = self.sendIq("<iq type='set' id='_session_auth_2' xmlns='jabber:client'><session xmlns='urn:ietf:params:xml:ns:xmpp-session'/></iq>")
+
+        #randomthing = res.body.iq['to']
+        #whatsitpart = randomthing.split('-')[0]
+
+        #print "other random bind thing: %s" % (randomthing)
+
+        # advertise preence to the jitsi room, with our nick
+        res = self.sendIq("<iq type='get' to='%s' xmlns='jabber:client' id='1:sendIQ'><services xmlns='urn:xmpp:extdisco:1'><service host='%s'/></services></iq><presence to='%s@%s/d98f6c40' xmlns='jabber:client'><x xmlns='http://jabber.org/protocol/muc'/><c xmlns='http://jabber.org/protocol/caps' hash='sha-1' node='http://jitsi.org/jitsimeet' ver='0WkSdhFnAUxrz4ImQQLdB80GFlE='/><nick xmlns='http://jabber.org/protocol/nick'>%s</nick></presence>" % (HOST, TURNSERVER, ROOMNAME, ROOMDOMAIN, self.userId))
+        self.muc = {'users': []}
+        for p in res.body.findAll('presence'):
+            u = {}
+            u['shortJid'] = p['from'].split('/')[1]
+            if p.c and p.c.nick:
+                u['nick'] = p.c.nick.string
+            self.muc['users'].append(u)
+        print "muc: ",self.muc
+
+        # wait for stuff
+        while True:
+            print "waiting..."
+            res = self.sendIq("")
+            print "got from stream: ",res
+            if res.body.iq:
+                jingles = res.body.iq.findAll('jingle')
+                if len(jingles):
+                    self.callfrom = res.body.iq['from']
+                    self.handleInvite(jingles[0])
+            elif 'type' in res.body and res.body['type'] == 'terminate':
+                self.running = False
+                del xmppClients[self.matrixRoom]
+                        return
+
+    def handleInvite(self, jingle):
+        self.initiator = jingle['initiator']
+        self.callsid = jingle['sid']
+        p = subprocess.Popen(['node', 'unjingle/unjingle.js', '--jingle'], stdin=subprocess.PIPE, stdout=subprocess.PIPE)
+        print "raw jingle invite",str(jingle)
+        sdp, out_err = p.communicate(str(jingle))
+        print "transformed remote offer sdp",sdp
+        inviteEvent = {
+            'offer': {
+                'type': 'offer',
+                'sdp': sdp
+            },
+            'call_id': self.matrixCallId,
+            'version': 0,
+            'lifetime': 30000
+        }
+        matrixCli.sendEvent(self.matrixRoom, 'm.call.invite', inviteEvent)
+
+matrixCli = TrivialMatrixClient(ACCESS_TOKEN)
+
+gevent.joinall([
+    gevent.spawn(matrixLoop)
+])
+

contrib/jitsimeetbridge/syweb-jitsi-conference.patch

@@ -0,0 +1,188 @@
+diff --git a/syweb/webclient/app/components/matrix/matrix-call.js b/syweb/webclient/app/components/matrix/matrix-call.js
+index 9fbfff0..dc68077 100644
+--- a/syweb/webclient/app/components/matrix/matrix-call.js
++++ b/syweb/webclient/app/components/matrix/matrix-call.js
+@@ -16,6 +16,45 @@ limitations under the License.
+ 
+ 'use strict';
+ 
++
++function sendKeyframe(pc) {
++    console.log('sendkeyframe', pc.iceConnectionState);
++    if (pc.iceConnectionState !== 'connected') return; // safe...
++    pc.setRemoteDescription(
++        pc.remoteDescription,
++        function () {
++            pc.createAnswer(
++                function (modifiedAnswer) {
++                    pc.setLocalDescription(
++                        modifiedAnswer,
++                        function () {
++                            // noop
++                        },
++                        function (error) {
++                            console.log('triggerKeyframe setLocalDescription failed', error);
++                            messageHandler.showError();
++                        }
++                    );
++                },
++                function (error) {
++                    console.log('triggerKeyframe createAnswer failed', error);
++                    messageHandler.showError();
++                }
++            );
++        },
++        function (error) {
++            console.log('triggerKeyframe setRemoteDescription failed', error);
++            messageHandler.showError();
++        }
++    );
++} 
++
++
++
++
++
++
++
+ var forAllVideoTracksOnStream = function(s, f) {
+     var tracks = s.getVideoTracks();
+     for (var i = 0; i < tracks.length; i++) {
+@@ -83,7 +122,7 @@ angular.module('MatrixCall', [])
+     }
+ 
+     // FIXME: we should prevent any calls from being placed or accepted before this has finished
+-    MatrixCall.getTurnServer();
++    //MatrixCall.getTurnServer();
+ 
+     MatrixCall.CALL_TIMEOUT = 60000;
+     MatrixCall.FALLBACK_STUN_SERVER = 'stun:stun.l.google.com:19302';
+@@ -132,6 +171,22 @@ angular.module('MatrixCall', [])
+         pc.onsignalingstatechange = function() { self.onSignallingStateChanged(); };
+         pc.onicecandidate = function(c) { self.gotLocalIceCandidate(c); };
+         pc.onaddstream = function(s) { self.onAddStream(s); };
++
++        var datachan = pc.createDataChannel('RTCDataChannel', {
++            reliable: false
++        });
++        console.log("data chan: "+datachan);
++        datachan.onopen = function() {
++            console.log("data channel open");
++        };
++        datachan.onmessage = function() {
++            console.log("data channel message");
++        };
++        pc.ondatachannel = function(event) {
++            console.log("have data channel");
++            event.channel.binaryType = 'blob';
++        };
++
+         return pc;
+     }
+ 
+@@ -200,6 +255,12 @@ angular.module('MatrixCall', [])
+         }, this.msg.lifetime - event.age);
+     };
+ 
++    MatrixCall.prototype.receivedInvite = function(event) {
++        console.log("Got second invite for call "+this.call_id);
++        this.peerConn.setRemoteDescription(new RTCSessionDescription(this.msg.offer), this.onSetRemoteDescriptionSuccess, this.onSetRemoteDescriptionError);
++    };
++    
++
+     // perverse as it may seem, sometimes we want to instantiate a call with a hangup message
+     // (because when getting the state of the room on load, events come in reverse order and
+     // we want to remember that a call has been hung up)
+@@ -349,7 +410,7 @@ angular.module('MatrixCall', [])
+             'mandatory': {
+                 'OfferToReceiveAudio': true,
+                 'OfferToReceiveVideo': this.type == 'video'
+-            },
++            }
+         };
+         this.peerConn.createAnswer(function(d) { self.createdAnswer(d); }, function(e) {}, constraints);
+         // This can't be in an apply() because it's called by a predecessor call under glare conditions :(
+@@ -359,8 +420,20 @@ angular.module('MatrixCall', [])
+     MatrixCall.prototype.gotLocalIceCandidate = function(event) {
+         if (event.candidate) {
+             console.log("Got local ICE "+event.candidate.sdpMid+" candidate: "+event.candidate.candidate);
+-            this.sendCandidate(event.candidate);
+-        }
++            //this.sendCandidate(event.candidate);
++        } else {
++            console.log("have all candidates, sending answer");
++            var content = {
++                version: 0,
++                call_id: this.call_id,
++                answer: this.peerConn.localDescription
++            };
++            this.sendEventWithRetry('m.call.answer', content);
++            var self = this;
++            $rootScope.$apply(function() {
++                self.state = 'connecting';
++            });
++	}
+     }
+ 
+     MatrixCall.prototype.gotRemoteIceCandidate = function(cand) {
+@@ -418,15 +491,6 @@ angular.module('MatrixCall', [])
+         console.log("Created answer: "+description);
+         var self = this;
+         this.peerConn.setLocalDescription(description, function() {
+-            var content = {
+-                version: 0,
+-                call_id: self.call_id,
+-                answer: self.peerConn.localDescription
+-            };
+-            self.sendEventWithRetry('m.call.answer', content);
+-            $rootScope.$apply(function() {
+-                self.state = 'connecting';
+-            });
+         }, function() { console.log("Error setting local description!"); } );
+     };
+ 
+@@ -448,6 +512,9 @@ angular.module('MatrixCall', [])
+             $rootScope.$apply(function() {
+                 self.state = 'connected';
+                 self.didConnect = true;
++		/*$timeout(function() {
++                    sendKeyframe(self.peerConn);
++                }, 1000);*/
+             });
+         } else if (this.peerConn.iceConnectionState == 'failed') {
+             this.hangup('ice_failed');
+@@ -518,6 +585,7 @@ angular.module('MatrixCall', [])
+ 
+     MatrixCall.prototype.onRemoteStreamEnded = function(event) {
+         console.log("Remote stream ended");
++        return;
+         var self = this;
+         $rootScope.$apply(function() {
+             self.state = 'ended';
+diff --git a/syweb/webclient/app/components/matrix/matrix-phone-service.js b/syweb/webclient/app/components/matrix/matrix-phone-service.js
+index 55dbbf5..272fa27 100644
+--- a/syweb/webclient/app/components/matrix/matrix-phone-service.js
++++ b/syweb/webclient/app/components/matrix/matrix-phone-service.js
+@@ -48,6 +48,13 @@ angular.module('matrixPhoneService', [])
+                 return;
+             }
+ 
++            // do we already have an entry for this call ID?
++            var existingEntry = matrixPhoneService.allCalls[msg.call_id];
++            if (existingEntry) {
++                 existingEntry.receivedInvite(msg);
++                 return;
++            }
++
+             var call = undefined;
+             if (!isLive) {
+                 // if this event wasn't live then this call may already be over
+@@ -108,7 +115,7 @@ angular.module('matrixPhoneService', [])
+                     call.hangup();
+                 }
+             } else {
+-                $rootScope.$broadcast(matrixPhoneService.INCOMING_CALL_EVENT, call);
++                 $rootScope.$broadcast(matrixPhoneService.INCOMING_CALL_EVENT, call);
+             }
+         } else if (event.type == 'm.call.answer') {
+             var call = matrixPhoneService.allCalls[msg.call_id];

contrib/jitsimeetbridge/unjingle/strophe.jingle.sdp.js

@@ -0,0 +1,712 @@
+/* jshint -W117 */
+// SDP STUFF
+function SDP(sdp) {
+    this.media = sdp.split('\r\nm=');
+    for (var i = 1; i < this.media.length; i++) {
+        this.media[i] = 'm=' + this.media[i];
+        if (i != this.media.length - 1) {
+            this.media[i] += '\r\n';
+        }
+    }
+    this.session = this.media.shift() + '\r\n';
+    this.raw = this.session + this.media.join('');
+}
+
+exports.SDP = SDP;
+
+var jsdom          = require("jsdom");
+var window = jsdom.jsdom().parentWindow;
+var $ = require('jquery')(window);
+
+var SDPUtil = require('./strophe.jingle.sdp.util.js').SDPUtil;
+
+/**
+ * Returns map of MediaChannel mapped per channel idx.
+ */
+SDP.prototype.getMediaSsrcMap = function() {
+    var self = this;
+    var media_ssrcs = {};
+    for (channelNum = 0; channelNum < self.media.length; channelNum++) {
+        modified = true;
+        tmp = SDPUtil.find_lines(self.media[channelNum], 'a=ssrc:');
+        var type = SDPUtil.parse_mid(SDPUtil.find_line(self.media[channelNum], 'a=mid:'));
+        var channel = new MediaChannel(channelNum, type);
+        media_ssrcs[channelNum] = channel;
+        tmp.forEach(function (line) {
+            var linessrc = line.substring(7).split(' ')[0];
+            // allocate new ChannelSsrc
+            if(!channel.ssrcs[linessrc]) {
+                channel.ssrcs[linessrc] = new ChannelSsrc(linessrc, type);
+            }
+            channel.ssrcs[linessrc].lines.push(line);
+        });
+        tmp = SDPUtil.find_lines(self.media[channelNum], 'a=ssrc-group:');
+        tmp.forEach(function(line){
+            var semantics = line.substr(0, idx).substr(13);
+            var ssrcs = line.substr(14 + semantics.length).split(' ');
+            if (ssrcs.length != 0) {
+                var ssrcGroup = new ChannelSsrcGroup(semantics, ssrcs);
+                channel.ssrcGroups.push(ssrcGroup);
+            }
+        });
+    }
+    return media_ssrcs;
+};
+/**
+ * Returns <tt>true</tt> if this SDP contains given SSRC.
+ * @param ssrc the ssrc to check.
+ * @returns {boolean} <tt>true</tt> if this SDP contains given SSRC.
+ */
+SDP.prototype.containsSSRC = function(ssrc) {
+    var channels = this.getMediaSsrcMap();
+    var contains = false;
+    Object.keys(channels).forEach(function(chNumber){
+        var channel = channels[chNumber];
+        //console.log("Check", channel, ssrc);
+        if(Object.keys(channel.ssrcs).indexOf(ssrc) != -1){
+            contains = true;
+        }
+    });
+    return contains;
+};
+
+/**
+ * Returns map of MediaChannel that contains only media not contained in <tt>otherSdp</tt>. Mapped by channel idx.
+ * @param otherSdp the other SDP to check ssrc with.
+ */
+SDP.prototype.getNewMedia = function(otherSdp) {
+
+    // this could be useful in Array.prototype.
+    function arrayEquals(array) {
+        // if the other array is a falsy value, return
+        if (!array)
+            return false;
+
+        // compare lengths - can save a lot of time
+        if (this.length != array.length)
+            return false;
+
+        for (var i = 0, l=this.length; i < l; i++) {
+            // Check if we have nested arrays
+            if (this[i] instanceof Array && array[i] instanceof Array) {
+                // recurse into the nested arrays
+                if (!this[i].equals(array[i]))
+                    return false;
+            }
+            else if (this[i] != array[i]) {
+                // Warning - two different object instances will never be equal: {x:20} != {x:20}
+                return false;
+            }
+        }
+        return true;
+    }
+
+    var myMedia = this.getMediaSsrcMap();
+    var othersMedia = otherSdp.getMediaSsrcMap();
+    var newMedia = {};
+    Object.keys(othersMedia).forEach(function(channelNum) {
+        var myChannel = myMedia[channelNum];
+        var othersChannel = othersMedia[channelNum];
+        if(!myChannel && othersChannel) {
+            // Add whole channel
+            newMedia[channelNum] = othersChannel;
+            return;
+        }
+        // Look for new ssrcs accross the channel
+        Object.keys(othersChannel.ssrcs).forEach(function(ssrc) {
+            if(Object.keys(myChannel.ssrcs).indexOf(ssrc) === -1) {
+                // Allocate channel if we've found ssrc that doesn't exist in our channel
+                if(!newMedia[channelNum]){
+                    newMedia[channelNum] = new MediaChannel(othersChannel.chNumber, othersChannel.mediaType);
+                }
+                newMedia[channelNum].ssrcs[ssrc] = othersChannel.ssrcs[ssrc];
+            }
+        });
+
+        // Look for new ssrc groups across the channels
+        othersChannel.ssrcGroups.forEach(function(otherSsrcGroup){
+
+            // try to match the other ssrc-group with an ssrc-group of ours
+            var matched = false;
+            for (var i = 0; i < myChannel.ssrcGroups.length; i++) {
+                var mySsrcGroup = myChannel.ssrcGroups[i];
+                if (otherSsrcGroup.semantics == mySsrcGroup.semantics
+                    && arrayEquals.apply(otherSsrcGroup.ssrcs, [mySsrcGroup.ssrcs])) {
+
+                    matched = true;
+                    break;
+                }
+            }
+
+            if (!matched) {
+                // Allocate channel if we've found an ssrc-group that doesn't
+                // exist in our channel
+
+                if(!newMedia[channelNum]){
+                    newMedia[channelNum] = new MediaChannel(othersChannel.chNumber, othersChannel.mediaType);
+                }
+                newMedia[channelNum].ssrcGroups.push(otherSsrcGroup);
+            }
+        });
+    });
+    return newMedia;
+};
+
+// remove iSAC and CN from SDP
+SDP.prototype.mangle = function () {
+    var i, j, mline, lines, rtpmap, newdesc;
+    for (i = 0; i < this.media.length; i++) {
+        lines = this.media[i].split('\r\n');
+        lines.pop(); // remove empty last element
+        mline = SDPUtil.parse_mline(lines.shift());
+        if (mline.media != 'audio')
+            continue;
+        newdesc = '';
+        mline.fmt.length = 0;
+        for (j = 0; j < lines.length; j++) {
+            if (lines[j].substr(0, 9) == 'a=rtpmap:') {
+                rtpmap = SDPUtil.parse_rtpmap(lines[j]);
+                if (rtpmap.name == 'CN' || rtpmap.name == 'ISAC')
+                    continue;
+                mline.fmt.push(rtpmap.id);
+                newdesc += lines[j] + '\r\n';
+            } else {
+                newdesc += lines[j] + '\r\n';
+            }
+        }
+        this.media[i] = SDPUtil.build_mline(mline) + '\r\n';
+        this.media[i] += newdesc;
+    }
+    this.raw = this.session + this.media.join('');
+};
+
+// remove lines matching prefix from session section
+SDP.prototype.removeSessionLines = function(prefix) {
+    var self = this;
+    var lines = SDPUtil.find_lines(this.session, prefix);
+    lines.forEach(function(line) {
+        self.session = self.session.replace(line + '\r\n', '');
+    });
+    this.raw = this.session + this.media.join('');
+    return lines;
+}
+// remove lines matching prefix from a media section specified by mediaindex
+// TODO: non-numeric mediaindex could match mid
+SDP.prototype.removeMediaLines = function(mediaindex, prefix) {
+    var self = this;
+    var lines = SDPUtil.find_lines(this.media[mediaindex], prefix);
+    lines.forEach(function(line) {
+        self.media[mediaindex] = self.media[mediaindex].replace(line + '\r\n', '');
+    });
+    this.raw = this.session + this.media.join('');
+    return lines;
+}
+
+// add content's to a jingle element
+SDP.prototype.toJingle = function (elem, thecreator) {
+    var i, j, k, mline, ssrc, rtpmap, tmp, line, lines;
+    var self = this;
+    // new bundle plan
+    if (SDPUtil.find_line(this.session, 'a=group:')) {
+        lines = SDPUtil.find_lines(this.session, 'a=group:');
+        for (i = 0; i < lines.length; i++) {
+            tmp = lines[i].split(' ');
+            var semantics = tmp.shift().substr(8);
+            elem.c('group', {xmlns: 'urn:xmpp:jingle:apps:grouping:0', semantics:semantics});
+            for (j = 0; j < tmp.length; j++) {
+                elem.c('content', {name: tmp[j]}).up();
+            }
+            elem.up();
+        }
+    }
+    // old bundle plan, to be removed
+    var bundle = [];
+    if (SDPUtil.find_line(this.session, 'a=group:BUNDLE')) {
+        bundle = SDPUtil.find_line(this.session, 'a=group:BUNDLE ').split(' ');
+        bundle.shift();
+    }
+    for (i = 0; i < this.media.length; i++) {
+        mline = SDPUtil.parse_mline(this.media[i].split('\r\n')[0]);
+        if (!(mline.media === 'audio' ||
+              mline.media === 'video' ||
+              mline.media === 'application'))
+        {
+            continue;
+        }
+        if (SDPUtil.find_line(this.media[i], 'a=ssrc:')) {
+            ssrc = SDPUtil.find_line(this.media[i], 'a=ssrc:').substring(7).split(' ')[0]; // take the first
+        } else {
+            ssrc = false;
+        }
+
+        elem.c('content', {creator: thecreator, name: mline.media});
+        if (SDPUtil.find_line(this.media[i], 'a=mid:')) {
+            // prefer identifier from a=mid if present
+            var mid = SDPUtil.parse_mid(SDPUtil.find_line(this.media[i], 'a=mid:'));
+            elem.attrs({ name: mid });
+
+            // old BUNDLE plan, to be removed
+            if (bundle.indexOf(mid) !== -1) {
+                elem.c('bundle', {xmlns: 'http://estos.de/ns/bundle'}).up();
+                bundle.splice(bundle.indexOf(mid), 1);
+            }
+        }
+
+        if (SDPUtil.find_line(this.media[i], 'a=rtpmap:').length)
+        {
+            elem.c('description',
+                {xmlns: 'urn:xmpp:jingle:apps:rtp:1',
+                    media: mline.media });
+            if (ssrc) {
+                elem.attrs({ssrc: ssrc});
+            }
+            for (j = 0; j < mline.fmt.length; j++) {
+                rtpmap = SDPUtil.find_line(this.media[i], 'a=rtpmap:' + mline.fmt[j]);
+                elem.c('payload-type', SDPUtil.parse_rtpmap(rtpmap));
+                // put any 'a=fmtp:' + mline.fmt[j] lines into <param name=foo value=bar/>
+                if (SDPUtil.find_line(this.media[i], 'a=fmtp:' + mline.fmt[j])) {
+                    tmp = SDPUtil.parse_fmtp(SDPUtil.find_line(this.media[i], 'a=fmtp:' + mline.fmt[j]));
+                    for (k = 0; k < tmp.length; k++) {
+                        elem.c('parameter', tmp[k]).up();
+                    }
+                }
+                this.RtcpFbToJingle(i, elem, mline.fmt[j]); // XEP-0293 -- map a=rtcp-fb
+
+                elem.up();
+            }
+            if (SDPUtil.find_line(this.media[i], 'a=crypto:', this.session)) {
+                elem.c('encryption', {required: 1});
+                var crypto = SDPUtil.find_lines(this.media[i], 'a=crypto:', this.session);
+                crypto.forEach(function(line) {
+                    elem.c('crypto', SDPUtil.parse_crypto(line)).up();
+                });
+                elem.up(); // end of encryption
+            }
+
+            if (ssrc) {
+                // new style mapping
+                elem.c('source', { ssrc: ssrc, xmlns: 'urn:xmpp:jingle:apps:rtp:ssma:0' });
+                // FIXME: group by ssrc and support multiple different ssrcs
+                var ssrclines = SDPUtil.find_lines(this.media[i], 'a=ssrc:');
+                ssrclines.forEach(function(line) {
+                    idx = line.indexOf(' ');
+                    var linessrc = line.substr(0, idx).substr(7);
+                    if (linessrc != ssrc) {
+                        elem.up();
+                        ssrc = linessrc;
+                        elem.c('source', { ssrc: ssrc, xmlns: 'urn:xmpp:jingle:apps:rtp:ssma:0' });
+                    }
+                    var kv = line.substr(idx + 1);
+                    elem.c('parameter');
+                    if (kv.indexOf(':') == -1) {
+                        elem.attrs({ name: kv });
+                    } else {
+                        elem.attrs({ name: kv.split(':', 2)[0] });
+                        elem.attrs({ value: kv.split(':', 2)[1] });
+                    }
+                    elem.up();
+                });
+                elem.up();
+
+                // old proprietary mapping, to be removed at some point
+                tmp = SDPUtil.parse_ssrc(this.media[i]);
+                tmp.xmlns = 'http://estos.de/ns/ssrc';
+                tmp.ssrc = ssrc;
+                elem.c('ssrc', tmp).up(); // ssrc is part of description
+
+                // XEP-0339 handle ssrc-group attributes
+                var ssrc_group_lines = SDPUtil.find_lines(this.media[i], 'a=ssrc-group:');
+                ssrc_group_lines.forEach(function(line) {
+                    idx = line.indexOf(' ');
+                    var semantics = line.substr(0, idx).substr(13);
+                    var ssrcs = line.substr(14 + semantics.length).split(' ');
+                    if (ssrcs.length != 0) {
+                        elem.c('ssrc-group', { semantics: semantics, xmlns: 'urn:xmpp:jingle:apps:rtp:ssma:0' });
+                        ssrcs.forEach(function(ssrc) {
+                            elem.c('source', { ssrc: ssrc })
+                                .up();
+                        });
+                        elem.up();
+                    }
+                });
+            }
+
+            if (SDPUtil.find_line(this.media[i], 'a=rtcp-mux')) {
+                elem.c('rtcp-mux').up();
+            }
+
+            // XEP-0293 -- map a=rtcp-fb:*
+            this.RtcpFbToJingle(i, elem, '*');
+
+            // XEP-0294
+            if (SDPUtil.find_line(this.media[i], 'a=extmap:')) {
+                lines = SDPUtil.find_lines(this.media[i], 'a=extmap:');
+                for (j = 0; j < lines.length; j++) {
+                    tmp = SDPUtil.parse_extmap(lines[j]);
+                    elem.c('rtp-hdrext', { xmlns: 'urn:xmpp:jingle:apps:rtp:rtp-hdrext:0',
+                        uri: tmp.uri,
+                        id: tmp.value });
+                    if (tmp.hasOwnProperty('direction')) {
+                        switch (tmp.direction) {
+                            case 'sendonly':
+                                elem.attrs({senders: 'responder'});
+                                break;
+                            case 'recvonly':
+                                elem.attrs({senders: 'initiator'});
+                                break;
+                            case 'sendrecv':
+                                elem.attrs({senders: 'both'});
+                                break;
+                            case 'inactive':
+                                elem.attrs({senders: 'none'});
+                                break;
+                        }
+                    }
+                    // TODO: handle params
+                    elem.up();
+                }
+            }
+            elem.up(); // end of description
+        }
+
+        // map ice-ufrag/pwd, dtls fingerprint, candidates
+        this.TransportToJingle(i, elem);
+
+        if (SDPUtil.find_line(this.media[i], 'a=sendrecv', this.session)) {
+            elem.attrs({senders: 'both'});
+        } else if (SDPUtil.find_line(this.media[i], 'a=sendonly', this.session)) {
+            elem.attrs({senders: 'initiator'});
+        } else if (SDPUtil.find_line(this.media[i], 'a=recvonly', this.session)) {
+            elem.attrs({senders: 'responder'});
+        } else if (SDPUtil.find_line(this.media[i], 'a=inactive', this.session)) {
+            elem.attrs({senders: 'none'});
+        }
+        if (mline.port == '0') {
+            // estos hack to reject an m-line
+            elem.attrs({senders: 'rejected'});
+        }
+        elem.up(); // end of content
+    }
+    elem.up();
+    return elem;
+};
+
+SDP.prototype.TransportToJingle = function (mediaindex, elem) {
+    var i = mediaindex;
+    var tmp;
+    var self = this;
+    elem.c('transport');
+
+    // XEP-0343 DTLS/SCTP
+    if (SDPUtil.find_line(this.media[mediaindex], 'a=sctpmap:').length)
+    {
+        var sctpmap = SDPUtil.find_line(
+            this.media[i], 'a=sctpmap:', self.session);
+        if (sctpmap)
+        {
+            var sctpAttrs = SDPUtil.parse_sctpmap(sctpmap);
+            elem.c('sctpmap',
+                {
+                    xmlns: 'urn:xmpp:jingle:transports:dtls-sctp:1',
+                    number: sctpAttrs[0], /* SCTP port */
+                    protocol: sctpAttrs[1], /* protocol */
+                });
+            // Optional stream count attribute
+            if (sctpAttrs.length > 2)
+                elem.attrs({ streams: sctpAttrs[2]});
+            elem.up();
+        }
+    }
+    // XEP-0320
+    var fingerprints = SDPUtil.find_lines(this.media[mediaindex], 'a=fingerprint:', this.session);
+    fingerprints.forEach(function(line) {
+        tmp = SDPUtil.parse_fingerprint(line);
+        tmp.xmlns = 'urn:xmpp:jingle:apps:dtls:0';
+        elem.c('fingerprint').t(tmp.fingerprint);
+        delete tmp.fingerprint;
+        line = SDPUtil.find_line(self.media[mediaindex], 'a=setup:', self.session);
+        if (line) {
+            tmp.setup = line.substr(8);
+        }
+        elem.attrs(tmp);
+        elem.up(); // end of fingerprint
+    });
+    tmp = SDPUtil.iceparams(this.media[mediaindex], this.session);
+    if (tmp) {
+        tmp.xmlns = 'urn:xmpp:jingle:transports:ice-udp:1';
+        elem.attrs(tmp);
+        // XEP-0176
+        if (SDPUtil.find_line(this.media[mediaindex], 'a=candidate:', this.session)) { // add any a=candidate lines
+            var lines = SDPUtil.find_lines(this.media[mediaindex], 'a=candidate:', this.session);
+            lines.forEach(function (line) {
+                elem.c('candidate', SDPUtil.candidateToJingle(line)).up();
+            });
+        }
+    }
+    elem.up(); // end of transport
+}
+
+SDP.prototype.RtcpFbToJingle = function (mediaindex, elem, payloadtype) { // XEP-0293
+    var lines = SDPUtil.find_lines(this.media[mediaindex], 'a=rtcp-fb:' + payloadtype);
+    lines.forEach(function (line) {
+        var tmp = SDPUtil.parse_rtcpfb(line);
+        if (tmp.type == 'trr-int') {
+            elem.c('rtcp-fb-trr-int', {xmlns: 'urn:xmpp:jingle:apps:rtp:rtcp-fb:0', value: tmp.params[0]});
+            elem.up();
+        } else {
+            elem.c('rtcp-fb', {xmlns: 'urn:xmpp:jingle:apps:rtp:rtcp-fb:0', type: tmp.type});
+            if (tmp.params.length > 0) {
+                elem.attrs({'subtype': tmp.params[0]});
+            }
+            elem.up();
+        }
+    });
+};
+
+SDP.prototype.RtcpFbFromJingle = function (elem, payloadtype) { // XEP-0293
+    var media = '';
+    var tmp = elem.find('>rtcp-fb-trr-int[xmlns="urn:xmpp:jingle:apps:rtp:rtcp-fb:0"]');
+    if (tmp.length) {
+        media += 'a=rtcp-fb:' + '*' + ' ' + 'trr-int' + ' ';
+        if (tmp.attr('value')) {
+            media += tmp.attr('value');
+        } else {
+            media += '0';
+        }
+        media += '\r\n';
+    }
+    tmp = elem.find('>rtcp-fb[xmlns="urn:xmpp:jingle:apps:rtp:rtcp-fb:0"]');
+    tmp.each(function () {
+        media += 'a=rtcp-fb:' + payloadtype + ' ' + $(this).attr('type');
+        if ($(this).attr('subtype')) {
+            media += ' ' + $(this).attr('subtype');
+        }
+        media += '\r\n';
+    });
+    return media;
+};
+
+// construct an SDP from a jingle stanza
+SDP.prototype.fromJingle = function (jingle) {
+    var self = this;
+    this.raw = 'v=0\r\n' +
+        'o=- ' + '1923518516' + ' 2 IN IP4 0.0.0.0\r\n' +// FIXME
+        's=-\r\n' +
+        't=0 0\r\n';
+    // http://tools.ietf.org/html/draft-ietf-mmusic-sdp-bundle-negotiation-04#section-8
+    if ($(jingle).find('>group[xmlns="urn:xmpp:jingle:apps:grouping:0"]').length) {
+        $(jingle).find('>group[xmlns="urn:xmpp:jingle:apps:grouping:0"]').each(function (idx, group) {
+            var contents = $(group).find('>content').map(function (idx, content) {
+                return content.getAttribute('name');
+            }).get();
+            if (contents.length > 0) {
+                self.raw += 'a=group:' + (group.getAttribute('semantics') || group.getAttribute('type')) + ' ' + contents.join(' ') + '\r\n';
+            }
+        });
+    } else if ($(jingle).find('>group[xmlns="urn:ietf:rfc:5888"]').length) {
+        // temporary namespace, not to be used. to be removed soon.
+        $(jingle).find('>group[xmlns="urn:ietf:rfc:5888"]').each(function (idx, group) {
+            var contents = $(group).find('>content').map(function (idx, content) {
+                return content.getAttribute('name');
+            }).get();
+            if (group.getAttribute('type') !== null && contents.length > 0) {
+                self.raw += 'a=group:' + group.getAttribute('type') + ' ' + contents.join(' ') + '\r\n';
+            }
+        });
+    } else {
+        // for backward compability, to be removed soon
+        // assume all contents are in the same bundle group, can be improved upon later
+        var bundle = $(jingle).find('>content').filter(function (idx, content) {
+            //elem.c('bundle', {xmlns:'http://estos.de/ns/bundle'});
+            return $(content).find('>bundle').length > 0;
+        }).map(function (idx, content) {
+                return content.getAttribute('name');
+            }).get();
+        if (bundle.length) {
+            this.raw += 'a=group:BUNDLE ' + bundle.join(' ') + '\r\n';
+        }
+    }
+
+    this.session = this.raw;
+    jingle.find('>content').each(function () {
+        var m = self.jingle2media($(this));
+        self.media.push(m);
+    });
+
+    // reconstruct msid-semantic -- apparently not necessary
+    /*
+     var msid = SDPUtil.parse_ssrc(this.raw);
+     if (msid.hasOwnProperty('mslabel')) {
+     this.session += "a=msid-semantic: WMS " + msid.mslabel + "\r\n";
+     }
+     */
+
+    this.raw = this.session + this.media.join('');
+};
+
+// translate a jingle content element into an an SDP media part
+SDP.prototype.jingle2media = function (content) {
+    var media = '',
+        desc = content.find('description'),
+        ssrc = desc.attr('ssrc'),
+        self = this,
+        tmp;
+    var sctp = content.find(
+        '>transport>sctpmap[xmlns="urn:xmpp:jingle:transports:dtls-sctp:1"]');
+
+    tmp = { media: desc.attr('media') };
+    tmp.port = '1';
+    if (content.attr('senders') == 'rejected') {
+        // estos hack to reject an m-line.
+        tmp.port = '0';
+    }
+    if (content.find('>transport>fingerprint').length || desc.find('encryption').length) {
+        if (sctp.length)
+            tmp.proto = 'DTLS/SCTP';
+        else
+            tmp.proto = 'RTP/SAVPF';
+    } else {
+        tmp.proto = 'RTP/AVPF';
+    }
+    if (!sctp.length)
+    {
+        tmp.fmt = desc.find('payload-type').map(
+            function () { return this.getAttribute('id'); }).get();
+        media += SDPUtil.build_mline(tmp) + '\r\n';
+    }
+    else
+    {
+        media += 'm=application 1 DTLS/SCTP ' + sctp.attr('number') + '\r\n';
+        media += 'a=sctpmap:' + sctp.attr('number') +
+            ' ' + sctp.attr('protocol');
+
+        var streamCount = sctp.attr('streams');
+        if (streamCount)
+            media += ' ' + streamCount + '\r\n';
+        else
+            media += '\r\n';
+    }
+
+    media += 'c=IN IP4 0.0.0.0\r\n';
+    if (!sctp.length)
+        media += 'a=rtcp:1 IN IP4 0.0.0.0\r\n';
+    //tmp = content.find('>transport[xmlns="urn:xmpp:jingle:transports:ice-udp:1"]');
+    tmp = content.find('>bundle>transport[xmlns="urn:xmpp:jingle:transports:ice-udp:1"]');
+    //console.log('transports: '+content.find('>transport[xmlns="urn:xmpp:jingle:transports:ice-udp:1"]').length);
+    //console.log('bundle.transports: '+content.find('>bundle>transport[xmlns="urn:xmpp:jingle:transports:ice-udp:1"]').length);
+    //console.log("tmp fingerprint: "+tmp.find('>fingerprint').innerHTML);
+    if (tmp.length) {
+        if (tmp.attr('ufrag')) {
+            media += SDPUtil.build_iceufrag(tmp.attr('ufrag')) + '\r\n';
+        }
+        if (tmp.attr('pwd')) {
+            media += SDPUtil.build_icepwd(tmp.attr('pwd')) + '\r\n';
+        }
+        tmp.find('>fingerprint').each(function () {
+            // FIXME: check namespace at some point
+            media += 'a=fingerprint:' + this.getAttribute('hash');
+            media += ' ' + $(this).text();
+            media += '\r\n';
+            //console.log("mline "+media);
+            if (this.getAttribute('setup')) {
+                media += 'a=setup:' + this.getAttribute('setup') + '\r\n';
+            }
+        });
+    }
+    switch (content.attr('senders')) {
+        case 'initiator':
+            media += 'a=sendonly\r\n';
+            break;
+        case 'responder':
+            media += 'a=recvonly\r\n';
+            break;
+        case 'none':
+            media += 'a=inactive\r\n';
+            break;
+        case 'both':
+            media += 'a=sendrecv\r\n';
+            break;
+    }
+    media += 'a=mid:' + content.attr('name') + '\r\n';
+    /*if (content.attr('name') == 'video') {
+        media += 'a=x-google-flag:conference' + '\r\n';
+    }*/
+
+    // <description><rtcp-mux/></description>
+    // see http://code.google.com/p/libjingle/issues/detail?id=309 -- no spec though
+    // and http://mail.jabber.org/pipermail/jingle/2011-December/001761.html
+    if (desc.find('rtcp-mux').length) {
+        media += 'a=rtcp-mux\r\n';
+    }
+
+    if (desc.find('encryption').length) {
+        desc.find('encryption>crypto').each(function () {
+            media += 'a=crypto:' + this.getAttribute('tag');
+            media += ' ' + this.getAttribute('crypto-suite');
+            media += ' ' + this.getAttribute('key-params');
+            if (this.getAttribute('session-params')) {
+                media += ' ' + this.getAttribute('session-params');
+            }
+            media += '\r\n';
+        });
+    }
+    desc.find('payload-type').each(function () {
+        media += SDPUtil.build_rtpmap(this) + '\r\n';
+        if ($(this).find('>parameter').length) {
+            media += 'a=fmtp:' + this.getAttribute('id') + ' ';
+            media += $(this).find('parameter').map(function () { return (this.getAttribute('name') ? (this.getAttribute('name') + '=') : '') + this.getAttribute('value'); }).get().join('; ');
+            media += '\r\n';
+        }
+        // xep-0293
+        media += self.RtcpFbFromJingle($(this), this.getAttribute('id'));
+    });
+
+    // xep-0293
+    media += self.RtcpFbFromJingle(desc, '*');
+
+    // xep-0294
+    tmp = desc.find('>rtp-hdrext[xmlns="urn:xmpp:jingle:apps:rtp:rtp-hdrext:0"]');
+    tmp.each(function () {
+        media += 'a=extmap:' + this.getAttribute('id') + ' ' + this.getAttribute('uri') + '\r\n';
+    });
+
+    content.find('>bundle>transport[xmlns="urn:xmpp:jingle:transports:ice-udp:1"]>candidate').each(function () {
+        media += SDPUtil.candidateFromJingle(this);
+    });
+
+    // XEP-0339 handle ssrc-group attributes
+    tmp = content.find('description>ssrc-group[xmlns="urn:xmpp:jingle:apps:rtp:ssma:0"]').each(function() {
+        var semantics = this.getAttribute('semantics');
+        var ssrcs = $(this).find('>source').map(function() {
+            return this.getAttribute('ssrc');
+        }).get();
+
+        if (ssrcs.length != 0) {
+            media += 'a=ssrc-group:' + semantics + ' ' + ssrcs.join(' ') + '\r\n';
+        }
+    });
+
+    tmp = content.find('description>source[xmlns="urn:xmpp:jingle:apps:rtp:ssma:0"]');
+    tmp.each(function () {
+        var ssrc = this.getAttribute('ssrc');
+        $(this).find('>parameter').each(function () {
+            media += 'a=ssrc:' + ssrc + ' ' + this.getAttribute('name');
+            if (this.getAttribute('value') && this.getAttribute('value').length)
+                media += ':' + this.getAttribute('value');
+            media += '\r\n';
+        });
+    });
+
+    if (tmp.length === 0) {
+        // fallback to proprietary mapping of a=ssrc lines
+        tmp = content.find('description>ssrc[xmlns="http://estos.de/ns/ssrc"]');
+        if (tmp.length) {
+            media += 'a=ssrc:' + ssrc + ' cname:' + tmp.attr('cname') + '\r\n';
+            media += 'a=ssrc:' + ssrc + ' msid:' + tmp.attr('msid') + '\r\n';
+            media += 'a=ssrc:' + ssrc + ' mslabel:' + tmp.attr('mslabel') + '\r\n';
+            media += 'a=ssrc:' + ssrc + ' label:' + tmp.attr('label') + '\r\n';
+        }
+    }
+    return media;
+};
+

contrib/jitsimeetbridge/unjingle/strophe.jingle.sdp.util.js

@@ -0,0 +1,408 @@
+/**
+ * Contains utility classes used in SDP class.
+ *
+ */
+
+/**
+ * Class holds a=ssrc lines and media type a=mid
+ * @param ssrc synchronization source identifier number(a=ssrc lines from SDP)
+ * @param type media type eg. "audio" or "video"(a=mid frm SDP)
+ * @constructor
+ */
+function ChannelSsrc(ssrc, type) {
+    this.ssrc = ssrc;
+    this.type = type;
+    this.lines = [];
+}
+
+/**
+ * Class holds a=ssrc-group: lines
+ * @param semantics
+ * @param ssrcs
+ * @constructor
+ */
+function ChannelSsrcGroup(semantics, ssrcs, line) {
+    this.semantics = semantics;
+    this.ssrcs = ssrcs;
+}
+
+/**
+ * Helper class represents media channel. Is a container for ChannelSsrc, holds channel idx and media type.
+ * @param channelNumber channel idx in SDP media array.
+ * @param mediaType media type(a=mid)
+ * @constructor
+ */
+function MediaChannel(channelNumber, mediaType) {
+    /**
+     * SDP channel number
+     * @type {*}
+     */
+    this.chNumber = channelNumber;
+    /**
+     * Channel media type(a=mid)
+     * @type {*}
+     */
+    this.mediaType = mediaType;
+    /**
+     * The maps of ssrc numbers to ChannelSsrc objects.
+     */
+    this.ssrcs = {};
+
+    /**
+     * The array of ChannelSsrcGroup objects.
+     * @type {Array}
+     */
+    this.ssrcGroups = [];
+}
+
+SDPUtil = {
+    iceparams: function (mediadesc, sessiondesc) {
+        var data = null;
+        if (SDPUtil.find_line(mediadesc, 'a=ice-ufrag:', sessiondesc) &&
+            SDPUtil.find_line(mediadesc, 'a=ice-pwd:', sessiondesc)) {
+            data = {
+                ufrag: SDPUtil.parse_iceufrag(SDPUtil.find_line(mediadesc, 'a=ice-ufrag:', sessiondesc)),
+                pwd: SDPUtil.parse_icepwd(SDPUtil.find_line(mediadesc, 'a=ice-pwd:', sessiondesc))
+            };
+        }
+        return data;
+    },
+    parse_iceufrag: function (line) {
+        return line.substring(12);
+    },
+    build_iceufrag: function (frag) {
+        return 'a=ice-ufrag:' + frag;
+    },
+    parse_icepwd: function (line) {
+        return line.substring(10);
+    },
+    build_icepwd: function (pwd) {
+        return 'a=ice-pwd:' + pwd;
+    },
+    parse_mid: function (line) {
+        return line.substring(6);
+    },
+    parse_mline: function (line) {
+        var parts = line.substring(2).split(' '),
+            data = {};
+        data.media = parts.shift();
+        data.port = parts.shift();
+        data.proto = parts.shift();
+        if (parts[parts.length - 1] === '') { // trailing whitespace
+            parts.pop();
+        }
+        data.fmt = parts;
+        return data;
+    },
+    build_mline: function (mline) {
+        return 'm=' + mline.media + ' ' + mline.port + ' ' + mline.proto + ' ' + mline.fmt.join(' ');
+    },
+    parse_rtpmap: function (line) {
+        var parts = line.substring(9).split(' '),
+            data = {};
+        data.id = parts.shift();
+        parts = parts[0].split('/');
+        data.name = parts.shift();
+        data.clockrate = parts.shift();
+        data.channels = parts.length ? parts.shift() : '1';
+        return data;
+    },
+    /**
+     * Parses SDP line "a=sctpmap:..." and extracts SCTP port from it.
+     * @param line eg. "a=sctpmap:5000 webrtc-datachannel"
+     * @returns [SCTP port number, protocol, streams]
+     */
+    parse_sctpmap: function (line)
+    {
+        var parts = line.substring(10).split(' ');
+        var sctpPort = parts[0];
+        var protocol = parts[1];
+        // Stream count is optional
+        var streamCount = parts.length > 2 ? parts[2] : null;
+        return [sctpPort, protocol, streamCount];// SCTP port
+    },
+    build_rtpmap: function (el) {
+        var line = 'a=rtpmap:' + el.getAttribute('id') + ' ' + el.getAttribute('name') + '/' + el.getAttribute('clockrate');
+        if (el.getAttribute('channels') && el.getAttribute('channels') != '1') {
+            line += '/' + el.getAttribute('channels');
+        }
+        return line;
+    },
+    parse_crypto: function (line) {
+        var parts = line.substring(9).split(' '),
+            data = {};
+        data.tag = parts.shift();
+        data['crypto-suite'] = parts.shift();
+        data['key-params'] = parts.shift();
+        if (parts.length) {
+            data['session-params'] = parts.join(' ');
+        }
+        return data;
+    },
+    parse_fingerprint: function (line) { // RFC 4572
+        var parts = line.substring(14).split(' '),
+            data = {};
+        data.hash = parts.shift();
+        data.fingerprint = parts.shift();
+        // TODO assert that fingerprint satisfies 2UHEX *(":" 2UHEX) ?
+        return data;
+    },
+    parse_fmtp: function (line) {
+        var parts = line.split(' '),
+            i, key, value,
+            data = [];
+        parts.shift();
+        parts = parts.join(' ').split(';');
+        for (i = 0; i < parts.length; i++) {
+            key = parts[i].split('=')[0];
+            while (key.length && key[0] == ' ') {
+                key = key.substring(1);
+            }
+            value = parts[i].split('=')[1];
+            if (key && value) {
+                data.push({name: key, value: value});
+            } else if (key) {
+                // rfc 4733 (DTMF) style stuff
+                data.push({name: '', value: key});
+            }
+        }
+        return data;
+    },
+    parse_icecandidate: function (line) {
+        var candidate = {},
+            elems = line.split(' ');
+        candidate.foundation = elems[0].substring(12);
+        candidate.component = elems[1];
+        candidate.protocol = elems[2].toLowerCase();
+        candidate.priority = elems[3];
+        candidate.ip = elems[4];
+        candidate.port = elems[5];
+        // elems[6] => "typ"
+        candidate.type = elems[7];
+        candidate.generation = 0; // default value, may be overwritten below
+        for (var i = 8; i < elems.length; i += 2) {
+            switch (elems[i]) {
+                case 'raddr':
+                    candidate['rel-addr'] = elems[i + 1];
+                    break;
+                case 'rport':
+                    candidate['rel-port'] = elems[i + 1];
+                    break;
+                case 'generation':
+                    candidate.generation = elems[i + 1];
+                    break;
+                case 'tcptype':
+                    candidate.tcptype = elems[i + 1];
+                    break;
+                default: // TODO
+                    console.log('parse_icecandidate not translating "' + elems[i] + '" = "' + elems[i + 1] + '"');
+            }
+        }
+        candidate.network = '1';
+        candidate.id = Math.random().toString(36).substr(2, 10); // not applicable to SDP -- FIXME: should be unique, not just random
+        return candidate;
+    },
+    build_icecandidate: function (cand) {
+        var line = ['a=candidate:' + cand.foundation, cand.component, cand.protocol, cand.priority, cand.ip, cand.port, 'typ', cand.type].join(' ');
+        line += ' ';
+        switch (cand.type) {
+            case 'srflx':
+            case 'prflx':
+            case 'relay':
+                if (cand.hasOwnAttribute('rel-addr') && cand.hasOwnAttribute('rel-port')) {
+                    line += 'raddr';
+                    line += ' ';
+                    line += cand['rel-addr'];
+                    line += ' ';
+                    line += 'rport';
+                    line += ' ';
+                    line += cand['rel-port'];
+                    line += ' ';
+                }
+                break;
+        }
+        if (cand.hasOwnAttribute('tcptype')) {
+            line += 'tcptype';
+            line += ' ';
+            line += cand.tcptype;
+            line += ' ';
+        }
+        line += 'generation';
+        line += ' ';
+        line += cand.hasOwnAttribute('generation') ? cand.generation : '0';
+        return line;
+    },
+    parse_ssrc: function (desc) {
+        // proprietary mapping of a=ssrc lines
+        // TODO: see "Jingle RTP Source Description" by Juberti and P. Thatcher on google docs
+        // and parse according to that
+        var lines = desc.split('\r\n'),
+            data = {};
+        for (var i = 0; i < lines.length; i++) {
+            if (lines[i].substring(0, 7) == 'a=ssrc:') {
+                var idx = lines[i].indexOf(' ');
+                data[lines[i].substr(idx + 1).split(':', 2)[0]] = lines[i].substr(idx + 1).split(':', 2)[1];
+            }
+        }
+        return data;
+    },
+    parse_rtcpfb: function (line) {
+        var parts = line.substr(10).split(' ');
+        var data = {};
+        data.pt = parts.shift();
+        data.type = parts.shift();
+        data.params = parts;
+        return data;
+    },
+    parse_extmap: function (line) {
+        var parts = line.substr(9).split(' ');
+        var data = {};
+        data.value = parts.shift();
+        if (data.value.indexOf('/') != -1) {
+            data.direction = data.value.substr(data.value.indexOf('/') + 1);
+            data.value = data.value.substr(0, data.value.indexOf('/'));
+        } else {
+            data.direction = 'both';
+        }
+        data.uri = parts.shift();
+        data.params = parts;
+        return data;
+    },
+    find_line: function (haystack, needle, sessionpart) {
+        var lines = haystack.split('\r\n');
+        for (var i = 0; i < lines.length; i++) {
+            if (lines[i].substring(0, needle.length) == needle) {
+                return lines[i];
+            }
+        }
+        if (!sessionpart) {
+            return false;
+        }
+        // search session part
+        lines = sessionpart.split('\r\n');
+        for (var j = 0; j < lines.length; j++) {
+            if (lines[j].substring(0, needle.length) == needle) {
+                return lines[j];
+            }
+        }
+        return false;
+    },
+    find_lines: function (haystack, needle, sessionpart) {
+        var lines = haystack.split('\r\n'),
+            needles = [];
+        for (var i = 0; i < lines.length; i++) {
+            if (lines[i].substring(0, needle.length) == needle)
+                needles.push(lines[i]);
+        }
+        if (needles.length || !sessionpart) {
+            return needles;
+        }
+        // search session part
+        lines = sessionpart.split('\r\n');
+        for (var j = 0; j < lines.length; j++) {
+            if (lines[j].substring(0, needle.length) == needle) {
+                needles.push(lines[j]);
+            }
+        }
+        return needles;
+    },
+    candidateToJingle: function (line) {
+        // a=candidate:2979166662 1 udp 2113937151 192.168.2.100 57698 typ host generation 0
+        //      <candidate component=... foundation=... generation=... id=... ip=... network=... port=... priority=... protocol=... type=.../>
+        if (line.indexOf('candidate:') === 0) {
+            line = 'a=' + line;
+        } else if (line.substring(0, 12) != 'a=candidate:') {
+            console.log('parseCandidate called with a line that is not a candidate line');
+            console.log(line);
+            return null;
+        }
+        if (line.substring(line.length - 2) == '\r\n') // chomp it
+            line = line.substring(0, line.length - 2);
+        var candidate = {},
+            elems = line.split(' '),
+            i;
+        if (elems[6] != 'typ') {
+            console.log('did not find typ in the right place');
+            console.log(line);
+            return null;
+        }
+        candidate.foundation = elems[0].substring(12);
+        candidate.component = elems[1];
+        candidate.protocol = elems[2].toLowerCase();
+        candidate.priority = elems[3];
+        candidate.ip = elems[4];
+        candidate.port = elems[5];
+        // elems[6] => "typ"
+        candidate.type = elems[7];
+
+        candidate.generation = '0'; // default, may be overwritten below
+        for (i = 8; i < elems.length; i += 2) {
+            switch (elems[i]) {
+                case 'raddr':
+                    candidate['rel-addr'] = elems[i + 1];
+                    break;
+                case 'rport':
+                    candidate['rel-port'] = elems[i + 1];
+                    break;
+                case 'generation':
+                    candidate.generation = elems[i + 1];
+                    break;
+                case 'tcptype':
+                    candidate.tcptype = elems[i + 1];
+                    break;
+                default: // TODO
+                    console.log('not translating "' + elems[i] + '" = "' + elems[i + 1] + '"');
+            }
+        }
+        candidate.network = '1';
+        candidate.id = Math.random().toString(36).substr(2, 10); // not applicable to SDP -- FIXME: should be unique, not just random
+        return candidate;
+    },
+    candidateFromJingle: function (cand) {
+        var line = 'a=candidate:';
+        line += cand.getAttribute('foundation');
+        line += ' ';
+        line += cand.getAttribute('component');
+        line += ' ';
+        line += cand.getAttribute('protocol'); //.toUpperCase(); // chrome M23 doesn't like this
+        line += ' ';
+        line += cand.getAttribute('priority');
+        line += ' ';
+        line += cand.getAttribute('ip');
+        line += ' ';
+        line += cand.getAttribute('port');
+        line += ' ';
+        line += 'typ';
+        line += ' ' + cand.getAttribute('type');
+        line += ' ';
+        switch (cand.getAttribute('type')) {
+            case 'srflx':
+            case 'prflx':
+            case 'relay':
+                if (cand.getAttribute('rel-addr') && cand.getAttribute('rel-port')) {
+                    line += 'raddr';
+                    line += ' ';
+                    line += cand.getAttribute('rel-addr');
+                    line += ' ';
+                    line += 'rport';
+                    line += ' ';
+                    line += cand.getAttribute('rel-port');
+                    line += ' ';
+                }
+                break;
+        }
+        if (cand.getAttribute('protocol').toLowerCase() == 'tcp') {
+            line += 'tcptype';
+            line += ' ';
+            line += cand.getAttribute('tcptype');
+            line += ' ';
+        }
+        line += 'generation';
+        line += ' ';
+        line += cand.getAttribute('generation') || '0';
+        return line + '\r\n';
+    }
+};
+
+exports.SDPUtil = SDPUtil;
+

contrib/jitsimeetbridge/unjingle/strophe/XMLHttpRequest.js

@@ -0,0 +1,254 @@
+/**
+ * Wrapper for built-in http.js to emulate the browser XMLHttpRequest object.
+ *
+ * This can be used with JS designed for browsers to improve reuse of code and
+ * allow the use of existing libraries.
+ *
+ * Usage: include("XMLHttpRequest.js") and use XMLHttpRequest per W3C specs.
+ *
+ * @todo SSL Support
+ * @author Dan DeFelippi <dan@driverdan.com>
+ * @license MIT
+ */
+
+var Url = require("url")
+	,sys = require("util");
+
+exports.XMLHttpRequest = function() {
+	/**
+	 * Private variables
+	 */
+	var self = this;
+	var http = require('http');
+	var https = require('https');
+
+	// Holds http.js objects
+	var client;
+	var request;
+	var response;
+	
+	// Request settings
+	var settings = {};
+	
+	// Set some default headers
+	var defaultHeaders = {
+		"User-Agent": "node.js",
+		"Accept": "*/*",
+	};
+	
+	var headers = defaultHeaders;
+	
+	/**
+	 * Constants
+	 */
+	this.UNSENT = 0;
+	this.OPENED = 1;
+	this.HEADERS_RECEIVED = 2;
+	this.LOADING = 3;
+	this.DONE = 4;
+
+	/**
+	 * Public vars
+	 */
+	// Current state
+	this.readyState = this.UNSENT;
+
+	// default ready state change handler in case one is not set or is set late
+	this.onreadystatechange = function() {};
+
+	// Result & response
+	this.responseText = "";
+	this.responseXML = "";
+	this.status = null;
+	this.statusText = null;
+		
+	/**
+	 * Open the connection. Currently supports local server requests.
+	 *
+	 * @param string method Connection method (eg GET, POST)
+	 * @param string url URL for the connection.
+	 * @param boolean async Asynchronous connection. Default is true.
+	 * @param string user Username for basic authentication (optional)
+	 * @param string password Password for basic authentication (optional)
+	 */
+	this.open = function(method, url, async, user, password) {
+		settings = {
+			"method": method,
+			"url": url,
+			"async": async || null,
+			"user": user || null,
+			"password": password || null
+		};
+		
+		this.abort();
+
+		setState(this.OPENED);
+	};
+	
+	/**
+	 * Sets a header for the request.
+	 *
+	 * @param string header Header name
+	 * @param string value Header value
+	 */
+	this.setRequestHeader = function(header, value) {
+		headers[header] = value;
+	};
+	
+	/**
+	 * Gets a header from the server response.
+	 *
+	 * @param string header Name of header to get.
+	 * @return string Text of the header or null if it doesn't exist.
+	 */
+	this.getResponseHeader = function(header) {
+		if (this.readyState > this.OPENED && response.headers[header]) {
+			return header + ": " + response.headers[header];
+		}
+		
+		return null;
+	};
+	
+	/**
+	 * Gets all the response headers.
+	 *
+	 * @return string 
+	 */
+	this.getAllResponseHeaders = function() {
+		if (this.readyState < this.HEADERS_RECEIVED) {
+			throw "INVALID_STATE_ERR: Headers have not been received.";
+		}
+		var result = "";
+		
+		for (var i in response.headers) {
+			result += i + ": " + response.headers[i] + "\r\n";
+		}
+		return result.substr(0, result.length - 2);
+	};
+
+	/**
+	 * Sends the request to the server.
+	 *
+	 * @param string data Optional data to send as request body.
+	 */
+	this.send = function(data) {
+		if (this.readyState != this.OPENED) {
+			throw "INVALID_STATE_ERR: connection must be opened before send() is called";
+		}
+		
+		var ssl = false;
+		var url = Url.parse(settings.url);
+		
+		// Determine the server
+		switch (url.protocol) {
+			case 'https:':
+				ssl = true;
+				// SSL & non-SSL both need host, no break here.
+			case 'http:':
+				var host = url.hostname;
+				break;
+			
+			case undefined:
+			case '':
+				var host = "localhost";
+				break;
+			
+			default:
+				throw "Protocol not supported.";
+		}
+
+		// Default to port 80. If accessing localhost on another port be sure
+		// to use http://localhost:port/path
+		var port = url.port || (ssl ? 443 : 80);
+		// Add query string if one is used
+		var uri = url.pathname + (url.search ? url.search : '');
+		
+		// Set the Host header or the server may reject the request
+		this.setRequestHeader("Host", host);
+		
+		// Set content length header
+		if (settings.method == "GET" || settings.method == "HEAD") {
+			data = null;
+		} else if (data) {
+			this.setRequestHeader("Content-Length", Buffer.byteLength(data));
+			
+			if (!headers["Content-Type"]) {
+				this.setRequestHeader("Content-Type", "text/plain;charset=UTF-8");
+			}
+		}
+
+		// Use the proper protocol
+		var doRequest = ssl ? https.request : http.request;
+
+		var options = {
+		    host: host,
+		    port: port,
+		    path: uri,
+		    method: settings.method,
+		    headers: headers, 
+                    agent: false
+		};
+		
+		var req = doRequest(options, function(res) {
+			response = res;
+			response.setEncoding("utf8");
+
+			setState(self.HEADERS_RECEIVED);
+			self.status = response.statusCode;
+
+			response.on('data', function(chunk) {
+				// Make sure there's some data
+				if (chunk) {
+					self.responseText += chunk;
+				}
+				setState(self.LOADING);
+			});
+
+			response.on('end', function() {
+				setState(self.DONE);
+			});
+
+			response.on('error', function() {
+				self.handleError(error);
+			});
+		}).on('error', function(error) {
+			self.handleError(error);
+		});
+
+		req.setHeader("Connection", "Close");
+
+		// Node 0.4 and later won't accept empty data. Make sure it's needed.
+		if (data) {
+			req.write(data);
+		}
+
+		req.end();
+	};
+
+	this.handleError = function(error) {
+		this.status = 503;
+		this.statusText = error;
+		this.responseText = error.stack;
+		setState(this.DONE);
+	};
+
+	/**
+	 * Aborts a request.
+	 */
+	this.abort = function() {
+		headers = defaultHeaders;
+		this.readyState = this.UNSENT;
+		this.responseText = "";
+		this.responseXML = "";
+	};
+	
+	/**
+	 * Changes readyState and calls onreadystatechange.
+	 *
+	 * @param int state New state
+	 */
+	var setState = function(state) {
+		self.readyState = state;
+		self.onreadystatechange();
+	}
+};

contrib/jitsimeetbridge/unjingle/strophe/base64.js

@@ -0,0 +1,83 @@
+// This code was written by Tyler Akins and has been placed in the
+// public domain.  It would be nice if you left this header intact.
+// Base64 code from Tyler Akins -- http://rumkin.com
+
+var Base64 = (function () {
+    var keyStr = "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/=";
+
+    var obj = {
+        /**
+         * Encodes a string in base64
+         * @param {String} input The string to encode in base64.
+         */
+        encode: function (input) {
+            var output = "";
+            var chr1, chr2, chr3;
+            var enc1, enc2, enc3, enc4;
+            var i = 0;
+
+            do {
+                chr1 = input.charCodeAt(i++);
+                chr2 = input.charCodeAt(i++);
+                chr3 = input.charCodeAt(i++);
+
+                enc1 = chr1 >> 2;
+                enc2 = ((chr1 & 3) << 4) | (chr2 >> 4);
+                enc3 = ((chr2 & 15) << 2) | (chr3 >> 6);
+                enc4 = chr3 & 63;
+
+                if (isNaN(chr2)) {
+                    enc3 = enc4 = 64;
+                } else if (isNaN(chr3)) {
+                    enc4 = 64;
+                }
+
+                output = output + keyStr.charAt(enc1) + keyStr.charAt(enc2) +
+                    keyStr.charAt(enc3) + keyStr.charAt(enc4);
+            } while (i < input.length);
+
+            return output;
+        },
+
+        /**
+         * Decodes a base64 string.
+         * @param {String} input The string to decode.
+         */
+        decode: function (input) {
+            var output = "";
+            var chr1, chr2, chr3;
+            var enc1, enc2, enc3, enc4;
+            var i = 0;
+
+            // remove all characters that are not A-Z, a-z, 0-9, +, /, or =
+            input = input.replace(/[^A-Za-z0-9\+\/\=]/g, '');
+
+            do {
+                enc1 = keyStr.indexOf(input.charAt(i++));
+                enc2 = keyStr.indexOf(input.charAt(i++));
+                enc3 = keyStr.indexOf(input.charAt(i++));
+                enc4 = keyStr.indexOf(input.charAt(i++));
+
+                chr1 = (enc1 << 2) | (enc2 >> 4);
+                chr2 = ((enc2 & 15) << 4) | (enc3 >> 2);
+                chr3 = ((enc3 & 3) << 6) | enc4;
+
+                output = output + String.fromCharCode(chr1);
+
+                if (enc3 != 64) {
+                    output = output + String.fromCharCode(chr2);
+                }
+                if (enc4 != 64) {
+                    output = output + String.fromCharCode(chr3);
+                }
+            } while (i < input.length);
+
+            return output;
+        }
+    };
+
+    return obj;
+})();
+
+// Nodify
+exports.Base64 = Base64;

contrib/jitsimeetbridge/unjingle/strophe/md5.js

@@ -0,0 +1,279 @@
+/*
+ * A JavaScript implementation of the RSA Data Security, Inc. MD5 Message
+ * Digest Algorithm, as defined in RFC 1321.
+ * Version 2.1 Copyright (C) Paul Johnston 1999 - 2002.
+ * Other contributors: Greg Holt, Andrew Kepert, Ydnar, Lostinet
+ * Distributed under the BSD License
+ * See http://pajhome.org.uk/crypt/md5 for more info.
+ */
+
+var MD5 = (function () {
+    /*
+     * Configurable variables. You may need to tweak these to be compatible with
+     * the server-side, but the defaults work in most cases.
+     */
+    var hexcase = 0;  /* hex output format. 0 - lowercase; 1 - uppercase */
+    var b64pad  = ""; /* base-64 pad character. "=" for strict RFC compliance */
+    var chrsz   = 8;  /* bits per input character. 8 - ASCII; 16 - Unicode */
+
+    /*
+     * Add integers, wrapping at 2^32. This uses 16-bit operations internally
+     * to work around bugs in some JS interpreters.
+     */
+    var safe_add = function (x, y) {
+        var lsw = (x & 0xFFFF) + (y & 0xFFFF);
+        var msw = (x >> 16) + (y >> 16) + (lsw >> 16);
+        return (msw << 16) | (lsw & 0xFFFF);
+    };
+
+    /*
+     * Bitwise rotate a 32-bit number to the left.
+     */
+    var bit_rol = function (num, cnt) {
+        return (num << cnt) | (num >>> (32 - cnt));
+    };
+
+    /*
+     * Convert a string to an array of little-endian words
+     * If chrsz is ASCII, characters >255 have their hi-byte silently ignored.
+     */
+    var str2binl = function (str) {
+        var bin = [];
+        var mask = (1 << chrsz) - 1;
+        for(var i = 0; i < str.length * chrsz; i += chrsz)
+        {
+            bin[i>>5] |= (str.charCodeAt(i / chrsz) & mask) << (i%32);
+        }
+        return bin;
+    };
+
+    /*
+     * Convert an array of little-endian words to a string
+     */
+    var binl2str = function (bin) {
+        var str = "";
+        var mask = (1 << chrsz) - 1;
+        for(var i = 0; i < bin.length * 32; i += chrsz)
+        {
+            str += String.fromCharCode((bin[i>>5] >>> (i % 32)) & mask);
+        }
+        return str;
+    };
+
+    /*
+     * Convert an array of little-endian words to a hex string.
+     */
+    var binl2hex = function (binarray) {
+        var hex_tab = hexcase ? "0123456789ABCDEF" : "0123456789abcdef";
+        var str = "";
+        for(var i = 0; i < binarray.length * 4; i++)
+        {
+            str += hex_tab.charAt((binarray[i>>2] >> ((i%4)*8+4)) & 0xF) +
+                hex_tab.charAt((binarray[i>>2] >> ((i%4)*8  )) & 0xF);
+        }
+        return str;
+    };
+
+    /*
+     * Convert an array of little-endian words to a base-64 string
+     */
+    var binl2b64 = function (binarray) {
+        var tab = "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/";
+        var str = "";
+        var triplet, j;
+        for(var i = 0; i < binarray.length * 4; i += 3)
+        {
+            triplet = (((binarray[i   >> 2] >> 8 * ( i   %4)) & 0xFF) << 16) |
+                (((binarray[i+1 >> 2] >> 8 * ((i+1)%4)) & 0xFF) << 8 ) |
+                ((binarray[i+2 >> 2] >> 8 * ((i+2)%4)) & 0xFF);
+            for(j = 0; j < 4; j++)
+            {
+                if(i * 8 + j * 6 > binarray.length * 32) { str += b64pad; }
+                else { str += tab.charAt((triplet >> 6*(3-j)) & 0x3F); }
+            }
+        }
+        return str;
+    };
+
+    /*
+     * These functions implement the four basic operations the algorithm uses.
+     */
+    var md5_cmn = function (q, a, b, x, s, t) {
+        return safe_add(bit_rol(safe_add(safe_add(a, q),safe_add(x, t)), s),b);
+    };
+
+    var md5_ff = function (a, b, c, d, x, s, t) {
+        return md5_cmn((b & c) | ((~b) & d), a, b, x, s, t);
+    };
+
+    var md5_gg = function (a, b, c, d, x, s, t) {
+        return md5_cmn((b & d) | (c & (~d)), a, b, x, s, t);
+    };
+
+    var md5_hh = function (a, b, c, d, x, s, t) {
+        return md5_cmn(b ^ c ^ d, a, b, x, s, t);
+    };
+
+    var md5_ii = function (a, b, c, d, x, s, t) {
+        return md5_cmn(c ^ (b | (~d)), a, b, x, s, t);
+    };
+
+    /*
+     * Calculate the MD5 of an array of little-endian words, and a bit length
+     */
+    var core_md5 = function (x, len) {
+        /* append padding */
+        x[len >> 5] |= 0x80 << ((len) % 32);
+        x[(((len + 64) >>> 9) << 4) + 14] = len;
+
+        var a =  1732584193;
+        var b = -271733879;
+        var c = -1732584194;
+        var d =  271733878;
+
+        var olda, oldb, oldc, oldd;
+        for (var i = 0; i < x.length; i += 16)
+        {
+            olda = a;
+            oldb = b;
+            oldc = c;
+            oldd = d;
+
+            a = md5_ff(a, b, c, d, x[i+ 0], 7 , -680876936);
+            d = md5_ff(d, a, b, c, x[i+ 1], 12, -389564586);
+            c = md5_ff(c, d, a, b, x[i+ 2], 17,  606105819);
+            b = md5_ff(b, c, d, a, x[i+ 3], 22, -1044525330);
+            a = md5_ff(a, b, c, d, x[i+ 4], 7 , -176418897);
+            d = md5_ff(d, a, b, c, x[i+ 5], 12,  1200080426);
+            c = md5_ff(c, d, a, b, x[i+ 6], 17, -1473231341);
+            b = md5_ff(b, c, d, a, x[i+ 7], 22, -45705983);
+            a = md5_ff(a, b, c, d, x[i+ 8], 7 ,  1770035416);
+            d = md5_ff(d, a, b, c, x[i+ 9], 12, -1958414417);
+            c = md5_ff(c, d, a, b, x[i+10], 17, -42063);
+            b = md5_ff(b, c, d, a, x[i+11], 22, -1990404162);
+            a = md5_ff(a, b, c, d, x[i+12], 7 ,  1804603682);
+            d = md5_ff(d, a, b, c, x[i+13], 12, -40341101);
+            c = md5_ff(c, d, a, b, x[i+14], 17, -1502002290);
+            b = md5_ff(b, c, d, a, x[i+15], 22,  1236535329);
+
+            a = md5_gg(a, b, c, d, x[i+ 1], 5 , -165796510);
+            d = md5_gg(d, a, b, c, x[i+ 6], 9 , -1069501632);
+            c = md5_gg(c, d, a, b, x[i+11], 14,  643717713);
+            b = md5_gg(b, c, d, a, x[i+ 0], 20, -373897302);
+            a = md5_gg(a, b, c, d, x[i+ 5], 5 , -701558691);
+            d = md5_gg(d, a, b, c, x[i+10], 9 ,  38016083);
+            c = md5_gg(c, d, a, b, x[i+15], 14, -660478335);
+            b = md5_gg(b, c, d, a, x[i+ 4], 20, -405537848);
+            a = md5_gg(a, b, c, d, x[i+ 9], 5 ,  568446438);
+            d = md5_gg(d, a, b, c, x[i+14], 9 , -1019803690);
+            c = md5_gg(c, d, a, b, x[i+ 3], 14, -187363961);
+            b = md5_gg(b, c, d, a, x[i+ 8], 20,  1163531501);
+            a = md5_gg(a, b, c, d, x[i+13], 5 , -1444681467);
+            d = md5_gg(d, a, b, c, x[i+ 2], 9 , -51403784);
+            c = md5_gg(c, d, a, b, x[i+ 7], 14,  1735328473);
+            b = md5_gg(b, c, d, a, x[i+12], 20, -1926607734);
+
+            a = md5_hh(a, b, c, d, x[i+ 5], 4 , -378558);
+            d = md5_hh(d, a, b, c, x[i+ 8], 11, -2022574463);
+            c = md5_hh(c, d, a, b, x[i+11], 16,  1839030562);
+            b = md5_hh(b, c, d, a, x[i+14], 23, -35309556);
+            a = md5_hh(a, b, c, d, x[i+ 1], 4 , -1530992060);
+            d = md5_hh(d, a, b, c, x[i+ 4], 11,  1272893353);
+            c = md5_hh(c, d, a, b, x[i+ 7], 16, -155497632);
+            b = md5_hh(b, c, d, a, x[i+10], 23, -1094730640);
+            a = md5_hh(a, b, c, d, x[i+13], 4 ,  681279174);
+            d = md5_hh(d, a, b, c, x[i+ 0], 11, -358537222);
+            c = md5_hh(c, d, a, b, x[i+ 3], 16, -722521979);
+            b = md5_hh(b, c, d, a, x[i+ 6], 23,  76029189);
+            a = md5_hh(a, b, c, d, x[i+ 9], 4 , -640364487);
+            d = md5_hh(d, a, b, c, x[i+12], 11, -421815835);
+            c = md5_hh(c, d, a, b, x[i+15], 16,  530742520);
+            b = md5_hh(b, c, d, a, x[i+ 2], 23, -995338651);
+
+            a = md5_ii(a, b, c, d, x[i+ 0], 6 , -198630844);
+            d = md5_ii(d, a, b, c, x[i+ 7], 10,  1126891415);
+            c = md5_ii(c, d, a, b, x[i+14], 15, -1416354905);
+            b = md5_ii(b, c, d, a, x[i+ 5], 21, -57434055);
+            a = md5_ii(a, b, c, d, x[i+12], 6 ,  1700485571);
+            d = md5_ii(d, a, b, c, x[i+ 3], 10, -1894986606);
+            c = md5_ii(c, d, a, b, x[i+10], 15, -1051523);
+            b = md5_ii(b, c, d, a, x[i+ 1], 21, -2054922799);
+            a = md5_ii(a, b, c, d, x[i+ 8], 6 ,  1873313359);
+            d = md5_ii(d, a, b, c, x[i+15], 10, -30611744);
+            c = md5_ii(c, d, a, b, x[i+ 6], 15, -1560198380);
+            b = md5_ii(b, c, d, a, x[i+13], 21,  1309151649);
+            a = md5_ii(a, b, c, d, x[i+ 4], 6 , -145523070);
+            d = md5_ii(d, a, b, c, x[i+11], 10, -1120210379);
+            c = md5_ii(c, d, a, b, x[i+ 2], 15,  718787259);
+            b = md5_ii(b, c, d, a, x[i+ 9], 21, -343485551);
+
+            a = safe_add(a, olda);
+            b = safe_add(b, oldb);
+            c = safe_add(c, oldc);
+            d = safe_add(d, oldd);
+        }
+        return [a, b, c, d];
+    };
+
+
+    /*
+     * Calculate the HMAC-MD5, of a key and some data
+     */
+    var core_hmac_md5 = function (key, data) {
+        var bkey = str2binl(key);
+        if(bkey.length > 16) { bkey = core_md5(bkey, key.length * chrsz); }
+
+        var ipad = new Array(16), opad = new Array(16);
+        for(var i = 0; i < 16; i++)
+        {
+            ipad[i] = bkey[i] ^ 0x36363636;
+            opad[i] = bkey[i] ^ 0x5C5C5C5C;
+        }
+
+        var hash = core_md5(ipad.concat(str2binl(data)), 512 + data.length * chrsz);
+        return core_md5(opad.concat(hash), 512 + 128);
+    };
+
+    var obj = {
+        /*
+         * These are the functions you'll usually want to call.
+         * They take string arguments and return either hex or base-64 encoded
+         * strings.
+         */
+        hexdigest: function (s) {
+            return binl2hex(core_md5(str2binl(s), s.length * chrsz));
+        },
+
+        b64digest: function (s) {
+            return binl2b64(core_md5(str2binl(s), s.length * chrsz));
+        },
+
+        hash: function (s) {
+            return binl2str(core_md5(str2binl(s), s.length * chrsz));
+        },
+
+        hmac_hexdigest: function (key, data) {
+            return binl2hex(core_hmac_md5(key, data));
+        },
+
+        hmac_b64digest: function (key, data) {
+            return binl2b64(core_hmac_md5(key, data));
+        },
+
+        hmac_hash: function (key, data) {
+            return binl2str(core_hmac_md5(key, data));
+        },
+
+        /*
+         * Perform a simple self-test to see if the VM is working
+         */
+        test: function () {
+            return MD5.hexdigest("abc") === "900150983cd24fb0d6963f7d28e17f72";
+        }
+    };
+
+    return obj;
+})();
+
+// Nodify
+exports.MD5 = MD5;

contrib/jitsimeetbridge/unjingle/unjingle.js

@@ -0,0 +1,48 @@
+var strophe = require("./strophe/strophe.js").Strophe;
+
+var Strophe = strophe.Strophe;
+var $iq     = strophe.$iq;
+var $msg    = strophe.$msg;
+var $build  = strophe.$build;
+var $pres   = strophe.$pres;
+
+var jsdom = require("jsdom");
+var window = jsdom.jsdom().parentWindow;
+var $ = require('jquery')(window);
+
+var stropheJingle = require("./strophe.jingle.sdp.js");
+
+
+var input = '';
+
+process.stdin.on('readable', function() {
+  var chunk = process.stdin.read();
+  if (chunk !== null) {
+    input += chunk;
+  }
+});
+
+process.stdin.on('end', function() {
+	if (process.argv[2] == '--jingle') {
+		var elem = $(input);
+		// app does:
+		// sess.setRemoteDescription($(iq).find('>jingle'), 'offer');
+		//console.log(elem.find('>content'));
+		var sdp = new stropheJingle.SDP('');
+		sdp.fromJingle(elem);
+		console.log(sdp.raw);
+	} else if (process.argv[2] == '--sdp') {
+		var sdp = new stropheJingle.SDP(input);
+		var accept = $iq({to: '%(tojid)s',
+			type: 'set'})
+			.c('jingle', {xmlns: 'urn:xmpp:jingle:1',
+			    //action: 'session-accept',
+			    action: '%(action)s',
+			    initiator: '%(initiator)s',
+			    responder: '%(responder)s',
+			    sid: '%(sid)s' });
+		sdp.toJingle(accept, 'responder');
+		console.log(Strophe.serialize(accept));
+	}
+});
+

contrib/vertobot/.gitignore

@@ -0,0 +1,2 @@
+vucbot.yaml
+vertobot.yaml

contrib/vertobot/bot.pl

@@ -0,0 +1,300 @@
+#!/usr/bin/env perl 
+
+use strict;
+use warnings;
+use 5.010; # //
+use IO::Socket::SSL qw(SSL_VERIFY_NONE);
+use IO::Async::Loop;
+use Net::Async::WebSocket::Client;
+use Net::Async::Matrix 0.11_002;
+use JSON;
+use YAML;
+use Data::UUID;
+use Getopt::Long;
+use Data::Dumper;
+
+binmode STDOUT, ":encoding(UTF-8)";
+binmode STDERR, ":encoding(UTF-8)";
+
+my $loop = IO::Async::Loop->new;
+# Net::Async::HTTP + SSL + IO::Poll doesn't play well. See
+#   https://rt.cpan.org/Ticket/Display.html?id=93107
+ref $loop eq "IO::Async::Loop::Poll" and
+    warn "Using SSL with IO::Poll causes known memory-leaks!!\n";
+
+GetOptions(
+   'C|config=s' => \my $CONFIG,
+   'eval-from=s' => \my $EVAL_FROM,
+) or exit 1;
+
+if( defined $EVAL_FROM ) {
+    # An emergency 'eval() this file' hack
+    $SIG{HUP} = sub {
+        my $code = do {
+            open my $fh, "<", $EVAL_FROM or warn( "Cannot read - $!" ), return;
+            local $/; <$fh>
+        };
+
+        eval $code or warn "Cannot eval() - $@";
+    };
+}
+
+defined $CONFIG or die "Must supply --config\n";
+
+my %CONFIG = %{ YAML::LoadFile( $CONFIG ) };
+
+my %MATRIX_CONFIG = %{ $CONFIG{matrix} };
+# No harm in always applying this
+$MATRIX_CONFIG{SSL_verify_mode} = SSL_VERIFY_NONE;
+
+# Track every Room object, so we can ->leave them all on shutdown
+my %bot_matrix_rooms;
+
+my $bridgestate = {};
+my $roomid_by_callid = {};
+
+my $bot_verto = Net::Async::WebSocket::Client->new(
+    on_frame => sub {
+          my ( $self, $frame ) = @_;
+          warn "[Verto] receiving $frame";
+          on_verto_json($frame);
+    },
+);
+$loop->add( $bot_verto );
+
+my $sessid = lc new Data::UUID->create_str();
+
+my $bot_matrix = Net::Async::Matrix->new(
+    %MATRIX_CONFIG,
+    on_log => sub { warn "log: @_\n" },
+    on_invite => sub {
+        my ($matrix, $invite) = @_;
+        warn "[Matrix] invited to: " . $invite->{room_id} . " by " . $invite->{inviter} . "\n";
+        
+        $matrix->join_room( $invite->{room_id} )->get;
+    },
+    on_room_new => sub {
+        my ($matrix, $room) = @_;
+
+        warn "[Matrix] have a room ID: " . $room->room_id . "\n";
+
+        $bot_matrix_rooms{$room->room_id} = $room;
+        
+        # log in to verto on behalf of this room
+        $bridgestate->{$room->room_id}->{sessid} = $sessid;
+         
+        $room->configure(
+            on_message => \&on_room_message,
+        );
+        
+        my $f = send_verto_json_request("login", {
+                'login' => $CONFIG{'verto-dialog-params'}{'login'},
+                'passwd' => $CONFIG{'verto-config'}{'passwd'},
+                'sessid' => $sessid,
+            });
+        $matrix->adopt_future($f);
+        
+        # we deliberately don't paginate the room, as we only care about
+        # new calls
+    },
+    on_unknown_event => \&on_unknown_event,
+    on_error => sub {
+        print STDERR "Matrix failure: @_\n";
+    },
+);
+$loop->add( $bot_matrix );
+
+sub on_unknown_event
+{
+    my ($matrix, $event) = @_;
+    print Dumper($event);
+    
+    my $room_id = $event->{room_id};
+    my %dp = %{$CONFIG{'verto-dialog-params'}};
+    $dp{callID} = $bridgestate->{$room_id}->{callid};
+    
+    if ($event->{type} eq 'm.call.invite') {
+        $bridgestate->{$room_id}->{matrix_callid} = $event->{content}->{call_id};
+        $bridgestate->{$room_id}->{callid} = lc new Data::UUID->create_str();
+        $bridgestate->{$room_id}->{offer} = $event->{content}->{offer}->{sdp};
+        $bridgestate->{$room_id}->{gathered_candidates} = 0;
+        $roomid_by_callid->{ $bridgestate->{$room_id}->{callid} } = $room_id;
+        # no trickle ICE in verto apparently
+    }
+    elsif ($event->{type} eq 'm.call.candidates') {
+        # XXX: compare call IDs
+        if (!$bridgestate->{$room_id}->{gathered_candidates}) {
+            $bridgestate->{$room_id}->{gathered_candidates} = 1;
+            my $offer = $bridgestate->{$room_id}->{offer};
+            my $candidate_block = "";
+            foreach (@{$event->{content}->{candidates}}) {
+                $candidate_block .= "a=" . $_->{candidate} . "\r\n";
+            }
+            # XXX: collate using the right m= line - for now assume audio call
+            $offer =~ s/(a=rtcp.*[\r\n]+)/$1$candidate_block/;
+            
+            my $f = send_verto_json_request("verto.invite", {
+                "sdp" => $offer,
+                "dialogParams" => \%dp,
+                "sessid" => $bridgestate->{$room_id}->{sessid},
+            });
+            $matrix->adopt_future($f);
+        }
+        else {
+            # ignore them, as no trickle ICE, although we might as well
+            # batch them up
+            # foreach (@{$event->{content}->{candidates}}) {
+            #     push @{$bridgestate->{$room_id}->{candidates}}, $_;
+            # }
+        }
+    }
+    elsif ($event->{type} eq 'm.call.hangup') {
+        if ($bridgestate->{$room_id}->{matrix_callid} eq $event->{content}->{call_id}) {
+            my $f = send_verto_json_request("verto.bye", {
+                "dialogParams" => \%dp,
+                "sessid" => $bridgestate->{$room_id}->{sessid},
+            });
+            $matrix->adopt_future($f);
+        }
+        else {
+            warn "Ignoring unrecognised callid: ".$event->{content}->{call_id};
+        }
+    }
+    else {
+        warn "Unhandled event: $event->{type}";
+    }
+}
+
+sub on_room_message
+{
+    my ($room, $from, $content) = @_;
+    my $room_id = $room->room_id;
+    warn "[Matrix] in $room_id: $from: " . $content->{body} . "\n";    
+}
+
+my $verto_connecting = $loop->new_future;
+$bot_verto->connect(
+    %{ $CONFIG{"verto-bot"} },
+    on_connected => sub {
+        warn("[Verto] connected to websocket");
+        $verto_connecting->done($bot_verto) if not $verto_connecting->is_done;
+    },
+    on_connect_error => sub { die "Cannot connect to verto - $_[-1]" },
+    on_resolve_error => sub { die "Cannot resolve to verto - $_[-1]" },        
+);
+
+Future->needs_all(
+    $bot_matrix->login( %{ $CONFIG{"matrix-bot"} } )->then( sub {
+        $bot_matrix->start;
+    }),
+    
+    $verto_connecting,
+)->get;
+
+$loop->attach_signal(
+    PIPE => sub { warn "pipe\n" }
+);
+$loop->attach_signal(
+    INT => sub { $loop->stop },
+);
+$loop->attach_signal(
+    TERM => sub { $loop->stop },
+);
+
+eval {
+   $loop->run;
+} or my $e = $@;
+
+# When the bot gets shut down, have it leave the rooms so it's clear to observers
+# that it is no longer running.
+# if( $CONFIG{"leave-on-shutdown"} // 1 ) {
+#     print STDERR "Removing bot from Matrix rooms...\n";
+#     Future->wait_all( map { $_->leave->else_done() } values %bot_matrix_rooms )->get;
+# }
+# else {
+#     print STDERR "Leaving bot users in Matrix rooms.\n";
+# }
+
+die $e if $e;
+
+exit 0;
+
+{    
+    my $json_id;
+    my $requests;
+
+    sub send_verto_json_request
+    {
+        $json_id ||= 1;
+        
+        my ($method, $params) = @_;
+        my $json = {
+            jsonrpc => "2.0",
+            method  => $method,
+            params  => $params,
+            id      => $json_id,
+        };
+        my $text = JSON->new->encode( $json );
+        warn "[Verto] sending $text";
+        $bot_verto->send_frame ( $text );
+        my $request = $loop->new_future;
+        $requests->{$json_id} = $request;
+        $json_id++;
+        return $request;
+    }
+    
+    sub send_verto_json_response
+    {
+        my ($result, $id) = @_;
+        my $json = {
+            jsonrpc => "2.0",
+            result  => $result,
+            id      => $id,
+        };
+        my $text = JSON->new->encode( $json );
+        warn "[Verto] sending $text";
+        $bot_verto->send_frame ( $text );
+    }
+    
+    sub on_verto_json
+    {
+        my $json = JSON->new->decode( $_[0] );
+        if ($json->{method}) {
+            if (($json->{method} eq 'verto.answer' && $json->{params}->{sdp}) ||
+                $json->{method} eq 'verto.media') {
+                                
+                my $room_id = $roomid_by_callid->{$json->{params}->{callID}};
+                my $room = $bot_matrix_rooms{$room_id};
+
+                if ($json->{params}->{sdp}) {
+                    # HACK HACK HACK HACK
+                    $room->_do_POST_json( "/send/m.call.answer", {
+                        call_id => $bridgestate->{$room_id}->{matrix_callid},
+                        version => 0,
+                        answer  => {
+                            sdp => $json->{params}->{sdp},
+                            type => "answer",
+                        },
+                    })->then( sub {
+                        send_verto_json_response( {
+                            method => $json->{method},
+                        }, $json->{id});
+                    })->get;
+                }
+            }
+            else {
+                warn ("[Verto] unhandled method: " . $json->{method});
+                send_verto_json_response( {
+                    method => $json->{method},
+                }, $json->{id});
+            }
+        }
+        elsif ($json->{result}) {
+            $requests->{$json->{id}}->done($json->{result});
+        }
+        elsif ($json->{error}) {
+            $requests->{$json->{id}}->fail($json->{error}->{message}, $json->{error});
+        }
+    }
+}
+

contrib/vertobot/bridge.pl

@@ -0,0 +1,489 @@
+#!/usr/bin/env perl 
+
+use strict;
+use warnings;
+use 5.010; # //
+use IO::Socket::SSL qw(SSL_VERIFY_NONE);
+use IO::Async::Loop;
+use Net::Async::WebSocket::Client;
+use Net::Async::HTTP;
+use Net::Async::HTTP::Server;
+use JSON;
+use YAML;
+use Data::UUID;
+use Getopt::Long;
+use Data::Dumper;
+use URI::Encode qw(uri_encode uri_decode);
+    
+binmode STDOUT, ":encoding(UTF-8)";
+binmode STDERR, ":encoding(UTF-8)";
+
+my $msisdn_to_matrix = {
+    '447417892400' => '@matthew:matrix.org',
+};
+
+my $matrix_to_msisdn = {};
+foreach (keys %$msisdn_to_matrix) {
+    $matrix_to_msisdn->{$msisdn_to_matrix->{$_}} = $_;
+}
+
+
+my $loop = IO::Async::Loop->new;
+# Net::Async::HTTP + SSL + IO::Poll doesn't play well. See
+#   https://rt.cpan.org/Ticket/Display.html?id=93107
+# ref $loop eq "IO::Async::Loop::Poll" and
+#     warn "Using SSL with IO::Poll causes known memory-leaks!!\n";
+
+GetOptions(
+   'C|config=s' => \my $CONFIG,
+   'eval-from=s' => \my $EVAL_FROM,
+) or exit 1;
+
+if( defined $EVAL_FROM ) {
+    # An emergency 'eval() this file' hack
+    $SIG{HUP} = sub {
+        my $code = do {
+            open my $fh, "<", $EVAL_FROM or warn( "Cannot read - $!" ), return;
+            local $/; <$fh>
+        };
+
+        eval $code or warn "Cannot eval() - $@";
+    };
+}
+
+defined $CONFIG or die "Must supply --config\n";
+
+my %CONFIG = %{ YAML::LoadFile( $CONFIG ) };
+
+my %MATRIX_CONFIG = %{ $CONFIG{matrix} };
+# No harm in always applying this
+$MATRIX_CONFIG{SSL_verify_mode} = SSL_VERIFY_NONE;
+
+my $bridgestate = {};
+my $roomid_by_callid = {};
+    
+my $sessid = lc new Data::UUID->create_str();    
+my $as_token = $CONFIG{"matrix-bot"}->{as_token};
+my $hs_domain = $CONFIG{"matrix-bot"}->{domain};
+
+my $http = Net::Async::HTTP->new();
+$loop->add( $http );
+
+sub create_virtual_user
+{
+    my ($localpart) = @_;
+    my ( $response ) = $http->do_request(
+        method => "POST",
+        uri => URI->new(
+            $CONFIG{"matrix"}->{server}.
+                "/_matrix/client/api/v1/register?".
+                "access_token=$as_token&user_id=$localpart"
+        ),
+        content_type => "application/json",
+        content => <<EOT
+{
+    "type": "m.login.application_service",
+    "user": "$localpart"
+}
+EOT
+    )->get;    
+    warn $response->as_string if ($response->code != 200);
+}
+    
+my $http_server =  Net::Async::HTTP::Server->new(
+    on_request => sub {
+        my $self = shift;
+        my ( $req ) = @_;
+
+        my $response;
+        my $path = uri_decode($req->path);
+        warn("request: $path");
+        if ($path =~ m#/users/\@(\+.*)#) {
+            # when queried about virtual users, auto-create them in the HS
+            my $localpart = $1;
+            create_virtual_user($localpart);
+            $response = HTTP::Response->new( 200 );
+            $response->add_content('{}');
+            $response->content_type( "application/json" );
+        }
+        elsif ($path =~ m#/transactions/(.*)#) {
+            my $event = JSON->new->decode($req->body);
+            print Dumper($event);
+
+            my $room_id = $event->{room_id};
+            my %dp = %{$CONFIG{'verto-dialog-params'}};
+            $dp{callID} = $bridgestate->{$room_id}->{callid};
+
+            if ($event->{type} eq 'm.room.membership') {
+                my $membership = $event->{content}->{membership};
+                my $state_key = $event->{state_key};
+                my $room_id = $event->{state_id};
+                
+                if ($membership eq 'invite') {
+                    # autojoin invites
+                    my ( $response ) = $http->do_request(
+                        method => "POST",
+                        uri => URI->new(
+                            $CONFIG{"matrix"}->{server}.
+                                "/_matrix/client/api/v1/rooms/$room_id/join?".
+                                "access_token=$as_token&user_id=$state_key"
+                        ),
+                        content_type => "application/json",
+                        content => "{}",
+                    )->get;
+                    warn $response->as_string if ($response->code != 200);
+                }
+            }
+            elsif ($event->{type} eq 'm.call.invite') {
+                my $room_id = $event->{room_id};
+                $bridgestate->{$room_id}->{matrix_callid} = $event->{content}->{call_id};
+                $bridgestate->{$room_id}->{callid} = lc new Data::UUID->create_str();
+                $bridgestate->{$room_id}->{sessid} = $sessid;                
+                # $bridgestate->{$room_id}->{offer} = $event->{content}->{offer}->{sdp};
+                my $offer = $event->{content}->{offer}->{sdp};
+                # $bridgestate->{$room_id}->{gathered_candidates} = 0;
+                $roomid_by_callid->{ $bridgestate->{$room_id}->{callid} } = $room_id;
+                # no trickle ICE in verto apparently
+
+                my $f = send_verto_json_request("verto.invite", {
+                    "sdp" => $offer,
+                    "dialogParams" => \%dp,
+                    "sessid" => $bridgestate->{$room_id}->{sessid},
+                });
+                $self->adopt_future($f);
+            }
+            # elsif ($event->{type} eq 'm.call.candidates') {
+            #     # XXX: this could fire for both matrix->verto and verto->matrix calls
+            #     # and races as it collects candidates. much better to just turn off
+            #     # candidate gathering in the webclient entirely for now
+            #     
+            #     my $room_id = $event->{room_id};
+            #     # XXX: compare call IDs
+            #     if (!$bridgestate->{$room_id}->{gathered_candidates}) {
+            #         $bridgestate->{$room_id}->{gathered_candidates} = 1;
+            #         my $offer = $bridgestate->{$room_id}->{offer};
+            #         my $candidate_block = "";
+            #         foreach (@{$event->{content}->{candidates}}) {
+            #             $candidate_block .= "a=" . $_->{candidate} . "\r\n";
+            #         }
+            #         # XXX: collate using the right m= line - for now assume audio call
+            #         $offer =~ s/(a=rtcp.*[\r\n]+)/$1$candidate_block/;
+            #     
+            #         my $f = send_verto_json_request("verto.invite", {
+            #             "sdp" => $offer,
+            #             "dialogParams" => \%dp,
+            #             "sessid" => $bridgestate->{$room_id}->{sessid},
+            #         });
+            #         $self->adopt_future($f);
+            #     }
+            #     else {
+            #         # ignore them, as no trickle ICE, although we might as well
+            #         # batch them up
+            #         # foreach (@{$event->{content}->{candidates}}) {
+            #         #     push @{$bridgestate->{$room_id}->{candidates}}, $_;
+            #         # }
+            #     }
+            # }
+            elsif ($event->{type} eq 'm.call.answer') {
+                # grab the answer and relay it to verto as a verto.answer
+                my $room_id = $event->{room_id};
+                
+                my $answer = $event->{content}->{answer}->{sdp};
+                my $f = send_verto_json_request("verto.answer", {
+                    "sdp" => $answer,
+                    "dialogParams" => \%dp,
+                    "sessid" => $bridgestate->{$room_id}->{sessid},
+                });
+                $self->adopt_future($f);
+            }
+            elsif ($event->{type} eq 'm.call.hangup') {
+                my $room_id = $event->{room_id};
+                if ($bridgestate->{$room_id}->{matrix_callid} eq $event->{content}->{call_id}) {
+                    my $f = send_verto_json_request("verto.bye", {
+                        "dialogParams" => \%dp,
+                        "sessid" => $bridgestate->{$room_id}->{sessid},
+                    });
+                    $self->adopt_future($f);
+                }
+                else {
+                    warn "Ignoring unrecognised callid: ".$event->{content}->{call_id};
+                }
+            }
+            else {
+                warn "Unhandled event: $event->{type}";
+            }
+            
+            $response = HTTP::Response->new( 200 );
+            $response->add_content('{}');
+            $response->content_type( "application/json" );            
+        }
+        else {
+            warn "Unhandled path: $path";
+            $response = HTTP::Response->new( 404 );
+        }
+
+        $req->respond( $response );
+    },
+);
+$loop->add( $http_server );
+
+$http_server->listen(
+    addr => { family => "inet", socktype => "stream", port => 8009 },
+    on_listen_error => sub { die "Cannot listen - $_[-1]\n" },
+);
+
+my $bot_verto = Net::Async::WebSocket::Client->new(
+    on_frame => sub {
+          my ( $self, $frame ) = @_;
+          warn "[Verto] receiving $frame";
+          on_verto_json($frame);
+    },
+);
+$loop->add( $bot_verto );
+
+my $verto_connecting = $loop->new_future;
+$bot_verto->connect(
+    %{ $CONFIG{"verto-bot"} },
+    on_connected => sub {
+        warn("[Verto] connected to websocket");
+        if (not $verto_connecting->is_done) {
+            $verto_connecting->done($bot_verto);
+
+            send_verto_json_request("login", {
+                'login' => $CONFIG{'verto-dialog-params'}{'login'},
+                'passwd' => $CONFIG{'verto-config'}{'passwd'},
+                'sessid' => $sessid,
+            });
+        }
+    },
+    on_connect_error => sub { die "Cannot connect to verto - $_[-1]" },
+    on_resolve_error => sub { die "Cannot resolve to verto - $_[-1]" },        
+);
+
+# die Dumper($verto_connecting);
+
+my $as_url = $CONFIG{"matrix-bot"}->{as_url};
+
+Future->needs_all(
+    $http->do_request(
+        method => "POST",
+        uri => URI->new( $CONFIG{"matrix"}->{server}."/_matrix/appservice/v1/register" ),
+        content_type => "application/json",
+        content => <<EOT
+{
+    "as_token": "$as_token",
+    "url": "$as_url",
+    "namespaces": { "users": ["\@\\\\+.*"] }
+}
+EOT
+    ),
+    $verto_connecting,
+)->get;
+
+$loop->attach_signal(
+    PIPE => sub { warn "pipe\n" }
+);
+$loop->attach_signal(
+    INT => sub { $loop->stop },
+);
+$loop->attach_signal(
+    TERM => sub { $loop->stop },
+);
+
+eval {
+   $loop->run;
+} or my $e = $@;
+
+die $e if $e;
+
+exit 0;
+
+{    
+    my $json_id;
+    my $requests;
+
+    sub send_verto_json_request
+    {
+        $json_id ||= 1;
+        
+        my ($method, $params) = @_;
+        my $json = {
+            jsonrpc => "2.0",
+            method  => $method,
+            params  => $params,
+            id      => $json_id,
+        };
+        my $text = JSON->new->encode( $json );
+        warn "[Verto] sending $text";
+        $bot_verto->send_frame ( $text );
+        my $request = $loop->new_future;
+        $requests->{$json_id} = $request;
+        $json_id++;
+        return $request;
+    }
+    
+    sub send_verto_json_response
+    {
+        my ($result, $id) = @_;
+        my $json = {
+            jsonrpc => "2.0",
+            result  => $result,
+            id      => $id,
+        };
+        my $text = JSON->new->encode( $json );
+        warn "[Verto] sending $text";
+        $bot_verto->send_frame ( $text );
+    }
+    
+    sub on_verto_json
+    {
+        my $json = JSON->new->decode( $_[0] );
+        if ($json->{method}) {
+            if (($json->{method} eq 'verto.answer' && $json->{params}->{sdp}) ||
+                $json->{method} eq 'verto.media') {
+
+                my $caller = $json->{dialogParams}->{caller_id_number};
+                my $callee = $json->{dialogParams}->{destination_number};
+                my $caller_user = '@+' . $caller . ':' . $hs_domain;
+                my $callee_user = $msisdn_to_matrix->{$callee} || warn "unrecogised callee: $callee";                                
+                my $room_id = $roomid_by_callid->{$json->{params}->{callID}};
+
+                if ($json->{params}->{sdp}) {
+                    $http->do_request(
+                        method => "POST",
+                        uri => URI->new(
+                            $CONFIG{"matrix"}->{server}.
+                                "/_matrix/client/api/v1/send/m.call.answer?".
+                                "access_token=$as_token&user_id=$caller_user"
+                        ),
+                        content_type => "application/json",
+                        content => JSON->new->encode({
+                            call_id => $bridgestate->{$room_id}->{matrix_callid},
+                            version => 0,
+                            answer  => {
+                                sdp => $json->{params}->{sdp},
+                                type => "answer",
+                            },
+                        }),
+                    )->then( sub {
+                        send_verto_json_response( {
+                            method => $json->{method},
+                        }, $json->{id});
+                    })->get;
+                }
+            }
+            elsif ($json->{method} eq 'verto.invite') {
+                my $caller = $json->{dialogParams}->{caller_id_number};
+                my $callee = $json->{dialogParams}->{destination_number};
+                my $caller_user = '@+' . $caller . ':' . $hs_domain;
+                my $callee_user = $msisdn_to_matrix->{$callee} || warn "unrecogised callee: $callee";
+                    
+                my $alias = ($caller lt $callee) ? ($caller.'-'.$callee) : ($callee.'-'.$caller);
+                my $room_id;
+
+                # create a virtual user for the caller if needed.
+                create_virtual_user($caller);
+                
+                # create a room of form #peer-peer and invite the callee
+                $http->do_request(
+                    method => "POST",
+                    uri => URI->new(
+                        $CONFIG{"matrix"}->{server}.
+                            "/_matrix/client/api/v1/createRoom?".
+                            "access_token=$as_token&user_id=$caller_user"
+                    ),
+                    content_type => "application/json",
+                    content => JSON->new->encode({
+                        room_alias_name => $alias,
+                        invite => [ $callee_user ],
+                    }),
+                )->then( sub {
+                    my ( $response ) = @_;
+                    my $resp = JSON->new->decode($response->content);
+                    $room_id = $resp->{room_id};
+                    $roomid_by_callid->{$json->{params}->{callID}} = $room_id;
+                })->get;
+
+                # join it
+                my ($response) = $http->do_request(
+                    method => "POST",
+                    uri => URI->new(
+                        $CONFIG{"matrix"}->{server}.
+                            "/_matrix/client/api/v1/join/$room_id?".
+                            "access_token=$as_token&user_id=$caller_user"
+                    ),
+                    content_type => "application/json",
+                    content => '{}',
+                )->get;
+
+                $bridgestate->{$room_id}->{matrix_callid} = lc new Data::UUID->create_str();
+                $bridgestate->{$room_id}->{callid} = $json->{dialogParams}->{callID};
+                $bridgestate->{$room_id}->{sessid} = $sessid;
+
+                # put the m.call.invite in there
+                $http->do_request(
+                    method => "POST",
+                    uri => URI->new(
+                        $CONFIG{"matrix"}->{server}.
+                            "/_matrix/client/api/v1/send/m.call.invite?".
+                            "access_token=$as_token&user_id=$caller_user"
+                    ),
+                    content_type => "application/json",
+                    content => JSON->new->encode({
+                        call_id => $bridgestate->{$room_id}->{matrix_callid},
+                        version => 0,
+                        answer  => {
+                            sdp => $json->{params}->{sdp},
+                            type => "offer",
+                        },
+                    }),
+                )->then( sub {
+                    # acknowledge the verto
+                    send_verto_json_response( {
+                        method => $json->{method},
+                    }, $json->{id});
+                })->get;
+            }
+            elsif ($json->{method} eq 'verto.bye') {
+                my $caller = $json->{dialogParams}->{caller_id_number};
+                my $callee = $json->{dialogParams}->{destination_number};
+                my $caller_user = '@+' . $caller . ':' . $hs_domain;
+                my $callee_user = $msisdn_to_matrix->{$callee} || warn "unrecogised callee: $callee";                                
+                my $room_id = $roomid_by_callid->{$json->{params}->{callID}};
+                
+                # put the m.call.hangup into the room
+                $http->do_request(
+                    method => "POST",
+                    uri => URI->new(
+                        $CONFIG{"matrix"}->{server}.
+                            "/_matrix/client/api/v1/send/m.call.hangup?".
+                            "access_token=$as_token&user_id=$caller_user"
+                    ),
+                    content_type => "application/json",
+                    content => JSON->new->encode({
+                        call_id => $bridgestate->{$room_id}->{matrix_callid},
+                        version => 0,
+                    }),
+                )->then( sub {
+                    # acknowledge the verto
+                    send_verto_json_response( {
+                        method => $json->{method},
+                    }, $json->{id});
+                })->get;
+            }
+            else {
+                warn ("[Verto] unhandled method: " . $json->{method});
+                send_verto_json_response( {
+                    method => $json->{method},
+                }, $json->{id});
+            }
+        }
+        elsif ($json->{result}) {
+            $requests->{$json->{id}}->done($json->{result});
+        }
+        elsif ($json->{error}) {
+            $requests->{$json->{id}}->fail($json->{error}->{message}, $json->{error});
+        }
+    }
+}
+

contrib/vertobot/config.yaml

@@ -0,0 +1,29 @@
+# Generic Matrix connection params
+matrix:
+  server: 'matrix.org'
+  SSL: 1
+
+# Bot-user connection details
+matrix-bot:
+  user_id: '@vertobot:matrix.org'
+  password: ''
+
+verto-bot:
+  host: webrtc.freeswitch.org
+  service: 8081
+  url: "ws://webrtc.freeswitch.org:8081/"
+
+verto-config:
+  passwd: 1234
+  
+verto-dialog-params:
+  useVideo: false
+  useStereo: false
+  tag: "webcam"
+  login: "1008@webrtc.freeswitch.org"
+  destination_number: "9664"
+  caller_id_name: "FreeSWITCH User"
+  caller_id_number: "1008"
+  callID: ""
+  remote_caller_id_name: "Outbound Call"
+  remote_caller_id_number: "9664"

contrib/vertobot/cpanfile

@@ -0,0 +1,17 @@
+requires 'parent', 0;
+requires 'Future', '>= 0.29';
+requires 'Net::Async::Matrix', '>= 0.11_002';
+requires 'Net::Async::Matrix::Utils';
+requires 'Net::Async::WebSocket::Protocol', 0;
+requires 'Data::UUID', 0;
+requires 'IO::Async', '>= 0.63';
+requires 'IO::Async::SSL', 0;
+requires 'IO::Socket::SSL', 0;
+requires 'YAML', 0;
+requires 'JSON', 0;
+requires 'Getopt::Long', 0;
+
+on 'test' => sub {
+	requires 'Test::More', '>= 0.98';
+};
+

contrib/vertobot/verto-example.json

@@ -0,0 +1,207 @@
+# JSON is shown in *reverse* chronological order.
+# Send v. Receive is implicit.
+
+{
+    "jsonrpc": "2.0",
+    "id": 7,
+    "result": {
+        "callID": "12795aa6-2a8d-84ee-ce63-2e82ffe825ef",
+        "message": "CALL ENDED",
+        "causeCode": 16,
+        "cause": "NORMAL_CLEARING",
+        "sessid": "03a11060-3e14-23b6-c620-51b892c52983"
+    }
+}
+
+{
+    "jsonrpc": "2.0",
+    "method": "verto.bye",
+    "params": {
+        "dialogParams": {
+            "useVideo": false,
+            "useStereo": true,
+            "tag": "webcam",
+            "login": "1008@webrtc.freeswitch.org",
+            "destination_number": "9664",
+            "caller_id_name": "FreeSWITCH User",
+            "caller_id_number": "1008",
+            "callID": "12795aa6-2a8d-84ee-ce63-2e82ffe825ef",
+            "remote_caller_id_name": "Outbound Call",
+            "remote_caller_id_number": "9664"
+        },
+        "sessid": "03a11060-3e14-23b6-c620-51b892c52983"
+    },
+    "id": 7
+}
+
+{
+    "jsonrpc": "2.0",
+    "id": 6,
+    "result": {
+        "callID": "12795aa6-2a8d-84ee-ce63-2e82ffe825ef",
+        "action": "toggleHold",
+        "holdState": "active",
+        "sessid": "03a11060-3e14-23b6-c620-51b892c52983"
+    }
+}
+
+{
+    "jsonrpc": "2.0",
+    "method": "verto.modify",
+    "params": {
+        "action": "toggleHold",
+        "dialogParams": {
+            "useVideo": false,
+            "useStereo": true,
+            "tag": "webcam",
+            "login": "1008@webrtc.freeswitch.org",
+            "destination_number": "9664",
+            "caller_id_name": "FreeSWITCH User",
+            "caller_id_number": "1008",
+            "callID": "12795aa6-2a8d-84ee-ce63-2e82ffe825ef",
+            "remote_caller_id_name": "Outbound Call",
+            "remote_caller_id_number": "9664"
+        },
+        "sessid": "03a11060-3e14-23b6-c620-51b892c52983"
+    },
+    "id": 6
+}
+
+{
+    "jsonrpc": "2.0",
+    "id": 5,
+    "result": {
+        "callID": "12795aa6-2a8d-84ee-ce63-2e82ffe825ef",
+        "action": "toggleHold",
+        "holdState": "held",
+        "sessid": "03a11060-3e14-23b6-c620-51b892c52983"
+    }
+}
+
+{
+    "jsonrpc": "2.0",
+    "method": "verto.modify",
+    "params": {
+        "action": "toggleHold",
+        "dialogParams": {
+            "useVideo": false,
+            "useStereo": true,
+            "tag": "webcam",
+            "login": "1008@webrtc.freeswitch.org",
+            "destination_number": "9664",
+            "caller_id_name": "FreeSWITCH User",
+            "caller_id_number": "1008",
+            "callID": "12795aa6-2a8d-84ee-ce63-2e82ffe825ef",
+            "remote_caller_id_name": "Outbound Call",
+            "remote_caller_id_number": "9664"
+        },
+        "sessid": "03a11060-3e14-23b6-c620-51b892c52983"
+    },
+    "id": 5
+}
+
+{
+    "jsonrpc": "2.0",
+    "id": 349819,
+    "result": {
+        "method": "verto.answer"
+    }
+}
+
+{
+    "jsonrpc": "2.0",
+    "id": 349819,
+    "method": "verto.answer",
+    "params": {
+        "callID": "12795aa6-2a8d-84ee-ce63-2e82ffe825ef",
+        "sdp": "v=0\no=FreeSWITCH 1417101432 1417101433 IN IP4 209.105.235.10\ns=FreeSWITCH\nc=IN IP4 209.105.235.10\nt=0 0\na=msid-semantic: WMS jA3rmwLVwUq1iE6TYEYHeLk2YTUlh1Vq\nm=audio 30134 RTP/SAVPF 111 126\na=rtpmap:111 opus/48000/2\na=fmtp:111 minptime=10; stereo=1\na=rtpmap:126 telephone-event/8000\na=silenceSupp:off - - - -\na=ptime:20\na=sendrecv\na=fingerprint:sha-256 F8:72:18:E9:72:89:99:22:5B:F8:B6:C6:C6:0D:C5:9B:B2:FB:BC:CA:8D:AB:13:8A:66:E1:37:38:A0:16:AA:41\na=rtcp-mux\na=rtcp:30134 IN IP4 209.105.235.10\na=ssrc:210967934 cname:rOIEajpw4FocakWY\na=ssrc:210967934 msid:jA3rmwLVwUq1iE6TYEYHeLk2YTUlh1Vq a0\na=ssrc:210967934 mslabel:jA3rmwLVwUq1iE6TYEYHeLk2YTUlh1Vq\na=ssrc:210967934 label:jA3rmwLVwUq1iE6TYEYHeLk2YTUlh1Vqa0\na=ice-ufrag:OKwTmGLapwmxn7OF\na=ice-pwd:MmaMwq8rVmtWxfLbQ7U2Ew3T\na=candidate:2372654928 1 udp 659136 209.105.235.10 30134 typ host generation 0\n"
+    }
+}
+
+{
+    "jsonrpc": "2.0",
+    "id": 4,
+    "result": {
+        "message": "CALL CREATED",
+        "callID": "12795aa6-2a8d-84ee-ce63-2e82ffe825ef",
+        "sessid": "03a11060-3e14-23b6-c620-51b892c52983"
+    }
+}
+
+{
+    "jsonrpc": "2.0",
+    "method": "verto.invite",
+    "params": {
+        "sdp": "v=0\r\no=- 1381685806032722557 2 IN IP4 127.0.0.1\r\ns=-\r\nt=0 0\r\na=group:BUNDLE audio\r\na=msid-semantic: WMS 6OOMyGAyJakjwaOOBtV7WcBCCuIW6PpuXsNg\r\nm=audio 63088 RTP/SAVPF 111 103 104 0 8 106 105 13 126\r\nc=IN IP4 81.138.8.249\r\na=rtcp:63088 IN IP4 81.138.8.249\r\na=candidate:460398169 1 udp 2122260223 10.10.79.10 49945 typ host generation 0\r\na=candidate:460398169 2 udp 2122260223 10.10.79.10 49945 typ host generation 0\r\na=candidate:3460887983 1 udp 2122194687 192.168.1.64 63088 typ host generation 0\r\na=candidate:3460887983 2 udp 2122194687 192.168.1.64 63088 typ host generation 0\r\na=candidate:945327227 1 udp 1685987071 81.138.8.249 63088 typ srflx raddr 192.168.1.64 rport 63088 generation 0\r\na=candidate:945327227 2 udp 1685987071 81.138.8.249 63088 typ srflx raddr 192.168.1.64 rport 63088 generation 0\r\na=candidate:1441981097 1 tcp 1518280447 10.10.79.10 0 typ host tcptype active generation 0\r\na=candidate:1441981097 2 tcp 1518280447 10.10.79.10 0 typ host tcptype active generation 0\r\na=candidate:2160789855 1 tcp 1518214911 192.168.1.64 0 typ host tcptype active generation 0\r\na=candidate:2160789855 2 tcp 1518214911 192.168.1.64 0 typ host tcptype active generation 0\r\na=ice-ufrag:cP4qeRhn0LpcpA88\r\na=ice-pwd:fREmgSkXsDLGUUH1bwfrBQhW\r\na=ice-options:google-ice\r\na=fingerprint:sha-256 AF:35:64:1B:62:8A:EF:27:AE:2B:88:2E:FE:78:29:0B:08:DA:64:6C:DE:02:57:E3:EE:B1:D7:86:B8:36:8F:B0\r\na=setup:actpass\r\na=mid:audio\r\na=extmap:1 urn:ietf:params:rtp-hdrext:ssrc-audio-level\r\na=extmap:3 http://www.webrtc.org/experiments/rtp-hdrext/abs-send-time\r\na=sendrecv\r\na=rtcp-mux\r\na=rtpmap:111 opus/48000/2\r\na=fmtp:111 minptime=10; stereo=1\r\na=rtpmap:103 ISAC/16000\r\na=rtpmap:104 ISAC/32000\r\na=rtpmap:0 PCMU/8000\r\na=rtpmap:8 PCMA/8000\r\na=rtpmap:106 CN/32000\r\na=rtpmap:105 CN/16000\r\na=rtpmap:13 CN/8000\r\na=rtpmap:126 telephone-event/8000\r\na=maxptime:60\r\na=ssrc:558827154 cname:vdKHBNqa17t2gmE3\r\na=ssrc:558827154 msid:6OOMyGAyJakjwaOOBtV7WcBCCuIW6PpuXsNg bf1303fb-9833-4d7d-b9e4-b32cfe04acc3\r\na=ssrc:558827154 mslabel:6OOMyGAyJakjwaOOBtV7WcBCCuIW6PpuXsNg\r\na=ssrc:558827154 label:bf1303fb-9833-4d7d-b9e4-b32cfe04acc3\r\n",
+        "dialogParams": {
+            "useVideo": false,
+            "useStereo": true,
+            "tag": "webcam",
+            "login": "1008@webrtc.freeswitch.org",
+            "destination_number": "9664",
+            "caller_id_name": "FreeSWITCH User",
+            "caller_id_number": "1008",
+            "callID": "12795aa6-2a8d-84ee-ce63-2e82ffe825ef",
+            "remote_caller_id_name": "Outbound Call",
+            "remote_caller_id_number": "9664"
+        },
+        "sessid": "03a11060-3e14-23b6-c620-51b892c52983"
+    },
+    "id": 4
+}
+
+{
+    "jsonrpc": "2.0",
+    "id": 3,
+    "result": {
+        "message": "logged in",
+        "sessid": "03a11060-3e14-23b6-c620-51b892c52983"
+    }
+}
+
+{
+    "jsonrpc": "2.0",
+    "id": 1,
+    "error": {
+        "code": -32000,
+        "message": "Authentication Required"
+    }
+}
+
+{
+    "jsonrpc": "2.0",
+    "method": "login",
+    "params": {
+        "login": "1008@webrtc.freeswitch.org",
+        "passwd": "1234",
+        "sessid": "03a11060-3e14-23b6-c620-51b892c52983"
+    },
+    "id": 3
+}
+
+{
+    "jsonrpc": "2.0",
+    "id": 2,
+    "error": {
+        "code": -32000,
+        "message": "Authentication Required"
+    }
+}
+
+{
+    "jsonrpc": "2.0",
+    "method": "login",
+    "params": {
+        "sessid": "03a11060-3e14-23b6-c620-51b892c52983"
+    },
+    "id": 1
+}
+
+{
+    "jsonrpc": "2.0",
+    "method": "login",
+    "params": {
+        "sessid": "03a11060-3e14-23b6-c620-51b892c52983"
+    },
+    "id": 2
+}

demo/clean.sh

@@ -14,3 +14,4 @@ fi
 find "$DIR" -name "*.log" -delete
 find "$DIR" -name "*.db" -delete
 
+rm -rf $DIR/etc

demo/demo.tls.dh

@@ -0,0 +1,9 @@
+2048-bit DH parameters taken from rfc3526
+-----BEGIN DH PARAMETERS-----
+MIIBCAKCAQEA///////////JD9qiIWjCNMTGYouA3BzRKQJOCIpnzHQCC76mOxOb
+IlFKCHmONATd75UZs806QxswKwpt8l8UN0/hNW1tUcJF5IW1dmJefsb0TELppjft
+awv/XLb0Brft7jhr+1qJn6WunyQRfEsf5kkoZlHs5Fs9wgB8uKFjvwWY2kg2HFXT
+mmkWP6j9JM9fg2VdI9yjrZYcYvNWIIVSu57VKQdwlpZtZww1Tkq8mATxdGwIyhgh
+fDKQXkYuNs474553LBgOhgObJ4Oi7Aeij7XFXfBvTFLJ3ivL9pVYFxg5lUl86pVq
+5RXSJhiY+gUQFXKOWoqsqmj//////////wIBAg==
+-----END DH PARAMETERS-----

demo/start.sh

@@ -6,20 +6,39 @@ CWD=$(pwd)
 
 cd "$DIR/.."
 
+mkdir -p demo/etc
+
+# Check the --no-rate-limit param
+PARAMS=""
+if [ $# -eq 1 ]; then
+    if [ $1 = "--no-rate-limit" ]; then
+	    PARAMS="--rc-messages-per-second 1000 --rc-message-burst-count 1000"
+    fi
+fi
+
 for port in 8080 8081 8082; do
     echo "Starting server on port $port... "
 
+    https_port=$((port + 400))
+
     python -m synapse.app.homeserver \
-        -p "$port" \
-        -H "localhost:$port" \
+        --generate-config \
+        --config-path "demo/etc/$port.config" \
+        -p "$https_port" \
+        --unsecure-port "$port" \
+        -H "localhost:$https_port" \
         -f "$DIR/$port.log" \
         -d "$DIR/$port.db" \
-        -vv \
         -D --pid-file "$DIR/$port.pid" \
-        --manhole $((port + 1000))
+        --manhole $((port + 1000)) \
+        --tls-dh-params-path "demo/demo.tls.dh" \
+        --media-store-path "demo/media_store.$port" \
+		$PARAMS $SYNAPSE_PARAMS \
+
+    python -m synapse.app.homeserver \
+        --config-path "demo/etc/$port.config" \
+        -vv \
+
 done
 
-echo "Starting webclient on port 8000..."
-python "demo/webserver.py" -p 8000 -P "$DIR/webserver.pid" "webclient"
-
 cd "$CWD"

docs/README.rst

@@ -0,0 +1,6 @@
+All matrix-generic documentation now lives in its own project at
+
+github.com/matrix-org/matrix-doc.git
+
+Only Synapse implementation-specific documentation lives here now
+(together with some older stuff will be shortly migrated over to matrix-doc)

docs/ancient_architecture_notes.rst

@@ -1,3 +1,9 @@
+.. WARNING::
+  These architecture notes are spectacularly old, and date back to when Synapse 
+  was just federation code in isolation.  This should be merged into the main
+  spec.
+  
+
 = Server to Server =
 
 == Server to Server Stack ==

docs/architecture.rst

@@ -0,0 +1,68 @@
+Synapse Architecture
+====================
+
+As of the end of Oct 2014, Synapse's overall architecture looks like::
+
+        synapse
+        .-----------------------------------------------------.
+        |                          Notifier                   |
+        |                            ^  |                     |
+        |                            |  |                     |
+        |                  .------------|------.              |
+        |                  | handlers/  |      |              |
+        |                  |            v      |              |
+        |                  | Event*Handler <--------> rest/* <=> Client
+        |                  | Rooms*Handler     |              |
+  HSes <=> federation/* <==> FederationHandler |              |
+        |      |           | PresenceHandler   |              |
+        |      |           | TypingHandler     |              |
+        |      |           '-------------------'              |
+        |      |                 |     |                      |
+        |      |              state/*  |                      |
+        |      |                 |     |                      |
+        |      |                 v     v                      |
+        |      `--------------> storage/*                     |
+        |                          |                          |
+        '--------------------------|--------------------------'
+                                   v
+                                .----.
+                                | DB |
+                                '----'
+
+* Handlers: business logic of synapse itself.  Follows a set contract of BaseHandler:
+
+  - BaseHandler gives us onNewRoomEvent which: (TODO: flesh this out and make it less cryptic):
+ 
+    + handle_state(event)
+    + auth(event)
+    + persist_event(event)
+    + notify notifier or federation(event)
+   
+  - PresenceHandler: use distributor to get EDUs out of Federation.  Very
+    lightweight logic built on the distributor
+  - TypingHandler: use distributor to get EDUs out of Federation.  Very
+    lightweight logic built on the distributor
+  - EventsHandler: handles the events stream...
+  - FederationHandler: - gets PDU from Federation Layer; turns into an event;
+    follows basehandler functionality.
+  - RoomsHandler: does all the room logic, including members - lots of classes in
+    RoomsHandler.
+  - ProfileHandler: talks to the storage to store/retrieve profile info.
+
+* EventFactory: generates events of particular event types.
+* Notifier: Backs the events handler
+* REST: Interfaces handlers and events to the outside world via HTTP/JSON.
+  Converts events back and forth from JSON.
+* Federation: holds the HTTP client & server to talk to other servers.  Does
+  replication to make sure there's nothing missing in the graph.  Handles
+  reliability.  Handles txns.
+* Distributor: generic event bus. used for presence & typing only currently. 
+  Notifier could be implemented using Distributor - so far we are only using for
+  things which actually /require/ dynamic pluggability however as it can
+  obfuscate the actual flow of control.
+* Auth: helper singleton to say whether a given event is allowed to do a given
+  thing  (TODO: put this on the diagram)
+* State: helper singleton: does state conflict resolution. You give it an event
+  and it tells you if it actually updates the state or not, and annotates the
+  event up properly and handles merge conflict resolution.
+* Storage: abstracts the storage engine.

docs/client-server/howto.rst

@@ -1,303 +0,0 @@
-TODO(kegan): Tweak joinalias API keys/path? Event stream historical > live needs
-a token (currently doesn't). im/sync responses include outdated event formats
-(room membership change messages). Room config (specifically: message history,
-public rooms). /register seems super simplistic compared to /login, maybe it
-would be better if /register used the same technique as /login? /register should
-be "user" not "user_id".
-
-
-How to use the client-server API
-================================
-
-This guide focuses on how the client-server APIs *provided by the reference 
-home server* can be used. Since this is specific to a home server 
-implementation, there may be variations in relation to registering/logging in
-which are not covered in extensive detail in this guide.
-
-If you haven't already, get a home server up and running on 
-``http://localhost:8080``.
-
-
-Accounts
-========
-Before you can send and receive messages, you must **register** for an account. 
-If you already have an account, you must **login** into it.
-
-**Try out the fiddle: http://jsfiddle.net/jrf1h02d/**
-
-Registration
-------------
-The aim of registration is to get a user ID and access token which you will need
-when accessing other APIs::
-
-    curl -XPOST -d '{"user_id":"example", "password":"wordpass"}' "http://localhost:8080/matrix/client/api/v1/register"
-
-    {
-        "access_token": "QGV4YW1wbGU6bG9jYWxob3N0.AqdSzFmFYrLrTmteXc", 
-        "home_server": "localhost", 
-        "user_id": "@example:localhost"
-    }
-
-NB: If a ``user_id`` is not specified, one will be randomly generated for you. 
-If you do not specify a ``password``, you will be unable to login to the account
-if you forget the ``access_token``.
-
-Implementation note: The matrix specification does not enforce how users 
-register with a server. It just specifies the URL path and absolute minimum 
-keys. The reference home server uses a username/password to authenticate user,
-but other home servers may use different methods.
-
-Login
------
-The aim when logging in is to get an access token for your existing user ID::
-
-    curl -XGET "http://localhost:8080/matrix/client/api/v1/login"
-
-    {
-        "type": "m.login.password"
-    }
-
-    curl -XPOST -d '{"type":"m.login.password", "user":"example", "password":"wordpass"}' "http://localhost:8080/matrix/client/api/v1/login"
-
-    {
-        "access_token": "QGV4YW1wbGU6bG9jYWxob3N0.vRDLTgxefmKWQEtgGd", 
-        "home_server": "localhost", 
-        "user_id": "@example:localhost"
-    }
-    
-Implementation note: Different home servers may implement different methods for 
-logging in to an existing account. In order to check that you know how to login 
-to this home server, you must perform a ``GET`` first and make sure you 
-recognise the login type. If you do not know how to login, you can 
-``GET /login/fallback`` which will return a basic webpage which you can use to 
-login. The reference home server implementation support username/password login,
-but other home servers may support different login methods (e.g. OAuth2).
-
-
-Communicating
-=============
-
-In order to communicate with another user, you must **create a room** with that 
-user and **send a message** to that room. 
-
-**Try out the fiddle: http://jsfiddle.net/jnwqcshc/**
-
-Creating a room
----------------
-If you want to send a message to someone, you have to be in a room with them. To
-create a room::
-
-    curl -XPOST -d '{"room_alias_name":"tutorial"}' "http://localhost:8080/matrix/client/api/v1/rooms?access_token=QGV4YW1wbGU6bG9jYWxob3N0.vRDLTgxefmKWQEtgGd"
-
-    {
-        "room_alias": "#tutorial:localhost", 
-        "room_id": "!CvcvRuDYDzTOzfKKgh:localhost"
-    }
-    
-The "room alias" is a human-readable string which can be shared with other users
-so they can join a room, rather than the room ID which is a randomly generated
-string. You can have multiple room aliases per room.
-
-TODO(kegan): How to add/remove aliases from an existing room.
-    
-
-Sending messages
-----------------
-You can now send messages to this room::
-
-    curl -XPUT -d '{"msgtype":"m.text", "body":"hello"}' "http://localhost:8080/matrix/client/api/v1/rooms/%21CvcvRuDYDzTOzfKKgh:localhost/messages/%40example%3Alocalhost/msgid1?access_token=QGV4YW1wbGU6bG9jYWxob3N0.vRDLTgxefmKWQEtgGd"
-    
-NB: There are no limitations to the types of messages which can be exchanged.
-The only requirement is that ``"msgtype"`` is specified.
-
-NB: Depending on the room config, users who join the room may be able to see
-message history from before they joined.
-
-Users and rooms
-===============
-
-Each room can be configured to allow or disallow certain rules. In particular,
-these rules may specify if you require an **invitation** from someone already in
-the room in order to **join the room**. In addition, you may also be able to 
-join a room **via a room alias** if one was set up.
-
-**Try out the fiddle: http://jsfiddle.net/og1xokcr/**
-
-Inviting a user to a room
--------------------------
-You can directly invite a user to a room like so::
-
-    curl -XPUT -d '{"membership":"invite"}' "http://localhost:8080/matrix/client/api/v1/rooms/%21CvcvRuDYDzTOzfKKgh:localhost/members/%40myfriend%3Alocalhost/state?access_token=QGV4YW1wbGU6bG9jYWxob3N0.vRDLTgxefmKWQEtgGd"
-    
-This informs ``@myfriend:localhost`` of the room ID 
-``!CvcvRuDYDzTOzfKKgh:localhost`` and allows them to join the room.
-
-Joining a room via an invite
-----------------------------
-If you receive an invite, you can join the room by changing the membership to
-join::
-
-    curl -XPUT -d '{"membership":"join"}' "http://localhost:8080/matrix/client/api/v1/rooms/%21CvcvRuDYDzTOzfKKgh:localhost/members/%40myfriend%3Alocalhost/state?access_token=QG15ZnJpZW5kOmxvY2FsaG9zdA...XKuGdVsovHmwMyDDvK"
-    
-NB: Only the person invited (``@myfriend:localhost``) can change the membership
-state to ``"join"``.
-
-Joining a room via an alias
----------------------------
-Alternatively, if you know the room alias for this room and the room config 
-allows it, you can directly join a room via the alias::
-
-    curl -XPUT -d '{}' "http://localhost:8080/matrix/client/api/v1/join/%23tutorial%3Alocalhost?access_token=QG15ZnJpZW5kOmxvY2FsaG9zdA...XKuGdVsovHmwMyDDvK"
-    
-    {
-        "room_id": "!CvcvRuDYDzTOzfKKgh:localhost"
-    }
-    
-You will need to use the room ID when sending messages, not the room alias.
-
-NB: If the room is configured to be an invite-only room, you will still require
-an invite in order to join the room even though you know the room alias. As a
-result, it is more common to see a room alias in relation to a public room, 
-which do not require invitations.
-
-Getting events
-==============
-An event is some interesting piece of data that a client may be interested in. 
-It can be a message in a room, a room invite, etc. There are many different ways
-of getting events, depending on what the client already knows.
-
-**Try out the fiddle: http://jsfiddle.net/5uk4dqe2/**
-
-Getting all state
------------------
-If the client doesn't know any information on the rooms the user is 
-invited/joined on, they can get all the user's state for all rooms::
-
-    curl -XGET "http://localhost:8080/matrix/client/api/v1/im/sync?access_token=QG15ZnJpZW5kOmxvY2FsaG9zdA...XKuGdVsovHmwMyDDvK"
-    
-    [
-        {
-            "membership": "join", 
-            "messages": {
-                "chunk": [
-                    {
-                        "content": {
-                            "body": "@example:localhost joined the room.", 
-                            "hsob_ts": 1408444664249, 
-                            "membership": "join", 
-                            "membership_source": "@example:localhost", 
-                            "membership_target": "@example:localhost", 
-                            "msgtype": "m.text"
-                        }, 
-                        "event_id": "lZjmmlrEvo", 
-                        "msg_id": "m1408444664249", 
-                        "room_id": "!CvcvRuDYDzTOzfKKgh:localhost", 
-                        "type": "m.room.message", 
-                        "user_id": "_homeserver_"
-                    }, 
-                    {
-                        "content": {
-                            "body": "hello", 
-                            "hsob_ts": 1408445405672, 
-                            "msgtype": "m.text"
-                        }, 
-                        "event_id": "BiBJqamISg", 
-                        "msg_id": "msgid1", 
-                        "room_id": "!CvcvRuDYDzTOzfKKgh:localhost", 
-                        "type": "m.room.message", 
-                        "user_id": "@example:localhost"
-                    }, 
-                    [...]
-                    {
-                        "content": {
-                            "body": "@myfriend:localhost joined the room.", 
-                            "hsob_ts": 1408446501661, 
-                            "membership": "join", 
-                            "membership_source": "@myfriend:localhost", 
-                            "membership_target": "@myfriend:localhost", 
-                            "msgtype": "m.text"
-                        }, 
-                        "event_id": "IMmXbOzFAa", 
-                        "msg_id": "m1408446501661", 
-                        "room_id": "!CvcvRuDYDzTOzfKKgh:localhost", 
-                        "type": "m.room.message", 
-                        "user_id": "_homeserver_"
-                    }
-                ], 
-                "end": "20", 
-                "start": "0"
-            }, 
-            "room_id": "!CvcvRuDYDzTOzfKKgh:localhost"
-        }
-    ]
-    
-This returns all the room IDs of rooms the user is invited/joined on, as well as
-all of the messages and feedback for these rooms. This can be a LOT of data. You
-may just want the most recent message for each room. This can be achieved by 
-applying pagination stream parameters to this request::
-
-    curl -XGET "http://localhost:8080/matrix/client/api/v1/im/sync?access_token=QG15ZnJpZW5kOmxvY2FsaG9zdA...XKuGdVsovHmwMyDDvK&from=END&to=START&limit=1"
-    
-    [
-        {
-            "membership": "join", 
-            "messages": {
-                "chunk": [
-                    {
-                        "content": {
-                            "body": "@myfriend:localhost joined the room.", 
-                            "hsob_ts": 1408446501661, 
-                            "membership": "join", 
-                            "membership_source": "@myfriend:localhost", 
-                            "membership_target": "@myfriend:localhost", 
-                            "msgtype": "m.text"
-                        }, 
-                        "event_id": "IMmXbOzFAa", 
-                        "msg_id": "m1408446501661", 
-                        "room_id": "!CvcvRuDYDzTOzfKKgh:localhost", 
-                        "type": "m.room.message", 
-                        "user_id": "_homeserver_"
-                    }
-                ], 
-                "end": "20", 
-                "start": "21"
-            }, 
-            "room_id": "!CvcvRuDYDzTOzfKKgh:localhost"
-        }
-    ]
-
-Getting live state
-------------------
-Once you know which rooms the client has previously interacted with, you need to
-listen for incoming events. This can be done like so::
-
-    curl -XGET "http://localhost:8080/matrix/client/api/v1/events?access_token=QG15ZnJpZW5kOmxvY2FsaG9zdA...XKuGdVsovHmwMyDDvK&from=END"
-    
-    {
-        "chunk": [], 
-        "end": "215", 
-        "start": "215"
-    }
-    
-This will block waiting for an incoming event, timing out after several seconds.
-Even if there are no new events (as in the example above), there will be some
-pagination stream response keys. The client should make subsequent requests 
-using the value of the ``"end"`` key (in this case ``215``) as the ``from`` 
-query parameter. This value should be stored so when the client reopens your app
-after a period of inactivity, you can resume from where you got up to in the 
-event stream. If it has been a long period of inactivity, there may be LOTS of 
-events waiting for the user. In this case, you may wish to get all state instead
-and then resume getting live state from a newer end token.
-
-NB: The timeout can be changed by adding a ``timeout`` query parameter, which is
-in milliseconds. A timeout of 0 will not block.
-
-
-Example application
--------------------
-The following example demonstrates registration and login, live event streaming,
-creating and joining rooms, sending messages, getting member lists and getting 
-historical messages for a room. This covers most functionality of a messaging
-application.
-
-**Try out the fiddle: http://jsfiddle.net/L8r3o1wr/**

docs/client-server/swagger_matrix/api-docs

@@ -1,46 +0,0 @@
-{
-  "apiVersion": "1.0.0",
-  "swaggerVersion": "1.2",
-  "apis": [
-    {
-      "path": "/login",
-      "description": "Login operations"
-    },
-    {
-      "path": "/registration",
-      "description": "Registration operations"
-    },
-    {
-      "path": "/rooms",
-      "description": "Room operations"
-    },
-    {
-      "path": "/profile",
-      "description": "Profile operations"
-    },
-    {
-      "path": "/presence",
-      "description": "Presence operations"
-    },
-    {
-      "path": "/events",
-      "description": "Event operations"
-    },
-    {
-      "path": "/directory",
-      "description": "Directory operations"
-    }
-  ],
-  "authorizations": {
-    "token": {
-      "scopes": []
-    }
-  },
-  "info": {
-    "title": "Matrix Client-Server API Reference",
-    "description": "This contains the client-server API for the reference implementation of the home server",
-    "termsOfServiceUrl": "http://matrix.org",
-    "license": "Apache 2.0",
-    "licenseUrl": "http://www.apache.org/licenses/LICENSE-2.0.html"
-  }
-}

docs/client-server/swagger_matrix/directory

@@ -1,83 +0,0 @@
-{
-  "apiVersion": "1.0.0",
-  "swaggerVersion": "1.2",
-  "basePath": "http://localhost:8080/matrix/client/api/v1",
-  "resourcePath": "/directory",
-  "produces": [
-    "application/json"
-  ],
-  "apis": [
-    {
-      "path": "/directory/room/{roomAlias}",
-      "operations": [
-        {
-          "method": "GET",
-          "summary": "Get the room ID corresponding to this room alias.",
-          "type": "DirectoryResponse",
-          "nickname": "get_room_id_for_alias",
-          "parameters": [
-            {
-              "name": "roomAlias",
-              "description": "The room alias.",
-              "required": true,
-              "type": "string",
-              "paramType": "path"
-            }
-          ]
-        },
-        {
-          "method": "PUT",
-          "summary": "Create a new mapping from room alias to room ID.",
-          "type": "void",
-          "nickname": "add_room_alias",
-          "parameters": [
-            {
-              "name": "roomAlias",
-              "description": "The room alias to set.",
-              "required": true,
-              "type": "string",
-              "paramType": "path"
-            },
-            {
-              "name": "body",
-              "description": "The room ID to set.",
-              "required": true,
-              "type": "RoomAliasRequest",
-              "paramType": "body"
-            }
-          ]
-        }
-      ]
-    }
-  ],
-  "models": {
-    "DirectoryResponse": {
-      "id": "DirectoryResponse",
-      "properties": {
-        "room_id": {
-          "type": "string",
-          "description": "The fully-qualified room ID.",
-          "required": true
-        },
-        "servers": {
-          "type": "array",
-          "items": {
-            "$ref": "string"
-          },
-          "description": "A list of servers that know about this room.",
-          "required": true
-        }
-      }
-    },
-    "RoomAliasRequest": {
-      "id": "RoomAliasRequest",
-      "properties": {
-        "room_id": {
-          "type": "string",
-          "description": "The room ID to map the alias to.",
-          "required": true
-        }
-      }
-    }
-  }
-}

docs/client-server/swagger_matrix/events

@@ -1,246 +0,0 @@
-{
-  "apiVersion": "1.0.0",
-  "swaggerVersion": "1.2",
-  "basePath": "http://localhost:8080/matrix/client/api/v1",
-  "resourcePath": "/events",
-  "produces": [
-    "application/json"
-  ],
-  "apis": [
-    {
-      "path": "/events",
-      "operations": [
-        {
-          "method": "GET",
-          "summary": "Listen on the event stream",
-          "notes": "This can only be done by the logged in user. This will block until an event is received, or until the timeout is reached.",
-          "type": "PaginationChunk",
-          "nickname": "get_event_stream"
-        }
-      ],
-      "parameters": [
-        {
-          "name": "from",
-          "description": "The token to stream from.",
-          "required": false,
-          "type": "string",
-          "paramType": "query"
-        },
-        {
-          "name": "timeout",
-          "description": "The maximum time in milliseconds to wait for an event.",
-          "required": false,
-          "type": "integer",
-          "paramType": "query"
-        }
-      ],
-      "responseMessages": [
-        {
-          "code": 400,
-          "message": "Bad pagination token."
-        }
-      ]
-    },
-    {
-      "path": "/events/{eventId}",
-      "operations": [
-        {
-          "method": "GET",
-          "summary": "Get information about a single event.",
-          "notes": "Get information about a single event.",
-          "type": "Event",
-          "nickname": "get_event",
-          "parameters": [
-            {
-              "name": "eventId",
-              "description": "The event ID to get.",
-              "required": true,
-              "type": "string",
-              "paramType": "path"
-            }
-          ],
-          "responseMessages": [
-            {
-              "code": 404,
-              "message": "Event not found."
-            }
-          ]
-        }
-      ]
-    },
-    {
-      "path": "/initialSync",
-      "operations": [
-        {
-          "method": "GET",
-          "summary": "Get this user's current state.",
-          "notes": "Get this user's current state.",
-          "type": "InitialSyncResponse",
-          "nickname": "initial_sync",
-          "parameters": [
-            {
-              "name": "limit",
-              "description": "The maximum number of messages to return for each room.",
-              "type": "integer",
-              "paramType": "query",
-              "required": false
-            }
-          ]
-        }
-      ]
-    },
-    {
-      "path": "/publicRooms",
-      "operations": [
-        {
-          "method": "GET",
-          "summary": "Get a list of publicly visible rooms.",
-          "type": "PublicRoomsPaginationChunk",
-          "nickname": "get_public_room_list"
-        }
-      ]
-    }
-  ],
-  "models": {
-    "PaginationChunk": {
-      "id": "PaginationChunk",
-      "properties": {
-        "start": {
-          "type": "string",
-          "description": "A token which correlates to the first value in \"chunk\" for paginating.",
-          "required": true
-        },
-        "end": {
-          "type": "string",
-          "description": "A token which correlates to the last value in \"chunk\" for paginating.",
-          "required": true
-        },
-        "chunk": {
-          "type": "array",
-          "description": "An array of events.",
-          "required": true,
-          "items": {
-            "$ref": "Event"
-          }
-        }
-      }
-    },
-    "Event": {
-      "id": "Event",
-      "properties": {
-        "event_id": {
-          "type": "string",
-          "description": "An ID which uniquely identifies this event.",
-          "required": true
-        },
-        "room_id": {
-          "type": "string",
-          "description": "The room in which this event occurred.",
-          "required": true
-        }
-      }
-    },
-    "PublicRoomInfo": {
-      "id": "PublicRoomInfo",
-      "properties": {
-        "aliases": {
-          "type": "array",
-          "description": "A list of room aliases for this room.",
-          "items": {
-            "$ref": "string"
-          }
-        },
-        "name": {
-          "type": "string",
-          "description": "The name of the room, as given by the m.room.name state event."
-        },
-        "room_id": {
-          "type": "string",
-          "description": "The room ID for this public room.",
-          "required": true
-        },
-        "topic": {
-          "type": "string",
-          "description": "The topic of this room, as given by the m.room.topic state event."
-        }
-      }
-    },
-    "PublicRoomsPaginationChunk": {
-      "id": "PublicRoomsPaginationChunk",
-      "properties": {
-        "start": {
-          "type": "string",
-          "description": "A token which correlates to the first value in \"chunk\" for paginating.",
-          "required": true
-        },
-        "end": {
-          "type": "string",
-          "description": "A token which correlates to the last value in \"chunk\" for paginating.",
-          "required": true
-        },
-        "chunk": {
-          "type": "array",
-          "description": "A list of public room data.",
-          "required": true,
-          "items": {
-            "$ref": "PublicRoomInfo"
-          }
-        }
-      }
-    },
-    "InitialSyncResponse": {
-      "id": "InitialSyncResponse",
-      "properties": {
-        "end": {
-          "type": "string",
-          "description": "A streaming token which can be used with /events to continue from this snapshot of data.",
-          "required": true
-        },
-        "presence": {
-          "type": "array",
-          "description": "A list of presence events.",
-          "items": {
-            "$ref": "Event"
-          },
-          "required": false
-        },
-        "rooms": {
-          "type": "array",
-          "description": "A list of initial sync room data.",
-          "required": false,
-          "items": {
-            "$ref": "InitialSyncRoomData"
-          }
-        }
-      }
-    },
-    "InitialSyncRoomData": {
-      "id": "InitialSyncRoomData",
-      "properties": {
-        "membership": {
-          "type": "string",
-          "description": "This user's membership state in this room.",
-          "required": true
-        },
-        "room_id": {
-          "type": "string",
-          "description": "The ID of this room.",
-          "required": true
-        },
-        "messages": {
-          "type": "PaginationChunk",
-          "description": "The most recent messages for this room, governed by the limit parameter.",
-          "required": false
-        },
-        "state": {
-          "type": "array",
-          "description": "A list of state events representing the current state of the room.",
-          "required": false,
-          "items": {
-            "$ref": "Event"
-          }
-        }
-      }
-    }
-  }
-}

docs/client-server/swagger_matrix/login

@@ -1,102 +0,0 @@
-{
-  "apiVersion": "1.0.0", 
-  "apis": [
-    {
-      "operations": [
-        {
-          "method": "GET", 
-          "nickname": "get_login_info", 
-          "notes": "All login stages MUST be mentioned if there is >1 login type.", 
-          "summary": "Get the login mechanism to use when logging in.", 
-          "type": "LoginInfo"
-        }, 
-        {
-          "method": "POST", 
-          "nickname": "submit_login", 
-          "notes": "If this is part of a multi-stage login, there MUST be a 'session' key.", 
-          "parameters": [
-            {
-              "description": "A login submission", 
-              "name": "body", 
-              "paramType": "body", 
-              "required": true, 
-              "type": "LoginSubmission"
-            }
-          ], 
-          "responseMessages": [
-            {
-              "code": 400, 
-              "message": "Bad login type"
-            }, 
-            {
-              "code": 400, 
-              "message": "Missing JSON keys"
-            }
-          ], 
-          "summary": "Submit a login action.", 
-          "type": "LoginResult"
-        }
-      ], 
-      "path": "/login"
-    }
-  ], 
-  "basePath": "http://localhost:8080/matrix/client/api/v1", 
-  "consumes": [
-    "application/json"
-  ], 
-  "models": {
-    "LoginInfo": {
-      "id": "LoginInfo", 
-      "properties": {
-        "stages": {
-          "description": "Multi-stage login only: An array of all the login types required to login.", 
-          "format": "string", 
-          "type": "array"
-        }, 
-        "type": {
-          "description": "The login type that must be used when logging in.", 
-          "type": "string"
-        }
-      }
-    }, 
-    "LoginResult": {
-      "id": "LoginResult", 
-      "properties": {
-        "access_token": {
-          "description": "The access token for this user's login if this is the final stage of the login process.", 
-          "type": "string"
-        }, 
-        "next": {
-          "description": "Multi-stage login only: The next login type to submit.", 
-          "type": "string"
-        },
-        "session": {
-          "description": "Multi-stage login only: The session token to send when submitting the next login type.",
-          "type": "string"
-        }
-      }
-    }, 
-    "LoginSubmission": {
-      "id": "LoginSubmission", 
-      "properties": {
-        "type": {
-          "description": "The type of login being submitted.", 
-          "type": "string"
-        },
-        "session": {
-          "description": "Multi-stage login only: The session token from an earlier login stage.",
-          "type": "string"
-        },
-        "_login_type_defined_keys_": {
-          "description": "Keys as defined by the specified login type, e.g. \"user\", \"password\""
-        }
-      }
-    }
-  }, 
-  "produces": [
-    "application/json"
-  ], 
-  "resourcePath": "/login", 
-  "swaggerVersion": "1.2"
-}
-

docs/client-server/swagger_matrix/presence

@@ -1,164 +0,0 @@
-{
-  "apiVersion": "1.0.0",
-  "swaggerVersion": "1.2",
-  "basePath": "http://localhost:8080/matrix/client/api/v1",
-  "resourcePath": "/presence",
-  "produces": [
-    "application/json"
-  ],
-  "consumes": [
-    "application/json"
-  ],
-  "apis": [
-    {
-      "path": "/presence/{userId}/status",
-      "operations": [
-        {
-          "method": "PUT",
-          "summary": "Update this user's presence state.",
-          "notes": "This can only be done by the logged in user.",
-          "type": "void",
-          "nickname": "update_presence",
-          "parameters": [
-            {
-              "name": "body",
-              "description": "The new presence state",
-              "required": true,
-              "type": "PresenceUpdate",
-              "paramType": "body"
-            },
-            {
-              "name": "userId",
-              "description": "The user whose presence to set.",
-              "required": true,
-              "type": "string",
-              "paramType": "path"
-            }
-          ]
-        },
-        {
-          "method": "GET",
-          "summary": "Get this user's presence state.",
-          "notes": "Get this user's presence state.",
-          "type": "PresenceUpdate",
-          "nickname": "get_presence",
-          "parameters": [
-            {
-              "name": "userId",
-              "description": "The user whose presence to get.",
-              "required": true,
-              "type": "string",
-              "paramType": "path"
-            }
-          ]
-        }
-      ]
-    },
-    {
-      "path": "/presence/list/{userId}",
-      "operations": [
-        {
-          "method": "GET",
-          "summary": "Retrieve a list of presences for all of this user's friends.",
-          "notes": "",
-          "type": "array",
-          "items": {
-            "$ref": "Presence"
-          },
-          "nickname": "get_presence_list",
-          "parameters": [
-            {
-              "name": "userId",
-              "description": "The user whose presence list to get.",
-              "required": true,
-              "type": "string",
-              "paramType": "path"
-            }
-          ]
-        },
-        {
-          "method": "POST",
-          "summary": "Add or remove users from this presence list.",
-          "notes": "Add or remove users from this presence list.",
-          "type": "void",
-          "nickname": "modify_presence_list",
-          "parameters": [
-            {
-              "name": "userId",
-              "description": "The user whose presence list is being modified.",
-              "required": true,
-              "type": "string",
-              "paramType": "path"
-            },
-            {
-              "name": "body",
-              "description": "The modifications to make to this presence list.",
-              "required": true,
-              "type": "PresenceListModifications",
-              "paramType": "body"
-            }
-          ]
-        }
-      ]
-    }
-  ],
-  "models": {
-    "PresenceUpdate": {
-      "id": "PresenceUpdate",
-      "properties": {
-        "state": {
-          "type": "string",
-          "description": "Enum: The presence state.",
-          "enum": [
-            "offline",
-            "unavailable",
-            "online",
-            "free_for_chat"
-          ]
-        },
-        "status_msg": {
-          "type": "string",
-          "description": "The user-defined message associated with this presence state."
-        }
-      },
-      "subTypes": [
-        "Presence"
-      ]
-    },
-    "Presence": {
-      "id": "Presence",
-      "properties": {
-        "mtime_age": {
-          "type": "integer",
-          "format": "int64",
-          "description": "The last time this user's presence state changed, in milliseconds."
-        },
-        "user_id": {
-          "type": "string",
-          "description": "The fully qualified user ID"
-        }
-      }
-    },
-    "PresenceListModifications": {
-      "id": "PresenceListModifications",
-      "properties": {
-        "invite": {
-          "type": "array",
-          "description": "A list of user IDs to add to the list.",
-          "items": {
-            "type": "string",
-            "description": "A fully qualified user ID."
-          }
-        },
-        "drop": {
-          "type": "array",
-          "description": "A list of user IDs to remove from the list.",
-          "items": {
-            "type": "string",
-            "description": "A fully qualified user ID."
-          }
-        }
-      }
-    }
-  }
-}

docs/client-server/swagger_matrix/profile

@@ -1,122 +0,0 @@
-{
-  "apiVersion": "1.0.0",
-  "swaggerVersion": "1.2",
-  "basePath": "http://localhost:8080/matrix/client/api/v1",
-  "resourcePath": "/profile",
-  "produces": [
-    "application/json"
-  ],
-  "consumes": [
-    "application/json"
-  ],
-  "apis": [
-    {
-      "path": "/profile/{userId}/displayname",
-      "operations": [
-        {
-          "method": "PUT",
-          "summary": "Set a display name.",
-          "notes": "This can only be done by the logged in user.",
-          "type": "void",
-          "nickname": "set_display_name",
-          "parameters": [
-            {
-              "name": "body",
-              "description": "The new display name for this user.",
-              "required": true,
-              "type": "DisplayName",
-              "paramType": "body"
-            },
-            {
-              "name": "userId",
-              "description": "The user whose display name to set.",
-              "required": true,
-              "type": "string",
-              "paramType": "path"
-            }
-          ]
-        },
-        {
-          "method": "GET",
-          "summary": "Get a display name.",
-          "notes": "This can be done by anyone.",
-          "type": "DisplayName",
-          "nickname": "get_display_name",
-          "parameters": [
-            {
-              "name": "userId",
-              "description": "The user whose display name to get.",
-              "required": true,
-              "type": "string",
-              "paramType": "path"
-            }
-          ]
-        }
-      ]
-    },
-    {
-      "path": "/profile/{userId}/avatar_url",
-      "operations": [
-        {
-          "method": "PUT",
-          "summary": "Set an avatar URL.",
-          "notes": "This can only be done by the logged in user.",
-          "type": "void",
-          "nickname": "set_avatar_url",
-          "parameters": [
-            {
-              "name": "body",
-              "description": "The new avatar url for this user.",
-              "required": true,
-              "type": "AvatarUrl",
-              "paramType": "body"
-            },
-            {
-              "name": "userId",
-              "description": "The user whose avatar url to set.",
-              "required": true,
-              "type": "string",
-              "paramType": "path"
-            }
-          ]
-        },
-        {
-          "method": "GET",
-          "summary": "Get an avatar url.",
-          "notes": "This can be done by anyone.",
-          "type": "AvatarUrl",
-          "nickname": "get_avatar_url",
-          "parameters": [
-            {
-              "name": "userId",
-              "description": "The user whose avatar url to get.",
-              "required": true,
-              "type": "string",
-              "paramType": "path"
-            }
-          ]
-        }
-      ]
-    }
-  ],
-  "models": {
-    "DisplayName": {
-      "id": "DisplayName",
-      "properties": {
-        "displayname": {
-          "type": "string",
-          "description": "The textual display name"
-        }
-      }
-    },
-    "AvatarUrl": {
-      "id": "AvatarUrl",
-      "properties": {
-        "avatar_url": {
-          "type": "string",
-          "description": "A url to an image representing an avatar."
-        }
-      }
-    }
-  }
-}

docs/client-server/swagger_matrix/registration

@@ -1,75 +0,0 @@
-{
-  "apiVersion": "1.0.0", 
-  "apis": [
-    {
-      "operations": [
-        {
-          "method": "POST", 
-          "nickname": "register", 
-          "notes": "Volatile: This API is likely to change.", 
-          "parameters": [
-            {
-              "description": "A registration request", 
-              "name": "body", 
-              "paramType": "body", 
-              "required": true, 
-              "type": "RegistrationRequest"
-            }
-          ], 
-          "responseMessages": [
-            {
-              "code": 400, 
-              "message": "No JSON object."
-            }, 
-            {
-              "code": 400, 
-              "message": "User ID must only contain characters which do not require url encoding."
-            },
-            {
-              "code": 400, 
-              "message": "User ID already taken."
-            }
-          ], 
-          "summary": "Register with the home server.", 
-          "type": "RegistrationResponse"
-        }
-      ], 
-      "path": "/register"
-    }
-  ], 
-  "basePath": "http://localhost:8080/matrix/client/api/v1", 
-  "consumes": [
-    "application/json"
-  ], 
-  "models": {
-    "RegistrationResponse": {
-      "id": "RegistrationResponse", 
-      "properties": {
-        "access_token": {
-          "description": "The access token for this user.", 
-          "type": "string"
-        }, 
-        "user_id": {
-          "description": "The fully-qualified user ID.", 
-          "type": "string"
-        }
-      }
-    }, 
-    "RegistrationRequest": {
-      "id": "RegistrationRequest", 
-      "properties": {
-        "user_id": {
-          "description": "The desired user ID. If not specified, a random user ID will be allocated.", 
-          "type": "string",
-          "required": false
-        }
-      }
-    }
-  }, 
-  "produces": [
-    "application/json"
-  ], 
-  "resourcePath": "/register", 
-  "swaggerVersion": "1.2"
-}
-

docs/client-server/swagger_matrix/rooms

@@ -1,863 +0,0 @@
-{
-  "apiVersion": "1.0.0",
-  "swaggerVersion": "1.2",
-  "basePath": "http://localhost:8080/matrix/client/api/v1", 
-  "resourcePath": "/rooms",
-  "produces": [
-    "application/json"
-  ],
-  "consumes": [
-    "application/json"
-  ],
-  "authorizations": {
-    "token": []
-  },
-  "apis": [
-    {
-      "path": "/rooms/{roomId}/send/{eventType}/{txnId}",
-      "operations": [
-        {
-          "method": "PUT",
-          "summary": "Send a generic non-state event to this room.",
-          "notes": "This operation can also be done as a POST to /rooms/{roomId}/send/{eventType}",
-          "type": "EventId",
-          "nickname": "send_non_state_event",
-          "consumes": [
-            "application/json"
-          ],
-          "parameters": [
-            {
-              "name": "body",
-              "description": "The event contents",
-              "required": true,
-              "type": "EventContent",
-              "paramType": "body"
-            },
-            {
-              "name": "roomId",
-              "description": "The room to send the message in.",
-              "required": true,
-              "type": "string",
-              "paramType": "path"
-            },
-            {
-              "name": "eventType",
-              "description": "The type of event to send.",
-              "required": true,
-              "type": "string",
-              "paramType": "path"
-            },
-            {
-              "name": "txnId",
-              "description": "A client transaction ID to ensure idempotency. This can only be omitted if the HTTP method becomes a POST.",
-              "required": true,
-              "type": "string",
-              "paramType": "path"
-            }
-          ]
-        }
-      ]
-    },
-    {
-      "path": "/rooms/{roomId}/state/{eventType}/{stateKey}",
-      "operations": [
-        {
-          "method": "PUT",
-          "summary": "Send a generic state event to this room.",
-          "notes": "The state key can be omitted, such that you can PUT to /rooms/{roomId}/state/{eventType}. The state key defaults to a 0 length string in this case.",
-          "type": "void",
-          "nickname": "send_state_event",
-          "consumes": [
-            "application/json"
-          ],
-          "parameters": [
-            {
-              "name": "body",
-              "description": "The event contents",
-              "required": true,
-              "type": "EventContent",
-              "paramType": "body"
-            },
-            {
-              "name": "roomId",
-              "description": "The room to send the message in.",
-              "required": true,
-              "type": "string",
-              "paramType": "path"
-            },
-            {
-              "name": "eventType",
-              "description": "The type of event to send.",
-              "required": true,
-              "type": "string",
-              "paramType": "path"
-            },
-            {
-              "name": "stateKey",
-              "description": "An identifier used to specify clobbering semantics. State events with the same (roomId, eventType, stateKey) will be replaced.",
-              "required": true,
-              "type": "string",
-              "paramType": "path"
-            }
-          ]
-        }
-      ]
-    },
-    {
-      "path": "/rooms/{roomId}/send/m.room.message/{txnId}",
-      "operations": [
-        {
-          "method": "PUT",
-          "summary": "Send a message in this room.",
-          "notes": "This operation can also be done as a POST to /rooms/{roomId}/send/m.room.message",
-          "type": "EventId",
-          "nickname": "send_message",
-          "consumes": [
-            "application/json"
-          ],
-          "parameters": [
-            {
-              "name": "body",
-              "description": "The message contents",
-              "required": true,
-              "type": "Message",
-              "paramType": "body"
-            },
-            {
-              "name": "roomId",
-              "description": "The room to send the message in.",
-              "required": true,
-              "type": "string",
-              "paramType": "path"
-            },
-            {
-              "name": "txnId",
-              "description": "A client transaction ID to ensure idempotency. This can only be omitted if the HTTP method becomes a POST.",
-              "required": true,
-              "type": "string",
-              "paramType": "path"
-            }
-          ]
-        }
-      ]
-    },
-    {
-      "path": "/rooms/{roomId}/state/m.room.topic",
-      "operations": [
-        {
-          "method": "PUT",
-          "summary": "Set the topic for this room.",
-          "notes": "Set the topic for this room.",
-          "type": "void",
-          "nickname": "set_topic",
-          "consumes": [
-            "application/json"
-          ],
-          "parameters": [
-            {
-              "name": "body",
-              "description": "The topic contents",
-              "required": true,
-              "type": "Topic",
-              "paramType": "body"
-            },
-            {
-              "name": "roomId",
-              "description": "The room to set the topic in.",
-              "required": true,
-              "type": "string",
-              "paramType": "path"
-            }
-          ]
-        },
-        {
-          "method": "GET",
-          "summary": "Get the topic for this room.",
-          "notes": "Get the topic for this room.",
-          "type": "Topic",
-          "nickname": "get_topic",
-          "parameters": [
-            {
-              "name": "roomId",
-              "description": "The room to get topic in.",
-              "required": true,
-              "type": "string",
-              "paramType": "path"
-            }
-          ],
-          "responseMessages": [
-            {
-              "code": 404,
-              "message": "Topic not found."
-            }
-          ]
-        }
-      ]
-    },
-    {
-      "path": "/rooms/{roomId}/send/m.room.message.feedback/{txnId}",
-      "operations": [
-        {
-          "method": "PUT",
-          "summary": "Send feedback to a message.",
-          "notes": "This operation can also be done as a POST to /rooms/{roomId}/send/m.room.message.feedback",
-          "type": "EventId",
-          "nickname": "send_feedback",
-          "consumes": [
-            "application/json"
-          ],
-          "parameters": [
-            {
-              "name": "body",
-              "description": "The feedback contents",
-              "required": true,
-              "type": "Feedback",
-              "paramType": "body"
-            },
-            {
-              "name": "roomId",
-              "description": "The room to send the feedback in.",
-              "required": true,
-              "type": "string",
-              "paramType": "path"
-            },
-            {
-              "name": "txnId",
-              "description": "A client transaction ID to ensure idempotency. This can only be omitted if the HTTP method becomes a POST.",
-              "required": true,
-              "type": "string",
-              "paramType": "path"
-            }
-          ],
-          "responseMessages": [
-            {
-              "code": 400,
-              "message": "Bad feedback type."
-            }
-          ]
-        }
-      ]
-    },
-    {
-      "path": "/rooms/{roomId}/invite/{txnId}",
-      "operations": [
-        {
-          "method": "PUT",
-          "summary": "Invite a user to this room.",
-          "notes": "This operation can also be done as a POST to /rooms/{roomId}/invite",
-          "type": "void",
-          "nickname": "invite",
-          "consumes": [
-            "application/json"
-          ],
-          "parameters": [
-            {
-              "name": "roomId",
-              "description": "The room which has this user.",
-              "required": true,
-              "type": "string",
-              "paramType": "path"
-            },
-            {
-              "name": "txnId",
-              "description": "A client transaction ID for this PUT to ensure idempotency. This can only be omitted if the HTTP method becomes a POST. ",
-              "required": false,
-              "type": "string",
-              "paramType": "path"
-            },
-            {
-              "name": "body",
-              "description": "The user to invite.",
-              "required": true,
-              "type": "InviteRequest",
-              "paramType": "body"
-            }
-          ]
-        }  
-      ]
-    },
-    {
-      "path": "/rooms/{roomId}/join/{txnId}",
-      "operations": [
-        {
-          "method": "PUT",
-          "summary": "Join this room.",
-          "notes": "This operation can also be done as a POST to /rooms/{roomId}/join",
-          "type": "void",
-          "nickname": "join_room",
-          "consumes": [
-            "application/json"
-          ],
-          "parameters": [
-            {
-              "name": "roomId",
-              "description": "The room to join.",
-              "required": true,
-              "type": "string",
-              "paramType": "path"
-            },
-            {
-              "name": "txnId",
-              "description": "A client transaction ID for this PUT to ensure idempotency. This can only be omitted if the HTTP method becomes a POST. ",
-              "required": false,
-              "type": "string",
-              "paramType": "path"
-            }
-          ]
-        }  
-      ]
-    },
-    {
-      "path": "/rooms/{roomId}/leave/{txnId}",
-      "operations": [
-        {
-          "method": "PUT",
-          "summary": "Leave this room.",
-          "notes": "This operation can also be done as a POST to /rooms/{roomId}/leave",
-          "type": "void",
-          "nickname": "leave",
-          "consumes": [
-            "application/json"
-          ],
-          "parameters": [
-            {
-              "name": "roomId",
-              "description": "The room to leave.",
-              "required": true,
-              "type": "string",
-              "paramType": "path"
-            },
-            {
-              "name": "txnId",
-              "description": "A client transaction ID for this PUT to ensure idempotency. This can only be omitted if the HTTP method becomes a POST. ",
-              "required": false,
-              "type": "string",
-              "paramType": "path"
-            }
-          ]
-        }  
-      ]
-    },
-    {
-      "path": "/rooms/{roomId}/state/m.room.member/{userId}",
-      "operations": [
-        {
-          "method": "PUT",
-          "summary": "Change the membership state for a user in a room.",
-          "notes": "Change the membership state for a user in a room.",
-          "type": "void",
-          "nickname": "set_membership",
-          "consumes": [
-            "application/json"
-          ],
-          "parameters": [
-            {
-              "name": "body",
-              "description": "The new membership state",
-              "required": true,
-              "type": "Member",
-              "paramType": "body"
-            },
-            {
-              "name": "userId",
-              "description": "The user whose membership is being changed.",
-              "required": true,
-              "type": "string",
-              "paramType": "path"
-            },
-            {
-              "name": "roomId",
-              "description": "The room which has this user.",
-              "required": true,
-              "type": "string",
-              "paramType": "path"
-            }
-          ],
-          "responseMessages": [
-            {
-              "code": 400,
-              "message": "No membership key."
-            },
-            {
-              "code": 400,
-              "message": "Bad membership value."
-            },
-            {
-              "code": 403,
-              "message": "When inviting: You are not in the room."
-            },
-            {
-              "code": 403,
-              "message": "When inviting: <target> is already in the room."
-            },
-            {
-              "code": 403,
-              "message": "When joining: Cannot force another user to join."
-            },
-            {
-              "code": 403,
-              "message": "When joining: You are not invited to this room."
-            }
-          ]
-        },
-        {
-          "method": "GET",
-          "summary": "Get the membership state of a user in a room.",
-          "notes": "Get the membership state of a user in a room.",
-          "type": "Member",
-          "nickname": "get_membership",
-          "parameters": [
-            {
-              "name": "userId",
-              "description": "The user whose membership state you want to get.",
-              "required": true,
-              "type": "string",
-              "paramType": "path"
-            },
-            {
-              "name": "roomId",
-              "description": "The room which has this user.",
-              "required": true,
-              "type": "string",
-              "paramType": "path"
-            }
-          ],
-          "responseMessages": [
-            {
-              "code": 404,
-              "message": "Member not found."
-            }
-          ]
-        }
-      ]
-    },
-    {
-      "path": "/join/{roomAliasOrId}",
-      "operations": [
-        {
-          "method": "PUT",
-          "summary": "Join a room via a room alias or room ID.",
-          "notes": "Join a room via a room alias or room ID.",
-          "type": "RoomInfo",
-          "nickname": "join",
-          "consumes": [
-            "application/json"
-          ],
-          "parameters": [
-            {
-              "name": "roomAliasOrId",
-              "description": "The room alias or room ID to join.",
-              "required": true,
-              "type": "string",
-              "paramType": "path"
-            }
-          ],
-          "responseMessages": [
-            {
-              "code": 400,
-              "message": "Bad room alias."
-            }
-          ]
-        }
-      ]
-    },
-    {
-      "path": "/createRoom",
-      "operations": [
-        {
-          "method": "POST",
-          "summary": "Create a room.",
-          "notes": "Create a room.",
-          "type": "RoomInfo",
-          "nickname": "create_room",
-          "consumes": [
-            "application/json"
-          ],
-          "parameters": [
-            {
-              "name": "body",
-              "description": "The desired configuration for the room.",
-              "required": true,
-              "type": "RoomConfig",
-              "paramType": "body"
-            }
-          ],
-          "responseMessages": [
-            {
-              "code": 400,
-              "message": "Body must be JSON."
-            },
-            {
-              "code": 400,
-              "message": "Room alias already taken."
-            }
-          ]
-        }
-      ]
-    },
-    {
-      "path": "/rooms/{roomId}/messages",
-      "operations": [
-        {
-          "method": "GET",
-          "summary": "Get a list of messages for this room.",
-          "notes": "Get a list of messages for this room.",
-          "type": "MessagePaginationChunk",
-          "nickname": "get_messages",
-          "parameters": [
-            {
-              "name": "roomId",
-              "description": "The room to get messages in.",
-              "required": true,
-              "type": "string",
-              "paramType": "path"
-            },
-            {
-              "name": "from",
-              "description": "The token to start getting results from.",
-              "required": false,
-              "type": "string",
-              "paramType": "query"
-            },
-            {
-              "name": "to",
-              "description": "The token to stop getting results at.",
-              "required": false,
-              "type": "string",
-              "paramType": "query"
-            },
-            {
-              "name": "limit",
-              "description": "The maximum number of messages to return.",
-              "required": false,
-              "type": "integer",
-              "paramType": "query"
-            }
-          ]
-        }
-      ]
-    },
-    {
-      "path": "/rooms/{roomId}/members",
-      "operations": [
-        {
-          "method": "GET",
-          "summary": "Get a list of members for this room.",
-          "notes": "Get a list of members for this room.",
-          "type": "MemberPaginationChunk",
-          "nickname": "get_members",
-          "parameters": [
-            {
-              "name": "roomId",
-              "description": "The room to get a list of members from.",
-              "required": true,
-              "type": "string",
-              "paramType": "path"
-            },
-            {
-              "name": "from",
-              "description": "The token to start getting results from.",
-              "required": false,
-              "type": "string",
-              "paramType": "query"
-            },
-            {
-              "name": "to",
-              "description": "The token to stop getting results at.",
-              "required": false,
-              "type": "string",
-              "paramType": "query"
-            },
-            {
-              "name": "limit",
-              "description": "The maximum number of members to return.",
-              "required": false,
-              "type": "integer",
-              "paramType": "query"
-            }
-          ]
-        }
-      ]
-    },
-    {
-      "path": "/rooms/{roomId}/state",
-      "operations": [
-        {
-          "method": "GET",
-          "summary": "Get a list of all the current state events for this room.",
-          "notes": "Get a list of all the current state events for this room.",
-          "type": "array",
-          "items": {
-            "$ref": "Event"
-          },
-          "nickname": "get_state_events",
-          "parameters": [
-            {
-              "name": "roomId",
-              "description": "The room to get a list of current state events from.",
-              "required": true,
-              "type": "string",
-              "paramType": "path"
-            }
-          ]
-        }
-      ]
-    },
-    {
-      "path": "/rooms/{roomId}/initialSync",
-      "operations": [
-        {
-          "method": "GET",
-          "summary": "Get all the current information for this room, including messages and state events.",
-          "notes": "Get all the current information for this room, including messages and state events.",
-          "type": "InitialSyncRoomData",
-          "nickname": "get_room_sync_data",
-          "parameters": [
-            {
-              "name": "roomId",
-              "description": "The room to get information for.",
-              "required": true,
-              "type": "string",
-              "paramType": "path"
-            }
-          ]
-        }
-      ]
-    }
-  ],
-  "models": {
-    "Topic": {
-      "id": "Topic",
-      "properties": {
-        "topic": {
-          "type": "string",
-          "description": "The topic text"
-        }
-      }
-    },
-    "Message": {
-      "id": "Message",
-      "properties": {
-        "msgtype": {
-          "type": "string",
-          "description": "The type of message being sent, e.g. \"m.text\"",
-          "required": true
-        },
-        "_msgtype_defined_keys_": {
-          "description": "Additional keys as defined by the msgtype, e.g. \"body\""
-        }
-      }
-    },
-    "Feedback": {
-      "id": "Feedback",
-      "properties": {
-      }
-    },
-    "Member": {
-      "id": "Member",
-      "properties": {
-        "membership": {
-          "type": "string",
-          "description": "Enum: The membership state of this member.",
-          "enum": [
-            "invite",
-            "join",
-            "leave",
-            "knock"
-          ]
-        }
-      }
-    },
-    "RoomInfo": {
-      "id": "RoomInfo",
-      "properties": {
-        "room_id": {
-          "type": "string",
-          "description": "The allocated room ID.",
-          "required": true
-        },
-        "room_alias": {
-          "type": "string",
-          "description": "The alias for the room.",
-          "required": false
-        }
-      }
-    },
-    "RoomConfig": {
-      "id": "RoomConfig",
-      "properties": {
-        "visibility": {
-          "type": "string",
-          "description": "Enum: The room visibility.",
-          "required": false,
-          "enum": [
-            "public",
-            "private"
-          ]
-        },
-        "room_alias_name": {
-          "type": "string",
-          "description": "The alias to give the new room.",
-          "required": false
-        }
-      }
-    },
-    "PaginationRequest": {
-      "id": "PaginationRequest",
-      "properties": {
-        "from": {
-          "type": "string",
-          "description": "The token to start getting results from."
-        },
-        "to": {
-          "type": "string",
-          "description": "The token to stop getting results at."
-        },
-        "limit": {
-          "type": "integer",
-          "description": "The maximum number of entries to return."
-        }
-      }
-    },
-    "PaginationChunk": {
-      "id": "PaginationChunk",
-      "properties": {
-        "start": {
-          "type": "string",
-          "description": "A token which correlates to the first value in \"chunk\" for paginating.",
-          "required": true
-        },
-        "end": {
-          "type": "string",
-          "description": "A token which correlates to the last value in \"chunk\" for paginating.",
-          "required": true
-        }
-      },
-      "subTypes": [
-        "MessagePaginationChunk"
-      ]
-    },
-    "MessagePaginationChunk": {
-      "id": "MessagePaginationChunk",
-      "properties": {
-        "chunk": {
-          "type": "array",
-          "description": "A list of message events.",
-          "items": {
-            "$ref": "MessageEvent"
-          },
-          "required": true
-        }
-      }
-    },
-    "MemberPaginationChunk": {
-      "id": "MemberPaginationChunk",
-      "properties": {
-        "chunk": {
-          "type": "array",
-          "description": "A list of member events.",
-          "items": {
-            "$ref": "MemberEvent"
-          },
-          "required": true
-        }
-      }
-    },
-    "Event": {
-      "id": "Event",
-      "properties": {
-        "event_id": {
-          "type": "string",
-          "description": "An ID which uniquely identifies this event. This is automatically set by the server.",
-          "required": true
-        },
-        "room_id": {
-          "type": "string",
-          "description": "The room in which this event occurred. This is automatically set by the server.",
-          "required": true
-        },
-        "type": {
-          "type": "string",
-          "description": "The event type.",
-          "required": true
-        }
-      },
-      "subTypes": [
-        "MessageEvent"
-      ]
-    },
-    "EventId": {
-      "id": "EventId",
-      "properties": {
-        "event_id": {
-          "type": "string",
-          "description": "The allocated event ID for this event.",
-          "required": true
-        }
-      }
-    },
-    "EventContent": {
-      "id": "EventContent",
-      "properties": {
-        "__event_content_keys__": {
-          "type": "string",
-          "description": "Event-specific content keys and values.",
-          "required": false
-        }
-      }
-    },
-    "MessageEvent": {
-      "id": "MessageEvent",
-      "properties": {
-        "content": {
-          "type": "Message"
-        }
-      }
-    },
-    "MemberEvent": {
-      "id": "MemberEvent",
-      "properties": {
-        "content": {
-          "type": "Member"
-        }
-      }
-    },
-    "InviteRequest": {
-      "id": "InviteRequest",
-      "properties": {
-        "user_id": {
-          "type": "string",
-          "description": "The fully-qualified user ID."
-        }
-      }
-    },
-    "InitialSyncRoomData": {
-      "id": "InitialSyncRoomData",
-      "properties": {
-        "membership": {
-          "type": "string",
-          "description": "This user's membership state in this room.",
-          "required": true
-        },
-        "room_id": {
-          "type": "string",
-          "description": "The ID of this room.",
-          "required": true
-        },
-        "messages": {
-          "type": "MessagePaginationChunk",
-          "description": "The most recent messages for this room, governed by the limit parameter.",
-          "required": false
-        },
-        "state": {
-          "type": "array",
-          "description": "A list of state events representing the current state of the room.",
-          "required": false,
-          "items": {
-            "$ref": "Event"
-          }
-        }
-      }
-    }
-  }
-}

docs/code_style.rst

@@ -0,0 +1,49 @@
+Basically, PEP8
+
+- NEVER tabs. 4 spaces to indent.
+- Max line width: 79 chars (with flexibility to overflow by a "few chars" if
+  the overflowing content is not semantically significant and avoids an
+  explosion of vertical whitespace).
+- Use camel case for class and type names
+- Use underscores for functions and variables.
+- Use double quotes.
+- Use parentheses instead of '\\' for line continuation where ever possible
+  (which is pretty much everywhere)
+- There should be max a single new line between:
+    - statements
+    - functions in a class
+- There should be two new lines between:
+    - definitions in a module (e.g., between different classes)
+- There should be spaces where spaces should be and not where there shouldn't be:
+    - a single space after a comma
+    - a single space before and after for '=' when used as assignment
+    - no spaces before and after for '=' for default values and keyword arguments.
+- Indenting must follow PEP8; either hanging indent or multiline-visual indent
+  depending on the size and shape of the arguments and what makes more sense to
+  the author. In other words, both this::
+
+    print("I am a fish %s" % "moo")
+
+  and this::
+
+    print("I am a fish %s" %
+          "moo")
+
+  and this::
+
+    print(
+        "I am a fish %s" %
+        "moo"
+    )
+
+  ...are valid, although given each one takes up 2x more vertical space than
+  the previous, it's up to the author's discretion as to which layout makes most
+  sense for their function invocation.  (e.g. if they want to add comments
+  per-argument, or put expressions in the arguments, or group related arguments
+  together, or want to deliberately extend or preserve vertical/horizontal
+  space)
+
+Comments should follow the google code style. This is so that we can generate
+documentation with sphinx (http://sphinxcontrib-napoleon.readthedocs.org/en/latest/) 
+
+Code should pass pep8 --max-line-length=100 without any warnings.

docs/implementation-notes/code_style.rst

@@ -1,18 +0,0 @@
-Basically, PEP8
-
-- Max line width: 80 chars.
-- Use camel case for class and type names
-- Use underscores for functions and variables.
-- Use double quotes.
-- Use parentheses instead of '\' for line continuation where ever possible (which is pretty much everywhere)
-- There should be max a single new line between:
-    - statements
-    - functions in a class
-- There should be two new lines between:
-    - definitions in a module (e.g., between different classes)
-- There should be spaces where spaces should be and not where there shouldn't be:
-    - a single space after a comma
-    - a single space before and after for '=' when used as assignment
-    - no spaces before and after for '=' for default values and keyword arguments.
-
-Comments should follow the google code style. This is so that we can generate documentation with sphinx (http://sphinxcontrib-napoleon.readthedocs.org/en/latest/) 

docs/implementation-notes/documentation_style.rst

@@ -1,43 +0,0 @@
-===================
-Documentation Style
-===================
-
-A brief single sentence to describe what this file contains; in this case a
-description of the style to write documentation in.
-
-
-Sections
-========
-
-Each section should be separated from the others by two blank lines. Headings
-should be underlined using a row of equals signs (===). Paragraphs should be
-separated by a single blank line, and wrap to no further than 80 columns.
-
-[[TODO(username): if you want to leave some unanswered questions, notes for
-further consideration, or other kinds of comment, use a TODO section. Make sure
-to notate it with your name so we know who to ask about it!]]
-
-Subsections
------------
-
-If required, subsections can use a row of dashes to underline their header. A
-single blank line between subsections of a single section.
-
-
-Bullet Lists
-============
-
- * Bullet lists can use asterisks with a single space either side.
- 
- * Another blank line between list elements.
-
-
-Definition Lists
-================
-
-Terms:
-  Start in the first column, ending with a colon
-
-Definitions:
-  Take a two space indent, following immediately from the term without a blank
-  line before it, but having a blank line afterwards.

docs/media_repository.rst

@@ -0,0 +1,27 @@
+Media Repository 
+================
+
+*Synapse implementation-specific details for the media repository*
+
+The media repository is where attachments and avatar photos are stored.
+It stores attachment content and thumbnails for media uploaded by local users.
+It caches attachment content and thumbnails for media uploaded by remote users.
+
+Storage
+-------
+
+Each item of media is assigned a ``media_id`` when it is uploaded.
+The ``media_id`` is a randomly chosen, URL safe 24 character string.
+Metadata such as the MIME type, upload time and length are stored in the
+sqlite3 database indexed by ``media_id``.
+Content is stored on the filesystem under a ``"local_content"`` directory.
+Thumbnails are stored under a ``"local_thumbnails"`` directory.
+The item with ``media_id`` ``"aabbccccccccdddddddddddd"`` is stored under
+``"local_content/aa/bb/ccccccccdddddddddddd"``. Its thumbnail with width
+``128`` and height ``96`` and type ``"image/jpeg"`` is stored under
+``"local_thumbnails/aa/bb/ccccccccdddddddddddd/128-96-image-jpeg"``
+Remote content is cached under ``"remote_content"`` directory. Each item of
+remote content is assigned a local "``filesystem_id``" to ensure that the
+directory structure ``"remote_content/server_name/aa/bb/ccccccccdddddddddddd"``
+is appropriate. Thumbnails for remote content are stored under
+``"remote_thumbnails/server_name/..."``

docs/model/presence.rst

@@ -1,249 +0,0 @@
-========
-Presence
-========
-
-A description of presence information and visibility between users.
-
-Overview
-========
-
-Each user has the concept of Presence information. This encodes a sense of the
-"availability" of that user, suitable for display on other user's clients.
-
-
-Presence Information
-====================
-
-The basic piece of presence information is an enumeration of a small set of
-state; such as "free to chat", "online", "busy", or "offline". The default state
-unless the user changes it is "online". Lower states suggest some amount of
-decreased availability from normal, which might have some client-side effect
-like muting notification sounds and suggests to other users not to bother them
-unless it is urgent. Equally, the "free to chat" state exists to let the user
-announce their general willingness to receive messages moreso than default.
-
-Home servers should also allow a user to set their state as "hidden" - a state
-which behaves as offline, but allows the user to see the client state anyway and
-generally interact with client features such as reading message history or
-accessing contacts in the address book.
-
-This basic state field applies to the user as a whole, regardless of how many
-client devices they have connected. The home server should synchronise this
-status choice among multiple devices to ensure the user gets a consistent
-experience.
-
-Idle Time
----------
-
-As well as the basic state field, the presence information can also show a sense
-of an "idle timer". This should be maintained individually by the user's
-clients, and the homeserver can take the highest reported time as that to
-report. Likely this should be presented in fairly coarse granularity; possibly
-being limited to letting the home server automatically switch from a "free to
-chat" or "online" mode into "idle".
-
-When a user is offline, the Home Server can still report when the user was last
-seen online, again perhaps in a somewhat coarse manner.
-
-Device Type
------------
-
-Client devices that may limit the user experience somewhat (such as "mobile"
-devices with limited ability to type on a real keyboard or read large amounts of
-text) should report this to the home server, as this is also useful information
-to report as "presence" if the user cannot be expected to provide a good typed
-response to messages.
-
-
-Presence List
-=============
-
-Each user's home server stores a "presence list" for that user. This stores a
-list of other user IDs the user has chosen to add to it (remembering any ACL
-Pointer if appropriate).
-
-To be added to a contact list, the user being added must grant permission. Once
-granted, both user's HS(es) store this information, as it allows the user who
-has added the contact some more abilities; see below. Since such subscriptions
-are likely to be bidirectional, HSes may wish to automatically accept requests
-when a reverse subscription already exists.
-
-As a convenience, presence lists should support the ability to collect users
-into groups, which could allow things like inviting the entire group to a new
-("ad-hoc") chat room, or easy interaction with the profile information ACL
-implementation of the HS.
-
-
-Presence and Permissions
-========================
-
-For a viewing user to be allowed to see the presence information of a target
-user, either
-
- * The target user has allowed the viewing user to add them to their presence
-   list, or
-
- * The two users share at least one room in common
-
-In the latter case, this allows for clients to display some minimal sense of
-presence information in a user list for a room.
-
-Home servers can also use the user's choice of presence state as a signal for
-how to handle new private one-to-one chat message requests. For example, it
-might decide:
-
-  "free to chat": accept anything
-  "online": accept from anyone in my addres book list
-  "busy": accept from anyone in this "important people" group in my address
-    book list
-
-
-API Efficiency
-==============
-
-A simple implementation of presence messaging has the ability to cause a large
-amount of Internet traffic relating to presence updates. In order to minimise
-the impact of such a feature, the following observations can be made:
-
- * There is no point in a Home Server polling status for peers in a user's
-   presence list if the user has no clients connected that care about it.
-
- * It is highly likely that most presence subscriptions will be symmetric - a
-   given user watching another is likely to in turn be watched by that user.
-
- * It is likely that most subscription pairings will be between users who share
-   at least one Room in common, and so their Home Servers are actively
-   exchanging message PDUs or transactions relating to that Room.
-
- * Presence update messages do not need realtime guarantees. It is acceptable to
-   delay delivery of updates for some small amount of time (10 seconds to a
-   minute).
-
-The general model of presence information is that of a HS registering its
-interest in receiving presence status updates from other HSes, which then
-promise to send them when required. Rather than actively polling for the
-currentt state all the time, HSes can rely on their relative stability to only
-push updates when required.
-
-A Home Server should not rely on the longterm validity of this presence
-information, however, as this would not cover such cases as a user's server
-crashing and thus failing to inform their peers that users it used to host are
-no longer available online. Therefore, each promise of future updates should
-carry with a timeout value (whether explicit in the message, or implicit as some
-defined default in the protocol), after which the receiving HS should consider
-the information potentially stale and request it again.
-
-However, because of the likelyhood that two home servers are exchanging messages
-relating to chat traffic in a room common to both of them, the ongoing receipt
-of these messages can be taken by each server as an implicit notification that
-the sending server is still up and running, and therefore that no status changes
-have happened; because if they had the server would have sent them. A second,
-larger timeout should be applied to this implicit inference however, to protect
-against implementation bugs or other reasons that the presence state cache may
-become invalid; eventually the HS should re-enquire the current state of users
-and update them with its own.
-
-The following workflows can therefore be used to handle presence updates:
-
- 1 When a user first appears online their HS sends a message to each other HS
-   containing at least one user to be watched; each message carrying both a
-   notification of the sender's new online status, and a request to obtain and
-   watch the target users' presence information. This message implicitly
-   promises the sending HS will now push updates to the target HSes.
-
- 2 The target HSes then respond a single message each, containing the current
-   status of the requested user(s). These messages too implicitly promise the
-   target HSes will themselves push updates to the sending HS.
-
-   As these messages arrive at the sending user's HS they can be pushed to the
-   user's client(s), possibly batched again to ensure not too many small
-   messages which add extra protocol overheads.
-
-At this point, all the user's clients now have the current presence status
-information for this moment in time, and have promised to send each other
-updates in future.
-
- 3 The HS maintains two watchdog timers per peer HS it is exchanging presence
-   information with. The first timer should have a relatively small expiry
-   (perhaps 1 minute), and the second timer should have a much longer time
-   (perhaps 1 hour).
-
- 4 Any time any kind of message is received from a peer HS, the short-term
-   presence timer associated with it is reset.
-
- 5 Whenever either of these timers expires, an HS should push a status reminder
-   to the target HS whose timer has now expired, and request again from that
-   server the status of the subscribed users.
-
- 6 On receipt of one of these presence status reminders, an HS can reset both
-   of its presence watchdog timers.
-
-To avoid bursts of traffic, implementations should attempt to stagger the expiry
-of the longer-term watchdog timers for different peer HSes.
-
-When individual users actively change their status (either by explicit requests
-from clients, or inferred changes due to idle timers or client timeouts), the HS
-should batch up any status changes for some reasonable amount of time (10
-seconds to a minute). This allows for reduced protocol overheads in the case of
-multiple messages needing to be sent to the same peer HS; as is the likely
-scenario in many cases, such as a given human user having multiple user
-accounts.
-
-
-API Requirements
-================
-
-The data model presented here puts the following requirements on the APIs:
-
-Client-Server
--------------
-
-Requests that a client can make to its Home Server
-
- * get/set current presence state
-   Basic enumeration + ability to set a custom piece of text
-
- * report per-device idle time
-   After some (configurable?) idle time the device should send a single message
-   to set the idle duration. The HS can then infer a "start of idle" instant and
-   use that to keep the device idleness up to date. At some later point the
-   device can cancel this idleness.
-
- * report per-device type
-   Inform the server that this device is a "mobile" device, or perhaps some
-   other to-be-defined category of reduced capability that could be presented to
-   other users.
-
- * start/stop presence polling for my presence list
-   It is likely that these messages could be implicitly inferred by other
-   messages, though having explicit control is always useful.
-
- * get my presence list
-   [implicit poll start?]
-   It is possible that the HS doesn't yet have current presence information when
-   the client requests this. There should be a "don't know" type too.
-
- * add/remove a user to my presence list
-
-Server-Server
--------------
-
-Requests that Home Servers make to others
-
- * request permission to add a user to presence list
-
- * allow/deny a request to add to a presence list
-
- * perform a combined presence state push and subscription request
-   For each sending user ID, the message contains their new status.
-   For each receiving user ID, the message should contain an indication on
-   whether the sending server is also interested in receiving status from that
-   user; either as an immediate update response now, or as a promise to send
-   future updates.
-
-Server to Client
-----------------
-
-[[TODO(paul): There also needs to be some way for a user's HS to push status
-updates of the presence list to clients, but the general server-client event
-model currently lacks a space to do that.]]

docs/model/profiles.rst

@@ -1,232 +0,0 @@
-========
-Profiles
-========
-
-A description of Synapse user profile metadata support.
-
-
-Overview
-========
-
-Internally within Synapse users are referred to by an opaque ID, which consists
-of some opaque localpart combined with the domain name of their home server.
-Obviously this does not yield a very nice user experience; users would like to
-see readable names for other users that are in some way meaningful to them.
-Additionally, users like to be able to publish "profile" details to inform other
-users of other information about them.
-
-It is also conceivable that since we are attempting to provide a
-worldwide-applicable messaging system, that users may wish to present different
-subsets of information in their profile to different other people, from a
-privacy and permissions perspective.
-
-A Profile consists of a display name, an (optional?) avatar picture, and a set
-of other metadata fields that the user may wish to publish (email address, phone
-numbers, website URLs, etc...). We put no requirements on the display name other
-than it being a valid Unicode string. Since it is likely that users will end up
-having multiple accounts (perhaps by necessity of being hosted in multiple
-places, perhaps by choice of wanting multiple distinct identifies), it would be
-useful that a metadata field type exists that can refer to another Synapse User
-ID, so that clients and HSes can make use of this information.
-
-Metadata Fields
----------------
-
-[[TODO(paul): Likely this list is incomplete; more fields can be defined as we
-think of them. At the very least, any sort of supported ID for the 3rd Party ID
-servers should be accounted for here.]]
-
- * Synapse Directory Server username(s)
-
- * Email address
-
- * Phone number - classify "home"/"work"/"mobile"/custom?
- 
- * Twitter/Facebook/Google+/... social networks
-
- * Location - keep this deliberately vague to allow people to choose how
-     granular it is
- 
- * "Bio" information - date of birth, etc...
-
- * Synapse User ID of another account
-
- * Web URL
-
- * Freeform description text
-
-
-Visibility Permissions
-======================
-
-A home server implementation could offer the ability to set permissions on
-limited visibility of those fields. When another user requests access to the
-target user's profile, their own identity should form part of that request. The
-HS implementation can then decide which fields to make available to the
-requestor.
-
-A particular detail of implementation could allow the user to create one or more
-ACLs; where each list is granted permission to see a given set of non-public
-fields (compare to Google+ Circles) and contains a set of other people allowed
-to use it. By giving these ACLs strong identities within the HS, they can be
-referenced in communications with it, granting other users who encounter these
-the "ACL Token" to use the details in that ACL.
-
-If we further allow an ACL Token to be present on Room join requests or stored
-by 3PID servers, then users of these ACLs gain the extra convenience of not
-having to manually curate people in the access list; anyone in the room or with
-knowledge of the 3rd Party ID is automatically granted access. Every HS and
-client implementation would have to be aware of the existence of these ACL
-Token, and include them in requests if present, but not every HS implementation
-needs to actually provide the full permissions model. This can be used as a
-distinguishing feature among competing implementations. However, servers MUST
-NOT serve profile information from a cache if there is a chance that its limited
-understanding could lead to information leakage.
-
-
-Client Concerns of Multiple Accounts
-====================================
-
-Because a given person may want to have multiple Synapse User accounts, client
-implementations should allow the use of multiple accounts simultaneously
-(especially in the field of mobile phone clients, which generally don't support
-running distinct instances of the same application). Where features like address
-books, presence lists or rooms are presented, the client UI should remember to
-make distinct with user account is in use for each.
-
-
-Directory Servers
-=================
-
-Directory Servers can provide a forward mapping from human-readable names to
-User IDs. These can provide a service similar to giving domain-namespaced names
-for Rooms; in this case they can provide a way for a user to reference their
-User ID in some external form (e.g. that can be printed on a business card).
-
-The format for Synapse user name will consist of a localpart specific to the
-directory server, and the domain name of that directory server:
-
-  @localname:some.domain.name
-
-The localname is separated from the domain name using a colon, so as to ensure
-the localname can still contain periods, as users may want this for similarity
-to email addresses or the like, which typically can contain them. The format is
-also visually quite distinct from email addresses, phone numbers, etc... so
-hopefully reasonably "self-describing" when written on e.g. a business card
-without surrounding context.
-
-[[TODO(paul): we might have to think about this one - too close to email?
-  Twitter? Also it suggests a format scheme for room names of
-  #localname:domain.name, which I quite like]]
-
-Directory server administrators should be able to make some kind of policy
-decision on how these are allocated. Servers within some "closed" domain (such
-as company-specific ones) may wish to verify the validity of a mapping using
-their own internal mechanisms; "public" naming servers can operate on a FCFS
-basis. There are overlapping concerns here with the idea of the 3rd party
-identity servers as well, though in this specific case we are creating a new
-namespace to allocate names into.
-
-It would also be nice from a user experience perspective if the profile that a
-given name links to can also declare that name as part of its metadata.
-Furthermore as a security and consistency perspective it would be nice if each
-end (the directory server and the user's home server) check the validity of the
-mapping in some way. This needs investigation from a security perspective to
-ensure against spoofing.
-
-One such model may be that the user starts by declaring their intent to use a
-given user name link to their home server, which then contacts the directory
-service. At some point later (maybe immediately for "public open FCFS servers",
-maybe after some kind of human intervention for verification) the DS decides to
-honour this link, and includes it in its served output. It should also tell the
-HS of this fact, so that the HS can present this as fact when requested for the
-profile information. For efficiency, it may further wish to provide the HS with
-a cryptographically-signed certificate as proof, so the HS serving the profile
-can provide that too when asked, avoiding requesting HSes from constantly having
-to contact the DS to verify this mapping. (Note: This is similar to the security
-model often applied in DNS to verify PTR <-> A bidirectional mappings).
-
-
-Identity Servers
-================
-
-The identity servers should support the concept of pointing a 3PID being able to
-store an ACL Token as well as the main User ID. It is however, beyond scope to
-do any kind of verification that any third-party IDs that the profile is
-claiming match up to the 3PID mappings.
-
-
-User Interface and Expectations Concerns
-========================================
-
-Given the weak "security" of some parts of this model as compared to what users
-might expect, some care should be taken on how it is presented to users,
-specifically in the naming or other wording of user interface components.
-
-Most notably mere knowledge of an ACL Pointer is enough to read the information
-stored in it. It is possible that Home or Identity Servers could leak this
-information, allowing others to see it. This is a security-vs-convenience
-balancing choice on behalf of the user who would choose, or not, to make use of
-such a feature to publish their information.
-
-Additionally, unless some form of strong end-to-end user-based encryption is
-used, a user of ACLs for information privacy has to trust other home servers not
-to lie about the identify of the user requesting access to the Profile.
-
-
-API Requirements
-================
-
-The data model presented here puts the following requirements on the APIs:
-
-Client-Server
--------------
-
-Requests that a client can make to its Home Server
-
- * get/set my Display Name
-   This should return/take a simple "text/plain" field
-
- * get/set my Avatar URL
-   The avatar image data itself is not stored by this API; we'll just store a
-   URL to let the clients fetch it. Optionally HSes could integrate this with
-   their generic content attacmhent storage service, allowing a user to set
-   upload their profile Avatar and update the URL to point to it.
-
- * get/add/remove my metadata fields
-   Also we need to actually define types of metadata
-
- * get another user's Display Name / Avatar / metadata fields
-
-[[TODO(paul): At some later stage we should consider the API for:
-
- * get/set ACL permissions on my metadata fields
-
- * manage my ACL tokens
-]]
-
-Server-Server
--------------
-
-Requests that Home Servers make to others
-
- * get a user's Display Name / Avatar
-
- * get a user's full profile - name/avatar + MD fields
-   This request must allow for specifying the User ID of the requesting user,
-   for permissions purposes. It also needs to take into account any ACL Tokens
-   the requestor has.
-
- * push a change of Display Name to observers (overlaps with the presence API)
-
-Room Event PDU Types
---------------------
-
-Events that are pushed from Home Servers to other Home Servers or clients.
-
- * user Display Name change
- 
- * user Avatar change
-   [[TODO(paul): should the avatar image itself be stored in all the room
-   histories? maybe this event should just be a hint to clients that they should
-   re-fetch the avatar image]]

docs/model/protocol_examples.rst

@@ -1,64 +0,0 @@
-PUT /send/abc/ HTTP/1.1
-Host: ...
-Content-Length: ...
-Content-Type: application/json
-
-{
-    "origin": "localhost:5000",
-    "pdus": [
-        {
-            "content": {},
-            "context": "tng",
-            "depth": 12,
-            "is_state": false,
-            "origin": "localhost:5000",
-            "pdu_id": 1404381396854,
-            "pdu_type": "feedback",
-            "prev_pdus": [
-                [
-                    "1404381395883",
-                    "localhost:6000"
-                ]
-            ],
-            "ts": 1404381427581
-        }
-    ],
-    "prev_ids": [
-        "1404381396852"
-    ],
-    "ts": 1404381427823
-}
-
-HTTP/1.1 200 OK
-...
-
-======================================
-
-GET /pull/-1/ HTTP/1.1
-Host: ...
-Content-Length: 0
-
-HTTP/1.1 200 OK
-Content-Length: ...
-Content-Type: application/json
-
-{
-    origin: ...,
-    prev_ids: ...,
-    data: [
-        {
-            data_id: ...,
-            prev_pdus: [...],
-            depth: ...,
-            ts: ...,
-            context: ...,
-            origin: ...,
-            content: {
-                ...
-            }
-        },
-        ...,
-    ]
-}
-
-

docs/model/room-join-workflow.rst

@@ -1,113 +0,0 @@
-==================
-Room Join Workflow
-==================
-
-An outline of the workflows required when a user joins a room.
-
-Discovery
-=========
-
-To join a room, a user has to discover the room by some mechanism in order to
-obtain the (opaque) Room ID and a candidate list of likely home servers that
-contain it.
-
-Sending an Invitation
----------------------
-
-The most direct way a user discovers the existence of a room is from a
-invitation from some other user who is a member of that room.
-
-The inviter's HS sets the membership status of the invitee to "invited" in the
-"m.members" state key by sending a state update PDU. The HS then broadcasts this
-PDU among the existing members in the usual way. An invitation message is also
-sent to the invited user, containing the Room ID and the PDU ID of this
-invitation state change and potentially a list of some other home servers to use
-to accept the invite. The user's client can then choose to display it in some
-way to alert the user.
-
-[[TODO(paul): At present, no API has been designed or described to actually send
-that invite to the invited user. Likely it will be some facet of the larger
-user-user API required for presence, profile management, etc...]]
-
-Directory Service
------------------
-
-Alternatively, the user may discover the channel via a directory service; either
-by performing a name lookup, or some kind of browse or search acitivty. However
-this is performed, the end result is that the user's home server requests the
-Room ID and candidate list from the directory service.
-
-[[TODO(paul): At present, no API has been designed or described for this
-directory service]]
-
-
-Joining
-=======
-
-Once the ID and home servers are obtained, the user can then actually join the
-room.
-
-Accepting an Invite
--------------------
-
-If a user has received and accepted an invitation to join a room, the invitee's
-home server can now send an invite acceptance message to a chosen candidate
-server from the list given in the invitation, citing also the PDU ID of the
-invitation as "proof" of their invite. (This is required as due to late message
-propagation it could be the case that the acceptance is received before the
-invite by some servers). If this message is allowed by the candidate server, it
-generates a new PDU that updates the invitee's membership status to "joined",
-referring back to the acceptance PDU, and broadcasts that as a state change in
-the usual way. The newly-invited user is now a full member of the room, and
-state propagation proceeds as usual.
-
-Joining a Public Room
----------------------
-
-If a user has discovered the existence of a room they wish to join but does not
-have an active invitation, they can request to join it directly by sending a
-join message to a candidate server on the list provided by the directory
-service. As this list may be out of date, the HS should be prepared to retry
-other candidates if the chosen one is no longer aware of the room, because it
-has no users as members in it.
-
-Once a candidate server that is aware of the room has been found, it can
-broadcast an update PDU to add the member into the "m.members" key setting their
-state directly to "joined" (i.e. bypassing the two-phase invite semantics),
-remembering to include the new user's HS in that list.
-
-Knocking on a Semi-Public Room
-------------------------------
-
-If a user requests to join a room but the join mode of the room is "knock", the
-join is not immediately allowed. Instead, if the user wishes to proceed, they
-can instead post a "knock" message, which informs other members of the room that
-the would-be joiner wishes to become a member and sets their membership value to
-"knocked". If any of them wish to accept this, they can then send an invitation
-in the usual way described above. Knowing that the user has already knocked and
-expressed an interest in joining, the invited user's home server should
-immediately accept that invitation on the user's behalf, and go on to join the
-room in the usual way.
-
-[[NOTE(Erik): Though this may confuse users who expect 'X has joined' to
-actually be a user initiated action, i.e. they may expect that 'X' is actually
-looking at synapse right now?]]
-
-[[NOTE(paul): Yes, a fair point maybe we should suggest HSes don't do that, and
-just offer an invite to the user as normal]]
-
-Private and Non-Existent Rooms
-------------------------------
-
-If a user requests to join a room but the room is either unknown by the home
-server receiving the request, or is known by the join mode is "invite" and the
-user has not been invited, the server must respond that the room does not exist.
-This is to prevent leaking information about the existence and identity of
-private rooms.
-
-
-Outstanding Questions
-=====================
-
- * Do invitations or knocks time out and expire at some point? If so when? Time
-   is hard in distributed systems.

docs/model/rooms.rst

@@ -1,274 +0,0 @@
-===========
-Rooms Model
-===========
-
-A description of the general data model used to implement Rooms, and the
-user-level visible effects and implications.
-
-
-Overview
-========
-
-"Rooms" in Synapse are shared messaging channels over which all the participant
-users can exchange messages. Rooms have an opaque persistent identify, a
-globally-replicated set of state (consisting principly of a membership set of
-users, and other management and miscellaneous metadata), and a message history.
-
-
-Room Identity and Naming
-========================
-
-Rooms can be arbitrarily created by any user on any home server; at which point
-the home server will sign the message that creates the channel, and the
-fingerprint of this signature becomes the strong persistent identify of the
-room. This now identifies the room to any home server in the network regardless
-of its original origin. This allows the identify of the room to outlive any
-particular server. Subject to appropriate permissions [to be discussed later],
-any current member of a room can invite others to join it, can post messages
-that become part of its history, and can change the persistent state of the room
-(including its current set of permissions).
-
-Home servers can provide a directory service, allowing a lookup from a
-convenient human-readable form of room label to a room ID. This mapping is
-scoped to the particular home server domain and so simply represents that server
-administrator's opinion of what room should take that label; it does not have to
-be globally replicated and does not form part of the stored state of that room.
-
-This room name takes the form
-
-  #localname:some.domain.name
-
-for similarity and consistency with user names on directories.
-
-To join a room (and therefore to be allowed to inspect past history, post new
-messages to it, and read its state), a user must become aware of the room's
-fingerprint ID. There are two mechanisms to allow this:
-
- * An invite message from someone else in the room
-
- * A referral from a room directory service
-
-As room IDs are opaque and ephemeral, they can serve as a mechanism to create
-"ad-hoc" rooms deliberately unnamed, for small group-chats or even private
-one-to-one message exchange.
-
-
-Stored State and Permissions
-============================
-
-Every room has a globally-replicated set of stored state. This state is a set of
-key/value or key/subkey/value pairs. The value of every (sub)key is a
-JSON-representable object. The main key of a piece of stored state establishes
-its meaning; some keys store sub-keys to allow a sub-structure within them [more
-detail below]. Some keys have special meaning to Synapse, as they relate to
-management details of the room itself, storing such details as user membership,
-and permissions of users to alter the state of the room itself. Other keys may
-store information to present to users, which the system does not directly rely
-on. The key space itself is namespaced, allowing 3rd party extensions, subject
-to suitable permission.
-
-Permission management is based on the concept of "power-levels". Every user
-within a room has an integer assigned, being their "power-level" within that
-room. Along with its actual data value, each key (or subkey) also stores the
-minimum power-level a user must have in order to write to that key, the
-power-level of the last user who actually did write to it, and the PDU ID of
-that state change.
-
-To be accepted as valid, a change must NOT:
-
- * Be made by a user having a power-level lower than required to write to the
-   state key
-
- * Alter the required power-level for that state key to a value higher than the
-   user has
-
- * Increase that user's own power-level
-
- * Grant any other user a power-level higher than the level of the user making
-   the change
-
-[[TODO(paul): consider if relaxations should be allowed; e.g. is the current
-outright-winner allowed to raise their own level, to allow for "inflation"?]]
-
-
-Room State Keys
-===============
-
-[[TODO(paul): if this list gets too big it might become necessary to move it
-into its own doc]]
-
-The following keys have special semantics or meaning to Synapse itself:
-
-m.member (has subkeys)
-  Stores a sub-key for every Synapse User ID which is currently a member of
-  this room. Its value gives the membership type ("knocked", "invited",
-  "joined").
-
-m.power_levels
-  Stores a mapping from Synapse User IDs to their power-level in the room. If
-  they are not present in this mapping, the default applies.
-
-  The reason to store this as a single value rather than a value with subkeys
-  is that updates to it are atomic; allowing a number of colliding-edit
-  problems to be avoided.
-
-m.default_level
-  Gives the default power-level for members of the room that do not have one
-  specified in their membership key.
-
-m.invite_level
-  If set, gives the minimum power-level required for members to invite others
-  to join, or to accept knock requests from non-members requesting access. If
-  absent, then invites are not allowed. An invitation involves setting their
-  membership type to "invited", in addition to sending the invite message.
-
-m.join_rules
-  Encodes the rules on how non-members can join the room. Has the following
-  possibilities:
-    "public" - a non-member can join the room directly
-    "knock" - a non-member cannot join the room, but can post a single "knock"
-        message requesting access, which existing members may approve or deny
-    "invite" - non-members cannot join the room without an invite from an
-        existing member
-    "private" - nobody who is not in the 'may_join' list or already a member
-        may join by any mechanism
-
-  In any of the first three modes, existing members with sufficient permission
-  can send invites to non-members if allowed by the "m.invite_level" key. A
-  "private" room is not allowed to have the "m.invite_level" set.
-
-  A client may use the value of this key to hint at the user interface
-  expectations to provide; in particular, a private chat with one other use
-  might warrant specific handling in the client.
-
-m.may_join
-  A list of User IDs that are always allowed to join the room, regardless of any
-  of the prevailing join rules and invite levels. These apply even to private
-  rooms. These are stored in a single list with normal update-powerlevel
-  permissions applied; users cannot arbitrarily remove themselves from the list.
-
-m.add_state_level
-  The power-level required for a user to be able to add new state keys.
-
-m.public_history
-  If set and true, anyone can request the history of the room, without needing
-  to be a member of the room.
-
-m.archive_servers
-  For "public" rooms with public history, gives a list of home servers that
-  should be included in message distribution to the room, even if no users on
-  that server are present. These ensure that a public room can still persist
-  even if no users are currently members of it. This list should be consulted by
-  the dirctory servers as the candidate list they respond with.
-
-The following keys are provided by Synapse for user benefit, but their value is
-not otherwise used by Synapse.
-
-m.name
-  Stores a short human-readable name for the room, such that clients can display
-  to a user to assist in identifying which room is which.
-  
-  This name specifically is not the strong ID used by the message transport
-  system to refer to the room, because it may be changed from time to time.
-
-m.topic
-  Stores the current human-readable topic
-
-
-Room Creation Templates
-=======================
-
-A client (or maybe home server?) could offer a few templates for the creation of
-new rooms. For example, for a simple private one-to-one chat the channel could
-assign the creator a power-level of 1, requiring a level of 1 to invite, and
-needing an invite before members can join. An invite is then sent to the other
-party, and if accepted and the other user joins, the creator's power-level can
-now be reduced to 0. This now leaves a room with two participants in it being
-unable to add more.
-
-
-Rooms that Continue History
-===========================
-
-An option that could be considered for room creation, is that when a new room is
-created the creator could specify a PDU ID into an existing room, as the history
-continuation point. This would be stored as an extra piece of meta-data on the
-initial PDU of the room's creation. (It does not appear in the normal previous
-PDU linkage).
-
-This would allow users in rooms to "fork" a room, if it is considered that the
-conversations in the room no longer fit its original purpose, and wish to
-diverge. Existing permissions on the original room would continue to apply of
-course, for viewing that history. If both rooms are considered "public" we might
-also want to define a message to post into the original room to represent this
-fork point, and give a reference to the new room.
-
-
-User Direct Message Rooms
-=========================
-
-There is no need to build a mechanism for directly sending messages between
-users, because a room can handle this ability. To allow direct user-to-user chat
-messaging we simply need to be able to create rooms with specific set of
-permissions to allow this direct messaging.
-
-Between any given pair of user IDs that wish to exchange private messages, there
-will exist a single shared Room, created lazily by either side. These rooms will
-need a certain amount of special handling in both home servers and display on
-clients, but as much as possible should be treated by the lower layers of code
-the same as other rooms.
-
-Specially, a client would likely offer a special menu choice associated with
-another user (in room member lists, presence list, etc..) as "direct chat". That
-would perform all the necessary steps to create the private chat room. Receiving
-clients should display these in a special way too as the room name is not
-important; instead it should distinguish them on the Display Name of the other
-party.
-
-Home Servers will need a client-API option to request setting up a new user-user
-chat room, which will then need special handling within the server. It will
-create a new room with the following 
-
-  m.member: the proposing user
-  m.join_rules: "private"
-  m.may_join: both users
-  m.power_levels: empty
-  m.default_level: 0
-  m.add_state_level: 0
-  m.public_history: False
-
-Having created the room, it can send an invite message to the other user in the
-normal way - the room permissions state that no users can be set to the invited
-state, but because they're in the may_join list then they'd be allowed to join
-anyway.
-
-In this arrangement there is now a room with both users may join but neither has
-the power to invite any others. Both users now have the confidence that (at
-least within the messaging system itself) their messages remain private and
-cannot later be provably leaked to a third party. They can freely set the topic
-or name if they choose and add or edit any other state of the room. The update
-powerlevel of each of these fixed properties should be 1, to lock out the users
-from being able to alter them.
-
-
-Anti-Glare
-==========
-
-There exists the possibility of a race condition if two users who have no chat
-history with each other simultaneously create a room and invite the other to it.
-This is called a "glare" situation. There are two possible ideas for how to
-resolve this:
-
- * Each Home Server should persist the mapping of (user ID pair) to room ID, so
-   that duplicate requests can be suppressed. On receipt of a room creation
-   request that the HS thinks there already exists a room for, the invitation to
-   join can be rejected if:
-      a) the HS believes the sending user is already a member of the room (and
-         maybe their HS has forgotten this fact), or
-      b) the proposed room has a lexicographically-higher ID than the existing
-         room (to resolve true race condition conflicts)
-      
- * The room ID for a private 1:1 chat has a special form, determined by
-   concatenting the User IDs of both members in a deterministic order, such that
-   it doesn't matter which side creates it first; the HSes can just ignore
-   (or merge?) received PDUs that create the room twice.

docs/model/terminology.rst

@@ -1,86 +0,0 @@
-===========
-Terminology
-===========
-
-A list of definitions of specific terminology used among these documents.
-These terms were originally taken from the server-server documentation, and may
-not currently match the exact meanings used in other places; though as a
-medium-term goal we should encourage the unification of this terminology.
-
-
-Terms
-=====
-
-Backfilling:
-  The process of synchronising historic state from one home server to another,
-  to backfill the event storage so that scrollback can be presented to the
-  client(s).  (Formerly, and confusingly, called 'pagination')
-
-Context:
-  A single human-level entity of interest (currently, a chat room)
-
-EDU (Ephemeral Data Unit):
-  A message that relates directly to a given pair of home servers that are
-  exchanging it. EDUs are short-lived messages that related only to one single
-  pair of servers; they are not persisted for a long time and are not forwarded
-  on to other servers. Because of this, they have no internal ID nor previous
-  EDUs reference chain.
-
-Event:
-  A record of activity that records a single thing that happened on to a context
-  (currently, a chat room). These are the "chat messages" that Synapse makes
-  available.
-  [[NOTE(paul): The current server-server implementation calls these simply
-  "messages" but the term is too ambiguous here; I've called them Events]]
-
-PDU (Persistent Data Unit):
-  A message that relates to a single context, irrespective of the server that
-  is communicating it. PDUs either encode a single Event, or a single State
-  change. A PDU is referred to by its PDU ID; the pair of its origin server
-  and local reference from that server.
-
-PDU ID:
-  The pair of PDU Origin and PDU Reference, that together globally uniquely
-  refers to a specific PDU.
-
-PDU Origin:
-  The name of the origin server that generated a given PDU. This may not be the
-  server from which it has been received, due to the way they are copied around
-  from server to server. The origin always records the original server that
-  created it.
-
-PDU Reference:
-  A local ID used to refer to a specific PDU from a given origin server. These
-  references are opaque at the protocol level, but may optionally have some
-  structured meaning within a given origin server or implementation.
-
-Presence:
-  The concept of whether a user is currently online, how available they declare
-  they are, and so on. See also: doc/model/presence
-
-Profile:
-  A set of metadata about a user, such as a display name, provided for the
-  benefit of other users. See also: doc/model/profiles
-
-Room ID:
-  An opaque string (of as-yet undecided format) that identifies a particular
-  room and used in PDUs referring to it.
-
-Room Alias:
-  A human-readable string of the form #name:some.domain that users can use as a
-  pointer to identify a room; a Directory Server will map this to its Room ID
-
-State:
-  A set of metadata maintained about a Context, which is replicated among the
-  servers in addition to the history of Events.
-
-User ID:
-  A string of the form @localpart:domain.name that identifies a user for
-  wire-protocol purposes. The localpart is meaningless outside of a particular
-  home server. This takes a human-readable form that end-users can use directly
-  if they so wish, avoiding the 3PIDs.
-
-Transaction:
-  A message which relates to the communication between a given pair of servers.
-  A transaction contains possibly-empty lists of PDUs and EDUs.
-

docs/model/third-party-id.rst

@@ -1,108 +0,0 @@
-======================
-Third Party Identities
-======================
-
-A description of how email addresses, mobile phone numbers and other third
-party identifiers can be used to authenticate and discover users in Matrix.
-
-
-Overview
-========
-
-New users need to authenticate their account. An email or SMS text message can 
-be a convenient form of authentication. Users already have email addresses 
-and phone numbers for contacts in their address book. They want to communicate
-with those contacts in Matrix without manually exchanging a Matrix User ID with 
-them.
-
-Third Party IDs
----------------
-
-[[TODO(markjh): Describe the format of a 3PID]]
-
-
-Third Party ID Associations
----------------------------
-
-An Associaton is a binding between a Matrix User ID and a Third Party ID (3PID).
-Each 3PID can be associated with one Matrix User ID at a time.
-
-[[TODO(markjh): JSON format of the association.]]
-
-Verification 
-------------
-
-An Assocation must be verified by a trusted Verification Server. Email 
-addresses and phone numbers can be verified by sending a token to the address 
-which a client can supply to the verifier to confirm ownership.
-
-An email Verification Server may be capable of verifying all email 3PIDs or may
-be restricted to verifying addresses for a particular domain. A phone number
-Verification Server may be capable of verifying all phone numbers or may be
-restricted to verifying numbers for a given country or phone prefix.
-
-Verification Servers fulfil a similar role to Certificate Authorities in PKI so
-a similar level of vetting should be required before clients trust their
-signatures.
-
-A Verification Server may wish to check for existing Associations for a 3PID 
-before creating a new Association.
-
-Discovery
----------
-
-Users can discover Associations using a trusted Identity Server. Each 
-Association will be signed by the Identity Server. An Identity Server may store
-the entire space of Associations or may delegate to other Identity Servers when
-looking up Associations.
-
-Each Association returned from an Identity Server must be signed by a 
-Verification Server. Clients should check these signatures.
-
-Identity Servers fulfil a similar role to DNS servers.
-
-Privacy
--------
-
-A User may publish the association between their phone number and Matrix User ID
-on the Identity Server without publishing the number in their Profile hosted on
-their Home Server.
-
-Identity Servers should refrain from publishing reverse mappings and should 
-take steps, such as rate limiting, to prevent attackers enumerating the space of
-mappings.
-
-Federation
-==========
-
-Delegation
-----------
-
-Verification Servers could delegate signing to another server by issuing 
-certificate to that server allowing it to verify and sign a subset of 3PID on 
-its behalf. It would be necessary to provide a language for describing which
-subset of 3PIDs that server had authority to validate. Alternatively it could 
-delegate the verification step to another server but sign the resulting
-association itself.
-
-The 3PID space will have a heirachical structure like DNS so Identity Servers
-can delegate lookups to other servers. An Identity Server should be prepared 
-to host or delegate any valid association within the subset of the 3PIDs it is 
-resonsible for.
-
-Multiple Root Verification Servers
-----------------------------------
-
-There can be multiple root Verification Servers and an Association could be
-signed by multiple servers if different clients trust different subsets of
-the verification servers.
-
-Multiple Root Identity Servers
-------------------------------
-
-There can be be multiple root Identity Servers. Clients will add each
-Association to all root Identity Servers.
-
-[[TODO(markjh): Describe how clients find the list of root Identity Servers]]
-
-

docs/server-server/protocol-format.rst

@@ -1,59 +0,0 @@
-
-Transaction
-===========
-
-Required keys:
-
-============ =================== ===============================================
-    Key            Type                         Description
-============ =================== ===============================================
-origin       String              DNS name of homeserver making this transaction.
-ts           Integer             Timestamp in milliseconds on originating 
-                                 homeserver when this transaction started.
-previous_ids List of Strings     List of transactions that were sent immediately
-                                 prior to this transaction.
-pdus         List of Objects     List of updates contained in this transaction.
-============ =================== ===============================================
-
-
-PDU
-===
-
-Required keys:
-
-============ ================== ================================================
-    Key            Type                         Description
-============ ================== ================================================
-context      String             Event context identifier
-origin       String             DNS name of homeserver that created this PDU.
-pdu_id       String             Unique identifier for PDU within the context for
-                                the originating homeserver.
-ts           Integer            Timestamp in milliseconds on originating 
-                                homeserver when this PDU was created.
-pdu_type     String             PDU event type.
-prev_pdus    List of Pairs      The originating homeserver and PDU ids of the
-             of Strings         most recent PDUs the homeserver was aware of for
-                                this context when it made this PDU.
-depth        Integer            The maximum depth of the previous PDUs plus one.
-============ ================== ================================================
-
-Keys for state updates:
-
-================== ============ ================================================
-    Key               Type                      Description
-================== ============ ================================================
-is_state           Boolean      True if this PDU is updating state.
-state_key          String       Optional key identifying the updated state within
-                                the context.
-power_level        Integer      The asserted power level of the user performing
-                                the update.
-min_update         Integer      The required power level needed to replace this
-                                update.
-prev_state_id      String       The homeserver of the update this replaces
-prev_state_origin  String       The PDU id of the update this replaces.
-user               String       The user updating the state.
-================== ============ ================================================
-
-
-
-

docs/server-server/security-threat-model.rst

@@ -1,141 +0,0 @@
-Overview
-========
-
-Scope
------
-
-This document considers threats specific to the server to server federation 
-synapse protocol.
-
-
-Attacker
---------
-
-It is assumed that the attacker can see and manipulate all network traffic 
-between any of the servers and may be in control of one or more homeservers 
-participating in the federation protocol.
-
-Threat Model
-============
-
-Denial of Service
------------------
-
-The attacker could attempt to prevent delivery of messages to or from the 
-victim in order to:
-
-    * Disrupt service or marketing campaign of a commercial competitor.
-    * Censor a discussion or censor a participant in a discussion.
-    * Perform general vandalism.
-
-Threat: Resource Exhaustion
-~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-An attacker could cause the victims server to exhaust a particular resource 
-(e.g. open TCP connections, CPU, memory, disk storage)
-
-Threat: Unrecoverable Consistency Violations
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-An attacker could send messages which created an unrecoverable "split-brain"
-state in the cluster such that the victim's servers could no longer dervive a
-consistent view of the chatroom state.
-
-Threat: Bad History
-~~~~~~~~~~~~~~~~~~~
-
-An attacker could convince the victim to accept invalid messages which the 
-victim would then include in their view of the chatroom history. Other servers
-in the chatroom would reject the invalid messages and potentially reject the
-victims messages as well since they depended on the invalid messages.
-
-Threat: Block Network Traffic
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-An attacker could try to firewall traffic between the victim's server and some
-or all of the other servers in the chatroom.
-
-Threat: High Volume of Messages
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-An attacker could send large volumes of messages to a chatroom with the victim
-making the chatroom unusable.
-
-Threat: Banning users without necessary authorisation
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-An attacker could attempt to ban a user from a chatroom with the necessary
-authorisation.
-
-Spoofing
---------
-
-An attacker could try to send a message claiming to be from the victim without 
-the victim having sent the message in order to:
-
-    * Impersonate the victim while performing illict activity.
-    * Obtain privileges of the victim.
-
-Threat: Altering Message Contents
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-An attacker could try to alter the contents of an existing message from the 
-victim.
-
-Threat: Fake Message "origin" Field
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-An attacker could try to send a new message purporting to be from the victim
-with a phony "origin" field.
-
-Spamming
---------
-
-The attacker could try to send a high volume of solicicted or unsolicted 
-messages to the victim in order to:
-    
-    * Find victims for scams.
-    * Market unwanted products.
-
-Threat: Unsoliticted Messages
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-An attacker could try to send messages to victims who do not wish to receive 
-them.
-
-Threat: Abusive Messages
-~~~~~~~~~~~~~~~~~~~~~~~~
-
-An attacker could send abusive or threatening messages to the victim
-
-Spying
-------
-
-The attacker could try to access message contents or metadata for messages sent
-by the victim or to the victim that were not intended to reach the attacker in
-order to:
-
-    * Gain sensitive personal or commercial information.
-    * Impersonate the victim using credentials contained in the messages.
-      (e.g. password reset messages)
-    * Discover who the victim was talking to and when.
-
-Threat: Disclosure during Transmission
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-An attacker could try to expose the message contents or metadata during 
-transmission between the servers.
-
-Threat: Disclosure to Servers Outside Chatroom
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-An attacker could try to convince servers within a chatroom to send messages to
-a server it controls that was not authorised to be within the chatroom.
-
-Threat: Disclosure to Servers Within Chatroom
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-An attacker could take control of a server within a chatroom to expose message
-contents or metadata for messages in that room.
-
-

docs/server-server/specification.rst

@@ -1,231 +0,0 @@
-===========================
-Matrix Server-to-Server API
-===========================
-
-A description of the protocol used to communicate between Matrix home servers;
-also known as Federation.
-
-
-Overview
-========
-
-The server-server API is a mechanism by which two home servers can exchange
-Matrix event messages, both as a real-time push of current events, and as a
-historic fetching mechanism to synchronise past history for clients to view. It
-uses HTTP connections between each pair of servers involved as the underlying
-transport. Messages are exchanged between servers in real-time by active pushing
-from each server's HTTP client into the server of the other. Queries to fetch
-historic data for the purpose of back-filling scrollback buffers and the like
-can also be performed.
-
-
-  { Matrix clients }                              { Matrix clients }
-     ^          |                                    ^          |
-     |  events  |                                    |  events  |
-     |          V                                    |          V
- +------------------+                            +------------------+
- |                  |---------( HTTP )---------->|                  |
- |   Home Server    |                            |   Home Server    |
- |                  |<--------( HTTP )-----------|                  |
- +------------------+                            +------------------+
-
-There are three main kinds of communication that occur between home servers:
-
- * Queries
-   These are single request/response interactions between a given pair of
-   servers, initiated by one side sending an HTTP request to obtain some
-   information, and responded by the other. They are not persisted and contain
-   no long-term significant history. They simply request a snapshot state at the
-   instant the query is made.
-
- * EDUs - Ephemeral Data Units
-   These are notifications of events that are pushed from one home server to
-   another. They are not persisted and contain no long-term significant history,
-   nor does the receiving home server have to reply to them.
-
- * PDUs - Persisted Data Units
-   These are notifications of events that are broadcast from one home server to
-   any others that are interested in the same "context" (namely, a Room ID).
-   They are persisted to long-term storage and form the record of history for
-   that context.
-
-Where Queries are presented directly across the HTTP connection as GET requests
-to specific URLs, EDUs and PDUs are further wrapped in an envelope called a
-Transaction, which is transferred from the origin to the destination home server
-using a PUT request.
-
-
-Transactions and EDUs/PDUs
-==========================
-
-The transfer of EDUs and PDUs between home servers is performed by an exchange
-of Transaction messages, which are encoded as JSON objects with a dict as the
-top-level element, passed over an HTTP PUT request. A Transaction is meaningful
-only to the pair of home servers that exchanged it; they are not globally-
-meaningful.
-
-Each transaction has an opaque ID and timestamp (UNIX epoch time in
-milliseconds) generated by its origin server, an origin and destination server
-name, a list of "previous IDs", and a list of PDUs - the actual message payload
-that the Transaction carries.
-
- {"transaction_id":"916d630ea616342b42e98a3be0b74113",
-  "ts":1404835423000,
-  "origin":"red",
-  "destination":"blue",
-  "prev_ids":["e1da392e61898be4d2009b9fecce5325"],
-  "pdus":[...],
-  "edus":[...]}
-
-The "previous IDs" field will contain a list of previous transaction IDs that
-the origin server has sent to this destination. Its purpose is to act as a
-sequence checking mechanism - the destination server can check whether it has
-successfully received that Transaction, or ask for a retransmission if not.
-
-The "pdus" field of a transaction is a list, containing zero or more PDUs.[*]
-Each PDU is itself a dict containing a number of keys, the exact details of
-which will vary depending on the type of PDU. Similarly, the "edus" field is
-another list containing the EDUs. This key may be entirely absent if there are
-no EDUs to transfer.
-
-(* Normally the PDU list will be non-empty, but the server should cope with
-receiving an "empty" transaction, as this is useful for informing peers of other
-transaction IDs they should be aware of. This effectively acts as a push
-mechanism to encourage peers to continue to replicate content.)
-
-All PDUs have an ID, a context, a declaration of their type, a list of other PDU
-IDs that have been seen recently on that context (regardless of which origin
-sent them), and a nested content field containing the actual event content.
-
-[[TODO(paul): Update this structure so that 'pdu_id' is a two-element
-[origin,ref] pair like the prev_pdus are]]
-
- {"pdu_id":"a4ecee13e2accdadf56c1025af232176",
-  "context":"#example.green",
-  "origin":"green",
-  "ts":1404838188000,
-  "pdu_type":"m.text",
-  "prev_pdus":[["blue","99d16afbc857975916f1d73e49e52b65"]],
-  "content":...
-  "is_state":false}
-
-In contrast to the transaction layer, it is important to note that the prev_pdus
-field of a PDU refers to PDUs that any origin server has sent, rather than
-previous IDs that this origin has sent. This list may refer to other PDUs sent
-by the same origin as the current one, or other origins.
-
-Because of the distributed nature of participants in a Matrix conversation, it
-is impossible to establish a globally-consistent total ordering on the events.
-However, by annotating each outbound PDU at its origin with IDs of other PDUs it
-has received, a partial ordering can be constructed allowing causallity
-relationships to be preserved. A client can then display these messages to the
-end-user in some order consistent with their content and ensure that no message
-that is semantically in reply of an earlier one is ever displayed before it.
-
-PDUs fall into two main categories: those that deliver Events, and those that
-synchronise State. For PDUs that relate to State synchronisation, additional
-keys exist to support this:
-
- {...,
-  "is_state":true,
-  "state_key":TODO
-  "power_level":TODO
-  "prev_state_id":TODO
-  "prev_state_origin":TODO}
-
-[[TODO(paul): At this point we should probably have a long description of how
-State management works, with descriptions of clobbering rules, power levels, etc
-etc... But some of that detail is rather up-in-the-air, on the whiteboard, and
-so on. This part needs refining. And writing in its own document as the details
-relate to the server/system as a whole, not specifically to server-server
-federation.]]
-
-EDUs, by comparison to PDUs, do not have an ID, a context, or a list of
-"previous" IDs. The only mandatory fields for these are the type, origin and
-destination home server names, and the actual nested content.
-
- {"edu_type":"m.presence",
-  "origin":"blue",
-  "destination":"orange",
-  "content":...}
-
-
-Protocol URLs
-=============
-
-All these URLs are namespaced within a prefix of 
-
-  /matrix/federation/v1/...
-
-For active pushing of messages representing live activity "as it happens":
-
-  PUT .../send/:transaction_id/
-    Body: JSON encoding of a single Transaction
-
-    Response: [[TODO(paul): I don't actually understand what
-    ReplicationLayer.on_transaction() is doing here, so I'm not sure what the
-    response ought to be]]
-
-  The transaction_id path argument will override any ID given in the JSON body.
-  The destination name will be set to that of the receiving server itself. Each
-  embedded PDU in the transaction body will be processed.
-
-
-To fetch a particular PDU:
-
-  GET .../pdu/:origin/:pdu_id/
-
-    Response: JSON encoding of a single Transaction containing one PDU
-
-  Retrieves a given PDU from the server. The response will contain a single new
-  Transaction, inside which will be the requested PDU.
-  
-
-To fetch all the state of a given context:
-
-  GET .../state/:context/
-
-    Response: JSON encoding of a single Transaction containing multiple PDUs
-
-  Retrieves a snapshot of the entire current state of the given context. The
-  response will contain a single Transaction, inside which will be a list of
-  PDUs that encode the state.
-
-
-To backfill events on a given context:
-
-  GET .../backfill/:context/
-    Query args: v, limit
-
-    Response: JSON encoding of a single Transaction containing multiple PDUs
-
-  Retrieves a sliding-window history of previous PDUs that occurred on the
-  given context. Starting from the PDU ID(s) given in the "v" argument, the
-  PDUs that preceeded it are retrieved, up to a total number given by the
-  "limit" argument. These are then returned in a new Transaction containing all
-  off the PDUs.
-
-
-To stream events all the events:
-
-  GET .../pull/
-    Query args: origin, v
-
-  Response: JSON encoding of a single Transaction consisting of multiple PDUs
-
-  Retrieves all of the transactions later than any version given by the "v"
-  arguments. [[TODO(paul): I'm not sure what the "origin" argument does because
-  I think at some point in the code it's got swapped around.]]
-
-
-To make a query:
-
-  GET .../query/:query_type
-    Query args: as specified by the individual query types
-
-  Response: JSON encoding of a response object
-
-  Performs a single query request on the receiving home server. The Query Type
-  part of the path specifies the kind of query being made, and its query
-  arguments have a meaning specific to that kind of query. The response is a
-  JSON-encoded object whose meaning also depends on the kind of query.

docs/server-server/versioning.rst

@@ -1,11 +0,0 @@
-Versioning is, like, hard for backfilling backwards because of the number of Home Servers involved.
-
-The way we solve this is by doing versioning as an acyclic directed graph of PDUs. For backfilling purposes, this is done on a per context basis. 
-When we send a PDU we include all PDUs that have been received for that context that hasn't been subsequently listed in a later PDU. The trivial case is a simple list of PDUs, e.g. A <- B <- C. However, if two servers send out a PDU at the same to, both B and C would point at A - a later PDU would then list both B and C.
-
-Problems with opaque version strings:
-    - How do you do clustering without mandating that a cluster can only have one transaction in flight to a given remote home server at a time. 
-      If you have multiple transactions sent at once, then you might drop one transaction, receive another with a version that is later than the dropped transaction and which point ARGH WE LOST A TRANSACTION.
-    - How do you do backfilling? A version string defines a point in a stream w.r.t. a single home server, not a point in the context.
-
-We only need to store the ends of the directed graph, we DO NOT need to do the whole one table of nodes and one of edges.

docs/specification.rst

@@ -1,839 +0,0 @@
-Matrix Specification
-====================
-
-TODO(Introduction) : Matthew
- - Similar to intro paragraph from README.
- - Explaining the overall mission, what this spec describes...
- - "What is Matrix?"
- - Draw parallels with email?
-
-Architecture
-============
-
-Clients transmit data to other clients through home servers (HSes). Clients do not communicate with each
-other directly.
-
-::
-
-                         How data flows between clients
-                         ==============================
-
-       { Matrix client A }                             { Matrix client B }
-           ^          |                                    ^          |
-           |  events  |                                    |  events  |
-           |          V                                    |          V
-       +------------------+                            +------------------+
-       |                  |---------( HTTP )---------->|                  |
-       |   Home Server    |                            |   Home Server    |
-       |                  |<--------( HTTP )-----------|                  |
-       +------------------+        Federation          +------------------+
-
-A "Client" is an end-user, typically a human using a web application or mobile app. Clients use the
-"Client-to-Server" (C-S) API to communicate with their home server. A single Client is usually
-responsible for a single user account. A user account is represented by their "User ID". This ID is
-namespaced to the home server which allocated the account and looks like::
-
-  @localpart:domain
-
-The ``localpart`` of a user ID may be a user name, or an opaque ID identifying this user.
-
-
-A "Home Server" is a server which provides C-S APIs and has the ability to federate with other HSes.
-It is typically responsible for multiple clients. "Federation" is the term used to describe the
-sharing of data between two or more home servers.
-
-Data in Matrix is encapsulated in an "Event". An event is an action within the system. Typically each
-action (e.g. sending a message) correlates with exactly one event. Each event has a ``type`` which is
-used to differentiate different kinds of data. ``type`` values SHOULD be namespaced according to standard
-Java package naming conventions, e.g. ``com.example.myapp.event``. Events are usually sent in the context
-of a "Room".
-
-Room structure
---------------
-
-A room is a conceptual place where users can send and receive events. Rooms 
-can be created, joined and left. Events are sent to a room, and all 
-participants in that room will receive the event. Rooms are uniquely 
-identified via a "Room ID", which look like::
-
-  !opaque_id:domain
-
-There is exactly one room ID for each room. Whilst the room ID does contain a
-domain, it is simply for namespacing room IDs. The room does NOT reside on the
-domain specified. Room IDs are not meant to be human readable.
-
-The following diagram shows an ``m.room.message`` event being sent in the room 
-``!qporfwt:matrix.org``::
-
-       { @alice:matrix.org }                             { @bob:domain.com }
-               |                                                 ^
-               |                                                 |
-      Room ID: !qporfwt:matrix.org                 Room ID: !qporfwt:matrix.org
-      Event type: m.room.message                   Event type: m.room.message
-      Content: { JSON object }                     Content: { JSON object }
-               |                                                 |
-               V                                                 |
-       +------------------+                          +------------------+
-       |   Home Server    |                          |   Home Server    |
-       |   matrix.org     |<-------Federation------->|   domain.com     |
-       +------------------+                          +------------------+
-                |       .................................        |
-                |______|     Partially Shared State      |_______|
-                       | Room ID: !qporfwt:matrix.org    |
-                       | Servers: matrix.org, domain.com |
-                       | Members:                        |
-                       |  - @alice:matrix.org            |
-                       |  - @bob:domain.com              |
-                       |.................................|
-
-Federation maintains shared state between multiple home servers, such that when an event is
-sent to a room, the home server knows where to forward the event on to, and how to process
-the event. Home servers do not need to have completely shared state in order to participate 
-in a room. State is scoped to a single room, and federation ensures that all home servers 
-have the information they need, even if that means the home server has to request more 
-information from another home server before processing the event.
-
-Room Aliases
-------------
-
-Each room can also have multiple "Room Aliases", which looks like::
-
-  #room_alias:domain
-
-A room alias "points" to a room ID. The room ID the alias is pointing to can be obtained
-by visiting the domain specified. Room aliases are designed to be human readable strings
-which can be used to publicise rooms. Note that the mapping from a room alias to a 
-room ID is not fixed, and may change over time to point to a different room ID. For this
-reason, Clients SHOULD resolve the room alias to a room ID once and then use that ID on
-subsequent requests.
-
-::
-
-          GET    
-   #matrix:domain.com      !aaabaa:matrix.org
-           |                    ^
-           |                    |
-    _______V____________________|____
-   |          domain.com            |
-   | Mappings:                      |
-   | #matrix >> !aaabaa:matrix.org  |
-   | #golf >> !wfeiofh:sport.com    |
-   | #bike >> !4rguxf:matrix.org    |
-   |________________________________|
-
-       
-Identity
---------
-- Identity in relation to 3PIDs. Discovery of users based on 3PIDs.
-- Identity servers; trusted clique of servers which replicate content.
-- They govern the mapping of 3PIDs to user IDs and the creation of said mappings.
-- Not strictly required in order to communicate.
-
-
-API Standards
--------------
-- All HTTP[S]
-- Uses JSON as HTTP bodies
-- Standard error response format { errcode: M_WHATEVER, error: "some message" }
-- C-S API provides POST for operations, or PUT with txn IDs. Explain txn IDs.
-
-Receiving live updates on a client
-----------------------------------
-- C-S longpoll event stream
-- Concept of start/end tokens.
-- Mention /initialSync to get token.
-
-
-Rooms
-=====
-- How are they created? PDU anchor point: "root of the tree".
-- Adding / removing aliases.
-- Invite/join dance
-- State and non-state data (+extensibility)
-
-TODO : Room permissions / config / power levels.
-
-Messages
-========
-
-This specification outlines several standard event types, all of which are
-prefixed with ``m.``
-
-State messages
---------------
-- m.room.name
-- m.room.topic
-- m.room.member
-- m.room.config
-- m.room.invite_join
-
-What are they, when are they used, what do they contain, how should they be used
-
-Non-state messages
-------------------
-- m.room.message
-- m.room.message.feedback (and compressed format)
-
-What are they, when are they used, what do they contain, how should they be used
-
-m.room.message msgtypes
------------------------
-Each ``m.room.message`` MUST have a ``msgtype`` key which identifies the type of
-message being sent. Each type has their own required and optional keys, as outlined
-below:
-
-``m.text``
-  Required keys:
-    - ``body`` : "string" - The body of the message.
-  Optional keys:
-    None.
-  Example:
-    ``{ "msgtype": "m.text", "body": "I am a fish" }``
-
-``m.emote``
-  Required keys:
-    - ``body`` : "string" - The emote action to perform.
-  Optional keys:
-    None.
-  Example:
-    ``{ "msgtype": "m.emote", "body": "tries to come up with a witty explanation" }``
-
-``m.image``
-  Required keys:
-    - ``url`` : "string" - The URL to the image.
-  Optional keys:
-    - ``info`` : "string" - info : JSON object (ImageInfo) - The image info for image 
-      referred to in ``url``.
-    - ``thumbnail_url`` : "string" - The URL to the thumbnail.
-    - ``thumbnail_info`` : JSON object (ImageInfo) - The image info for the image 
-      referred to in ``thumbnail_url``.
-    - ``body`` : "string" - The alt text of the image, or some kind of content 
-      description for accessibility e.g. "image attachment".
-
-  ImageInfo: 
-    Information about an image::
-    
-      { 
-        "size" : integer (size of image in bytes),
-        "w" : integer (width of image in pixels),
-        "h" : integer (height of image in pixels),
-        "mimetype" : "string (e.g. image/jpeg)",
-      }
-
-``m.audio``
-  Required keys:
-    - ``url`` : "string" - The URL to the audio.
-  Optional keys:
-    - ``info`` : JSON object (AudioInfo) - The audio info for the audio referred to in 
-      ``url``.
-    - ``body`` : "string" - A description of the audio e.g. "Bee Gees - 
-      Stayin' Alive", or some kind of content description for accessibility e.g. 
-      "audio attachment".
-  AudioInfo: 
-    Information about a piece of audio::
-
-      {
-        "mimetype" : "string (e.g. audio/aac)",
-        "size" : integer (size of audio in bytes),
-        "duration" : integer (duration of audio in milliseconds),
-      }
-
-``m.video``
-  Required keys:
-    - ``url`` : "string" - The URL to the video.
-  Optional keys:
-    - ``info`` : JSON object (VideoInfo) - The video info for the video referred to in 
-      ``url``.
-    - ``body`` : "string" - A description of the video e.g. "Gangnam style", 
-      or some kind of content description for accessibility e.g. "video attachment".
-
-  VideoInfo: 
-    Information about a video::
-
-      {
-        "mimetype" : "string (e.g. video/mp4)",
-        "size" : integer (size of video in bytes),
-        "duration" : integer (duration of video in milliseconds),
-        "w" : integer (width of video in pixels),
-        "h" : integer (height of video in pixels),
-        "thumbnail_url" : "string (URL to image)",
-        "thumbanil_info" : JSON object (ImageInfo)
-      }
-
-``m.location``
-  Required keys:
-    - ``geo_uri`` : "string" - The geo URI representing the location.
-  Optional keys:
-    - ``thumbnail_url`` : "string" - The URL to a thumnail of the location being 
-      represented.
-    - ``thumbnail_info`` : JSON object (ImageInfo) - The image info for the image 
-      referred to in ``thumbnail_url``.
-    - ``body`` : "string" - A description of the location e.g. "Big Ben, 
-      London, UK", or some kind of content description for accessibility e.g. 
-      "location attachment".
-
-The following keys can be attached to any ``m.room.message``:
-
-  Optional keys:
-    - ``sender_ts`` : integer - A timestamp (ms resolution) representing the 
-      wall-clock time when the message was sent from the client.
-
-Presence
-========
-
-Each user has the concept of presence information. This encodes the
-"availability" of that user, suitable for display on other user's clients. This
-is transmitted as an ``m.presence`` event and is one of the few events which
-are sent *outside the context of a room*. The basic piece of presence information 
-is represented by the ``state`` key, which is an enum of one of the following:
-
-  - ``online`` : The default state when the user is connected to an event stream.
-  - ``unavailable`` : The user is not reachable at this time.
-  - ``offline`` : The user is not connected to an event stream.
-  - ``free_for_chat`` : The user is generally willing to receive messages 
-    moreso than default.
-  - ``hidden`` : TODO. Behaves as offline, but allows the user to see the client 
-    state anyway and generally interact with client features.
-
-This basic ``state`` field applies to the user as a whole, regardless of how many
-client devices they have connected. The home server should synchronise this
-status choice among multiple devices to ensure the user gets a consistent
-experience.
-
-Idle Time
----------
-As well as the basic ``state`` field, the presence information can also show a sense
-of an "idle timer". This should be maintained individually by the user's
-clients, and the home server can take the highest reported time as that to
-report. When a user is offline, the home server can still report when the user was last
-seen online.
-
-Transmission
-------------
-- Transmitted as an EDU.
-- Presence lists determine who to send to.
-
-Presence List
--------------
-Each user's home server stores a "presence list" for that user. This stores a
-list of other user IDs the user has chosen to add to it. To be added to this 
-list, the user being added must receive permission from the list owner. Once
-granted, both user's HS(es) store this information. Since such subscriptions
-are likely to be bidirectional, HSes may wish to automatically accept requests
-when a reverse subscription already exists.
-
-Presence and Permissions
-------------------------
-For a viewing user to be allowed to see the presence information of a target
-user, either:
-
- - The target user has allowed the viewing user to add them to their presence
-   list, or
- - The two users share at least one room in common
-
-In the latter case, this allows for clients to display some minimal sense of
-presence information in a user list for a room.
-
-Typing notifications
-====================
-
-TODO : Leo
-
-Voice over IP
-=============
-
-TODO : Dave
-
-Profiles
-========
-
-Internally within Matrix users are referred to by their user ID, which is not a
-human-friendly string. Profiles grant users the ability to see human-readable 
-names for other users that are in some way meaningful to them. Additionally, 
-profiles can publish additional information, such as the user's age or location.
-
-A Profile consists of a display name, an avatar picture, and a set of other 
-metadata fields that the user may wish to publish (email address, phone
-numbers, website URLs, etc...). This specification puts no requirements on the 
-display name other than it being a valid unicode string.
-
-- Metadata extensibility
-- Bundled with which events? e.g. m.room.member
-- Generate own events? What type?
-
-Registration and login
-======================
-
-Clients must register with a home server in order to use Matrix. After 
-registering, the client will be given an access token which must be used in ALL
-requests to that home server as a query parameter 'access_token'.
-
-- TODO Kegan : Make registration like login (just omit the "user" key on the 
-  initial request?)
-
-If the client has already registered, they need to be able to login to their
-account. The home server may provide many different ways of logging in, such
-as user/password auth, login via a social network (OAuth2), login by confirming 
-a token sent to their email address, etc. This specification does not define how
-home servers should authorise their users who want to login to their existing 
-accounts, but instead defines the standard interface which implementations 
-should follow so that ANY client can login to ANY home server.
-
-The login process breaks down into the following:
-  1. Determine the requirements for logging in.
-  2. Submit the login stage credentials.
-  3. Get credentials or be told the next stage in the login process and repeat 
-     step 2.
-     
-As each home server may have different ways of logging in, the client needs to know how
-they should login. All distinct login stages MUST have a corresponding ``type``.
-A ``type`` is a namespaced string which details the mechanism for logging in.
-
-A client may be able to login via multiple valid login flows, and should choose a single
-flow when logging in. A flow is a series of login stages. The home server MUST respond 
-with all the valid login flows when requested::
-
-  The client can login via 3 paths: 1a and 1b, 2a and 2b, or 3. The client should
-  select one of these paths.
-  
-  {
-    "flows": [
-      {
-        "type": "<login type1a>",
-        "stages": [ "<login type 1a>", "<login type 1b>" ]
-      },
-      {
-        "type": "<login type2a>",
-        "stages": [ "<login type 2a>", "<login type 2b>" ]
-      },
-      {
-        "type": "<login type3>"
-      }
-    ]
-  }
-
-After the login is completed, the client's fully-qualified user ID and a new access 
-token MUST be returned::
-
-  {
-    "user_id": "@user:matrix.org",
-    "access_token": "abcdef0123456789"
-  }
-
-The ``user_id`` key is particularly useful if the home server wishes to support 
-localpart entry of usernames (e.g. "user" rather than "@user:matrix.org"), as the
-client may not be able to determine its ``user_id`` in this case.
-
-If a login has multiple requests, the home server may wish to create a session. If
-a home server responds with a 'session' key to a request, clients MUST submit it in 
-subsequent requests until the login is completed::
-
-  {
-    "session": "<session id>"
-  }
-
-This specification defines the following login types:
- - ``m.login.password``
- - ``m.login.oauth2``
- - ``m.login.email.code``
- - ``m.login.email.url``
-
-
-Password-based
---------------
-:Type: 
-  m.login.password
-:Description: 
-  Login is supported via a username and password.
-
-To respond to this type, reply with::
-
-  {
-    "type": "m.login.password",
-    "user": "<user_id or user localpart>",
-    "password": "<password>"
-  }
-
-The home server MUST respond with either new credentials, the next stage of the login
-process, or a standard error response.
-
-OAuth2-based
-------------
-:Type: 
-  m.login.oauth2
-:Description:
-  Login is supported via OAuth2 URLs. This login consists of multiple requests.
-
-To respond to this type, reply with::
-
-  {
-    "type": "m.login.oauth2",
-    "user": "<user_id or user localpart>"
-  }
-
-The server MUST respond with::
-
-  {
-    "uri": <Authorization Request URI OR service selection URI>
-  }
-
-The home server acts as a 'confidential' client for the purposes of OAuth2.
-If the uri is a ``sevice selection URI``, it MUST point to a webpage which prompts the 
-user to choose which service to authorize with. On selection of a service, this
-MUST link through to an ``Authorization Request URI``. If there is only 1 service which the
-home server accepts when logging in, this indirection can be skipped and the
-"uri" key can be the ``Authorization Request URI``. 
-
-The client then visits the ``Authorization Request URI``, which then shows the OAuth2 
-Allow/Deny prompt. Hitting 'Allow' returns the ``redirect URI`` with the auth code. 
-Home servers can choose any path for the ``redirect URI``. The client should visit 
-the ``redirect URI``, which will then finish the OAuth2 login process, granting the 
-home server an access token for the chosen service. When the home server gets 
-this access token, it verifies that the cilent has authorised with the 3rd party, and 
-can now complete the login. The OAuth2 ``redirect URI`` (with auth code) MUST respond 
-with either new credentials, the next stage of the login process, or a standard error 
-response.
-    
-For example, if a home server accepts OAuth2 from Google, it would return the 
-Authorization Request URI for Google::
-
-  {
-    "uri": "https://accounts.google.com/o/oauth2/auth?response_type=code&
-    client_id=CLIENT_ID&redirect_uri=REDIRECT_URI&scope=photos"
-  }
-
-The client then visits this URI and authorizes the home server. The client then
-visits the REDIRECT_URI with the auth code= query parameter which returns::
-
-  {
-    "user_id": "@user:matrix.org",
-    "access_token": "0123456789abcdef"
-  }
-
-Email-based (code)
-------------------
-:Type: 
-  m.login.email.code
-:Description:
-  Login is supported by typing in a code which is sent in an email. This login 
-  consists of multiple requests.
-
-To respond to this type, reply with::
-
-  {
-    "type": "m.login.email.code",
-    "user": "<user_id or user localpart>",
-    "email": "<email address>"
-  }
-
-After validating the email address, the home server MUST send an email containing
-an authentication code and return::
-
-  {
-    "type": "m.login.email.code",
-    "session": "<session id>"
-  }
-
-The second request in this login stage involves sending this authentication code::
-
-  {
-    "type": "m.login.email.code",
-    "session": "<session id>",
-    "code": "<code in email sent>"
-  }
-
-The home server MUST respond to this with either new credentials, the next stage of 
-the login process, or a standard error response.
-
-Email-based (url)
------------------
-:Type: 
-  m.login.email.url
-:Description:
-  Login is supported by clicking on a URL in an email. This login consists of 
-  multiple requests.
-
-To respond to this type, reply with::
-
-  {
-    "type": "m.login.email.url",
-    "user": "<user_id or user localpart>",
-    "email": "<email address>"
-  }
-
-After validating the email address, the home server MUST send an email containing
-an authentication URL and return::
-
-  {
-    "type": "m.login.email.url",
-    "session": "<session id>"
-  }
-
-The email contains a URL which must be clicked. After it has been clicked, the
-client should perform another request::
-
-  {
-    "type": "m.login.email.url",
-    "session": "<session id>"
-  }
-
-The home server MUST respond to this with either new credentials, the next stage of 
-the login process, or a standard error response. 
-
-A common client implementation will be to periodically poll until the link is clicked.
-If the link has not been visited yet, a standard error response with an errcode of 
-``M_LOGIN_EMAIL_URL_NOT_YET`` should be returned.
-
-
-N-Factor Authentication
------------------------
-Multiple login stages can be combined to create N-factor authentication during login.
-
-This can be achieved by responding with the ``next`` login type on completion of a 
-previous login stage::
-
-  {
-    "next": "<next login type>"
-  }
-
-If a home server implements N-factor authentication, it MUST respond with all 
-``stages`` when initially queried for their login requirements::
-
-  {
-    "type": "<1st login type>",
-    "stages": [ <1st login type>, <2nd login type>, ... , <Nth login type> ]
-  }
-
-This can be represented conceptually as::
-
-   _______________________
-  |    Login Stage 1      |
-  | type: "<login type1>" |
-  |  ___________________  |
-  | |_Request_1_________| | <-- Returns "session" key which is used throughout.
-  |  ___________________  |     
-  | |_Request_2_________| | <-- Returns a "next" value of "login type2"
-  |_______________________|
-            |
-            |
-   _________V_____________
-  |    Login Stage 2      |
-  | type: "<login type2>" |
-  |  ___________________  |
-  | |_Request_1_________| |
-  |  ___________________  |
-  | |_Request_2_________| |
-  |  ___________________  |
-  | |_Request_3_________| | <-- Returns a "next" value of "login type3"
-  |_______________________|
-            |
-            |
-   _________V_____________
-  |    Login Stage 3      |
-  | type: "<login type3>" |
-  |  ___________________  |
-  | |_Request_1_________| | <-- Returns user credentials
-  |_______________________|
-
-Fallback
---------
-Clients cannot be expected to be able to know how to process every single
-login type. If a client determines it does not know how to handle a given
-login type, it should request a login fallback page::
-
-  GET matrix/client/api/v1/login/fallback
-
-This MUST return an HTML page which can perform the entire login process.
-
-Identity
-========
-
-TODO : Dave
-- 3PIDs and identity server, functions
-
-Federation
-==========
-
-Federation is the term used to describe how to communicate between Matrix home 
-servers. Federation is a mechanism by which two home servers can exchange
-Matrix event messages, both as a real-time push of current events, and as a
-historic fetching mechanism to synchronise past history for clients to view. It
-uses HTTP connections between each pair of servers involved as the underlying
-transport. Messages are exchanged between servers in real-time by active pushing
-from each server's HTTP client into the server of the other. Queries to fetch
-historic data for the purpose of back-filling scrollback buffers and the like
-can also be performed.
-
-There are three main kinds of communication that occur between home servers:
-
-:Queries:
-   These are single request/response interactions between a given pair of
-   servers, initiated by one side sending an HTTP GET request to obtain some
-   information, and responded by the other. They are not persisted and contain
-   no long-term significant history. They simply request a snapshot state at the
-   instant the query is made.
-
-:Ephemeral Data Units (EDUs):
-   These are notifications of events that are pushed from one home server to
-   another. They are not persisted and contain no long-term significant history,
-   nor does the receiving home server have to reply to them.
-
-:Persisted Data Units (PDUs):
-   These are notifications of events that are broadcast from one home server to
-   any others that are interested in the same "context" (namely, a Room ID).
-   They are persisted to long-term storage and form the record of history for
-   that context.
-
-EDUs and PDUs are further wrapped in an envelope called a Transaction, which is 
-transferred from the origin to the destination home server using an HTTP PUT request.
-
-
-Transactions
-------------
-The transfer of EDUs and PDUs between home servers is performed by an exchange
-of Transaction messages, which are encoded as JSON objects, passed over an 
-HTTP PUT request. A Transaction is meaningful only to the pair of home servers that 
-exchanged it; they are not globally-meaningful.
-
-Each transaction has:
- - An opaque transaction ID.
- - A timestamp (UNIX epoch time in milliseconds) generated by its origin server.
- - An origin and destination server name.
- - A list of "previous IDs".
- - A list of PDUs and EDUs - the actual message payload that the Transaction carries.
-
-::
-
- {
-  "transaction_id":"916d630ea616342b42e98a3be0b74113",
-  "ts":1404835423000,
-  "origin":"red",
-  "destination":"blue",
-  "prev_ids":["e1da392e61898be4d2009b9fecce5325"],
-  "pdus":[...],
-  "edus":[...]
- }
-
-The ``prev_ids`` field contains a list of previous transaction IDs that
-the ``origin`` server has sent to this ``destination``. Its purpose is to act as a
-sequence checking mechanism - the destination server can check whether it has
-successfully received that Transaction, or ask for a retransmission if not.
-
-The ``pdus`` field of a transaction is a list, containing zero or more PDUs.[*]
-Each PDU is itself a JSON object containing a number of keys, the exact details of
-which will vary depending on the type of PDU. Similarly, the ``edus`` field is
-another list containing the EDUs. This key may be entirely absent if there are
-no EDUs to transfer.
-
-(* Normally the PDU list will be non-empty, but the server should cope with
-receiving an "empty" transaction, as this is useful for informing peers of other
-transaction IDs they should be aware of. This effectively acts as a push
-mechanism to encourage peers to continue to replicate content.)
-
-PDUs and EDUs
--------------
-
-All PDUs have:
- - An ID
- - A context
- - A declaration of their type
- - A list of other PDU IDs that have been seen recently on that context (regardless of which origin
-   sent them)
-
-[[TODO(paul): Update this structure so that 'pdu_id' is a two-element
-[origin,ref] pair like the prev_pdus are]]
-
-::
-
- {
-  "pdu_id":"a4ecee13e2accdadf56c1025af232176",
-  "context":"#example.green",
-  "origin":"green",
-  "ts":1404838188000,
-  "pdu_type":"m.text",
-  "prev_pdus":[["blue","99d16afbc857975916f1d73e49e52b65"]],
-  "content":...
-  "is_state":false
- }
-
-In contrast to Transactions, it is important to note that the ``prev_pdus``
-field of a PDU refers to PDUs that any origin server has sent, rather than
-previous IDs that this ``origin`` has sent. This list may refer to other PDUs sent
-by the same origin as the current one, or other origins.
-
-Because of the distributed nature of participants in a Matrix conversation, it
-is impossible to establish a globally-consistent total ordering on the events.
-However, by annotating each outbound PDU at its origin with IDs of other PDUs it
-has received, a partial ordering can be constructed allowing causallity
-relationships to be preserved. A client can then display these messages to the
-end-user in some order consistent with their content and ensure that no message
-that is semantically in reply of an earlier one is ever displayed before it.
-
-PDUs fall into two main categories: those that deliver Events, and those that
-synchronise State. For PDUs that relate to State synchronisation, additional
-keys exist to support this:
-
-::
-
- {...,
-  "is_state":true,
-  "state_key":TODO
-  "power_level":TODO
-  "prev_state_id":TODO
-  "prev_state_origin":TODO}
-
-[[TODO(paul): At this point we should probably have a long description of how
-State management works, with descriptions of clobbering rules, power levels, etc
-etc... But some of that detail is rather up-in-the-air, on the whiteboard, and
-so on. This part needs refining. And writing in its own document as the details
-relate to the server/system as a whole, not specifically to server-server
-federation.]]
-
-EDUs, by comparison to PDUs, do not have an ID, a context, or a list of
-"previous" IDs. The only mandatory fields for these are the type, origin and
-destination home server names, and the actual nested content.
-
-::
-
- {"edu_type":"m.presence",
-  "origin":"blue",
-  "destination":"orange",
-  "content":...}
-
-Backfilling
------------
-- What it is, when is it used, how is it done
-
-SRV Records
------------
-- Why it is needed
-
-Security
-========
-- rate limiting
-- crypto (s-s auth)
-- E2E
-- Lawful intercept + Key Escrow
-
-TODO Mark
-
-Policy Servers
-==============
-TODO
-
-Content repository
-==================
-- thumbnail paths
-
-Address book repository
-=======================
-- format
-
-
-Glossary
-========
-- domain specific words/acronyms with definitions
-
-User ID:
-  An opaque ID which identifies an end-user, which consists of some opaque 
-  localpart combined with the domain name of their home server. 

docs/sphinx/README.rst

@@ -0,0 +1 @@
+TODO: how (if at all) is this actually maintained?

docs/turn-howto.rst

@@ -0,0 +1,93 @@
+How to enable VoIP relaying on your Home Server with TURN
+
+Overview
+--------
+The synapse Matrix Home Server supports integration with TURN server via the
+TURN server REST API
+(http://tools.ietf.org/html/draft-uberti-behave-turn-rest-00). This allows
+the Home Server to generate credentials that are valid for use on the TURN
+server through the use of a secret shared between the Home Server and the
+TURN server.
+
+This document described how to install coturn
+(https://code.google.com/p/coturn/) which also supports the TURN REST API,
+and integrate it with synapse.
+
+coturn Setup
+============
+
+ 1. Check out coturn::
+      svn checkout http://coturn.googlecode.com/svn/trunk/ coturn
+      cd coturn
+
+ 2. Configure it::
+      ./configure
+
+    You may need to install libevent2: if so, you should do so
+    in the way recommended by your operating system.
+    You can ignore warnings about lack of database support: a
+    database is unnecessary for this purpose.
+
+ 3. Build and install it::
+      make
+      make install
+
+ 4. Make a config file in /etc/turnserver.conf. You can customise
+    a config file from turnserver.conf.default. The relevant
+    lines, with example values, are::
+
+      lt-cred-mech
+      use-auth-secret
+      static-auth-secret=[your secret key here]
+      realm=turn.myserver.org
+
+    See turnserver.conf.default for explanations of the options.
+    One way to generate the static-auth-secret is with pwgen::
+
+       pwgen -s 64 1
+
+ 5. Ensure youe firewall allows traffic into the TURN server on
+    the ports you've configured it to listen on (remember to allow
+    both TCP and UDP if you've enabled both).
+
+ 6. If you've configured coturn to support TLS/DTLS, generate or
+    import your private key and certificate.
+
+ 7. Start the turn server::
+       bin/turnserver -o
+
+
+synapse Setup
+=============
+
+Your home server configuration file needs the following extra keys:
+
+ 1. "turn_uris": This needs to be a yaml list
+    of public-facing URIs for your TURN server to be given out 
+    to your clients. Add separate entries for each transport your
+    TURN server supports.
+
+ 2. "turn_shared_secret": This is the secret shared between your Home
+    server and your TURN server, so you should set it to the same
+    string you used in turnserver.conf.
+
+ 3. "turn_user_lifetime": This is the amount of time credentials
+    generated by your Home Server are valid for (in milliseconds).
+    Shorter times offer less potential for abuse at the expense
+    of increased traffic between web clients and your home server
+    to refresh credentials. The TURN REST API specification recommends
+    one day (86400000).
+
+As an example, here is the relevant section of the config file for
+matrix.org::
+
+    turn_uris: [ "turn:turn.matrix.org:3478?transport=udp", "turn:turn.matrix.org:3478?transport=tcp" ]
+    turn_shared_secret: n0t4ctuAllymatr1Xd0TorgSshar3d5ecret4obvIousreAsons
+    turn_user_lifetime: 86400000
+
+Now, restart synapse::
+
+    cd /where/you/run/synapse
+    ./synctl restart
+
+...and your Home Server now supports VoIP relaying!

jsfiddles/create_room_send_msg/demo.css

@@ -1,17 +0,0 @@
-.loggedin {
-    visibility: hidden;
-}
-
-p {
-    font-family: monospace;
-}
-
-table
-{
-    border-spacing:5px;
-}
-
-th,td
-{
-    padding:5px;
-}

jsfiddles/create_room_send_msg/demo.html

@@ -1,30 +0,0 @@
-<div>
-    <p>This room creation / message sending demo requires a home server to be running on http://localhost:8080</p>
-</div>
-<form class="loginForm">
-    <input type="text" id="userLogin" placeholder="Username"></input>
-    <input type="password" id="passwordLogin" placeholder="Password"></input>
-    <input type="button" class="login" value="Login"></input>
-</form>
-<div class="loggedin">
-    <form class="createRoomForm">
-        <input type="text" id="roomAlias" placeholder="Room alias (optional)"></input>
-        <input type="button" class="createRoom" value="Create Room"></input>
-    </form>
-    <form class="sendMessageForm">
-        <input type="text" id="roomId" placeholder="Room ID"></input>
-        <input type="text" id="messageBody" placeholder="Message body"></input>
-        <input type="button" class="sendMessage" value="Send Message"></input>
-    </form>
-    <table id="rooms">
-        <tbody>
-            <tr>
-                <th>Room ID</th>
-                <th>My state</th>
-                <th>Room Alias</th>
-                <th>Latest message</th>
-            </tr>
-        </tbody>
-    </table>
-</div>
-    

jsfiddles/create_room_send_msg/demo.js

@@ -1,109 +0,0 @@
-var accountInfo = {};
-
-var showLoggedIn = function(data) {
-    accountInfo = data;
-    getCurrentRoomList();
-    $(".loggedin").css({visibility: "visible"});
-};
-
-$('.login').live('click', function() {
-    var user = $("#userLogin").val();
-    var password = $("#passwordLogin").val();
-    $.ajax({
-        url: "http://localhost:8080/matrix/client/api/v1/login",
-        type: "POST",
-        contentType: "application/json; charset=utf-8",
-        data: JSON.stringify({ user: user, password: password, type: "m.login.password" }),
-        dataType: "json",
-        success: function(data) {
-            showLoggedIn(data);
-        },
-        error: function(err) {
-            alert(JSON.stringify($.parseJSON(err.responseText)));  
-        }
-    }); 
-});
-
-var getCurrentRoomList = function() {
-    var url = "http://localhost:8080/matrix/client/api/v1/im/sync?access_token=" + accountInfo.access_token + "&from=END&to=START&limit=1";
-    $.getJSON(url, function(data) {
-        for (var i=0; i<data.length; ++i) {
-            data[i].latest_message = data[i].messages.chunk[0].content.body;
-            addRoom(data[i]);   
-        }
-    }).fail(function(err) {
-        alert(JSON.stringify($.parseJSON(err.responseText)));
-    });
-};
-
-$('.createRoom').live('click', function() {
-    var roomAlias = $("#roomAlias").val();
-    var data = {};
-    if (roomAlias.length > 0) {
-        data.room_alias_name = roomAlias;   
-    }
-    $.ajax({
-        url: "http://localhost:8080/matrix/client/api/v1/rooms?access_token="+accountInfo.access_token,
-        type: "POST",
-        contentType: "application/json; charset=utf-8",
-        data: JSON.stringify(data),
-        dataType: "json",
-        success: function(data) {
-            data.membership = "join"; // you are automatically joined into every room you make.
-            data.latest_message = "";
-            addRoom(data);
-        },
-        error: function(err) {
-            alert(JSON.stringify($.parseJSON(err.responseText)));  
-        }
-    }); 
-});
-
-var addRoom = function(data) {
-    row = "<tr>" +
-        "<td>"+data.room_id+"</td>" +
-        "<td>"+data.membership+"</td>" +
-        "<td>"+data.room_alias+"</td>" +
-        "<td>"+data.latest_message+"</td>" +
-        "</tr>";
-    $("#rooms").append(row);
-};
-
-$('.sendMessage').live('click', function() {
-    var roomId = $("#roomId").val();
-    var body = $("#messageBody").val();
-    var msgId = $.now();
-    
-    if (roomId.length === 0 || body.length === 0) {
-        return;
-    }
-    
-    var url = "http://localhost:8080/matrix/client/api/v1/rooms/$roomid/messages/$user/$msgid?access_token=$token";
-    url = url.replace("$token", accountInfo.access_token);
-    url = url.replace("$roomid", encodeURIComponent(roomId));
-    url = url.replace("$user", encodeURIComponent(accountInfo.user_id));
-    url = url.replace("$msgid", msgId);
-    
-    var data = {
-        msgtype: "m.text",
-        body: body
-    };
-    
-    $.ajax({
-        url: url,
-        type: "PUT",
-        contentType: "application/json; charset=utf-8",
-        data: JSON.stringify(data),
-        dataType: "json",
-        success: function(data) {
-            $("#messageBody").val("");
-            // wipe the table and reload it. Using the event stream would be the best
-            // solution but that is out of scope of this fiddle.
-            $("#rooms").find("tr:gt(0)").remove();
-            getCurrentRoomList();
-        },
-        error: function(err) {
-            alert(JSON.stringify($.parseJSON(err.responseText)));  
-        }
-    }); 
-});

jsfiddles/event_stream/demo.css

@@ -1,17 +0,0 @@
-.loggedin {
-    visibility: hidden;
-}
-
-p {
-    font-family: monospace;
-}
-
-table
-{
-    border-spacing:5px;
-}
-
-th,td
-{
-    padding:5px;
-}

jsfiddles/event_stream/demo.html

@@ -1,23 +0,0 @@
-<div>
-    <p>This event stream demo requires a home server to be running on http://localhost:8080</p>
-</div>
-<form class="loginForm">
-    <input type="text" id="userLogin" placeholder="Username"></input>
-    <input type="password" id="passwordLogin" placeholder="Password"></input>
-    <input type="button" class="login" value="Login"></input>
-</form>
-<div class="loggedin">
-    <form class="sendMessageForm">
-        <input type="button" class="sendMessage" value="Send random message"></input>
-    </form>
-    <p id="streamErrorText"></p>
-    <table id="rooms">
-        <tbody>
-            <tr>
-                <th>Room ID</th>
-                <th>Latest message</th>
-            </tr>
-        </tbody>
-    </table>
-</div>
-    

jsfiddles/event_stream/demo.js

@@ -1,142 +0,0 @@
-var accountInfo = {};
-
-var eventStreamInfo = {
-    from: "END"
-};
-
-var roomInfo = [];
-
-var longpollEventStream = function() {
-    var url = "http://localhost:8080/matrix/client/api/v1/events?access_token=$token&from=$from";
-    url = url.replace("$token", accountInfo.access_token);
-    url = url.replace("$from", eventStreamInfo.from);
-    
-    $.getJSON(url, function(data) {
-        eventStreamInfo.from = data.end;
-        
-        var hasNewLatestMessage = false;
-        for (var i=0; i<data.chunk.length; ++i) {
-            if (data.chunk[i].type === "m.room.message") {
-                for (var j=0; j<roomInfo.length; ++j) {
-                    if (roomInfo[j].room_id === data.chunk[i].room_id) {
-                        roomInfo[j].latest_message = data.chunk[i].content.body;
-                        hasNewLatestMessage = true;
-                    }
-                }
-            }
-        }
-        
-        if (hasNewLatestMessage) {
-           setRooms(roomInfo);
-        }
-        $("#streamErrorText").text("");
-        longpollEventStream();
-    }).fail(function(err) {
-        $("#streamErrorText").text("Event stream error: "+JSON.stringify($.parseJSON(err.responseText)));
-        setTimeout(longpollEventStream, 5000);
-    });
-};
-
-var showLoggedIn = function(data) {
-    accountInfo = data;
-    longpollEventStream();
-    getCurrentRoomList();
-    $(".loggedin").css({visibility: "visible"});
-};
-
-$('.login').live('click', function() {
-    var user = $("#userLogin").val();
-    var password = $("#passwordLogin").val();
-    $.ajax({
-        url: "http://localhost:8080/matrix/client/api/v1/login",
-        type: "POST",
-        contentType: "application/json; charset=utf-8",
-        data: JSON.stringify({ user: user, password: password, type: "m.login.password" }),
-        dataType: "json",
-        success: function(data) {
-            $("#rooms").find("tr:gt(0)").remove();
-            showLoggedIn(data);
-        },
-        error: function(err) {
-            alert(JSON.stringify($.parseJSON(err.responseText)));  
-        }
-    }); 
-});
-
-var getCurrentRoomList = function() {
-    $("#roomId").val("");
-    var url = "http://localhost:8080/matrix/client/api/v1/im/sync?access_token=" + accountInfo.access_token + "&from=END&to=START&limit=1";
-    $.getJSON(url, function(data) {
-        for (var i=0; i<data.length; ++i) {
-            if ("messages" in data[i]) {
-                data[i].latest_message = data[i].messages.chunk[0].content.body;   
-            }
-        }
-        roomInfo = data;
-        setRooms(roomInfo);  
-    }).fail(function(err) {
-        alert(JSON.stringify($.parseJSON(err.responseText)));
-    });
-};
-
-$('.sendMessage').live('click', function() {
-    if (roomInfo.length === 0) {
-        alert("There is no room to send a message to!");
-        return;
-    }
-    
-    var index = Math.floor(Math.random() * roomInfo.length);
-    
-    sendMessage(roomInfo[index].room_id);
-});
-
-var sendMessage = function(roomId) {
-    var body = "jsfiddle message @" + $.now();
-    var msgId = $.now();
-    
-    if (roomId.length === 0) {
-        return;
-    }
-    
-    var url = "http://localhost:8080/matrix/client/api/v1/rooms/$roomid/messages/$user/$msgid?access_token=$token";
-    url = url.replace("$token", accountInfo.access_token);
-    url = url.replace("$roomid", encodeURIComponent(roomId));
-    url = url.replace("$user", encodeURIComponent(accountInfo.user_id));
-    url = url.replace("$msgid", msgId);
-    
-    var data = {
-        msgtype: "m.text",
-        body: body
-    };
-    
-    $.ajax({
-        url: url,
-        type: "PUT",
-        contentType: "application/json; charset=utf-8",
-        data: JSON.stringify(data),
-        dataType: "json",
-        success: function(data) {
-            $("#messageBody").val("");
-        },
-        error: function(err) {
-            alert(JSON.stringify($.parseJSON(err.responseText)));  
-        }
-    }); 
-};
-
-var setRooms = function(roomList) {
-    // wipe existing entries
-    $("#rooms").find("tr:gt(0)").remove();
-    
-    var rows = "";
-    for (var i=0; i<roomList.length; ++i) {
-        row = "<tr>" +
-            "<td>"+roomList[i].room_id+"</td>" +
-            "<td>"+roomList[i].latest_message+"</td>" +
-            "</tr>";  
-        rows += row;
-    }
-    
-    $("#rooms").append(rows);
-};
-

jsfiddles/example_app/demo.css

@@ -1,43 +0,0 @@
-.roomListDashboard, .roomContents, .sendMessageForm {
-    visibility: hidden;
-}
-
-.roomList {
-    background-color: #909090;
-}
-
-.messageWrapper {
-    background-color: #EEEEEE;
-    height: 400px;
-    overflow: scroll;
-}
-
-.membersWrapper {
-    background-color: #EEEEEE;
-    height: 200px;
-    width: 50%;
-    overflow: scroll;
-}
-
-.textEntry {
-    width: 100%
-}
-
-p {
-    font-family: monospace;
-}
-
-table
-{
-    border-spacing:5px;
-}
-
-th,td
-{
-    padding:5px;
-}
-
-.roomList tr:not(:first-child):hover {
-    background-color: orange;
-    cursor: pointer;
-}

jsfiddles/example_app/demo.html

@@ -1,56 +0,0 @@
-<div class="signUp">
-    <p>Matrix example application: Requires a local home server running at http://localhost:8080</p>
-    <form class="registrationForm">
-        <p>No account? Register:</p>
-        <input type="text" id="userReg" placeholder="Username"></input>
-        <input type="password" id="passwordReg" placeholder="Password"></input>
-        <input type="button" class="register" value="Register"></input>
-    </form>
-    <form class="loginForm">
-        <p>Got an account? Login:</p>
-        <input type="text" id="userLogin" placeholder="Username"></input>
-        <input type="password" id="passwordLogin" placeholder="Password"></input>
-        <input type="button" class="login" value="Login"></input>
-    </form>
-</div>
-
-<div class="roomListDashboard">
-    <form class="createRoomForm">
-        <input type="text" id="roomAlias" placeholder="Room alias"></input>
-        <input type="button" class="createRoom" value="Create Room"></input>
-    </form>
-    <table id="rooms" class="roomList">
-        <tbody>
-            <tr>
-                <th>Room</th>
-                <th>My state</th>
-                <th>Latest message</th>
-            </tr>
-        </tbody>
-    </table>
-</div>
-
-<div class="roomContents">
-    <p id="roomName">Select a room</p>
-    <div class="messageWrapper">
-        <table id="messages">
-            <tbody>
-            </tbody>
-        </table>
-    </div>
-    <form class="sendMessageForm">
-        <input type="text" class="textEntry" id="body" placeholder="Enter text here..." onkeydown="javascript:if (event.keyCode == 13) document.getElementById('sendMsg').focus()"></input>
-        <input type="button" class="sendMessage" id="sendMsg" value="Send"></input>
-    </form>
-</div>
-
-<div>
-    <p>Member list:</p>
-    <div class="membersWrapper">
-        <table id="members">
-            <tbody>
-            </tbody>
-        </table>
-    </div>
-</div>
-    

jsfiddles/example_app/demo.js

@@ -1,303 +0,0 @@
-var accountInfo = {};
-
-var eventStreamInfo = {
-    from: "END"
-};
-
-var roomInfo = [];
-var memberInfo = [];
-var viewingRoomId;
-
-// ************** Event Streaming **************
-var longpollEventStream = function() {
-    var url = "http://localhost:8080/matrix/client/api/v1/events?access_token=$token&from=$from";
-    url = url.replace("$token", accountInfo.access_token);
-    url = url.replace("$from", eventStreamInfo.from);
-
-    $.getJSON(url, function(data) {
-        eventStreamInfo.from = data.end;
-        
-        var hasNewLatestMessage = false;
-        var updatedMemberList = false;
-        var i=0;
-        var j=0;
-        for (i=0; i<data.chunk.length; ++i) {
-            if (data.chunk[i].type === "m.room.message") {
-                console.log("Got new message: " + JSON.stringify(data.chunk[i]));
-                if (viewingRoomId === data.chunk[i].room_id) {
-                    addMessage(data.chunk[i]);
-                }
-                
-                for (j=0; j<roomInfo.length; ++j) {
-                    if (roomInfo[j].room_id === data.chunk[i].room_id) {
-                        roomInfo[j].latest_message = data.chunk[i].content.body;
-                        hasNewLatestMessage = true;
-                    }
-                }
-            }
-            else if (data.chunk[i].type === "m.room.member") {
-                if (viewingRoomId === data.chunk[i].room_id) {
-                    console.log("Got new member: " + JSON.stringify(data.chunk[i]));
-                    for (j=0; j<memberInfo.length; ++j) {
-                        if (memberInfo[j].target_user_id === data.chunk[i].target_user_id) {
-                            memberInfo[j] = data.chunk[i];
-                            updatedMemberList = true;
-                            break;
-                        }
-                    }
-                    if (!updatedMemberList) {
-                        memberInfo.push(data.chunk[i]);  
-                        updatedMemberList = true;
-                    }
-                }
-                if (data.chunk[i].target_user_id === accountInfo.user_id) {
-                    getCurrentRoomList(); // update our join/invite list
-                }
-            }
-            else {
-                console.log("Discarding: " + JSON.stringify(data.chunk[i]));
-            }
-        }
-        
-        if (hasNewLatestMessage) {
-           setRooms(roomInfo);
-        }
-        if (updatedMemberList) {
-            $("#members").empty();
-            for (i=0; i<memberInfo.length; ++i) { 
-                addMember(memberInfo[i]);
-            }
-        }
-        longpollEventStream();
-    }).fail(function(err) {
-        setTimeout(longpollEventStream, 5000);
-    });
-};
-
-// ************** Registration and Login **************
-var onLoggedIn = function(data) {
-    accountInfo = data;
-    longpollEventStream();
-    getCurrentRoomList();
-    $(".roomListDashboard").css({visibility: "visible"});
-    $(".roomContents").css({visibility: "visible"});
-    $(".signUp").css({display: "none"});
-};
-
-$('.login').live('click', function() {
-    var user = $("#userLogin").val();
-    var password = $("#passwordLogin").val();
-    $.ajax({
-        url: "http://localhost:8080/matrix/client/api/v1/login",
-        type: "POST",
-        contentType: "application/json; charset=utf-8",
-        data: JSON.stringify({ user: user, password: password, type: "m.login.password" }),
-        dataType: "json",
-        success: function(data) {
-            onLoggedIn(data);
-        },
-        error: function(err) {
-            alert("Unable to login: is the home server running?");  
-        }
-    }); 
-});
-
-$('.register').live('click', function() {
-    var user = $("#userReg").val();
-    var password = $("#passwordReg").val();
-    $.ajax({
-        url: "http://localhost:8080/matrix/client/api/v1/register",
-        type: "POST",
-        contentType: "application/json; charset=utf-8",
-        data: JSON.stringify({ user_id: user, password: password }),
-        dataType: "json",
-        success: function(data) {
-            onLoggedIn(data);
-        },
-        error: function(err) {
-            var msg = "Is the home server running?";
-            var errJson = $.parseJSON(err.responseText);
-            if (errJson !== null) {
-                msg = errJson.error;   
-            }
-            alert("Unable to register: "+msg);  
-        }
-    });
-});
-
-// ************** Creating a room ******************
-$('.createRoom').live('click', function() {
-    var roomAlias = $("#roomAlias").val();
-    var data = {};
-    if (roomAlias.length > 0) {
-        data.room_alias_name = roomAlias;   
-    }
-    $.ajax({
-        url: "http://localhost:8080/matrix/client/api/v1/rooms?access_token="+accountInfo.access_token,
-        type: "POST",
-        contentType: "application/json; charset=utf-8",
-        data: JSON.stringify(data),
-        dataType: "json",
-        success: function(response) {
-            $("#roomAlias").val("");
-            response.membership = "join"; // you are automatically joined into every room you make.
-            response.latest_message = "";
-            
-            roomInfo.push(response);
-            setRooms(roomInfo);
-        },
-        error: function(err) {
-            alert(JSON.stringify($.parseJSON(err.responseText)));  
-        }
-    }); 
-});
-
-// ************** Getting current state **************
-var getCurrentRoomList = function() {
-    var url = "http://localhost:8080/matrix/client/api/v1/im/sync?access_token=" + accountInfo.access_token + "&from=END&to=START&limit=1";
-    $.getJSON(url, function(data) {
-        for (var i=0; i<data.length; ++i) {
-            if ("messages" in data[i]) {
-                data[i].latest_message = data[i].messages.chunk[0].content.body;   
-            }
-        }
-        roomInfo = data;
-        setRooms(roomInfo);  
-    }).fail(function(err) {
-        alert(JSON.stringify($.parseJSON(err.responseText)));
-    });
-};
-
-var loadRoomContent = function(roomId) {
-    console.log("loadRoomContent " + roomId);
-    viewingRoomId = roomId;
-    $("#roomName").text("Room: "+roomId);
-    $(".sendMessageForm").css({visibility: "visible"});
-    getMessages(roomId);
-    getMemberList(roomId);
-};
-
-var getMessages = function(roomId) {
-    $("#messages").empty();
-    var url = "http://localhost:8080/matrix/client/api/v1/rooms/" + roomId + "/messages/list?access_token=" + accountInfo.access_token + "&from=END&to=START&limit=10";
-    $.getJSON(url, function(data) {
-        for (var i=data.chunk.length-1; i>=0; --i) {
-            addMessage(data.chunk[i]);   
-        }
-    });
-};
-
-var getMemberList = function(roomId) {
-    $("#members").empty();
-    memberInfo = [];
-    var url = "http://localhost:8080/matrix/client/api/v1/rooms/" + roomId + "/members/list?access_token=" + accountInfo.access_token;
-    $.getJSON(url, function(data) {
-        for (var i=0; i<data.chunk.length; ++i) {
-            memberInfo.push(data.chunk[i]);
-            addMember(data.chunk[i]);   
-        }
-    });
-};
-
-// ************** Sending messages **************
-$('.sendMessage').live('click', function() {
-    if (viewingRoomId === undefined) {
-        alert("There is no room to send a message to!");
-        return;
-    }
-    var body = $("#body").val();
-    sendMessage(viewingRoomId, body);
-});
-
-var sendMessage = function(roomId, body) {
-    var msgId = $.now();
-    
-    var url = "http://localhost:8080/matrix/client/api/v1/rooms/$roomid/messages/$user/$msgid?access_token=$token";
-    url = url.replace("$token", accountInfo.access_token);
-    url = url.replace("$roomid", encodeURIComponent(roomId));
-    url = url.replace("$user", encodeURIComponent(accountInfo.user_id));
-    url = url.replace("$msgid", msgId);
-    
-    var data = {
-        msgtype: "m.text",
-        body: body
-    };
-    
-    $.ajax({
-        url: url,
-        type: "PUT",
-        contentType: "application/json; charset=utf-8",
-        data: JSON.stringify(data),
-        dataType: "json",
-        success: function(data) {
-            $("#body").val("");
-        },
-        error: function(err) {
-            alert(JSON.stringify($.parseJSON(err.responseText)));  
-        }
-    });
-};
-
-// ************** Navigation and DOM manipulation **************
-var setRooms = function(roomList) {
-    // wipe existing entries
-    $("#rooms").find("tr:gt(0)").remove();
-    
-    var rows = "";
-    for (var i=0; i<roomList.length; ++i) {
-        row = "<tr>" +
-              "<td>"+roomList[i].room_id+"</td>" +
-              "<td>"+roomList[i].membership+"</td>" +
-              "<td>"+roomList[i].latest_message+"</td>" +
-              "</tr>";  
-        rows += row;
-    }
-    
-    $("#rooms").append(rows);
-    
-    $('#rooms').find("tr").click(function(){
-        var roomId = $(this).find('td:eq(0)').text();
-        var membership = $(this).find('td:eq(1)').text();
-        if (membership !== "join") {
-            console.log("Joining room " + roomId); 
-            var url = "http://localhost:8080/matrix/client/api/v1/rooms/$roomid/members/$user/state?access_token=$token";
-            url = url.replace("$token", accountInfo.access_token);
-            url = url.replace("$roomid", encodeURIComponent(roomId));
-            url = url.replace("$user", encodeURIComponent(accountInfo.user_id));
-            $.ajax({
-                url: url,
-                type: "PUT",
-                contentType: "application/json; charset=utf-8",
-                data: JSON.stringify({membership: "join"}),
-                dataType: "json",
-                success: function(data) {
-                    loadRoomContent(roomId);
-                    getCurrentRoomList();
-                },
-                error: function(err) {
-                    alert(JSON.stringify($.parseJSON(err.responseText)));  
-                }
-            });
-        }
-        else {
-            loadRoomContent(roomId);
-        }
-    });
-};
-
-var addMessage = function(data) {
-    var row = "<tr>" +
-              "<td>"+data.user_id+"</td>" +
-              "<td>"+data.content.body+"</td>" +
-              "</tr>"; 
-    $("#messages").append(row);
-};
-
-var addMember = function(data) {
-    var row = "<tr>" +
-              "<td>"+data.target_user_id+"</td>" +
-              "<td>"+data.content.membership+"</td>" +
-              "</tr>"; 
-    $("#members").append(row);
-};
-

jsfiddles/register_login/demo.css

@@ -1,7 +0,0 @@
-.loggedin {
-    visibility: hidden;
-}
-
-p {
-    font-family: monospace;
-}

jsfiddles/register_login/demo.html

@@ -1,20 +0,0 @@
-<div>
-    <p>This registration/login demo requires a home server to be running on http://localhost:8080</p>
-</div>
-<form class="registrationForm">
-    <input type="text" id="user" placeholder="Username"></input>
-    <input type="password" id="password" placeholder="Password"></input>
-    <input type="button" class="register" value="Register"></input>
-</form>
-<form class="loginForm">
-    <input type="text" id="userLogin" placeholder="Username"></input>
-    <input type="password" id="passwordLogin" placeholder="Password"></input>
-    <input type="button" class="login" value="Login"></input>
-</form>
-<div class="loggedin">
-    <p id="welcomeText"></p>
-    <input type="button" class="testToken" value="Test token"></input>
-    <input type="button" class="logout" value="Logout"></input>
-    <p id="imSyncText"></p>
-</div>
-    

jsfiddles/register_login/demo.js

@@ -1,69 +0,0 @@
-var accountInfo = {};
-
-var showLoggedIn = function(data) {
-    accountInfo = data;
-    $(".loggedin").css({visibility: "visible"});
-    $("#welcomeText").text("Welcome " + accountInfo.user_id+". Your access token is:  " +
-                           accountInfo.access_token);    
-};
-
-$('.register').live('click', function() {
-    var user = $("#user").val();
-    var password = $("#password").val();
-    $.ajax({
-        url: "http://localhost:8080/matrix/client/api/v1/register",
-        type: "POST",
-        contentType: "application/json; charset=utf-8",
-        data: JSON.stringify({ user_id: user, password: password }),
-        dataType: "json",
-        success: function(data) {
-            showLoggedIn(data);
-        },
-        error: function(err) {
-            alert(JSON.stringify($.parseJSON(err.responseText)));  
-        }
-    });
-});
-
-var login = function(user, password) {
-    $.ajax({
-        url: "http://localhost:8080/matrix/client/api/v1/login",
-        type: "POST",
-        contentType: "application/json; charset=utf-8",
-        data: JSON.stringify({ user: user, password: password, type: "m.login.password" }),
-        dataType: "json",
-        success: function(data) {
-            showLoggedIn(data);
-        },
-        error: function(err) {
-            alert(JSON.stringify($.parseJSON(err.responseText)));  
-        }
-    });  
-};
-
-$('.login').live('click', function() {
-    var user = $("#userLogin").val();
-    var password = $("#passwordLogin").val();
-    $.getJSON("http://localhost:8080/matrix/client/api/v1/login", function(data) {
-        if (data.type !== "m.login.password") {
-            alert("I don't know how to login with this type: " + data.type);
-            return;
-        }
-        login(user, password);
-    });
-});
-
-$('.logout').live('click', function() {
-    accountInfo = {};
-    $("#imSyncText").text("");
-    $(".loggedin").css({visibility: "hidden"});
-});
-
-$('.testToken').live('click', function() {
-    var url = "http://localhost:8080/matrix/client/api/v1/im/sync?access_token=" + accountInfo.access_token + "&from=END&to=START&limit=1";
-    $.getJSON(url, function(data) {
-         $("#imSyncText").text(JSON.stringify(data, undefined, 2));
-    }).fail(function(err) {
-        $("#imSyncText").text(JSON.stringify($.parseJSON(err.responseText)));
-    });
-});

jsfiddles/room_memberships/demo.css

@@ -1,17 +0,0 @@
-.loggedin {
-    visibility: hidden;
-}
-
-p {
-    font-family: monospace;
-}
-
-table
-{
-    border-spacing:5px;
-}
-
-th,td
-{
-    padding:5px;
-}

jsfiddles/room_memberships/demo.html

@@ -1,37 +0,0 @@
-<div>
-    <p>This room membership demo requires a home server to be running on http://localhost:8080</p>
-</div>
-<form class="loginForm">
-    <input type="text" id="userLogin" placeholder="Username"></input>
-    <input type="password" id="passwordLogin" placeholder="Password"></input>
-    <input type="button" class="login" value="Login"></input>
-</form>
-<div class="loggedin">
-    <form class="createRoomForm">
-        <input type="button" class="createRoom" value="Create Room"></input>
-    </form>
-    <form class="changeMembershipForm">
-        <input type="text" id="roomId" placeholder="Room ID"></input>
-        <input type="text" id="targetUser" placeholder="Target User ID"></input>
-        <select id="membership">
-            <option value="invite">Invite</option>
-            <option value="join">Join</option>
-            <option value="leave">Leave</option>
-        </select>
-        <input type="button" class="changeMembership" value="Change Membership"></input>
-    </form>
-    <form class="joinAliasForm">
-        <input type="text" id="roomAlias" placeholder="Room Alias (#name:domain)"></input>
-        <input type="button" class="joinAlias" value="Join via Alias"></input>
-    </form>
-    <table id="rooms">
-        <tbody>
-            <tr>
-                <th>Room ID</th>
-                <th>My state</th>
-                <th>Room Alias</th>
-            </tr>
-        </tbody>
-    </table>
-</div>
-    

jsfiddles/room_memberships/demo.js

@@ -1,139 +0,0 @@
-var accountInfo = {};
-
-var showLoggedIn = function(data) {
-    accountInfo = data;
-    getCurrentRoomList();
-    $(".loggedin").css({visibility: "visible"});
-};
-
-$('.login').live('click', function() {
-    var user = $("#userLogin").val();
-    var password = $("#passwordLogin").val();
-    $.ajax({
-        url: "http://localhost:8080/matrix/client/api/v1/login",
-        type: "POST",
-        contentType: "application/json; charset=utf-8",
-        data: JSON.stringify({ user: user, password: password, type: "m.login.password" }),
-        dataType: "json",
-        success: function(data) {
-            $("#rooms").find("tr:gt(0)").remove();
-            showLoggedIn(data);
-        },
-        error: function(err) {
-            alert(JSON.stringify($.parseJSON(err.responseText)));  
-        }
-    }); 
-});
-
-var getCurrentRoomList = function() {
-    $("#roomId").val("");
-    // wipe the table and reload it. Using the event stream would be the best
-    // solution but that is out of scope of this fiddle.
-    $("#rooms").find("tr:gt(0)").remove();
-    
-    var url = "http://localhost:8080/matrix/client/api/v1/im/sync?access_token=" + accountInfo.access_token + "&from=END&to=START&limit=1";
-    $.getJSON(url, function(data) {
-        for (var i=0; i<data.length; ++i) {
-            addRoom(data[i]);   
-        }
-    }).fail(function(err) {
-        alert(JSON.stringify($.parseJSON(err.responseText)));
-    });
-};
-
-$('.createRoom').live('click', function() {
-    var data = {};
-    $.ajax({
-        url: "http://localhost:8080/matrix/client/api/v1/rooms?access_token="+accountInfo.access_token,
-        type: "POST",
-        contentType: "application/json; charset=utf-8",
-        data: JSON.stringify(data),
-        dataType: "json",
-        success: function(data) {
-            data.membership = "join"; // you are automatically joined into every room you make.
-            data.latest_message = "";
-            addRoom(data);
-        },
-        error: function(err) {
-            alert(JSON.stringify($.parseJSON(err.responseText)));  
-        }
-    }); 
-});
-
-var addRoom = function(data) {
-    row = "<tr>" +
-        "<td>"+data.room_id+"</td>" +
-        "<td>"+data.membership+"</td>" +
-        "<td>"+data.room_alias+"</td>" +
-        "</tr>";
-    $("#rooms").append(row);
-};
-
-$('.changeMembership').live('click', function() {
-    var roomId = $("#roomId").val();
-    var member = $("#targetUser").val();
-    var membership = $("#membership").val();
-    
-    if (roomId.length === 0) {
-        return;
-    }
-    
-    var url = "http://localhost:8080/matrix/client/api/v1/rooms/$roomid/members/$user/state?access_token=$token";
-    url = url.replace("$token", accountInfo.access_token);
-    url = url.replace("$roomid", encodeURIComponent(roomId));
-    url = url.replace("$user", encodeURIComponent(member));
-    
-    if (membership === "leave") {
-        $.ajax({
-            url: url,
-            type: "DELETE",
-            contentType: "application/json; charset=utf-8",
-            dataType: "json",
-            success: function(data) {
-                getCurrentRoomList();
-            },
-            error: function(err) {
-                alert(JSON.stringify($.parseJSON(err.responseText)));  
-            }
-        });
-    }
-    else {
-        var data = {
-            membership: membership
-        };
-        
-        $.ajax({
-            url: url,
-            type: "PUT",
-            contentType: "application/json; charset=utf-8",
-            data: JSON.stringify(data),
-            dataType: "json",
-            success: function(data) {
-                getCurrentRoomList();
-            },
-            error: function(err) {
-                alert(JSON.stringify($.parseJSON(err.responseText)));  
-            }
-        }); 
-    }
-});
-
-$('.joinAlias').live('click', function() {
-    var roomAlias = $("#roomAlias").val();
-    var url = "http://localhost:8080/matrix/client/api/v1/join/$roomalias?access_token=$token";
-    url = url.replace("$token", accountInfo.access_token);
-    url = url.replace("$roomalias", encodeURIComponent(roomAlias));
-    $.ajax({
-        url: url,
-        type: "PUT",
-        contentType: "application/json; charset=utf-8",
-        data: JSON.stringify({}),
-        dataType: "json",
-        success: function(data) {
-            getCurrentRoomList();
-        },
-        error: function(err) {
-            alert(JSON.stringify($.parseJSON(err.responseText)));  
-        }
-    }); 
-});

pylint.cfg

@@ -0,0 +1,280 @@
+[MASTER]
+
+# Specify a configuration file.
+#rcfile=
+
+# Python code to execute, usually for sys.path manipulation such as
+# pygtk.require().
+#init-hook=
+
+# Profiled execution.
+profile=no
+
+# Add files or directories to the blacklist. They should be base names, not
+# paths.
+ignore=CVS
+
+# Pickle collected data for later comparisons.
+persistent=yes
+
+# List of plugins (as comma separated values of python modules names) to load,
+# usually to register additional checkers.
+load-plugins=
+
+
+[MESSAGES CONTROL]
+
+# Enable the message, report, category or checker with the given id(s). You can
+# either give multiple identifier separated by comma (,) or put this option
+# multiple time. See also the "--disable" option for examples.
+#enable=
+
+# Disable the message, report, category or checker with the given id(s). You
+# can either give multiple identifiers separated by comma (,) or put this
+# option multiple times (only on the command line, not in the configuration
+# file where it should appear only once).You can also use "--disable=all" to
+# disable everything first and then reenable specific checks. For example, if
+# you want to run only the similarities checker, you can use "--disable=all
+# --enable=similarities". If you want to run only the classes checker, but have
+# no Warning level messages displayed, use"--disable=all --enable=classes
+# --disable=W"
+disable=missing-docstring
+
+
+[REPORTS]
+
+# Set the output format. Available formats are text, parseable, colorized, msvs
+# (visual studio) and html. You can also give a reporter class, eg
+# mypackage.mymodule.MyReporterClass.
+output-format=text
+
+# Put messages in a separate file for each module / package specified on the
+# command line instead of printing them on stdout. Reports (if any) will be
+# written in a file name "pylint_global.[txt|html]".
+files-output=no
+
+# Tells whether to display a full report or only the messages
+reports=yes
+
+# Python expression which should return a note less than 10 (10 is the highest
+# note). You have access to the variables errors warning, statement which
+# respectively contain the number of errors / warnings messages and the total
+# number of statements analyzed. This is used by the global evaluation report
+# (RP0004).
+evaluation=10.0 - ((float(5 * error + warning + refactor + convention) / statement) * 10)
+
+# Add a comment according to your evaluation note. This is used by the global
+# evaluation report (RP0004).
+comment=no
+
+# Template used to display messages. This is a python new-style format string
+# used to format the message information. See doc for all details
+#msg-template=
+
+
+[TYPECHECK]
+
+# Tells whether missing members accessed in mixin class should be ignored. A
+# mixin class is detected if its name ends with "mixin" (case insensitive).
+ignore-mixin-members=yes
+
+# List of classes names for which member attributes should not be checked
+# (useful for classes with attributes dynamically set).
+ignored-classes=SQLObject
+
+# When zope mode is activated, add a predefined set of Zope acquired attributes
+# to generated-members.
+zope=no
+
+# List of members which are set dynamically and missed by pylint inference
+# system, and so shouldn't trigger E0201 when accessed. Python regular
+# expressions are accepted.
+generated-members=REQUEST,acl_users,aq_parent
+
+
+[MISCELLANEOUS]
+
+# List of note tags to take in consideration, separated by a comma.
+notes=FIXME,XXX,TODO
+
+
+[SIMILARITIES]
+
+# Minimum lines number of a similarity.
+min-similarity-lines=4
+
+# Ignore comments when computing similarities.
+ignore-comments=yes
+
+# Ignore docstrings when computing similarities.
+ignore-docstrings=yes
+
+# Ignore imports when computing similarities.
+ignore-imports=no
+
+
+[VARIABLES]
+
+# Tells whether we should check for unused import in __init__ files.
+init-import=no
+
+# A regular expression matching the beginning of the name of dummy variables
+# (i.e. not used).
+dummy-variables-rgx=_$|dummy
+
+# List of additional names supposed to be defined in builtins. Remember that
+# you should avoid to define new builtins when possible.
+additional-builtins=
+
+
+[BASIC]
+
+# Required attributes for module, separated by a comma
+required-attributes=
+
+# List of builtins function names that should not be used, separated by a comma
+bad-functions=map,filter,apply,input
+
+# Regular expression which should only match correct module names
+module-rgx=(([a-z_][a-z0-9_]*)|([A-Z][a-zA-Z0-9]+))$
+
+# Regular expression which should only match correct module level names
+const-rgx=(([A-Z_][A-Z0-9_]*)|(__.*__))$
+
+# Regular expression which should only match correct class names
+class-rgx=[A-Z_][a-zA-Z0-9]+$
+
+# Regular expression which should only match correct function names
+function-rgx=[a-z_][a-z0-9_]{2,30}$
+
+# Regular expression which should only match correct method names
+method-rgx=[a-z_][a-z0-9_]{2,30}$
+
+# Regular expression which should only match correct instance attribute names
+attr-rgx=[a-z_][a-z0-9_]{2,30}$
+
+# Regular expression which should only match correct argument names
+argument-rgx=[a-z_][a-z0-9_]{2,30}$
+
+# Regular expression which should only match correct variable names
+variable-rgx=[a-z_][a-z0-9_]{2,30}$
+
+# Regular expression which should only match correct attribute names in class
+# bodies
+class-attribute-rgx=([A-Za-z_][A-Za-z0-9_]{2,30}|(__.*__))$
+
+# Regular expression which should only match correct list comprehension /
+# generator expression variable names
+inlinevar-rgx=[A-Za-z_][A-Za-z0-9_]*$
+
+# Good variable names which should always be accepted, separated by a comma
+good-names=i,j,k,ex,Run,_
+
+# Bad variable names which should always be refused, separated by a comma
+bad-names=foo,bar,baz,toto,tutu,tata
+
+# Regular expression which should only match function or class names that do
+# not require a docstring.
+no-docstring-rgx=__.*__
+
+# Minimum line length for functions/classes that require docstrings, shorter
+# ones are exempt.
+docstring-min-length=-1
+
+
+[FORMAT]
+
+# Maximum number of characters on a single line.
+max-line-length=80
+
+# Regexp for a line that is allowed to be longer than the limit.
+ignore-long-lines=^\s*(# )?<?https?://\S+>?$
+
+# Allow the body of an if to be on the same line as the test if there is no
+# else.
+single-line-if-stmt=no
+
+# List of optional constructs for which whitespace checking is disabled
+no-space-check=trailing-comma,dict-separator
+
+# Maximum number of lines in a module
+max-module-lines=1000
+
+# String used as indentation unit. This is usually " " (4 spaces) or "\t" (1
+# tab).
+indent-string='    '
+
+
+[DESIGN]
+
+# Maximum number of arguments for function / method
+max-args=5
+
+# Argument names that match this expression will be ignored. Default to name
+# with leading underscore
+ignored-argument-names=_.*
+
+# Maximum number of locals for function / method body
+max-locals=15
+
+# Maximum number of return / yield for function / method body
+max-returns=6
+
+# Maximum number of branch for function / method body
+max-branches=12
+
+# Maximum number of statements in function / method body
+max-statements=50
+
+# Maximum number of parents for a class (see R0901).
+max-parents=7
+
+# Maximum number of attributes for a class (see R0902).
+max-attributes=7
+
+# Minimum number of public methods for a class (see R0903).
+min-public-methods=2
+
+# Maximum number of public methods for a class (see R0904).
+max-public-methods=20
+
+
+[IMPORTS]
+
+# Deprecated modules which should not be used, separated by a comma
+deprecated-modules=regsub,TERMIOS,Bastion,rexec
+
+# Create a graph of every (i.e. internal and external) dependencies in the
+# given file (report RP0402 must not be disabled)
+import-graph=
+
+# Create a graph of external dependencies in the given file (report RP0402 must
+# not be disabled)
+ext-import-graph=
+
+# Create a graph of internal dependencies in the given file (report RP0402 must
+# not be disabled)
+int-import-graph=
+
+
+[CLASSES]
+
+# List of interface methods to ignore, separated by a comma. This is used for
+# instance to not check methods defines in Zope's Interface base class.
+ignore-iface-methods=isImplementedBy,deferred,extends,names,namesAndDescriptions,queryDescriptionFor,getBases,getDescriptionFor,getDoc,getName,getTaggedValue,getTaggedValueTags,isEqualOrExtendedBy,setTaggedValue,isImplementedByInstancesOf,adaptWith,is_implemented_by
+
+# List of method names used to declare (i.e. assign) instance attributes.
+defining-attr-methods=__init__,__new__,setUp
+
+# List of valid names for the first argument in a class method.
+valid-classmethod-first-arg=cls
+
+# List of valid names for the first argument in a metaclass class method.
+valid-metaclass-classmethod-first-arg=mcs
+
+
+[EXCEPTIONS]
+
+# Exceptions that will emit a warning when being caught. Defaults to
+# "Exception"
+overgeneral-exceptions=Exception

scripts/check_auth.py

@@ -0,0 +1,65 @@
+from synapse.events import FrozenEvent
+from synapse.api.auth import Auth
+
+from mock import Mock
+
+import argparse
+import itertools
+import json
+import sys
+
+
+def check_auth(auth, auth_chain, events):
+    auth_chain.sort(key=lambda e: e.depth)
+
+    auth_map = {
+        e.event_id: e
+        for e in auth_chain
+    }
+
+    create_events = {}
+    for e in auth_chain:
+        if e.type == "m.room.create":
+            create_events[e.room_id] = e
+
+    for e in itertools.chain(auth_chain, events):
+        auth_events_list = [auth_map[i] for i, _ in e.auth_events]
+
+        auth_events = {
+            (e.type, e.state_key): e
+            for e in auth_events_list
+        }
+
+        auth_events[("m.room.create", "")] = create_events[e.room_id]
+
+        try:
+            auth.check(e, auth_events=auth_events)
+        except Exception as ex:
+            print "Failed:", e.event_id, e.type, e.state_key
+            print "Auth_events:", auth_events
+            print ex
+            print json.dumps(e.get_dict(), sort_keys=True, indent=4)
+            # raise
+        print "Success:", e.event_id, e.type, e.state_key
+
+if __name__ == '__main__':
+    parser = argparse.ArgumentParser()
+
+    parser.add_argument(
+        'json',
+        nargs='?',
+        type=argparse.FileType('r'),
+        default=sys.stdin,
+    )
+
+    args = parser.parse_args()
+
+    js = json.load(args.json)
+
+
+    auth = Auth(Mock())
+    check_auth(
+        auth,
+        [FrozenEvent(d) for d in js["auth_chain"]],
+        [FrozenEvent(d) for d in js["pdus"]],
+    )

scripts/check_event_hash.py

@@ -0,0 +1,50 @@
+from synapse.crypto.event_signing import *
+from syutil.base64util import encode_base64
+
+import argparse
+import hashlib
+import sys
+import json
+
+
+class dictobj(dict):
+    def __init__(self, *args, **kargs):
+        dict.__init__(self, *args, **kargs)
+        self.__dict__ = self
+
+    def get_dict(self):
+        return dict(self)
+
+    def get_full_dict(self):
+        return dict(self)
+
+    def get_pdu_json(self):
+        return dict(self)
+
+
+def main():
+    parser = argparse.ArgumentParser()
+    parser.add_argument("input_json", nargs="?", type=argparse.FileType('r'),
+                        default=sys.stdin)
+    args = parser.parse_args()
+    logging.basicConfig()
+
+    event_json = dictobj(json.load(args.input_json))
+
+    algorithms = {
+        "sha256": hashlib.sha256,
+    }
+
+    for alg_name in event_json.hashes:
+        if check_event_content_hash(event_json, algorithms[alg_name]):
+            print "PASS content hash %s" % (alg_name,)
+        else:
+            print "FAIL content hash %s" % (alg_name,)
+
+    for algorithm in algorithms.values():
+        name, h_bytes = compute_event_reference_hash(event_json, algorithm)
+        print "Reference hash %s: %s" % (name, encode_base64(h_bytes))
+
+if __name__=="__main__":
+    main()
+

scripts/check_signature.py

@@ -0,0 +1,73 @@
+
+from syutil.crypto.jsonsign import verify_signed_json
+from syutil.crypto.signing_key import (
+    decode_verify_key_bytes, write_signing_keys
+)
+from syutil.base64util import decode_base64
+
+import urllib2
+import json
+import sys
+import dns.resolver
+import pprint
+import argparse
+import logging
+
+def get_targets(server_name):
+    if ":" in server_name:
+        target, port = server_name.split(":")
+        yield (target, int(port))
+        return
+    try:
+        answers = dns.resolver.query("_matrix._tcp." + server_name, "SRV")
+        for srv in answers:
+            yield (srv.target, srv.port)
+    except dns.resolver.NXDOMAIN:
+        yield (server_name, 8448)
+
+def get_server_keys(server_name, target, port):
+    url = "https://%s:%i/_matrix/key/v1" % (target, port)
+    keys = json.load(urllib2.urlopen(url))
+    verify_keys = {}
+    for key_id, key_base64 in keys["verify_keys"].items():
+        verify_key = decode_verify_key_bytes(key_id, decode_base64(key_base64))
+        verify_signed_json(keys, server_name, verify_key)
+        verify_keys[key_id] = verify_key
+    return verify_keys
+
+def main():
+
+    parser = argparse.ArgumentParser()
+    parser.add_argument("signature_name")
+    parser.add_argument("input_json", nargs="?", type=argparse.FileType('r'),
+                        default=sys.stdin)
+
+    args = parser.parse_args()
+    logging.basicConfig()
+
+    server_name = args.signature_name
+    keys = {}
+    for target, port in get_targets(server_name):
+        try:
+            keys = get_server_keys(server_name, target, port)
+            print "Using keys from https://%s:%s/_matrix/key/v1" % (target, port)
+            write_signing_keys(sys.stdout, keys.values())
+            break
+        except:
+            logging.exception("Error talking to %s:%s", target, port)
+
+    json_to_check = json.load(args.input_json)
+    print "Checking JSON:"
+    for key_id in json_to_check["signatures"][args.signature_name]:
+        try:
+            key = keys[key_id]
+            verify_signed_json(json_to_check, args.signature_name, key)
+            print "PASS %s" % (key_id,)
+        except:
+            logging.exception("Check for key %s failed" % (key_id,))
+            print "FAIL %s" % (key_id,)
+
+
+if __name__ == '__main__':
+    main()
+

scripts/copyrighter-sql.pl

@@ -0,0 +1,33 @@
+#!/usr/bin/perl -pi
+# Copyright 2015 OpenMarket Ltd
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+$copyright = <<EOT;
+/* Copyright 2015 OpenMarket Ltd
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+EOT
+
+s/^(# -\*- coding: utf-8 -\*-\n)?/$1$copyright/ if ($. == 1);

scripts/copyrighter.pl

@@ -1,5 +1,5 @@
 #!/usr/bin/perl -pi
-# Copyright 2014 matrix.org
+# Copyright 2014 OpenMarket Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -14,7 +14,7 @@
 # limitations under the License.
 
 $copyright = <<EOT;
-# Copyright 2014 matrix.org
+# Copyright 2015 OpenMarket Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.

scripts/database-prepare-for-0.5.0.sh

@@ -0,0 +1,21 @@
+#!/bin/bash
+
+# This is will prepare a synapse database for running with v0.5.0 of synapse. 
+# It will store all the user information, but will *delete* all messages and
+# room data.
+
+set -e
+
+cp "$1" "$1.bak"
+
+DUMP=$(sqlite3 "$1" << 'EOF'
+.dump users
+.dump access_tokens
+.dump presence
+.dump profiles
+EOF
+)
+
+rm "$1"
+
+sqlite3 "$1" <<< "$DUMP" 

scripts/federation_client.py

@@ -0,0 +1,146 @@
+import nacl.signing
+import json
+import base64
+import requests
+import sys
+import srvlookup
+
+
+def encode_base64(input_bytes):
+    """Encode bytes as a base64 string without any padding."""
+
+    input_len = len(input_bytes)
+    output_len = 4 * ((input_len + 2) // 3) + (input_len + 2) % 3 - 2
+    output_bytes = base64.b64encode(input_bytes)
+    output_string = output_bytes[:output_len].decode("ascii")
+    return output_string
+
+
+def decode_base64(input_string):
+    """Decode a base64 string to bytes inferring padding from the length of the
+    string."""
+
+    input_bytes = input_string.encode("ascii")
+    input_len = len(input_bytes)
+    padding = b"=" * (3 - ((input_len + 3) % 4))
+    output_len = 3 * ((input_len + 2) // 4) + (input_len + 2) % 4 - 2
+    output_bytes = base64.b64decode(input_bytes + padding)
+    return output_bytes[:output_len]
+
+
+def encode_canonical_json(value):
+    return json.dumps(
+         value,
+         # Encode code-points outside of ASCII as UTF-8 rather than \u escapes
+         ensure_ascii=False,
+         # Remove unecessary white space.
+         separators=(',',':'),
+         # Sort the keys of dictionaries.
+         sort_keys=True,
+         # Encode the resulting unicode as UTF-8 bytes.
+     ).encode("UTF-8")
+
+
+def sign_json(json_object, signing_key, signing_name):
+    signatures = json_object.pop("signatures", {})
+    unsigned = json_object.pop("unsigned", None)
+
+    signed = signing_key.sign(encode_canonical_json(json_object))
+    signature_base64 = encode_base64(signed.signature)
+
+    key_id = "%s:%s" % (signing_key.alg, signing_key.version)
+    signatures.setdefault(signing_name, {})[key_id] = signature_base64
+
+    json_object["signatures"] = signatures
+    if unsigned is not None:
+        json_object["unsigned"] = unsigned
+
+    return json_object
+
+
+NACL_ED25519 = "ed25519"
+
+def decode_signing_key_base64(algorithm, version, key_base64):
+    """Decode a base64 encoded signing key
+    Args:
+        algorithm (str): The algorithm the key is for (currently "ed25519").
+        version (str): Identifies this key out of the keys for this entity.
+        key_base64 (str): Base64 encoded bytes of the key.
+    Returns:
+        A SigningKey object.
+    """
+    if algorithm == NACL_ED25519:
+        key_bytes = decode_base64(key_base64)
+        key = nacl.signing.SigningKey(key_bytes)
+        key.version = version
+        key.alg = NACL_ED25519
+        return key
+    else:
+        raise ValueError("Unsupported algorithm %s" % (algorithm,))
+
+
+def read_signing_keys(stream):
+    """Reads a list of keys from a stream
+    Args:
+        stream : A stream to iterate for keys.
+    Returns:
+        list of SigningKey objects.
+    """
+    keys = []
+    for line in stream:
+        algorithm, version, key_base64 = line.split()
+        keys.append(decode_signing_key_base64(algorithm, version, key_base64))
+    return keys
+
+
+def lookup(destination, path):
+    if ":" in destination:
+        return "https://%s%s" % (destination, path)
+    else:
+        try:
+            srv = srvlookup.lookup("matrix", "tcp", destination)[0]
+            return "https://%s:%d%s" % (srv.host, srv.port, path)
+        except:
+            return "https://%s:%d%s" % (destination, 8448, path)
+
+def get_json(origin_name, origin_key, destination, path):
+    request_json = {
+        "method": "GET",
+        "uri": path,
+        "origin": origin_name,
+        "destination": destination,
+    }
+
+    signed_json = sign_json(request_json, origin_key, origin_name)
+
+    authorization_headers = []
+
+    for key, sig in signed_json["signatures"][origin_name].items():
+        authorization_headers.append(bytes(
+            "X-Matrix origin=%s,key=\"%s\",sig=\"%s\"" % (
+                origin_name, key, sig,
+            )
+        ))
+
+    result = requests.get(
+        lookup(destination, path),
+        headers={"Authorization": authorization_headers[0]},
+        verify=False,
+    )
+    return result.json()
+
+
+def main():
+    origin_name, keyfile, destination, path = sys.argv[1:]
+
+    with open(keyfile) as f:
+        key = read_signing_keys(f)[0]
+
+    result = get_json(
+        origin_name, key, destination, "/_matrix/federation/v1/" + path
+    )
+
+    json.dump(result, sys.stdout)
+
+if __name__ == "__main__":
+    main()

scripts/graph_tracer.py

@@ -0,0 +1,213 @@
+import fileinput
+import pydot
+import sys
+import itertools
+import json
+
+
+def pairwise(iterable):
+    "s -> (s0,s1), (s1,s2), (s2, s3), ..."
+    a, b = itertools.tee(iterable)
+    next(b, None)
+    return itertools.izip(a, b)
+
+
+nodes = {}
+edges = set()
+
+graph = pydot.Dot(graph_name="call_graph", graph_type="digraph")
+
+names = {}
+starts = {}
+ends = {}
+deferreds = set()
+deferreds_map = {}
+deferred_edges = set()
+
+root_id = None
+
+for line in fileinput.input():
+    line = line.strip()
+    try:
+        if " calls " in line:
+            start, end = line.split(" calls ")
+            start, end = start.strip(), end.strip()
+            edges.add((start, end))
+            # print start, end
+        if " named " in line:
+            node_id, name = line.split(" named ")
+            names[node_id.strip()] = name.strip()
+
+            if name.strip() == "synapse.rest.client.v1.room.RoomSendEventRestServlet.on_PUT":
+                root_id = node_id
+        if " in " in line:
+            node_id, d = line.split(" in ")
+            deferreds_map[node_id.strip()] = d.strip()
+        if " is deferred" in line:
+            node_id, _ = line.split(" is deferred")
+            deferreds.add(node_id)
+        if " start " in line:
+            node_id, ms = line.split(" start ")
+            starts[node_id.strip()] = int(ms.strip())
+        if " end " in line:
+            node_id, ms = line.split(" end ")
+            ends[node_id.strip()] = int(ms.strip())
+        if " waits on " in line:
+            start, end = line.split(" waits on ")
+            start, end = start.strip(), end.strip()
+            deferred_edges.add((start, end))
+            # print start, end
+    except Exception as e:
+        sys.stderr.write("failed %s to parse '%s'\n" % (e.message, line))
+
+if not root_id:
+    sys.stderr.write("Could not find root")
+    sys.exit(1)
+
+
+# deferreds_root = set(deferreds.values())
+# for parent, child in deferred_edges:
+#     deferreds_root.discard(child)
+#
+# deferred_tree = {
+#     d: {}
+#     for d in deferreds_root
+# }
+#
+# def populate(root, tree):
+#     for leaf in deferred_edges.get(root, []):
+#         populate(leaf, tree.setdefault(leaf, {}))
+#
+#
+# for d in deferreds_root:
+#     tree = deferred_tree.setdefault(d, {})
+#     populate(d, tree)
+
+# print deferred_edges
+# print root_id
+
+def is_in_deferred(d):
+    while True:
+        if d == root_id:
+            return True
+
+        for start, end in deferred_edges:
+            if d == end:
+                d = start
+                break
+        else:
+            return False
+
+
+def walk_graph(d):
+    res = [d]
+    while d != root_id:
+        for start, end in edges:
+            if d == end:
+                d = start
+                res.append(d)
+                break
+        else:
+            return res
+    return res
+
+
+def make_tree_el(node_id):
+    return {
+        "id": node_id,
+        "name": names[node_id],
+        "children": [],
+        "start": starts[node_id],
+        "end": ends[node_id],
+        "size": ends[node_id] - starts[node_id],
+    }
+
+tree = make_tree_el(root_id)
+
+tree_index = {
+    root_id: tree,
+}
+
+
+viz_out = {
+    "nodes": [],
+    "edges": [],
+}
+
+for node_id, name in names.items():
+    # if times.get(node_id, 100) < 5:
+    #     continue
+
+    walk = walk_graph(node_id)
+    # print walk
+    if root_id not in walk:
+        continue
+
+    if node_id in deferreds:
+        if not is_in_deferred(node_id):
+            continue
+    elif node_id in deferreds_map:
+        if not is_in_deferred(deferreds_map[node_id]):
+            continue
+
+    walk_names = [
+        names[w].split("synapse.", 1)[1] for w in walk
+    ]
+
+    for child, parent in reversed(list(pairwise(walk))):
+        if parent in tree_index and child not in tree_index:
+            el = make_tree_el(child)
+            tree_index[parent]["children"].append(el)
+            tree_index[child] = el
+
+    # print "-".join(reversed(["end"] + walk_names)) + ", " + str(ends[node_id] - starts[node_id])
+    # print "%d,%s,%s,%s" % (len(walk), walk_names[0], starts[node_id], ends[node_id])
+
+    viz_out["nodes"].append({
+        "id": node_id,
+        "label": names[node_id].split("synapse.", 1)[1],
+        "value": ends[node_id] - starts[node_id],
+        "level": len(walk),
+    })
+
+    node = pydot.Node(node_id, label=name)
+
+    # if node_id in deferreds:
+    #     clusters[deferreds[node_id]].add_node(node)
+    # elif node_id in clusters:
+    #     clusters[node_id].add_node(node)
+    # else:
+    #     graph.add_node(node)
+    graph.add_node(node)
+    nodes[node_id] = node
+
+    # print node_id
+
+# for el in tree_index.values():
+#     el["children"].sort(key=lambda e: e["start"])
+#
+# print json.dumps(tree)
+
+for parent, child in edges:
+    if child not in nodes:
+        # sys.stderr.write(child + " not a node\n")
+        continue
+
+    if parent not in nodes:
+        # sys.stderr.write(parent + " not a node\n")
+        continue
+
+    viz_out["edges"].append({
+        "from": parent,
+        "to": child,
+        "value": ends[child] - starts[child],
+    })
+
+    edge = pydot.Edge(nodes[parent], nodes[child])
+    graph.add_edge(edge)
+
+print json.dumps(viz_out)
+
+file_prefix = "call_graph_out"
+graph.write('%s.dot' % file_prefix, format='raw', prog='dot')
+graph.write_svg("%s.svg" % file_prefix, prog='dot')

scripts/hash_history.py

@@ -0,0 +1,69 @@
+from synapse.storage.pdu import PduStore
+from synapse.storage.signatures import SignatureStore
+from synapse.storage._base import SQLBaseStore
+from synapse.federation.units import Pdu
+from synapse.crypto.event_signing import (
+    add_event_pdu_content_hash, compute_pdu_event_reference_hash
+)
+from synapse.api.events.utils import prune_pdu
+from syutil.base64util import encode_base64, decode_base64
+from syutil.jsonutil import encode_canonical_json
+import sqlite3
+import sys
+
+class Store(object):
+    _get_pdu_tuples = PduStore.__dict__["_get_pdu_tuples"]
+    _get_pdu_content_hashes_txn = SignatureStore.__dict__["_get_pdu_content_hashes_txn"]
+    _get_prev_pdu_hashes_txn = SignatureStore.__dict__["_get_prev_pdu_hashes_txn"]
+    _get_pdu_origin_signatures_txn = SignatureStore.__dict__["_get_pdu_origin_signatures_txn"]
+    _store_pdu_content_hash_txn = SignatureStore.__dict__["_store_pdu_content_hash_txn"]
+    _store_pdu_reference_hash_txn = SignatureStore.__dict__["_store_pdu_reference_hash_txn"]
+    _store_prev_pdu_hash_txn = SignatureStore.__dict__["_store_prev_pdu_hash_txn"]
+    _simple_insert_txn = SQLBaseStore.__dict__["_simple_insert_txn"]
+
+
+store = Store()
+
+
+def select_pdus(cursor):
+    cursor.execute(
+        "SELECT pdu_id, origin FROM pdus ORDER BY depth ASC"
+    )
+
+    ids = cursor.fetchall()
+
+    pdu_tuples = store._get_pdu_tuples(cursor, ids)
+
+    pdus = [Pdu.from_pdu_tuple(p) for p in pdu_tuples]
+
+    reference_hashes = {}
+
+    for pdu in pdus:
+        try:
+            if pdu.prev_pdus:
+                print "PROCESS", pdu.pdu_id, pdu.origin, pdu.prev_pdus
+                for pdu_id, origin, hashes in pdu.prev_pdus:
+                    ref_alg, ref_hsh = reference_hashes[(pdu_id, origin)]
+                    hashes[ref_alg] = encode_base64(ref_hsh)
+                    store._store_prev_pdu_hash_txn(cursor,  pdu.pdu_id, pdu.origin, pdu_id, origin, ref_alg, ref_hsh)
+                print "SUCCESS", pdu.pdu_id, pdu.origin, pdu.prev_pdus
+            pdu = add_event_pdu_content_hash(pdu)
+            ref_alg, ref_hsh = compute_pdu_event_reference_hash(pdu)
+            reference_hashes[(pdu.pdu_id, pdu.origin)] = (ref_alg, ref_hsh)
+            store._store_pdu_reference_hash_txn(cursor, pdu.pdu_id, pdu.origin, ref_alg, ref_hsh)
+
+            for alg, hsh_base64 in pdu.hashes.items():
+                print alg, hsh_base64
+                store._store_pdu_content_hash_txn(cursor, pdu.pdu_id, pdu.origin, alg, decode_base64(hsh_base64))
+
+        except:
+            print "FAILED_", pdu.pdu_id, pdu.origin, pdu.prev_pdus
+
+def main():
+    conn = sqlite3.connect(sys.argv[1])
+    cursor = conn.cursor()
+    select_pdus(cursor)
+    conn.commit()
+
+if __name__=='__main__':
+    main()

scripts/make_identicons.pl

@@ -0,0 +1,39 @@
+#!/usr/bin/env perl
+
+use strict;
+use warnings;
+
+use DBI;
+use DBD::SQLite;
+use JSON;
+use Getopt::Long;
+
+my $db; # = "homeserver.db";
+my $server = "http://localhost:8008";
+my $size = 320;
+
+GetOptions("db|d=s",     \$db,
+           "server|s=s", \$server,
+           "width|w=i",  \$size) or usage();
+
+usage() unless $db;
+
+my $dbh = DBI->connect("dbi:SQLite:dbname=$db","","") || die $DBI::errstr;
+
+my $res = $dbh->selectall_arrayref("select token, name from access_tokens, users where access_tokens.user_id = users.id group by user_id") || die $DBI::errstr;
+
+foreach (@$res) {
+    my ($token, $mxid) = ($_->[0], $_->[1]);
+    my ($user_id) = ($mxid =~ m/@(.*):/);
+    my ($url) = $dbh->selectrow_array("select avatar_url from profiles where user_id=?", undef, $user_id);
+    if (!$url || $url =~ /#auto$/) {
+        `curl -s -o tmp.png "$server/_matrix/media/v1/identicon?name=${mxid}&width=$size&height=$size"`;
+        my $json = `curl -s -X POST -H "Content-Type: image/png" -T "tmp.png" $server/_matrix/media/v1/upload?access_token=$token`;
+        my $content_uri = from_json($json)->{content_uri};
+        `curl -X PUT -H "Content-Type: application/json" --data '{ "avatar_url": "${content_uri}#auto"}' $server/_matrix/client/api/v1/profile/${mxid}/avatar_url?access_token=$token`;
+    }
+}
+
+sub usage {
+    die "usage: ./make-identicons.pl\n\t-d database [e.g. homeserver.db]\n\t-s homeserver (default: http://localhost:8008)\n\t-w identicon size in pixels (default 320)";
+}

scripts/upgrade_db_to_v0.6.0.py

@@ -0,0 +1,331 @@
+
+from synapse.storage import SCHEMA_VERSION, read_schema
+from synapse.storage._base import SQLBaseStore
+from synapse.storage.signatures import SignatureStore
+from synapse.storage.event_federation import EventFederationStore
+
+from syutil.base64util import encode_base64, decode_base64
+
+from synapse.crypto.event_signing import compute_event_signature
+
+from synapse.events.builder import EventBuilder
+from synapse.events.utils import prune_event
+
+from synapse.crypto.event_signing import check_event_content_hash
+
+from syutil.crypto.jsonsign import (
+    verify_signed_json, SignatureVerifyException,
+)
+from syutil.crypto.signing_key import decode_verify_key_bytes
+
+from syutil.jsonutil import encode_canonical_json
+
+import argparse
+# import dns.resolver
+import hashlib
+import httplib
+import json
+import sqlite3
+import syutil
+import urllib2
+
+
+delta_sql = """
+CREATE TABLE IF NOT EXISTS event_json(
+    event_id TEXT NOT NULL,
+    room_id TEXT NOT NULL,
+    internal_metadata NOT NULL,
+    json BLOB NOT NULL,
+    CONSTRAINT ev_j_uniq UNIQUE (event_id)
+);
+
+CREATE INDEX IF NOT EXISTS event_json_id ON event_json(event_id);
+CREATE INDEX IF NOT EXISTS event_json_room_id ON event_json(room_id);
+
+PRAGMA user_version = 10;
+"""
+
+
+class Store(object):
+    _get_event_signatures_txn = SignatureStore.__dict__["_get_event_signatures_txn"]
+    _get_event_content_hashes_txn = SignatureStore.__dict__["_get_event_content_hashes_txn"]
+    _get_event_reference_hashes_txn = SignatureStore.__dict__["_get_event_reference_hashes_txn"]
+    _get_prev_event_hashes_txn = SignatureStore.__dict__["_get_prev_event_hashes_txn"]
+    _get_prev_events_and_state = EventFederationStore.__dict__["_get_prev_events_and_state"]
+    _get_auth_events = EventFederationStore.__dict__["_get_auth_events"]
+    cursor_to_dict = SQLBaseStore.__dict__["cursor_to_dict"]
+    _simple_select_onecol_txn = SQLBaseStore.__dict__["_simple_select_onecol_txn"]
+    _simple_select_list_txn = SQLBaseStore.__dict__["_simple_select_list_txn"]
+    _simple_insert_txn = SQLBaseStore.__dict__["_simple_insert_txn"]
+
+    def _generate_event_json(self, txn, rows):
+        events = []
+        for row in rows:
+            d = dict(row)
+
+            d.pop("stream_ordering", None)
+            d.pop("topological_ordering", None)
+            d.pop("processed", None)
+
+            if "origin_server_ts" not in d:
+                d["origin_server_ts"] = d.pop("ts", 0)
+            else:
+                d.pop("ts", 0)
+
+            d.pop("prev_state", None)
+            d.update(json.loads(d.pop("unrecognized_keys")))
+
+            d["sender"] = d.pop("user_id")
+
+            d["content"] = json.loads(d["content"])
+
+            if "age_ts" not in d:
+                # For compatibility
+                d["age_ts"] = d.get("origin_server_ts", 0)
+
+            d.setdefault("unsigned", {})["age_ts"] = d.pop("age_ts")
+
+            outlier = d.pop("outlier", False)
+
+            # d.pop("membership", None)
+
+            d.pop("state_hash", None)
+
+            d.pop("replaces_state", None)
+
+            b = EventBuilder(d)
+            b.internal_metadata.outlier = outlier
+
+            events.append(b)
+
+        for i, ev in enumerate(events):
+            signatures = self._get_event_signatures_txn(
+                txn, ev.event_id,
+            )
+
+            ev.signatures = {
+                n: {
+                    k: encode_base64(v) for k, v in s.items()
+                }
+                for n, s in signatures.items()
+            }
+
+            hashes = self._get_event_content_hashes_txn(
+                txn, ev.event_id,
+            )
+
+            ev.hashes = {
+                k: encode_base64(v) for k, v in hashes.items()
+            }
+
+            prevs = self._get_prev_events_and_state(txn, ev.event_id)
+
+            ev.prev_events = [
+                (e_id, h)
+                for e_id, h, is_state in prevs
+                if is_state == 0
+            ]
+
+            # ev.auth_events = self._get_auth_events(txn, ev.event_id)
+
+            hashes = dict(ev.auth_events)
+
+            for e_id, hash in ev.prev_events:
+                if e_id in hashes and not hash:
+                    hash.update(hashes[e_id])
+            #
+            # if hasattr(ev, "state_key"):
+            #     ev.prev_state = [
+            #         (e_id, h)
+            #         for e_id, h, is_state in prevs
+            #         if is_state == 1
+            #     ]
+
+        return [e.build() for e in events]
+
+
+store = Store()
+
+
+# def get_key(server_name):
+#     print "Getting keys for: %s" % (server_name,)
+#     targets = []
+#     if ":" in server_name:
+#         target, port = server_name.split(":")
+#         targets.append((target, int(port)))
+#     try:
+#         answers = dns.resolver.query("_matrix._tcp." + server_name, "SRV")
+#         for srv in answers:
+#             targets.append((srv.target, srv.port))
+#     except dns.resolver.NXDOMAIN:
+#         targets.append((server_name, 8448))
+#     except:
+#         print "Failed to lookup keys for %s" % (server_name,)
+#         return {}
+#
+#     for target, port in targets:
+#         url = "https://%s:%i/_matrix/key/v1" % (target, port)
+#         try:
+#             keys = json.load(urllib2.urlopen(url, timeout=2))
+#             verify_keys = {}
+#             for key_id, key_base64 in keys["verify_keys"].items():
+#                 verify_key = decode_verify_key_bytes(
+#                     key_id, decode_base64(key_base64)
+#                 )
+#                 verify_signed_json(keys, server_name, verify_key)
+#                 verify_keys[key_id] = verify_key
+#             print "Got keys for: %s" % (server_name,)
+#             return verify_keys
+#         except urllib2.URLError:
+#             pass
+#         except urllib2.HTTPError:
+#             pass
+#         except httplib.HTTPException:
+#             pass
+#
+#     print "Failed to get keys for %s" % (server_name,)
+#     return {}
+
+
+def reinsert_events(cursor, server_name, signing_key):
+    print "Running delta: v10"
+
+    cursor.executescript(delta_sql)
+
+    cursor.execute(
+        "SELECT * FROM events ORDER BY rowid ASC"
+    )
+
+    print "Getting events..."
+
+    rows = store.cursor_to_dict(cursor)
+
+    events = store._generate_event_json(cursor, rows)
+
+    print "Got events from DB."
+
+    algorithms = {
+        "sha256": hashlib.sha256,
+    }
+
+    key_id = "%s:%s" % (signing_key.alg, signing_key.version)
+    verify_key = signing_key.verify_key
+    verify_key.alg = signing_key.alg
+    verify_key.version = signing_key.version
+
+    server_keys = {
+        server_name: {
+            key_id: verify_key
+        }
+    }
+
+    i = 0
+    N = len(events)
+
+    for event in events:
+        if i % 100 == 0:
+            print "Processed: %d/%d events" % (i,N,)
+        i += 1
+
+        # for alg_name in event.hashes:
+        #     if check_event_content_hash(event, algorithms[alg_name]):
+        #         pass
+        #     else:
+        #         pass
+        #         print "FAIL content hash %s %s" % (alg_name, event.event_id, )
+
+        have_own_correctly_signed = False
+        for host, sigs in event.signatures.items():
+            pruned = prune_event(event)
+
+            for key_id in sigs:
+                if host not in server_keys:
+                    server_keys[host] = {}  # get_key(host)
+                if key_id in server_keys[host]:
+                    try:
+                        verify_signed_json(
+                            pruned.get_pdu_json(),
+                            host,
+                            server_keys[host][key_id]
+                        )
+
+                        if host == server_name:
+                            have_own_correctly_signed = True
+                    except SignatureVerifyException:
+                        print "FAIL signature check %s %s" % (
+                            key_id, event.event_id
+                        )
+
+        # TODO: Re sign with our own server key
+        if not have_own_correctly_signed:
+            sigs = compute_event_signature(event, server_name, signing_key)
+            event.signatures.update(sigs)
+
+            pruned = prune_event(event)
+
+            for key_id in event.signatures[server_name]:
+                verify_signed_json(
+                    pruned.get_pdu_json(),
+                    server_name,
+                    server_keys[server_name][key_id]
+                )
+
+        event_json = encode_canonical_json(
+            event.get_dict()
+        ).decode("UTF-8")
+
+        metadata_json = encode_canonical_json(
+            event.internal_metadata.get_dict()
+        ).decode("UTF-8")
+
+        store._simple_insert_txn(
+            cursor,
+            table="event_json",
+            values={
+                "event_id": event.event_id,
+                "room_id": event.room_id,
+                "internal_metadata": metadata_json,
+                "json": event_json,
+            },
+            or_replace=True,
+        )
+
+
+def main(database, server_name, signing_key):
+    conn = sqlite3.connect(database)
+    cursor = conn.cursor()
+
+    # Do other deltas:
+    cursor.execute("PRAGMA user_version")
+    row = cursor.fetchone()
+
+    if row and row[0]:
+        user_version = row[0]
+        # Run every version since after the current version.
+        for v in range(user_version + 1, 10):
+            print "Running delta: %d" % (v,)
+            sql_script = read_schema("delta/v%d" % (v,))
+            cursor.executescript(sql_script)
+
+    reinsert_events(cursor, server_name, signing_key)
+
+    conn.commit()
+
+    print "Success!"
+
+
+if __name__ == "__main__":
+    parser = argparse.ArgumentParser()
+
+    parser.add_argument("database")
+    parser.add_argument("server_name")
+    parser.add_argument(
+        "signing_key", type=argparse.FileType('r'),
+    )
+    args = parser.parse_args()
+
+    signing_key = syutil.crypto.signing_key.read_signing_keys(
+        args.signing_key
+    )
+
+    main(args.database, args.server_name, signing_key[0])

setup.cfg

@@ -8,3 +8,11 @@ test = trial
 
 [trial]
 test_suite = tests
+
+[check-manifest]
+ignore =
+    contrib
+    contrib/*
+    docs/*
+    pylint.cfg
+    tox.ini

setup.py

@@ -1,6 +1,6 @@
 #!/usr/bin/env python
 
-# Copyright 2014 matrix.org
+# Copyright 2014 OpenMarket Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -18,39 +18,42 @@ import os
 from setuptools import setup, find_packages
 
 
-# Utility function to read the README file.
-# Used for the long_description.  It's nice, because now 1) we have a top level
-# README file and 2) it's easier to type in the README file than to put a raw
-# string in below ...
-def read(fname):
-    return open(os.path.join(os.path.dirname(__file__), fname)).read()
+here = os.path.abspath(os.path.dirname(__file__))
+
+
+def read_file(path_segments):
+    """Read a file from the package. Takes a list of strings to join to
+    make the path"""
+    file_path = os.path.join(here, *path_segments)
+    with open(file_path) as f:
+        return f.read()
+
+
+def exec_file(path_segments):
+    """Execute a single python file to get the variables defined in it"""
+    result = {}
+    code = read_file(path_segments)
+    exec(code, result)
+    return result
+
+version = exec_file(("synapse", "__init__.py"))["__version__"]
+dependencies = exec_file(("synapse", "python_dependencies.py"))
+long_description = read_file(("README.rst",))
 
 setup(
-    name="SynapseHomeServer",
-    version="0.0.1",
-    packages=find_packages(exclude=["tests"]),
+    name="matrix-synapse",
+    version=version,
+    packages=find_packages(exclude=["tests", "tests.*"]),
     description="Reference Synapse Home Server",
-    install_requires=[
-        "syutil==0.0.1",
-        "Twisted>=14.0.0",
-        "service_identity>=1.0.0",
-        "pyasn1",
-        "pynacl",
-        "daemonize",
-        "py-bcrypt",
-    ],
-    dependency_links=[
-        "git+ssh://git@github.com/matrix-org/syutil.git#egg=syutil-0.0.1",
-    ],
+    install_requires=dependencies["REQUIREMENTS"].keys(),
     setup_requires=[
+        "Twisted==14.0.2", # Here to override setuptools_trial's dependency on Twisted>=2.4.0
         "setuptools_trial",
-        "setuptools>=1.0.0", # Needs setuptools that supports git+ssh. It's not obvious when support for this was introduced.
         "mock"
     ],
+    dependency_links=dependencies["DEPENDENCY_LINKS"],
     include_package_data=True,
-    long_description=read("README.rst"),
-    entry_points="""
-    [console_scripts]
-    synapse-homeserver=synapse.app.homeserver:run
-    """
+    zip_safe=False,
+    long_description=long_description,
+    scripts=["synctl"],
 )

static/client/register/index.html

@@ -0,0 +1,32 @@
+<html>
+<head>
+<title> Registration </title>
+<meta name='viewport' content='width=device-width, initial-scale=1, user-scalable=no, minimum-scale=1.0, maximum-scale=1.0'> 
+<link rel="stylesheet" href="style.css">
+<script src="js/jquery-2.1.3.min.js"></script>
+<script src="js/recaptcha_ajax.js"></script>
+<script src="register_config.js"></script>
+<script src="js/register.js"></script>
+</head>
+<body onload="matrixRegistration.onLoad()">
+<form id="registrationForm" onsubmit="matrixRegistration.signUp(); return false;">
+    <div>
+        Create account:<br/>
+        
+        <div style="text-align: center">
+            <input id="desired_user_id" size="32" type="text" placeholder="Matrix ID (e.g. bob)" autocapitalize="off" autocorrect="off" />
+            <br/>
+            <input id="pwd1" size="32" type="password" placeholder="Type a password"/>
+            <br/>
+            <input id="pwd2" size="32" type="password" placeholder="Confirm your password"/>
+            <br/>
+            <span id="feedback" style="color: #f00"></span>
+            <br/>
+            <div id="regcaptcha"></div>
+
+            <button type="submit" style="margin: 10px">Sign up</button>
+        </div>
+    </div>
+</form>
+</body>
+</html>

static/client/register/js/register.js

@@ -0,0 +1,117 @@
+window.matrixRegistration = {
+    endpoint: location.origin + "/_matrix/client/api/v1/register"
+};
+
+var setupCaptcha = function() {
+    if (!window.matrixRegistrationConfig) {
+        return;
+    }
+    $.get(matrixRegistration.endpoint, function(response) {
+        var serverExpectsCaptcha = false;
+        for (var i=0; i<response.flows.length; i++) {
+            var flow = response.flows[i];
+            if ("m.login.recaptcha" === flow.type) {
+                serverExpectsCaptcha = true;
+                break;
+            }
+        }
+        if (!serverExpectsCaptcha) {
+            console.log("This server does not require a captcha.");
+            return;
+        }
+        console.log("Setting up ReCaptcha for "+matrixRegistration.endpoint);
+        var public_key = window.matrixRegistrationConfig.recaptcha_public_key;
+        if (public_key === undefined) {
+            console.error("No public key defined for captcha!");
+            setFeedbackString("Misconfigured captcha for server. Contact server admin.");
+            return;
+        }
+        Recaptcha.create(public_key,
+        "regcaptcha",
+        {
+            theme: "red",
+            callback: Recaptcha.focus_response_field
+        });
+        window.matrixRegistration.isUsingRecaptcha = true;
+    }).error(errorFunc);
+    
+};
+
+var submitCaptcha = function(user, pwd) {
+    var challengeToken = Recaptcha.get_challenge();
+    var captchaEntry = Recaptcha.get_response();
+    var data = {
+        type: "m.login.recaptcha",
+        challenge: challengeToken,
+        response: captchaEntry
+    };
+    console.log("Submitting captcha");
+    $.post(matrixRegistration.endpoint, JSON.stringify(data), function(response) {
+        console.log("Success -> "+JSON.stringify(response));
+        submitPassword(user, pwd, response.session);
+    }).error(function(err) {
+        Recaptcha.reload();
+        errorFunc(err);
+    });
+};
+
+var submitPassword = function(user, pwd, session) {
+    console.log("Registering...");
+    var data = {
+        type: "m.login.password",
+        user: user,
+        password: pwd,
+        session: session
+    };
+    $.post(matrixRegistration.endpoint, JSON.stringify(data), function(response) {
+        matrixRegistration.onRegistered(
+            response.home_server, response.user_id, response.access_token
+        );
+    }).error(errorFunc);
+};
+
+var errorFunc = function(err) {
+    if (err.responseJSON && err.responseJSON.error) {
+        setFeedbackString(err.responseJSON.error + " (" + err.responseJSON.errcode + ")");
+    }
+    else {
+        setFeedbackString("Request failed: " + err.status);
+    }
+};
+
+var setFeedbackString = function(text) {
+    $("#feedback").text(text);
+};
+
+matrixRegistration.onLoad = function() {
+    setupCaptcha();
+};
+
+matrixRegistration.signUp = function() {
+    var user = $("#desired_user_id").val();
+    if (user.length == 0) {
+        setFeedbackString("Must specify a username.");
+        return;
+    }
+    var pwd1 = $("#pwd1").val();
+    var pwd2 = $("#pwd2").val();
+    if (pwd1.length < 6) {
+        setFeedbackString("Password: min. 6 characters.");
+        return;
+    }
+    if (pwd1 != pwd2) {
+        setFeedbackString("Passwords do not match.");
+        return;
+    }
+    if (window.matrixRegistration.isUsingRecaptcha) {
+        submitCaptcha(user, pwd1);
+    }
+    else {
+        submitPassword(user, pwd1);
+    }
+};
+
+matrixRegistration.onRegistered = function(hs_url, user_id, access_token) {
+    // clobber this function
+    console.log("onRegistered - This function should be replaced to proceed.");
+};

static/client/register/register_config.sample.js

@@ -0,0 +1,3 @@
+window.matrixRegistrationConfig = {
+  recaptcha_public_key: "YOUR_PUBLIC_KEY"
+};

static/client/register/style.css

@@ -0,0 +1,56 @@
+html {
+    height: 100%;
+}
+
+body {
+    height: 100%;
+    font-family: "Myriad Pro", "Myriad", Helvetica, Arial, sans-serif;
+    font-size: 12pt;
+    margin: 0px;
+}
+
+h1 {
+    font-size: 20pt;
+}
+
+a:link    { color: #666; }
+a:visited { color: #666; }
+a:hover   { color: #000; }
+a:active  { color: #000; }
+
+input {
+   width: 100%
+}
+
+textarea, input {
+   font-family: inherit;
+   font-size: inherit;
+}
+
+.smallPrint {
+    color: #888;
+    font-size: 9pt ! important;
+    font-style: italic ! important;
+}
+
+#recaptcha_area { 
+    margin: auto
+}
+
+#registrationForm {
+    text-align: left;
+    padding: 1em;
+    margin-bottom: 40px;
+    display: inline-block;
+    
+    -webkit-border-radius: 10px;
+    -moz-border-radius: 10px;
+    border-radius: 10px;
+    
+    -webkit-box-shadow: 0px 0px 20px 0px rgba(0,0,0,0.15);
+    -moz-box-shadow: 0px 0px 20px 0px rgba(0,0,0,0.15);
+    box-shadow: 0px 0px 20px 0px rgba(0,0,0,0.15);
+    
+    background-color: #f8f8f8;
+    border: 1px #ccc solid;
+}