Pin the twisted version so that it doesn't pull in twisted 15.

Hotfixes v0.6.1e

CHANGES.rst

@@ -1,7 +1,21 @@
+Changes in synapse 0.6.1 (2015-01-07)
+=====================================
+
+ * Major optimizations to improve performance of initial sync and event sending
+   in large rooms (by up to 10x)
+ * Media repository now includes a Content-Length header on media downloads.
+ * Improve quality of thumbnails by changing resizing algorithm.
+
 Changes in synapse 0.6.0 (2014-12-16)
 =====================================
 
  * Add new API for media upload and download that supports thumbnailing.
+ * Replicate media uploads over multiple homeservers so media is always served
+   to clients from their local homeserver.  This obsoletes the
+   --content-addr parameter and confusion over accessing content directly
+   from remote homeservers.
+ * Implement exponential backoff when retrying federation requests when
+   sending to remote homeservers which are offline.
  * Implement typing notifications.
  * Fix bugs where we sent events with invalid signatures due to bugs where
    we incorrectly persisted events.

README.rst

@@ -95,18 +95,33 @@ Installing prerequisites on Ubuntu or Debian::
 
     $ sudo apt-get install build-essential python2.7-dev libffi-dev \
                            python-pip python-setuptools sqlite3 \
-                           libssl-dev
+                           libssl-dev python-virtualenv libjpeg-dev
 
 Installing prerequisites on Mac OS X::
 
     $ xcode-select --install
+    $ sudo pip install virtualenv
     
 To install the synapse homeserver run::
 
-    $ pip install --user --process-dependency-links https://github.com/matrix-org/synapse/tarball/master
+    $ virtualenv ~/.synapse
+    $ source ~/.synapse/bin/activate
+    $ pip install --process-dependency-links https://github.com/matrix-org/synapse/tarball/master
 
-This installs synapse, along with the libraries it uses, into
-``$HOME/.local/lib/`` on Linux or ``$HOME/Library/Python/2.7/lib/`` on OSX.
+This installs synapse, along with the libraries it uses, into a virtual
+environment under ``~/.synapse``.
+
+To set up your homeserver, run (in your virtualenv, as before)::
+
+    $ python -m synapse.app.homeserver \
+        --server-name machine.my.domain.name \
+        --config-path homeserver.yaml \
+        --generate-config
+
+Substituting your host and domain name as appropriate.
+
+For reliable VoIP calls to be routed via this homeserver, you MUST configure
+a TURN server.  See docs/turn-howto.rst for details.
 
 Troubleshooting Installation
 ----------------------------
@@ -116,19 +131,19 @@ you get errors about ``error: no such option: --process-dependency-links`` you
 may need to manually upgrade it::
 
     $ sudo pip install --upgrade pip
-    
+
 If pip crashes mid-installation for reason (e.g. lost terminal), pip may
 refuse to run until you remove the temporary installation directory it
 created. To reset the installation::
 
     $ rm -rf /tmp/pip_install_matrix
-    
+
 pip seems to leak *lots* of memory during installation.  For instance, a Linux 
 host with 512MB of RAM may run out of memory whilst installing Twisted.  If this 
 happens, you will have to individually install the dependencies which are 
 failing, e.g.::
 
-    $ pip install --user twisted
+    $ pip install twisted
 
 On OSX, if you encounter clang: error: unknown argument: '-mno-fused-madd' you
 will need to export CFLAGS=-Qunused-arguments.
@@ -143,7 +158,7 @@ Synapse can be installed on Cygwin. It requires the following Cygwin packages:
  - openssl (and openssl-devel, python-openssl)
  - python
  - python-setuptools
- 
+
 The content repository requires additional packages and will be unable to process
 uploads without them:
  - libjpeg8
@@ -170,23 +185,13 @@ Running Your Homeserver
 To actually run your new homeserver, pick a working directory for Synapse to run 
 (e.g. ``~/.synapse``), and::
 
-    $ mkdir ~/.synapse
     $ cd ~/.synapse
-    
-    $ # on Linux
-    $ ~/.local/bin/synctl start
-    
-    $ # on OSX
-    $ ~/Library/Python/2.7/bin/synctl start
+    $ source ./bin/activate
+    $ synctl start
 
 Troubleshooting Running
 -----------------------
 
-If ``synctl`` fails with ``pkg_resources.DistributionNotFound`` errors you may 
-need a newer version of setuptools than that provided by your OS.::
-
-    $ sudo pip install setuptools --upgrade
-
 If synapse fails with ``missing "sodium.h"`` crypto errors, you may need 
 to manually upgrade PyNaCL, as synapse uses NaCl (http://nacl.cr.yp.to/) for 
 encryption and digital signatures.
@@ -213,13 +218,15 @@ directory of your choice::
     $ cd synapse
 
 The homeserver has a number of external dependencies, that are easiest
-to install by making setup.py do so, in --user mode::
+to install using pip and a virtualenv::
 
-    $ python setup.py develop --user
+    $ virtualenv env
+    $ source env/bin/activate
+    $ python synapse/python_dependencies.py | xargs -n1 pip install
+    $ pip install setuptools_trial mock
 
-This will run a process of downloading and installing into your
-user's .local/lib directory all of the required dependencies that are
-missing.
+This will run a process of downloading and installing all the needed
+dependencies into a virtual env.
 
 Once this is done, you may wish to run the homeserver's unit tests, to
 check that everything is installed as it should be::
@@ -239,6 +246,11 @@ Upgrading an existing homeserver
 IMPORTANT: Before upgrading an existing homeserver to a new version, please
 refer to UPGRADE.rst for any additional instructions.
 
+Otherwise, simply re-install the new codebase over the current one - e.g.
+by ``pip install --process-dependency-links
+https://github.com/matrix-org/synapse/tarball/master``
+if using pip, or by ``git pull`` if running off a git working copy.
+
 
 Setting up Federation
 =====================
@@ -262,9 +274,9 @@ For the first form, simply pass the required hostname (of the machine) as the
 
     $ python -m synapse.app.homeserver \
         --server-name machine.my.domain.name \
-        --config-path homeserver.config \
+        --config-path homeserver.yaml \
         --generate-config
-    $ python -m synapse.app.homeserver --config-path homeserver.config
+    $ python -m synapse.app.homeserver --config-path homeserver.yaml
 
 Alternatively, you can run ``synctl start`` to guide you through the process.
 
@@ -284,9 +296,9 @@ SRV record, as that is the name other machines will expect it to have::
     $ python -m synapse.app.homeserver \
         --server-name YOURDOMAIN \
         --bind-port 8448 \
-        --config-path homeserver.config \
+        --config-path homeserver.yaml \
         --generate-config
-    $ python -m synapse.app.homeserver --config-path homeserver.config
+    $ python -m synapse.app.homeserver --config-path homeserver.yaml
 
 
 You may additionally want to pass one or more "-v" options, in order to

VERSION

@@ -1 +1 @@
-0.6.0
+0.6.1e

docs/media_repository.rst

@@ -1,6 +1,8 @@
-Media Repository
+Media Repository 
 ================
 
+*Synapse implementation-specific details for the media repository*
+
 The media repository is where attachments and avatar photos are stored.
 It stores attachment content and thumbnails for media uploaded by local users.
 It caches attachment content and thumbnails for media uploaded by remote users.

scripts/copyrighter.pl

@@ -14,7 +14,7 @@
 # limitations under the License.
 
 $copyright = <<EOT;
-# Copyright 2014 OpenMarket Ltd
+# Copyright 2015 OpenMarket Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.

setup.py

@@ -32,8 +32,8 @@ setup(
     description="Reference Synapse Home Server",
     install_requires=[
         "syutil==0.0.2",
-        "matrix_angular_sdk==0.6.0",
-        "Twisted>=14.0.0",
+        "matrix_angular_sdk>=0.6.1",
+        "Twisted==14.0.2",
         "service_identity>=1.0.0",
         "pyopenssl>=0.14",
         "pyyaml",
@@ -47,9 +47,10 @@ setup(
     dependency_links=[
         "https://github.com/matrix-org/syutil/tarball/v0.0.2#egg=syutil-0.0.2",
         "https://github.com/pyca/pynacl/tarball/d4d3175589b892f6ea7c22f466e0e223853516fa#egg=pynacl-0.3.0",
-        "https://github.com/matrix-org/matrix-angular-sdk/tarball/v0.6.0/#egg=matrix_angular_sdk-0.6.0",
+        "https://github.com/matrix-org/matrix-angular-sdk/tarball/v0.6.1/#egg=matrix_angular_sdk-0.6.1",
     ],
     setup_requires=[
+        "Twisted==14.0.2", # Here to override setuptools_trial's dependency on Twisted>=2.4.0
         "setuptools_trial",
         "setuptools>=1.0.0", # Needs setuptools that supports git+ssh.
                              # TODO: Do we need this now? we don't use git+ssh.

synapse/__init__.py

@@ -1,5 +1,5 @@
 # -*- coding: utf-8 -*-
-# Copyright 2014 OpenMarket Ltd
+# Copyright 2014, 2015 OpenMarket Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -16,4 +16,4 @@
 """ This is a reference implementation of a synapse home server.
 """
 
-__version__ = "0.6.0"
+__version__ = "0.6.1e"

synapse/api/__init__.py

@@ -1,5 +1,5 @@
 # -*- coding: utf-8 -*-
-# Copyright 2014 OpenMarket Ltd
+# Copyright 2014, 2015 OpenMarket Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.

synapse/api/auth.py

@@ -1,5 +1,5 @@
 # -*- coding: utf-8 -*-
-# Copyright 2014 OpenMarket Ltd
+# Copyright 2014, 2015 OpenMarket Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.

synapse/api/constants.py

@@ -1,5 +1,5 @@
 # -*- coding: utf-8 -*-
-# Copyright 2014 OpenMarket Ltd
+# Copyright 2014, 2015 OpenMarket Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.

synapse/api/errors.py

@@ -1,5 +1,5 @@
 # -*- coding: utf-8 -*-
-# Copyright 2014 OpenMarket Ltd
+# Copyright 2014, 2015 OpenMarket Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.

synapse/api/ratelimiting.py

@@ -1,4 +1,4 @@
-# Copyright 2014 OpenMarket Ltd
+# Copyright 2014, 2015 OpenMarket Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.

synapse/api/urls.py

@@ -1,5 +1,5 @@
 # -*- coding: utf-8 -*-
-# Copyright 2014 OpenMarket Ltd
+# Copyright 2014, 2015 OpenMarket Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.

synapse/app/__init__.py

@@ -1,5 +1,5 @@
 # -*- coding: utf-8 -*-
-# Copyright 2014 OpenMarket Ltd
+# Copyright 2014, 2015 OpenMarket Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.

synapse/app/homeserver.py

@@ -1,6 +1,6 @@
 #!/usr/bin/env python
 # -*- coding: utf-8 -*-
-# Copyright 2014 OpenMarket Ltd
+# Copyright 2014, 2015 OpenMarket Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -234,7 +234,7 @@ def setup():
     except UpgradeDatabaseException:
         sys.stderr.write(
             "\nFailed to upgrade database.\n"
-            "Have you followed any instructions in UPGRADES.rst?\n"
+            "Have you checked for version specific instructions in UPGRADES.rst?\n"
         )
         sys.exit(1)
 

synapse/app/synctl.py

@@ -1,6 +1,6 @@
 #!/usr/bin/env python
 # -*- coding: utf-8 -*-
-# Copyright 2014 OpenMarket Ltd
+# Copyright 2014, 2015 OpenMarket Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.

synapse/config/__init__.py

@@ -1,5 +1,5 @@
 # -*- coding: utf-8 -*-
-# Copyright 2014 OpenMarket Ltd
+# Copyright 2014, 2015 OpenMarket Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.

synapse/config/_base.py

@@ -1,5 +1,5 @@
 # -*- coding: utf-8 -*-
-# Copyright 2014 OpenMarket Ltd
+# Copyright 2014, 2015 OpenMarket Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.

synapse/config/captcha.py

@@ -1,4 +1,4 @@
-# Copyright 2014 OpenMarket Ltd
+# Copyright 2014, 2015 OpenMarket Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.

synapse/config/database.py

@@ -1,5 +1,5 @@
 # -*- coding: utf-8 -*-
-# Copyright 2014 OpenMarket Ltd
+# Copyright 2014, 2015 OpenMarket Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.

synapse/config/email.py

@@ -1,5 +1,5 @@
 # -*- coding: utf-8 -*-
-# Copyright 2014 OpenMarket Ltd
+# Copyright 2014, 2015 OpenMarket Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.

synapse/config/homeserver.py

@@ -1,5 +1,5 @@
 # -*- coding: utf-8 -*-
-# Copyright 2014 OpenMarket Ltd
+# Copyright 2014, 2015 OpenMarket Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.

synapse/config/logger.py

@@ -1,5 +1,5 @@
 # -*- coding: utf-8 -*-
-# Copyright 2014 OpenMarket Ltd
+# Copyright 2014, 2015 OpenMarket Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.

synapse/config/ratelimiting.py

@@ -1,4 +1,4 @@
-# Copyright 2014 OpenMarket Ltd
+# Copyright 2014, 2015 OpenMarket Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.

synapse/config/repository.py

@@ -1,5 +1,5 @@
 # -*- coding: utf-8 -*-
-# Copyright 2014 matrix.org
+# Copyright 2014, 2015 matrix.org
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -37,7 +37,7 @@ class ContentRepositoryConfig(Config):
         super(ContentRepositoryConfig, cls).add_arguments(parser)
         db_group = parser.add_argument_group("content_repository")
         db_group.add_argument(
-            "--max-upload-size", default="1M"
+            "--max-upload-size", default="10M"
         )
         db_group.add_argument(
             "--media-store-path", default=cls.default_path("media_store")

synapse/config/server.py

@@ -1,5 +1,5 @@
 # -*- coding: utf-8 -*-
-# Copyright 2014 OpenMarket Ltd
+# Copyright 2014, 2015 OpenMarket Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.

synapse/config/tls.py

@@ -1,5 +1,5 @@
 # -*- coding: utf-8 -*-
-# Copyright 2014 OpenMarket Ltd
+# Copyright 2014, 2015 OpenMarket Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.

synapse/config/voip.py

@@ -1,4 +1,4 @@
-# Copyright 2014 OpenMarket Ltd
+# Copyright 2014, 2015 OpenMarket Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.

synapse/crypto/__init__.py

@@ -1,5 +1,5 @@
 # -*- coding: utf-8 -*-
-# Copyright 2014 OpenMarket Ltd
+# Copyright 2014, 2015 OpenMarket Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.

synapse/crypto/context_factory.py

@@ -1,4 +1,4 @@
-# Copyright 2014 OpenMarket Ltd
+# Copyright 2014, 2015 OpenMarket Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.

synapse/crypto/event_signing.py

@@ -1,6 +1,6 @@
 # -*- coding: utf-8 -*-
 
-# Copyright 2014 OpenMarket Ltd
+# Copyright 2014, 2015 OpenMarket Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.

synapse/crypto/keyclient.py

@@ -1,5 +1,5 @@
 # -*- coding: utf-8 -*-
-# Copyright 2014 OpenMarket Ltd
+# Copyright 2014, 2015 OpenMarket Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.

synapse/crypto/keyring.py

@@ -1,5 +1,5 @@
 # -*- coding: utf-8 -*-
-# Copyright 2014 OpenMarket Ltd
+# Copyright 2014, 2015 OpenMarket Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.

synapse/events/__init__.py

@@ -1,5 +1,5 @@
 # -*- coding: utf-8 -*-
-# Copyright 2014 OpenMarket Ltd
+# Copyright 2014, 2015 OpenMarket Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -15,12 +15,10 @@
 
 from synapse.util.frozenutils import freeze, unfreeze
 
-import copy
-
 
 class _EventInternalMetadata(object):
     def __init__(self, internal_metadata_dict):
-        self.__dict__ = copy.deepcopy(internal_metadata_dict)
+        self.__dict__ = internal_metadata_dict
 
     def get_dict(self):
         return dict(self.__dict__)
@@ -49,10 +47,10 @@ def _event_dict_property(key):
 class EventBase(object):
     def __init__(self, event_dict, signatures={}, unsigned={},
                  internal_metadata_dict={}):
-        self.signatures = copy.deepcopy(signatures)
-        self.unsigned = copy.deepcopy(unsigned)
+        self.signatures = signatures
+        self.unsigned = unsigned
 
-        self._event_dict = copy.deepcopy(event_dict)
+        self._event_dict = event_dict
 
         self.internal_metadata = _EventInternalMetadata(
             internal_metadata_dict
@@ -112,10 +110,16 @@ class EventBase(object):
 
 class FrozenEvent(EventBase):
     def __init__(self, event_dict, internal_metadata_dict={}):
-        event_dict = copy.deepcopy(event_dict)
+        event_dict = dict(event_dict)
 
-        signatures = copy.deepcopy(event_dict.pop("signatures", {}))
-        unsigned = copy.deepcopy(event_dict.pop("unsigned", {}))
+        # Signatures is a dict of dicts, and this is faster than doing a
+        # copy.deepcopy
+        signatures = {
+            name: {sig_id: sig for sig_id, sig in sigs.items()}
+            for name, sigs in event_dict.pop("signatures", {}).items()
+        }
+
+        unsigned = dict(event_dict.pop("unsigned", {}))
 
         frozen_dict = freeze(event_dict)
 

synapse/events/builder.py

@@ -1,5 +1,5 @@
 # -*- coding: utf-8 -*-
-# Copyright 2014 OpenMarket Ltd
+# Copyright 2014, 2015 OpenMarket Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.

synapse/events/snapshot.py

@@ -1,5 +1,5 @@
 # -*- coding: utf-8 -*-
-# Copyright 2014 OpenMarket Ltd
+# Copyright 2014, 2015 OpenMarket Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.

synapse/events/utils.py

@@ -1,5 +1,5 @@
 # -*- coding: utf-8 -*-
-# Copyright 2014 OpenMarket Ltd
+# Copyright 2014, 2015 OpenMarket Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -98,7 +98,7 @@ def serialize_event(hs, e):
     d = {k: v for k, v in e.get_dict().items()}
     if "age_ts" in d["unsigned"]:
         now = int(hs.get_clock().time_msec())
-        d["unsigned"]["age"] = now - d["unsigned"]["age_ts"]
+        d["age"] = now - d["unsigned"]["age_ts"]
         del d["unsigned"]["age_ts"]
 
     d["user_id"] = d.pop("sender", None)

synapse/events/validator.py

@@ -1,5 +1,5 @@
 # -*- coding: utf-8 -*-
-# Copyright 2014 OpenMarket Ltd
+# Copyright 2014, 2015 OpenMarket Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.

synapse/federation/__init__.py

@@ -1,5 +1,5 @@
 # -*- coding: utf-8 -*-
-# Copyright 2014 OpenMarket Ltd
+# Copyright 2014, 2015 OpenMarket Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.

synapse/federation/persistence.py

@@ -1,5 +1,5 @@
 # -*- coding: utf-8 -*-
-# Copyright 2014 OpenMarket Ltd
+# Copyright 2014, 2015 OpenMarket Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.

synapse/federation/replication.py

@@ -1,5 +1,5 @@
 # -*- coding: utf-8 -*-
-# Copyright 2014 OpenMarket Ltd
+# Copyright 2014, 2015 OpenMarket Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -730,6 +730,7 @@ class _TransactionQueue(object):
 
         destinations = set(destinations)
         destinations.discard(self.server_name)
+        destinations.discard("localhost")
 
         logger.debug("Sending to: %s", str(destinations))
 
@@ -814,6 +815,8 @@ class _TransactionQueue(object):
             else:
                 logger.info("TX [%s] is ready for retry", destination)
 
+        logger.info("TX [%s] _attempt_new_transaction", destination)
+        
         if destination in self.pending_transactions:
             # XXX: pending_transactions can get stuck on by a never-ending
             # request at which point pending_pdus_by_dest just keeps growing.
@@ -826,6 +829,9 @@ class _TransactionQueue(object):
         pending_edus = self.pending_edus_by_dest.pop(destination, [])
         pending_failures = self.pending_failures_by_dest.pop(destination, [])
 
+        if pending_pdus:
+            logger.info("TX [%s] len(pending_pdus_by_dest[dest]) = %d", destination, len(pending_pdus))
+
         if not pending_pdus and not pending_edus and not pending_failures:
             return
 

synapse/federation/transport.py

@@ -1,5 +1,5 @@
 # -*- coding: utf-8 -*-
-# Copyright 2014 OpenMarket Ltd
+# Copyright 2014, 2015 OpenMarket Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.

synapse/federation/units.py

@@ -1,5 +1,5 @@
 # -*- coding: utf-8 -*-
-# Copyright 2014 OpenMarket Ltd
+# Copyright 2014, 2015 OpenMarket Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.

synapse/handlers/__init__.py

@@ -1,5 +1,5 @@
 # -*- coding: utf-8 -*-
-# Copyright 2014 OpenMarket Ltd
+# Copyright 2014, 2015 OpenMarket Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.

synapse/handlers/_base.py

@@ -1,5 +1,5 @@
 # -*- coding: utf-8 -*-
-# Copyright 2014 OpenMarket Ltd
+# Copyright 2014, 2015 OpenMarket Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.

synapse/handlers/admin.py

@@ -1,5 +1,5 @@
 # -*- coding: utf-8 -*-
-# Copyright 2014 OpenMarket Ltd
+# Copyright 2014, 2015 OpenMarket Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.

synapse/handlers/directory.py

@@ -1,5 +1,5 @@
 # -*- coding: utf-8 -*-
-# Copyright 2014 OpenMarket Ltd
+# Copyright 2014, 2015 OpenMarket Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -155,4 +155,4 @@ class DirectoryHandler(BaseHandler):
             "room_id": room_id,
             "sender": user_id,
             "content": {"aliases": aliases},
-        })
+        }, ratelimit=False)

synapse/handlers/events.py

@@ -1,5 +1,5 @@
 # -*- coding: utf-8 -*-
-# Copyright 2014 OpenMarket Ltd
+# Copyright 2014, 2015 OpenMarket Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.

synapse/handlers/federation.py

@@ -1,5 +1,5 @@
 # -*- coding: utf-8 -*-
-# Copyright 2014 OpenMarket Ltd
+# Copyright 2014, 2015 OpenMarket Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.

synapse/handlers/login.py

@@ -1,5 +1,5 @@
 # -*- coding: utf-8 -*-
-# Copyright 2014 OpenMarket Ltd
+# Copyright 2014, 2015 OpenMarket Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.

synapse/handlers/message.py

@@ -1,5 +1,5 @@
 # -*- coding: utf-8 -*-
-# Copyright 2014 OpenMarket Ltd
+# Copyright 2014, 2015 OpenMarket Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -19,6 +19,7 @@ from synapse.api.constants import EventTypes, Membership
 from synapse.api.errors import RoomError
 from synapse.streams.config import PaginationConfig
 from synapse.events.validator import EventValidator
+from synapse.util.logcontext import PreserveLoggingContext
 
 from ._base import BaseHandler
 
@@ -106,7 +107,7 @@ class MessageHandler(BaseHandler):
         defer.returnValue(chunk)
 
     @defer.inlineCallbacks
-    def create_and_send_event(self, event_dict):
+    def create_and_send_event(self, event_dict, ratelimit=True):
         """ Given a dict from a client, create and handle a new event.
 
         Creates an FrozenEvent object, filling out auth_events, prev_events,
@@ -123,7 +124,8 @@ class MessageHandler(BaseHandler):
 
         self.validator.validate_new(builder)
 
-        self.ratelimit(builder.user_id)
+        if ratelimit:
+            self.ratelimit(builder.user_id)
         # TODO(paul): Why does 'event' not have a 'user' object?
         user = self.hs.parse_userid(builder.user_id)
         assert self.hs.is_mine(user), "User must be our own: %s" % (user,)
@@ -152,6 +154,11 @@ class MessageHandler(BaseHandler):
                 context=context,
             )
 
+        if event.type == EventTypes.Message:
+            presence = self.hs.get_handlers().presence_handler
+            with PreserveLoggingContext():
+                presence.bump_presence_active_time(user)
+
         defer.returnValue(event)
 
     @defer.inlineCallbacks
@@ -256,7 +263,7 @@ class MessageHandler(BaseHandler):
             }
 
             if event.membership == Membership.INVITE:
-                d["inviter"] = event.user_id
+                d["inviter"] = event.sender
 
             rooms_ret.append(d)
 

synapse/handlers/presence.py

@@ -1,5 +1,5 @@
 # -*- coding: utf-8 -*-
-# Copyright 2014 OpenMarket Ltd
+# Copyright 2014, 2015 OpenMarket Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.

synapse/handlers/profile.py

@@ -1,5 +1,5 @@
 # -*- coding: utf-8 -*-
-# Copyright 2014 OpenMarket Ltd
+# Copyright 2014, 2015 OpenMarket Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -16,7 +16,7 @@
 from twisted.internet import defer
 
 from synapse.api.errors import SynapseError, AuthError, CodeMessageException
-from synapse.api.constants import Membership
+from synapse.api.constants import EventTypes, Membership
 from synapse.util.logcontext import PreserveLoggingContext
 
 from ._base import BaseHandler
@@ -194,6 +194,8 @@ class ProfileHandler(BaseHandler):
         if not self.hs.is_mine(user):
             return
 
+        self.ratelimit(user.to_string())
+
         joins = yield self.store.get_rooms_for_user_where_membership_is(
             user.to_string(),
             [Membership.JOIN],
@@ -201,7 +203,7 @@ class ProfileHandler(BaseHandler):
 
         for j in joins:
             content = {
-                "membership": j.content["membership"],
+                "membership": Membership.JOIN,
             }
 
             yield self.distributor.fire(
@@ -210,9 +212,9 @@ class ProfileHandler(BaseHandler):
 
             msg_handler = self.hs.get_handlers().message_handler
             yield msg_handler.create_and_send_event({
-                "type": j.type,
+                "type": EventTypes.Member,
                 "room_id": j.room_id,
-                "state_key": j.state_key,
+                "state_key": user.to_string(),
                 "content": content,
-                "sender": j.state_key,
-            })
+                "sender": user.to_string()
+            }, ratelimit=False)

synapse/handlers/register.py

@@ -1,5 +1,5 @@
 # -*- coding: utf-8 -*-
-# Copyright 2014 OpenMarket Ltd
+# Copyright 2014, 2015 OpenMarket Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -163,7 +163,7 @@ class RegistrationHandler(BaseHandler):
         # each request
         httpCli = SimpleHttpClient(self.hs)
         # XXX: make this configurable!
-        trustedIdServers = ['matrix.org:8090']
+        trustedIdServers = ['matrix.org:8090', 'matrix.org']
         if not creds['idServer'] in trustedIdServers:
             logger.warn('%s is not a trusted ID server: rejecting 3pid ' +
                         'credentials', creds['idServer'])

synapse/handlers/room.py

@@ -1,5 +1,5 @@
 # -*- coding: utf-8 -*-
-# Copyright 2014 OpenMarket Ltd
+# Copyright 2014, 2015 OpenMarket Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -130,6 +130,7 @@ class RoomCreationHandler(BaseHandler):
                 "type": EventTypes.Name,
                 "room_id": room_id,
                 "sender": user_id,
+                "state_key": "",
                 "content": {"name": name},
             })
 
@@ -139,6 +140,7 @@ class RoomCreationHandler(BaseHandler):
                 "type": EventTypes.Topic,
                 "room_id": room_id,
                 "sender": user_id,
+                "state_key": "",
                 "content": {"topic": topic},
             })
 
@@ -243,14 +245,12 @@ class RoomMemberHandler(BaseHandler):
         self.distributor.declare("user_left_room")
 
     @defer.inlineCallbacks
-    def get_room_members(self, room_id, membership=Membership.JOIN):
+    def get_room_members(self, room_id):
         hs = self.hs
 
-        memberships = yield self.store.get_room_members(
-            room_id=room_id, membership=membership
-        )
+        users = yield self.store.get_users_in_room(room_id)
 
-        defer.returnValue([hs.parse_userid(m.user_id) for m in memberships])
+        defer.returnValue([hs.parse_userid(u) for u in users])
 
     @defer.inlineCallbacks
     def fetch_room_distributions_into(self, room_id, localusers=None,
@@ -529,11 +529,10 @@ class RoomListHandler(BaseHandler):
     def get_public_room_list(self):
         chunk = yield self.store.get_rooms(is_public=True)
         for room in chunk:
-            joined_members = yield self.store.get_room_members(
+            joined_users = yield self.store.get_users_in_room(
                 room_id=room["room_id"],
-                membership=Membership.JOIN
             )
-            room["num_joined_members"] = len(joined_members)
+            room["num_joined_members"] = len(joined_users)
         # FIXME (erikj): START is no longer a valid value
         defer.returnValue({"start": "START", "end": "END", "chunk": chunk})
 

synapse/handlers/typing.py

@@ -1,5 +1,5 @@
 # -*- coding: utf-8 -*-
-# Copyright 2014 OpenMarket Ltd
+# Copyright 2014, 2015 OpenMarket Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.

synapse/http/__init__.py

@@ -1,5 +1,5 @@
 # -*- coding: utf-8 -*-
-# Copyright 2014 OpenMarket Ltd
+# Copyright 2014, 2015 OpenMarket Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.

synapse/http/agent_name.py

@@ -0,0 +1,18 @@
+# -*- coding: utf-8 -*-
+# Copyright 2014, 2015 OpenMarket Ltd
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from synapse import __version__
+
+AGENT_NAME = ("Synapse/%s" % (__version__,)).encode("ascii")

synapse/http/client.py

@@ -1,5 +1,5 @@
 # -*- coding: utf-8 -*-
-# Copyright 2014 OpenMarket Ltd
+# Copyright 2014, 2015 OpenMarket Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -14,6 +14,7 @@
 # limitations under the License.
 
 
+from synapse.http.agent_name import AGENT_NAME
 from twisted.internet import defer, reactor
 from twisted.web.client import (
     Agent, readBody, FileBodyProducer, PartialDownloadError
@@ -51,7 +52,8 @@ class SimpleHttpClient(object):
             "POST",
             uri.encode("ascii"),
             headers=Headers({
-                "Content-Type": ["application/x-www-form-urlencoded"]
+                b"Content-Type": [b"application/x-www-form-urlencoded"],
+                b"User-Agent": [AGENT_NAME],
             }),
             bodyProducer=FileBodyProducer(StringIO(query_bytes))
         )
@@ -86,6 +88,9 @@ class SimpleHttpClient(object):
         response = yield self.agent.request(
             "GET",
             uri.encode("ascii"),
+            headers=Headers({
+                b"User-Agent": [AGENT_NAME],
+            })
         )
 
         body = yield readBody(response)
@@ -108,7 +113,8 @@ class CaptchaServerHttpClient(SimpleHttpClient):
             url.encode("ascii"),
             bodyProducer=FileBodyProducer(StringIO(query_bytes)),
             headers=Headers({
-                "Content-Type": ["application/x-www-form-urlencoded"]
+                b"Content-Type": [b"application/x-www-form-urlencoded"],
+                b"User-Agent": [AGENT_NAME],
             })
         )
 

synapse/http/endpoint.py

@@ -1,5 +1,5 @@
 # -*- coding: utf-8 -*-
-# Copyright 2014 OpenMarket Ltd
+# Copyright 2014, 2015 OpenMarket Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.

synapse/http/matrixfederationclient.py

@@ -1,5 +1,5 @@
 # -*- coding: utf-8 -*-
-# Copyright 2014 OpenMarket Ltd
+# Copyright 2014, 2015 OpenMarket Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -20,6 +20,7 @@ from twisted.web.client import readBody, _AgentBase, _URI
 from twisted.web.http_headers import Headers
 from twisted.web._newclient import ResponseDone
 
+from synapse.http.agent_name import AGENT_NAME
 from synapse.http.endpoint import matrix_federation_endpoint
 from synapse.util.async import sleep
 from synapse.util.logcontext import PreserveLoggingContext
@@ -71,6 +72,7 @@ class MatrixFederationHttpClient(object):
             requests.
     """
 
+
     def __init__(self, hs):
         self.hs = hs
         self.signing_key = hs.config.signing_key[0]
@@ -83,7 +85,7 @@ class MatrixFederationHttpClient(object):
                         query_bytes=b"", retry_on_dns_fail=True):
         """ Creates and sends a request to the given url
         """
-        headers_dict[b"User-Agent"] = [b"Synapse"]
+        headers_dict[b"User-Agent"] = [AGENT_NAME]
         headers_dict[b"Host"] = [destination]
 
         url_bytes = urlparse.urlunparse(

synapse/http/server.py

@@ -1,5 +1,5 @@
 # -*- coding: utf-8 -*-
-# Copyright 2014 OpenMarket Ltd
+# Copyright 2014, 2015 OpenMarket Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -14,14 +14,16 @@
 # limitations under the License.
 
 
-from syutil.jsonutil import (
-    encode_canonical_json, encode_pretty_printed_json
-)
+from synapse.http.agent_name import AGENT_NAME
 from synapse.api.errors import (
     cs_exception, SynapseError, CodeMessageException
 )
 from synapse.util.logcontext import LoggingContext
 
+from syutil.jsonutil import (
+    encode_canonical_json, encode_pretty_printed_json
+)
+
 from twisted.internet import defer, reactor
 from twisted.web import server, resource
 from twisted.web.server import NOT_DONE_YET
@@ -230,6 +232,8 @@ def respond_with_json_bytes(request, code, json_bytes, send_cors=False,
 
     request.setResponseCode(code, message=response_code_message)
     request.setHeader(b"Content-Type", b"application/json")
+    request.setHeader(b"Server", AGENT_NAME)
+    request.setHeader(b"Content-Length", b"%d" % (len(json_bytes),))
 
     if send_cors:
         request.setHeader("Access-Control-Allow-Origin", "*")

synapse/http/server_key_resource.py

@@ -1,5 +1,5 @@
 # -*- coding: utf-8 -*-
-# Copyright 2014 OpenMarket Ltd
+# Copyright 2014, 2015 OpenMarket Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.

synapse/media/v0/content_repository.py

@@ -1,5 +1,5 @@
 # -*- coding: utf-8 -*-
-# Copyright 2014 OpenMarket Ltd
+# Copyright 2014, 2015 OpenMarket Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.

synapse/media/v1/__init__.py

@@ -0,0 +1,43 @@
+# -*- coding: utf-8 -*-
+# Copyright 2014, 2015 OpenMarket Ltd
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import PIL.Image
+
+# check for JPEG support.
+try:
+    PIL.Image._getdecoder("rgb", "jpeg", None)
+except IOError as e:
+    if str(e).startswith("decoder jpeg not available"):
+        raise Exception(
+            "FATAL: jpeg codec not supported. Install pillow correctly! "
+            " 'sudo apt-get install libjpeg-dev' then 'pip install -I pillow'"
+        )
+except Exception:
+    # any other exception is fine
+    pass
+
+
+# check for PNG support.
+try:
+    PIL.Image._getdecoder("rgb", "zip", None)
+except IOError as e:
+    if str(e).startswith("decoder zip not available"):
+        raise Exception(
+            "FATAL: zip codec not supported. Install pillow correctly! "
+            " 'sudo apt-get install libjpeg-dev' then 'pip install -I pillow'"
+        )
+except Exception:
+    # any other exception is fine
+    pass

synapse/media/v1/base_resource.py

@@ -1,5 +1,5 @@
 # -*- coding: utf-8 -*-
-# Copyright 2014 OpenMarket Ltd
+# Copyright 2014, 2015 OpenMarket Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -139,6 +139,7 @@ class BaseMediaResource(Resource):
             @download.addBoth
             def callback(media_info):
                 del self.downloads[key]
+                return media_info
         return download
 
     @defer.inlineCallbacks
@@ -201,7 +202,8 @@ class BaseMediaResource(Resource):
         defer.returnValue(media_info)
 
     @defer.inlineCallbacks
-    def _respond_with_file(self, request, media_type, file_path):
+    def _respond_with_file(self, request, media_type, file_path,
+                           file_size=None):
         logger.debug("Responding with %r", file_path)
 
         if os.path.isfile(file_path):
@@ -215,13 +217,20 @@ class BaseMediaResource(Resource):
             request.setHeader(
                 b"Cache-Control", b"public,max-age=86400,s-maxage=86400"
             )
+            if file_size is None:
+                stat = os.stat(file_path)
+                file_size = stat.st_size
+
+            request.setHeader(
+                b"Content-Length", b"%d" % (file_size,)
+            )
 
             with open(file_path, "rb") as f:
                 yield FileSender().beginFileTransfer(f, request)
 
             request.finish()
         else:
-            self._respond_404()
+            self._respond_404(request)
 
     def _get_thumbnail_requirements(self, media_type):
         if media_type == "image/jpeg":

synapse/media/v1/download_resource.py

@@ -1,5 +1,5 @@
 # -*- coding: utf-8 -*-
-# Copyright 2014 OpenMarket Ltd
+# Copyright 2014, 2015 OpenMarket Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -46,23 +46,29 @@ class DownloadResource(BaseMediaResource):
     def _respond_local_file(self, request, media_id):
         media_info = yield self.store.get_local_media(media_id)
         if not media_info:
-            self._respond_404()
+            self._respond_404(request)
             return
 
         media_type = media_info["media_type"]
+        media_length = media_info["media_length"]
         file_path = self.filepaths.local_media_filepath(media_id)
 
-        yield self._respond_with_file(request, media_type, file_path)
+        yield self._respond_with_file(
+            request, media_type, file_path, media_length
+        )
 
     @defer.inlineCallbacks
     def _respond_remote_file(self, request, server_name, media_id):
         media_info = yield self._get_remote_media(server_name, media_id)
 
         media_type = media_info["media_type"]
+        media_length = media_info["media_length"]
         filesystem_id = media_info["filesystem_id"]
 
         file_path = self.filepaths.remote_media_filepath(
             server_name, filesystem_id
         )
 
-        yield self._respond_with_file(request, media_type, file_path)
+        yield self._respond_with_file(
+            request, media_type, file_path, media_length
+        )

synapse/media/v1/filepath.py

@@ -1,5 +1,5 @@
 # -*- coding: utf-8 -*-
-# Copyright 2014 OpenMarket Ltd
+# Copyright 2014, 2015 OpenMarket Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.

synapse/media/v1/media_repository.py

@@ -1,5 +1,5 @@
 # -*- coding: utf-8 -*-
-# Copyright 2014 OpenMarket Ltd
+# Copyright 2014, 2015 OpenMarket Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.

synapse/media/v1/thumbnail_resource.py

@@ -1,5 +1,5 @@
 # -*- coding: utf-8 -*-
-# Copyright 2014 OpenMarket Ltd
+# Copyright 2014, 2015 OpenMarket Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -100,11 +100,12 @@ class ThumbnailResource(BaseMediaResource):
             t_type = thumbnail_info["thumbnail_type"]
             t_method = thumbnail_info["thumbnail_method"]
             file_id = thumbnail_info["filesystem_id"]
+            t_length = thumbnail_info["thumbnail_length"]
 
             file_path = self.filepaths.remote_media_thumbnail(
                 server_name, file_id, t_width, t_height, t_type, t_method,
             )
-            yield self._respond_with_file(request, t_type, file_path)
+            yield self._respond_with_file(request, t_type, file_path, t_length)
         else:
             yield self._respond_default_thumbnail(
                 request, media_info, width, height, method, m_type,
@@ -139,11 +140,12 @@ class ThumbnailResource(BaseMediaResource):
         t_height = thumbnail_info["thumbnail_height"]
         t_type = thumbnail_info["thumbnail_type"]
         t_method = thumbnail_info["thumbnail_method"]
+        t_length = thumbnail_info["thumbnail_length"]
 
         file_path = self.filepaths.default_thumbnail(
             top_level_type, sub_type, t_width, t_height, t_type, t_method,
         )
-        yield self.respond_with_file(request, t_type, file_path)
+        yield self.respond_with_file(request, t_type, file_path, t_length)
 
     def _select_thumbnail(self, desired_width, desired_height, desired_method,
                           desired_type, thumbnail_infos):

synapse/media/v1/thumbnailer.py

@@ -1,5 +1,5 @@
 # -*- coding: utf-8 -*-
-# Copyright 2014 OpenMarket Ltd
+# Copyright 2014, 2015 OpenMarket Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -48,7 +48,7 @@ class Thumbnailer(object):
 
     def scale(self, output_path, width, height, output_type):
         """Rescales the image to the given dimensions"""
-        scaled = self.image.resize((width, height), Image.BILINEAR)
+        scaled = self.image.resize((width, height), Image.ANTIALIAS)
         return self.save_image(scaled, output_type, output_path)
 
     def crop(self, output_path, width, height, output_type):
@@ -65,7 +65,7 @@ class Thumbnailer(object):
         if width * self.height > height * self.width:
             scaled_height = (width * self.height) // self.width
             scaled_image = self.image.resize(
-                (width, scaled_height), Image.BILINEAR
+                (width, scaled_height), Image.ANTIALIAS
             )
             crop_top = (scaled_height - height) // 2
             crop_bottom = height + crop_top
@@ -73,7 +73,7 @@ class Thumbnailer(object):
         else:
             scaled_width = (height * self.width) // self.height
             scaled_image = self.image.resize(
-                (scaled_width, height), Image.BILINEAR
+                (scaled_width, height), Image.ANTIALIAS
             )
             crop_left = (scaled_width - width) // 2
             crop_right = width + crop_left

synapse/media/v1/upload_resource.py

@@ -1,5 +1,5 @@
 # -*- coding: utf-8 -*-
-# Copyright 2014 OpenMarket Ltd
+# Copyright 2014, 2015 OpenMarket Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.

synapse/notifier.py

@@ -1,5 +1,5 @@
 # -*- coding: utf-8 -*-
-# Copyright 2014 OpenMarket Ltd
+# Copyright 2014, 2015 OpenMarket Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.

synapse/python_dependencies.py

@@ -0,0 +1,122 @@
+import logging
+from distutils.version import LooseVersion
+
+logger = logging.getLogger(__name__)
+
+REQUIREMENTS = {
+    "syutil==0.0.2": ["syutil"],
+    "matrix_angular_sdk==0.6.0": ["syweb>=0.6.0"],
+    "Twisted==14.0.2": ["twisted==14.0.2"],
+    "service_identity>=1.0.0": ["service_identity>=1.0.0"],
+    "pyopenssl>=0.14": ["OpenSSL>=0.14"],
+    "pyyaml": ["yaml"],
+    "pyasn1": ["pyasn1"],
+    "pynacl": ["nacl"],
+    "daemonize": ["daemonize"],
+    "py-bcrypt": ["bcrypt"],
+    "frozendict>=0.4": ["frozendict"],
+    "pillow": ["PIL"],
+    "pydenticon": ["pydenticon"],
+}
+
+def github_link(project, version, egg):
+    return "https://github.com/%s/tarball/%s/#egg=%s" % (project, version, egg)
+
+DEPENDENCY_LINKS=[
+    github_link(
+        project="matrix-org/syutil",
+        version="v0.0.2",
+        egg="syutil-0.0.2",
+    ),
+    github_link(
+        project="matrix-org/matrix-angular-sdk",
+        version="v0.6.0",
+        egg="matrix_angular_sdk-0.6.0",
+    ),
+    github_link(
+        project="pyca/pynacl",
+        version="d4d3175589b892f6ea7c22f466e0e223853516fa",
+        egg="pynacl-0.3.0",
+    )
+]
+
+
+class MissingRequirementError(Exception):
+    pass
+
+
+def check_requirements():
+    """Checks that all the modules needed by synapse have been correctly
+    installed and are at the correct version"""
+    for dependency, module_requirements in REQUIREMENTS.items():
+        for module_requirement in module_requirements:
+            if ">=" in module_requirement:
+                module_name, required_version = module_requirement.split(">=")
+                version_test = ">="
+            elif "==" in module_requirement:
+                module_name, required_version = module_requirement.split("==")
+                version_test = "=="
+            else:
+                module_name = module_requirement
+                version_test = None
+
+            try:
+                module = __import__(module_name)
+            except ImportError:
+                logging.exception(
+                    "Can't import %r which is part of %r",
+                    module_name, dependency
+                )
+                raise MissingRequirementError(
+                    "Can't import %r which is part of %r"
+                    % (module_name, dependency)
+                )
+            version = getattr(module, "__version__", None)
+            file_path = getattr(module, "__file__", None)
+            logger.info(
+                "Using %r version %r from %r to satisfy %r",
+                module_name, version, file_path, dependency
+            )
+
+            if version_test == ">=":
+                if version is None:
+                    raise MissingRequirementError(
+                        "Version of %r isn't set as __version__ of module %r"
+                        % (dependency, module_name)
+                    )
+                if LooseVersion(version) < LooseVersion(required_version):
+                    raise MissingRequirementError(
+                        "Version of %r in %r is too old. %r < %r"
+                        % (dependency, file_path, version, required_version)
+                    )
+            elif version_test == "==":
+                if version is None:
+                    raise MissingRequirementError(
+                        "Version of %r isn't set as __version__ of module %r"
+                        % (dependency, module_name)
+                    )
+                if LooseVersion(version) != LooseVersion(required_version):
+                    raise MissingRequirementError(
+                        "Unexpected version of %r in %r. %r != %r"
+                        % (dependency, file_path, version, required_version)
+                    )
+
+def list_requirements():
+    result = []
+    linked = []
+    for link in DEPENDENCY_LINKS:
+        egg = link.split("#egg=")[1]
+        linked.append(egg.split('-')[0])
+        result.append(link)
+    for requirement in REQUIREMENTS:
+        is_linked = False
+        for link in linked:
+            if requirement.replace('-','_').startswith(link):
+                is_linked = True
+        if not is_linked:
+            result.append(requirement)
+    return result
+
+if __name__ == "__main__":
+    import sys
+    sys.stdout.writelines(req + "\n" for req in list_requirements())

synapse/rest/__init__.py

@@ -1,5 +1,5 @@
 # -*- coding: utf-8 -*-
-# Copyright 2014 OpenMarket Ltd
+# Copyright 2014, 2015 OpenMarket Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.

synapse/rest/admin.py

@@ -1,5 +1,5 @@
 # -*- coding: utf-8 -*-
-# Copyright 2014 OpenMarket Ltd
+# Copyright 2014, 2015 OpenMarket Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.

synapse/rest/base.py

@@ -1,5 +1,5 @@
 # -*- coding: utf-8 -*-
-# Copyright 2014 OpenMarket Ltd
+# Copyright 2014, 2015 OpenMarket Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.

synapse/rest/directory.py

@@ -1,5 +1,5 @@
 # -*- coding: utf-8 -*-
-# Copyright 2014 OpenMarket Ltd
+# Copyright 2014, 2015 OpenMarket Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.

synapse/rest/events.py

@@ -1,5 +1,5 @@
 # -*- coding: utf-8 -*-
-# Copyright 2014 OpenMarket Ltd
+# Copyright 2014, 2015 OpenMarket Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.

synapse/rest/initial_sync.py

@@ -1,5 +1,5 @@
 # -*- coding: utf-8 -*-
-# Copyright 2014 OpenMarket Ltd
+# Copyright 2014, 2015 OpenMarket Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.

synapse/rest/login.py

@@ -1,5 +1,5 @@
 # -*- coding: utf-8 -*-
-# Copyright 2014 OpenMarket Ltd
+# Copyright 2014, 2015 OpenMarket Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.

synapse/rest/presence.py

@@ -1,5 +1,5 @@
 # -*- coding: utf-8 -*-
-# Copyright 2014 OpenMarket Ltd
+# Copyright 2014, 2015 OpenMarket Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.

synapse/rest/profile.py

@@ -1,5 +1,5 @@
 # -*- coding: utf-8 -*-
-# Copyright 2014 OpenMarket Ltd
+# Copyright 2014, 2015 OpenMarket Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.

synapse/rest/register.py

@@ -1,5 +1,5 @@
 # -*- coding: utf-8 -*-
-# Copyright 2014 OpenMarket Ltd
+# Copyright 2014, 2015 OpenMarket Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.

synapse/rest/room.py

@@ -1,5 +1,5 @@
 # -*- coding: utf-8 -*-
-# Copyright 2014 OpenMarket Ltd
+# Copyright 2014, 2015 OpenMarket Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.

synapse/rest/transactions.py

@@ -1,5 +1,5 @@
 # -*- coding: utf-8 -*-
-# Copyright 2014 OpenMarket Ltd
+# Copyright 2014, 2015 OpenMarket Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.

synapse/rest/voip.py

@@ -1,5 +1,5 @@
 # -*- coding: utf-8 -*-
-# Copyright 2014 OpenMarket Ltd
+# Copyright 2014, 2015 OpenMarket Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.

synapse/server.py

@@ -1,5 +1,5 @@
 # -*- coding: utf-8 -*-
-# Copyright 2014 OpenMarket Ltd
+# Copyright 2014, 2015 OpenMarket Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.

synapse/state.py

@@ -1,5 +1,5 @@
 # -*- coding: utf-8 -*-
-# Copyright 2014 OpenMarket Ltd
+# Copyright 2014, 2015 OpenMarket Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.

synapse/storage/__init__.py

@@ -1,5 +1,5 @@
 # -*- coding: utf-8 -*-
-# Copyright 2014 OpenMarket Ltd
+# Copyright 2014, 2015 OpenMarket Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -66,7 +66,7 @@ SCHEMAS = [
 
 # Remember to update this number every time an incompatible change is made to
 # database schema files, so the users will be informed on server restarts.
-SCHEMA_VERSION = 10
+SCHEMA_VERSION = 11
 
 
 class _RollbackButIsFineException(Exception):

synapse/storage/_base.py

@@ -1,5 +1,5 @@
 # -*- coding: utf-8 -*-
-# Copyright 2014 OpenMarket Ltd
+# Copyright 2014, 2015 OpenMarket Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -64,7 +64,7 @@ class LoggingTransaction(object):
             # Don't let logging failures stop SQL from working
             pass
 
-        start = time.clock() * 1000
+        start = time.time() * 1000
         try:
             return self.txn.execute(
                 sql, *args, **kwargs
@@ -73,7 +73,7 @@ class LoggingTransaction(object):
                 logger.exception("[SQL FAIL] {%s}", self.name)
                 raise
         finally:
-            end = time.clock() * 1000
+            end = time.time() * 1000
             sql_logger.debug("[SQL time] {%s} %f", self.name, end - start)
 
 
@@ -93,7 +93,7 @@ class SQLBaseStore(object):
         def inner_func(txn, *args, **kwargs):
             with LoggingContext("runInteraction") as context:
                 current_context.copy_to(context)
-                start = time.clock() * 1000
+                start = time.time() * 1000
                 txn_id = SQLBaseStore._TXN_ID
 
                 # We don't really need these to be unique, so lets stop it from
@@ -109,7 +109,7 @@ class SQLBaseStore(object):
                     logger.exception("[TXN FAIL] {%s}", name)
                     raise
                 finally:
-                    end = time.clock() * 1000
+                    end = time.time() * 1000
                     transaction_logger.debug(
                         "[TXN END] {%s} %f",
                         name, end - start
@@ -434,23 +434,31 @@ class SQLBaseStore(object):
 
         return self.runInteraction("_simple_max_id", func)
 
-    def _get_events(self, event_ids):
+    def _get_events(self, event_ids, check_redacted=True,
+                    get_prev_content=False):
         return self.runInteraction(
-            "_get_events", self._get_events_txn, event_ids
+            "_get_events", self._get_events_txn, event_ids,
+            check_redacted=check_redacted, get_prev_content=get_prev_content,
         )
 
-    def _get_events_txn(self, txn, event_ids):
-        events = []
-        for e_id in event_ids:
-            ev = self._get_event_txn(txn, e_id)
+    def _get_events_txn(self, txn, event_ids, check_redacted=True,
+                        get_prev_content=False):
+        if not event_ids:
+            return []
 
-            if ev:
-                events.append(ev)
+        events = [
+            self._get_event_txn(
+                txn, event_id,
+                check_redacted=check_redacted,
+                get_prev_content=get_prev_content
+            )
+            for event_id in event_ids
+        ]
 
-        return events
+        return [e for e in events if e]
 
     def _get_event_txn(self, txn, event_id, check_redacted=True,
-                       get_prev_content=True):
+                       get_prev_content=False):
         sql = (
             "SELECT internal_metadata, json, r.event_id FROM event_json as e "
             "LEFT JOIN redactions as r ON e.event_id = r.redacts "
@@ -467,6 +475,14 @@ class SQLBaseStore(object):
 
         internal_metadata, js, redacted = res
 
+        return self._get_event_from_row_txn(
+            txn, internal_metadata, js, redacted,
+            check_redacted=check_redacted,
+            get_prev_content=get_prev_content,
+        )
+
+    def _get_event_from_row_txn(self, txn, internal_metadata, js, redacted,
+                                check_redacted=True, get_prev_content=False):
         d = json.loads(js)
         internal_metadata = json.loads(internal_metadata)
 

synapse/storage/directory.py

@@ -1,5 +1,5 @@
 # -*- coding: utf-8 -*-
-# Copyright 2014 OpenMarket Ltd
+# Copyright 2014, 2015 OpenMarket Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.

synapse/storage/event_federation.py

@@ -1,5 +1,5 @@
 # -*- coding: utf-8 -*-
-# Copyright 2014 OpenMarket Ltd
+# Copyright 2014, 2015 OpenMarket Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.

synapse/storage/feedback.py

@@ -1,5 +1,5 @@
 # -*- coding: utf-8 -*-
-# Copyright 2014 OpenMarket Ltd
+# Copyright 2014, 2015 OpenMarket Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.

synapse/storage/keys.py

@@ -1,5 +1,5 @@
 # -*- coding: utf-8 -*-
-# Copyright 2014 OpenMarket Ltd
+# Copyright 2014, 2015 OpenMarket Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.

synapse/storage/media_repository.py

@@ -1,5 +1,5 @@
 # -*- coding: utf-8 -*-
-# Copyright 2014 OpenMarket Ltd
+# Copyright 2014, 2015 OpenMarket Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.

synapse/storage/presence.py

@@ -1,5 +1,5 @@
 # -*- coding: utf-8 -*-
-# Copyright 2014 OpenMarket Ltd
+# Copyright 2014, 2015 OpenMarket Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.

synapse/storage/profile.py

@@ -1,5 +1,5 @@
 # -*- coding: utf-8 -*-
-# Copyright 2014 OpenMarket Ltd
+# Copyright 2014, 2015 OpenMarket Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.

synapse/storage/registration.py

@@ -1,5 +1,5 @@
 # -*- coding: utf-8 -*-
-# Copyright 2014 OpenMarket Ltd
+# Copyright 2014, 2015 OpenMarket Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.

synapse/storage/room.py

@@ -1,5 +1,5 @@
 # -*- coding: utf-8 -*-
-# Copyright 2014 OpenMarket Ltd
+# Copyright 2014, 2015 OpenMarket Ltd
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -135,26 +135,26 @@ class RoomStore(SQLBaseStore):
         defer.returnValue(ret)
 
     def _store_room_topic_txn(self, txn, event):
-        if hasattr(event, "topic"):
+        if hasattr(event, "content") and "topic" in event.content:
             self._simple_insert_txn(
                 txn,
                 "topics",
                 {
                     "event_id": event.event_id,
                     "room_id": event.room_id,
-                    "topic": event.topic,
+                    "topic": event.content["topic"],
                 }
             )
 
     def _store_room_name_txn(self, txn, event):
-        if hasattr(event, "name"):
+        if hasattr(event, "content") and "name" in event.content:
             self._simple_insert_txn(
                 txn,
                 "room_names",
                 {
                     "event_id": event.event_id,
                     "room_id": event.room_id,
-                    "name": event.name,
+                    "name": event.content["name"],
                 }
             )