Reverse Proxy
The task is to have a secure reverse proxy server for Synapse Matrix Protocol Server to handle external traffic, as to not expose Synapse directly for security purposes. There are several documented FOSS web servers to choose from, but since we had lighttpd for use aside from Apache, and it was undocumented, it became a challenge to undertake, and contribute back to upstream documentation.
While many use nginx as the go to, there is still purpose and utility for other web servers such as lighttpd. lighttpd has very low CPU usage and extremely low memory usage, potentially using less memory than all other web servers, which makes it ideal for embedded or memory constrained environments. While CPU and memory were not the primary factors, they were a consideration, and are selling factors for using lighttpd vs another.
The problem still remained that lighttpd was an undocumented reverse proxy for Synapse, and some things could not be done as they were in other web servers, such as nginx ability to serve up JSON data from configuration, and it is unknown if lighttpd can do the same. Therefore, a novel work around solution was produced using URL rewrite and external JSON files on disk.
Now we cannot just stop there, you can proxy one of two ways, TCP/IP port or Unix file socket, with Unix socket file communication being much faster and lower overhead than port. Therefore, we configured lighttpd to be a reverse proxy for Synapse via Unix file socket communication. The following is an overview of the configuration.
lighttpd configuration
The following lighttpd configuration snippets document Matrix protocol homeserver proxy services over IPv4 and IPv6, with HTTP for certificate renewal and some homeserver location URL endpoints, and with primary communication over secure HTTPS using TLS and ultra-fast socket file communication to Synapse.
Configuration starts with common settings in /etc/lighttpd/lighttpd.conf for including
necessary modules and permissions for lighttpd. There are more settings, but they are command unrelated
to the task at hand, so excluded for brevity.
server.modules = (
"mod_rewrite",
"mod_redirect",
"mod_access",
"mod_setenv",
"mod_openssl",
"mod_proxy",
"mod_accesslog"
)
server.username = "lighttpd"
server.groupname = "lighttpd"
TCP/IP
We set server.use-ipv6 to “disable” and use IPv6 [::] explicitly
below, in order to listen on all incoming IPv6 addresses. If you only want to listen on specific IPv6
addresses, set this to “enable” and specify said addresses below.
server.use-ipv6 = "disable"Common file settings for secure socket layer (SSL), mostly transport layer security (TLS) these days.
ssl.pemfile = "/etc/lighttpd/cert+privkey.pem"
ssl.ca-file = "/etc/lighttpd/fullchain.pem"
Port settings, for both insecure and secure communication, where communication on insecure ports is
redirected to communication on secure, forcing all communication to be secure, which requires a valid
certificate, even for renewals with Let's Encrypt certbot; something to keep in mind.
Otherwise enabling TLS on a IPv4 or IPv6 port is rather straight forward and trivial common settings, with the addition of port 8448 for Matrix protocol Federation with others Matrix homeservers.
# redirect HTTP traffic to HTTPS, same for IPv6 below
$SERVER["socket"] == "0.0.0.0:80" {
url.redirect = (
"" => "https://${url.authority.noport}${url.path}${qsa}"
)
}
$SERVER["socket"] == "0.0.0.0:443" { ssl.engine = "enable" }
$SERVER["socket"] == "0.0.0.0:8448" { ssl.engine = "enable" }
$SERVER["socket"] == "[::]:80" {
url.redirect = (
"" => "https://${url.authority.noport}${url.path}${qsa}"
)
}
$SERVER["socket"] == "[::]:443" { ssl.engine = "enable" }
$SERVER["socket"] == "[::]:8448" { ssl.engine = "enable" }
Unix Socket Communication
There are two ways to reverse proxy traffic to and/from Synapse Matrix protocol server, either using
regular TCP/IP ports, or via ultra-fast Unix File Socket, which eliminates the TCP/IP overhead between
lighttpd and Synapse. This is the most efficient way to reverse proxy, and makes it such that the proxy
server and application are one. The primary URL endpoints used by the Synapse are
/_matrix/, /_synapse/admin/, and /_synapse/client/, all traffic
on those URL endpoints are mapped to Synapse via Unix file socket.
# both lighttpd and synapse need permissions for socket r/w
$HTTP["url"] =~ "(/_matrix|_synapse/admin|/_synapse/client)" {
proxy.balance = "hash"
proxy.server = (
"" => (
"backend-socket" => (
"host" => "/var/lib/synapse/main_public.sock",
"port" => 0
)
)
)
proxy.forwarded = (
"for" => 1,
"proto" => 1,
"host" => 1,
)
}
Usage of Unix file socket does require the socket file to have permissions for both lighttpd and Synapse to have read and write access, which can be accomplished by adding either to the other's group and ensuring the socket is created with the proper group permissions. In this case, lighttpd was added to synapse user group, with the socket file residing under a synapse managed directory.
Protect Admin
The Matrix protocol has an administration API that is available on the following endpoint,
/_synapse/admin/, and rather than allow full access from anywhere, it has been restricted
to only allow access from non-publicly routable, private, internal IPv6 ULA only. This makes
the API access extremely secure, as it requires a valid internal ULA to connect.
# protect admin access IPv6 ULA only
$HTTP["remoteip"] !="fd00::/8" {
$HTTP["url"] =~ "^/_synapse/admin/" {
url.access-deny = ( "" )
}
}
This is optional and not required, but access to the admin API should be restricted in some form.
Delegation of incoming federation traffic
The following is way to serve JSON formatted data from files without the file extension. Some web
servers such as nginx have the ability to serve up data directly from a configuration file, as if it
existed in a file. With lighttpd lacking such, or lacking the knowledge to do such with lighttpd, an
alternative method was derived using URL rewriting. In this case, the data would reside in JSON
formatted datafiles as normal, and those files will be provided as urls without the .json
file extension, and when the URL without file extension is visited, the URL is rewritten to the the
valid server destination that includes the .json file extension, providing the JSON
formatted data.
# serve json files as urls. is there a better way?
url.rewrite-once = (
"^/\.well-known/matrix/client$" => "/.well-known/matrix/client.json",
"^/\.well-known/matrix/server$" => "/.well-known/matrix/server.json"
)
That addresses serving up the data on specific URLs, but does not handle the file MIME type, which is the type of data, something that can be determined via the first few bites or the file extension that was removed. Therefore, we must set the mime type, and we do so after the URL has been rewritten. There is a final entry of access control to allow the content to be accessed from any origin.
# This condition intentionally matches the post-rewrite URLs.
$HTTP["url"] =~ "^/\.well-known/matrix/(client|server)\.json$" {
mimetype.assign = ( ".json" => "application/json" )
setenv.set-response-header = ( "Access-Control-Allow-Origin" => "*" )
}
Completed
Task completed, the JaxLUG's Synapse Matrix Protocol server, matrix.jaxlug.ngo, is online and ready for usage, come join and interact with us on our Matrix server and explore the Matrix universe!
Such contributions were provided to upstream in a Pull Request (PR), in Synapse PR #19875, as an example of how the JaxLUG contributes to FOSS, not just consuming! This documentation addition PR has been merged and will be part of future documentation. Thanks to the JaxLUG, lighttpd is no longer an undocumented reverse proxy for Synapse Matrix protocol server!