I want to expose different containers on specific URL paths, possibly on different hosts, and nghttpx, from the nghttp2 library by Tatsuhiro Tsujikawa, does this in an intuitive way, and does a lot more besides.
sudo apt-get install nghttp2-proxy
An example configuration is installed in
/etc/nghttpx/nghttpx.conf, which configures nghttpx to listen for cleartext
http on port
localhost:3000 and proxies (e.g. forwards to)
The example includes a second configuration, shown below, that listens on port
3000 on all interfaces, forwarding again to
localhost:80. The default behaviour of a frontend is to listen for both
https traffic, and the latter requires that a certificate & key be specified. If these files cannot be read (e.g. missing either files, or required permissions), then the proxy will not start.
private-key-file=/path/to/server.key certificate-file=/path/to/server.crt frontend=0.0.0.0,3000 backend=127.0.0.1,80
backend entry specifies the IP and port number of the web server that will handle requests. You can specify one or more patterns that must match on the request URL for this particular back-end to match (and therefore receive traffic). So, the line: –
would send requests that match either of the two given patterns (separated by
: colon ) to the back-end at
80 . Note however that nghttpx applies priorities when matching – a pattern that includes a host will match before one that has only a path. Wildcards can be used for hosts and paths, but more specific non-wildcard patterns will match first. Also, longer paths will match before shorter paths. You should check the man page to get a concise description of matching priorities.
The other option that’s likely to be useful is
errorlog-syslog=yes, particularly when getting things up and running. By default, errors are written to /dev/stderr.
So realistically, we need our TLS/SSL keys and so we’ll use Let’s Encrypt via Neil Pang’s excellent acme.sh script. My nghttpx configuration is shown below, and you can see I install my TLS certificate files right beside the reverse-proxy configuration file.
# Define keys for frontend TLS private-key-file=/etc/nghttpx/key.pem certificate-file=/etc/nghttpx/fullchain.pem # Two front ends, one http, the other https frontend=203.0.113.27,80;no-tls frontend=203.0.113.27,443 # Two backends, one for roundcube, and one catch-all backend=10.197.43.155,80;/roundcube/ backend=127.0.0.1,80 # Logs are useful, and we don't need many workers errorlog-syslog=yes workers=1
To solve the problem of issuing certificates for servers in multiple containers, I use standalone mode on port 88, and configure nghttpx to forward all traffic matching ‘/.well-known/’ to this port on localhost.
./acme.sh --issue --standalone -d twitbook.uk -d www.twitbook.uk -d blog.twitbook.uk -d webmail.twitbook.uk --httpport 88
This is quite neat because it’s so simple, and because certificate issue and renewal does not interfere with any existing servers. All that’s required in
nghttpx.conf in order to support the above
issue command: –
The above line says that any request containing
/.well-known/should be handled by port 88 on localhost (the standalone server).
However, where two possible back-ends can match, any rule with a specific host would take priority. Therefore, it’s more complete to include a
backend for each host that will be challenged by the Let’ s Encrypt ACME protocol (as defined when we requested the certificate be issued).
# Each host in the issue command is defined here for completeness. backend=127.0.0.1,88;twitbook.uk/.well-known/ backend=127.0.0.1,88;www.twitbook.uk/.well-known/ backend=127.0.0.1,88;blog.twitbook.uk/.well-known/ backend=127.0.0.1,88;webmail.twitbook.uk/.well-known/
This way, regardless of what other back-ends are configured for the above hosts, anything with the
/.well-known/ path will be handled by the right back-end.
A reload or restart is needed for changes to take effect. I haven’t noticed any way to test a configuration without applying it, so it’s probably worth having a failsafe copy of your configuration file to hand if you’re working on a production server.
Forward real IP address to back ends
With a reverse proxy, the connection to the back end web-server comes from the proxy process, not from the user’s client. Therefore, it’s our proxy server’s IP address that shows up in the logs, which is rarely useful.
To fix this,
nghttpx can add extra headers to declare the original IP address for a request. The configuration options below tell
nghttpx to strip out existing headers (giving us a ‘clean slate’), and to add headers to the request to declare the client’s IP address.
# Add these to /etc/nghttpx/nghttpx.conf strip-incoming-x-forwarded-for=yes strip-incoming-forwarded=yes add-x-forwarded-for=yes add-forwarded=for forwarded-for=ip
Then, we need to modify the
LogFormat definition in
/etc/apache2/apache2.conf for the format named
combined (assuming this is the log format you are using for your site).
If you modify that format string, changing ‘
%h‘ to ‘
%a‘, then Apache will log the address that
nghttpx gives us in the ‘forwarded’ headers. Note that you need to enable
sudo a2enmod remoteip) to use this feature. See the Apache mod_remoteip documentation for more detailed information.
LetsEncrypt for the host and containers.
Using a reverse proxy to direct traffic to containers complicates our certificate handling, since there may be multiple servers of different kinds that need to be kept up to date. The way I chose to address this, as mentioned above, is to use the acme.sh standalone server option when issuing and renewing. This certificate is needed on the host for the proxy, but also for SMTP and IMAP servers.
So, here are the configuration files that define place to copy the certificate files when renewing. Note that there’s no longer a need to apply the certificate to Apache or Nginx – the proxy speaks cleartext to the back-ends.
/etc/dovecot/conf.d/10-ssl.conf: ssl_cert = </etc/ssl/certs/mail.twitbook.uk.crt ssl_key = </etc/ssl/private/mail.twitbook.uk.key /etc/smtpd.conf: pki mail.twitbook.uk certificate "/etc/ssl/certs/mail.twitbook.uk.crt" pki mail.twitbook.uk key "/etc/ssl/private/mail.twitbook.uk.key"
In any case, all installation can be handled by deployhooks that acme.sh offers, or even just a custom script to handle installation via the reloadcmd parameter.
- Man mage for nghttpx – https://nghttp2.org/documentation/nghttpx.1.html
- Handy commands to query certs – https://www.sslshopper.com/article-most-common-openssl-commands.html
- Mini HTTP Guide for Developers – useful summary. Covers headers relevant to a reverse proxy.