The nghttpx Reverse Proxy

I want to expose different containers on specific URL paths, possibly on different hosts, and nghttpx, from the nghttp2 library by Tatsuhiro Tsujikawa, does this in an intuitive way, and does a lot more besides.

    sudo apt-get install nghttp2-proxy

An example configuration is installed in /etc/nghttpx/nghttpx.conf, which configures nghttpx to listen for cleartext http on port localhost:3000 and proxies (e.g. forwards to) localhost:80.

    frontend=127.0.0.1,3000;no-tls
    backend=127.0.0.1,80

The example includes a second configuration, shown below, that listens on port 3000 on all interfaces, forwarding again to localhost:80. The default behaviour of a frontend is to listen for both http and https traffic, and the latter requires that a certificate & key be specified. If these files cannot be read (e.g. missing either files, or required permissions), then the proxy will not start.

    private-key-file=/path/to/server.key
    certificate-file=/path/to/server.crt

    frontend=0.0.0.0,3000
    backend=127.0.0.1,80

The backend entry specifies the IP and port number of the web server that will handle requests. You can specify one or more patterns that must match on the request URL for this particular back-end to match (and therefore receive traffic). So, the line: –

    backend=127.0.0.1,80;www.twitbook.uk/roundcube/:webmail.twitbook.uk

would send requests that match either of the two given patterns (separated by : colon ) to the back-end at 127.0.0.1, port 80 . Note however that nghttpx applies priorities when matching – a pattern that includes a host will match before one that has only a path. Wildcards can be used for hosts and paths, but more specific non-wildcard patterns will match first. Also, longer paths will match before shorter paths. You should check the man page to get a concise description of matching priorities.

The other option that’s likely to be useful is errorlog-syslog=yes, particularly when getting things up and running. By default, errors are written to /dev/stderr.

So realistically, we need our TLS/SSL keys and so we’ll use Let’s Encrypt via Neil Pang’s excellent acme.sh script. My nghttpx configuration is shown below, and you can see I install my TLS certificate files right beside the reverse-proxy configuration file.

   # Define keys for frontend TLS
   private-key-file=/etc/nghttpx/key.pem
   certificate-file=/etc/nghttpx/fullchain.pem

   # Two front ends, one http, the other https
   frontend=203.0.113.27,80;no-tls
   frontend=203.0.113.27,443

   # Two backends, one for roundcube, and one catch-all
   backend=10.197.43.155,80;/roundcube/
   backend=127.0.0.1,80

   # Logs are useful, and we don't need many workers
   errorlog-syslog=yes
   workers=1

To solve the problem of issuing certificates for servers in multiple containers, I use standalone mode on port 88, and configure nghttpx to forward all traffic matching ‘/.well-known/’ to this port on localhost.

./acme.sh --issue --standalone -d twitbook.uk -d www.twitbook.uk -d blog.twitbook.uk -d webmail.twitbook.uk --httpport 88

This is quite neat because it’s so simple, and because certificate issue and renewal does not interfere with any existing servers. All that’s required in nghttpx.conf in order to support the above issue command: –

    backend=127.0.0.1,88;/.well-known/

The above line says that any request containing /.well-known/should be handled by port 88 on localhost (the standalone server).

However, where two possible back-ends can match, any rule with a specific host would take priority. Therefore, it’s more complete to include a backend for each host that will be challenged by the Let’ s Encrypt ACME protocol (as defined when we requested the certificate be issued).

# Each host in the issue command is defined here for completeness.
backend=127.0.0.1,88;twitbook.uk/.well-known/
backend=127.0.0.1,88;www.twitbook.uk/.well-known/
backend=127.0.0.1,88;blog.twitbook.uk/.well-known/
backend=127.0.0.1,88;webmail.twitbook.uk/.well-known/

This way, regardless of what other back-ends are configured for the above hosts, anything with the /.well-known/ path will be handled by the right back-end.

A reload or restart is needed for changes to take effect. I haven’t noticed any way to test a configuration without applying it, so it’s probably worth having a failsafe copy of your configuration file to hand if you’re working on a production server.

Forward real IP address to back ends

With a reverse proxy, the connection to the back end web-server comes from the proxy process, not from the user’s client. Therefore, it’s our proxy server’s IP address that shows up in the logs, which is rarely useful.

To fix this, nghttpx can add extra headers to declare the original IP address for a request. The configuration options below tell nghttpx to strip out existing headers (giving us a ‘clean slate’), and to add headers to the request to declare the client’s IP address.

  # Add these to /etc/nghttpx/nghttpx.conf
  strip-incoming-x-forwarded-for=yes
  strip-incoming-forwarded=yes
  add-x-forwarded-for=yes
  add-forwarded=for
  forwarded-for=ip

Then, we need to modify the LogFormat definition in /etc/apache2/apache2.conf for the format named combined (assuming this is the log format you are using for your site).

If you modify that format string, changing ‘%h‘ to ‘%a‘, then Apache will log the address that nghttpx gives us in the ‘forwarded’ headers. Note that you need to enable mod_remoteip (e.g. sudo a2enmod remoteip) to use this feature. See the Apache mod_remoteip documentation for more detailed information.

LetsEncrypt for the host and containers.

Using a reverse proxy to direct traffic to containers complicates our certificate handling, since there may be multiple servers of different kinds that need to be kept up to date. The way I chose to address this, as mentioned above, is to use the acme.sh standalone server option when issuing and renewing. This certificate is needed on the host for the proxy, but also for SMTP and IMAP servers.

So, here are the configuration files that define place to copy the certificate files when renewing. Note that there’s no longer a need to apply the certificate to Apache or Nginx – the proxy speaks cleartext to the back-ends.

/etc/dovecot/conf.d/10-ssl.conf:
  ssl_cert = </etc/ssl/certs/mail.twitbook.uk.crt
  ssl_key = </etc/ssl/private/mail.twitbook.uk.key

/etc/smtpd.conf:
   pki mail.twitbook.uk certificate "/etc/ssl/certs/mail.twitbook.uk.crt"
   pki mail.twitbook.uk key "/etc/ssl/private/mail.twitbook.uk.key"

In any case, all installation can be handled by deployhooks that acme.sh offers, or even just a custom script to handle installation via the reloadcmd parameter.

References

Leave a Reply

Your email address will not be published. Required fields are marked *