Set up a Reverse Proxy in a Raspberry Pi with Apache

05/03/2023 14:00 digital life

For any reason the concept of a reverse Proxy always sounded like black magic in my head. Maybe I just jumped into programming too early and didn't get to set up one. Now the need knocks my door and I must jump into the fun of understanding it and updating my infrastructure, and the key piece is a Raspbery Pi as a Reverse Proxy.

Why do I need a reverse proxy?

Long story short, I have diverse apps delivering web interfaces in my cluster of Raspberri Pi home hosts and I want to move away from interacting with them via different ports, as I want to deploy some more apps that will bring more webs, including several SSLs. This is the perfect scenario (and excuse) to install a reverse proxy and manage every web app by their host.

The current setup

Let me explain the current infrastructure I have:

The registered domains: what will be seen.

Let's say that I have 3 domains registered. The three of them are 3 hosts that contain one or more web apps:

alderaan.mydomain.com is a dev environment for this blog, a Grav CMS. I try here things before "pushing them to prod". It is hosted in a Raspberry Pi at home
home.mydomain.com is a Nextcloud for personal purposes. It is hosted also in a Raspberry Pi at home.
mymastodon.social is a Mastodon instance meant to handle some amount of users. It is hosted in a droplet in DigitalOcean.

The mymastodon domain has setup the DNS of DigitalOcean, so that everything else is served and managed from there. By now let's park it. The alderaan and home domains have a CNAME pointing to another pair of domains in a dynamic DNS service, I explain this in the section below. This CNAME entries are set up in the DNS service of the registered domains, in my case Directnic.

The dynamic DNS service: the forwarding domains.

I use NoIp.com as a dynamic DNS service. Knowing that the external IPs assigned to the routers are usually not static, it is very handy to have a service that maintains the changing IP under a static name, so that you can always refer to that name and will point to your home, whatever the IP is in this moment. For this reason your computer (or your router) has to update the dynamic DNS service periodically. Nowadays routers and other out-of-the-box solutions already have a feature to keep the IP updated in the dynamic DNS service. The cool thing of NoIp is that they have a client that you can install in your host and does this job anyways. I've been using their Linux dynamic DNS client version for my Raspberries smoothly since ever.

So what do I have here?

myhome.dyn-dns.com which points to the external IP of my home's router

So, one more time, the alderaan.mydomain.com and the home.mydomain.com from the section above point to the myhome.dyn-dns.com from this section via a CNAME entry, and its time the myhome.dyn-dns.com point to my router's external IP.

The router: a port mess

And here is where I had to patch the solutions until now. I am hosting 4 apps in 2 web servers and all of them would love to publish through the default HTTP port 80, plus one of the apps even handles SSL. What I did here is to set up the port forwarding in the following way:

Ports 80 (HTTP) and 443 (HTTPS) forward to Tatooine, the host that allocates the Nextcloud.
Ports 8080, 81 and 82 (all HTTP) forward to Alderaan, the host that allocates the Grav CMS and the other 2 minor apps.

The internal hosts: a Raspberry Pi cluster

And finally we arrive to the Rasberry Pi cluster I have:

Dagobah is the DNS Server.
Alderaan is my dev environment and has an Apache to serve my blog's dev's Grav CMS plus something more.
Tatooine hosts the Nextcloud Docker instance.

The problem

The more web apps I intend to add, the more ports I will use, until a moment that I want to use any android client that wants to communicate to one of these web apps and does not allow me to configure the port to use, or on a web app that requires SSL and the 443 is already taken...

The solution: a reverse proxy

According to the Wikipedia,

a reverse proxy is an application that sits in front of back-end applications and forwards client (e.g. browser) requests to those applications

What we want to do here is to identify the host in the user's request and forward this request to the appropriate machine that serves that web app.

Let's look at this other diagram: Infra with reverse proxy

Domains setup stay untouched

We don't need to change anything in the external part of the setup. The domains keep chaining to finally drive the user into my router.

The router: forward everything to one machine

As we'll see in a minute, the port forwarding now only delivers to one machine which hosts the reverse proxy and will forward to the related host. I set up in the router the following ports to forward:

Port 80: Will be used by mostly all the HTTP requests
Port 443: Will be used by all the HTTPS requests
Port 81 and 82: Will be used by 2 specific HTTP web apps.

The internal hosts: spawning a reverse proxy

Here is where the main change happens:

1. Dagobah becomes also the reverse proxy

It was hosting the DNS Server and it was quite bored. So now has an Apache that will handle the reverse proxy. Also, he's going to be the responsible to handle the SSL, so we'll have to set up a Let's Encrypt certificate in it. In a section below we'll see how to set it all up.

2. Alderaan needs to change the listening port

It was listening the port 8080 (a part of the 81 and 82) because the router had already booked the 80, but now that we have a reverse proxy, it can happily listen for the default 80. We'll see in a section below how to do so.

3. Tatooine does not handle SSL anymore

One of the docker containers in this Nextcloud setup is the Nginx that also works as a reverse proxy, redirecting the HTTP to HTTPS and handling the SSL requests with the installed certificate to then drive the user to the Nextcloud container. This is not needed anymore, as Dagobah will handle all this SSL responsibility.

Technically we should get rid of the Nginx container, but due to my huge experience in Docker and its networking (ehem) I will simply deactivate the SSL handling and just chain the proxies until the request hits the Nextcloud. I will also explain this in a later section.

4. What is Corellia?

At this point, don't pay too much attention to it. It's another Raspberry Pi that has a goal that will be revealed at the end of the article 😉

What actions do we need to do, then?

Now that we see the benefits of having a reverse proxy in place and what is the change in behaviour of our internal hosts, let's go step by step:

Change the port forwarding in the router
Change the port configuration to listen in Alderaan
Install an Apache into Dagobah and set it up as a reverse proxy
Set up the SSL handling in Dagobah
Stop the SSL handling in the Nginx container in Tatooine

1. Change the port forwarding in the router

We start from here so that we can see small changes every time we do an action in the inner parts. We need to be aware that by doing this we'll temporary make unavailable anything that we have behind the router, so be sure that noone is expecting a stable connection right now.

Every router is different, so I won't enter into details in this section. Just make sure that:

The port sharing or port forwarding that is currently set up for alderaan and tatooine is deactivated or removed
Create a new port sharing or port forwarding for the ports 80, 443, 81 and 82 to the host dagobah (remember that I use hostnames, you may want to use here the IP of your future reverse proxy)
Apply the changes in the router. Some need to be restarted.

At this point, all your internal web servers should not be accessible from internet.

2. Change the port configuration to listen in the internal hosts

So I had alderaan listening to the 8080 port, so from the outside I was able to hit that web server by browsing to http://alderaan.mydomain.com:8080. This is because I had the Apache's Virtual Host for alderaan set up like the following. Let's say that the file that defines my Virtual Hosts in alderaan is /etc/apache2/sites-available/010-alderaan.mydomain.com.conf:

Listen 8080
Listen 81
Listen 82
<VirtualHost *:8080>
...
</VirtualHost>
<VirtualHost *:81>
...
</VirtualHost>
<VirtualHost *:82>
...
</VirtualHost>

So what I need to do is:

SSH into the alderaan host:
```
ssh xavier@alderaan
```

Edit the Virtual Host file:

sudo nano /etc/apache2/sites-available/010-alderaan.mydomain.com.conf

Remove the line Listen 8080. The port 80 is listened by default and does not need to be specified.
Change the line <VirtualHost *:8080> to <VirtualHost *:80>, as we want now to react to the requests entering through the port 80.
Save it with ctrl + o and exit with ctrl + x
Restart the apache server
```
sudo systemctl restart apache2
```

At this point won't see any change from outside the router. We could run curl --head http://alderaan/ in any Raspberry Pi from inside the local network and should answer us with a HTTP 200 OK because now alderaan is set up to listen the port 80 (and not 8080 anymore.

3. Setup a Reverse Proxy with Apache

So a reverse proxy is nothing else than a webserver that performs a different role. We'll see a bit below that it is as simple as proxying the request from outbound to the internal host that will handle it. So first of all we should choose which webserver we'll use. I personally feel way more comfortable playing around with Apache than with Nginx. For what I heard, Nginx is more powerful, but for this project Apache will work perfectly good. I took the following article as reference: How to Set Up a Reverse Proxy With Apache

Assumptions

I'm starting from the article Quick DNS server on a Raspberry Pi that I did a month ago. We have a Raspberry Pi behaving as a local DNS server. It has an official Raspberry OS (which is a Raspbian or Debian) and is up and running in my local network.

I will assume the username xavier and the local hostnames dagobah for this host, alderaan for the Raspberry Pi that hosts the Grav CMS and tatooine for the one that hosts the Nextcloud setup. You can replace these hostnames with the IPs in the network.

Install Apache in the Raspberry Pi and have it ready

SSH into dagobah
```
ssh xavier@dagobah
```
Update the repos
```
sudo apt update
```
Install Apache2
```
sudo apt install apache2
```

Enable the Proxy modules

sudo a2enmod proxy
sudo a2enmod proxy_http

Enable the SSL and its related modules
```
sudo a2enmod ssl
sudo a2enmod header
```
Restart the apache server
```
sudo systemctl restart apache2
```

Set up the proxied Virtual Hosts

Create a new file that will hold the apache configuration for the proxied virtual hosts
```
sudo nano /etc/apache2/sites-available/010-reverse-proxy.conf
```
Add the following content (remember that I use hostnames from my local DNS server, update what you need, most likely the hostnames to your IPs and the ServerName values to your domains). When you finish save it with ctrl + o and exit with ctrl + x.

Listen 81
Listen 82

<VirtualHost *:80>
    ServerName home.mydomain.com

    ProxyPreserveHost On
    ProxyPass / http://tatooine/ nocanon
    ProxyPassReverse / http://tatooine/
    ProxyRequests Off
</VirtualHost>
<VirtualHost *:443>
    ServerName home.mydomain.com

    ProxyPreserveHost On
    ProxyPass / http://tatooine:80/ nocanon
    ProxyPassReverse / http://tatooine:80/
    ProxyRequests Off
</VirtualHost>
<VirtualHost *:80>
    ServerName alderaan.mydomain.com

    ProxyPreserveHost On
    ProxyPass / http://alderaan/ nocanon
    ProxyPassReverse / http://alderaan/
    ProxyRequests Off
</VirtualHost>
<VirtualHost *:81>
    ServerName alderaan.mydomain.com

    ProxyPreserveHost On
    ProxyPass / http://alderaan:81/ nocanon
    ProxyPassReverse / http://alderaan:81/
    ProxyRequests Off
</VirtualHost>
<VirtualHost *:82>
    ServerName alderaan.mydomain.com

    ProxyPreserveHost On
    ProxyPass / http://alderaan:82/ nocanon
    ProxyPassReverse / http://alderaan:82/
    ProxyRequests Off
</VirtualHost>

I want to explain some things in this file:

The `Listen` statements

Apache listens by default the port 80 and the port 443, as can be seen in /etc/apache2/ports.conf. For all the rest ports that we want to listen we need to specify them.

The typical `VirtualHost` section

So let's take a typical basic VirtualHost:

<VirtualHost *:80>
    ServerName alderaan.mydomain.com

    ProxyPreserveHost On
    ProxyPass / http://alderaan/ nocanon
    ProxyPassReverse / http://alderaan/
    ProxyRequests Off
</VirtualHost>

The *:80 defines that we'll react on the requests coming from the port 80
The ServerName defines that we'll react only for requests for this host. And this is the key for distributing the requests through hosts.
The ProxyPreserveHost will keep alderaan.mydomain.com as the host in the request. By default Apache replaces it by the host specified in the ProxyPass statement (see more in the oficial doc: ProxyPreserveHost)
The ProxyPass defines that all requests in the given path or deeper (/) will be forwarded to the given url (attending the protocol too!). The nocanon avoids to "canonicalise" the URL which would add some security but it's incompatible with some backends. This directive is very powerful and full of parameters, also used for load balancing, so I recommend to take a look at the official doc: ProxyPass.
The ProxyPassReverse adjusts some related headers on the responses, which is mandatory in our cases to avoid bypassing the reverse proxy. We apply the same configuration as the ProxyPass. Read more in the official doc: ProxyPassReverse.
The ProxyRequests prevents Apache to work as a forward proxy server, which makes the network secure.

There are more directives that we can use here to pick what requests to redirect. One that may be interesting to take a look is the ProxyPassMatch.

The VirtualHosts for a site that supports SSL

The sites that support SSL include some other directives relating to certificates, HTTP/2 and HTST.

To be honest, the certificates part is easier if handled by the certbot process that automatically sets up the directives and redirections for us, and later on we can revisit the VirtualHost definitions to add the rest of our SSL settings. This process is explained in a section below, and we'll return back to this file there too.

Continuing with the Virtual Hosts process

So we added the virtual hosts into the /etc/apache2/sites-available/010-reverse-proxy.conf file. Let's continue:

Enable the "site" we just create in this file:
```
sudo a2ensite 010-reverse-proxy.conf
```
And restart apache again:
```
sudo service apache2 restart
```

At this point we should be able to see alderaan.mydomain.com working in the browser for all supported ports, as all the chain is wired: alderaan.mydomain.com > alderaan.dyn-dns.com > router > reverse proxy in dagobah > webserver in alderaan

What still won't work is the Nextcloud, as we still need to work in the SSL part.

4. Set up the SSL handling in the reverse proxy host

What we want here is:

To install the certificate needed so that we can serve valid HTTPS content
To add some extra security into the connection by forcing HTTP2 and HTST

Install a Let's encrypt certificate

To be honest, this is way more simple than expected. This is all done by a bot that just asks us a couple of things and then it does the modifications needed. Still, there are some steps to do:

As usual, SSH into dagobah
```
ssh xavier@dagobah`
```
Stop the web server. The certbot that we'll use needs to have exclusive access to ports 80 and 443.
```
sudo service apache2 stop
```

Update the system and Install the certbot

sudo apt update && sudo apt install certbot

Also install the module that will perform the change to our Apache automatically
```
sudo apt install python3-certbot-apache
```
Now let's start with the configuration of the certification. The command can be simpler, but these extra parameters just avoid some steps:
```
sudo certbot --agree-tos -d home.mydomain.com --apache
```
There will be some questions that the bot will ask:
- Decide if you want to share your mail for news and updates (up to you)
- Be sure that the domain we're registering matches with the one that it tells
- It asks if we want to redirect the HTTP traffic to HTTPS, we'll say yes. It will make some extra changes into our Virtual Host.

At this point we should have set up the certificate and have some extra changes in our Virtual Hosts file. If we're curious, these are the changes it introduced (focusing now only to home.mydomain.com):

<VirtualHost *:80>
    ServerName home.arnaus.net

    ProxyPreserveHost On
    ProxyPass / http://tatooine/ nocanon
    ProxyPassReverse / http://tatooine/
    ProxyRequests Off

    RewriteEngine on
    RewriteCond %{SERVER_NAME} =home.arnaus.net
    RewriteRule ^ https://%{SERVER_NAME}%{REQUEST_URI} [END,NE,R=permanent]
</VirtualHost>
<VirtualHost *:443>
    ServerName home.arnaus.net

    ProxyPreserveHost On
    ProxyPass / http://tatooine:80/ nocanon
    ProxyPassReverse / http://tatooine:80/
    ProxyRequests Off

    SSLCertificateFile /etc/letsencrypt/live/home.arnaus.net/fullchain.pem
    SSLCertificateKeyFile /etc/letsencrypt/live/home.arnaus.net/privkey.pem
    Include /etc/letsencrypt/options-ssl-apache.conf
</VirtualHost>

We have some Rewrite directives in the Virtual Host for the port 80 that redirects the traffic to the port 443
We have the SSL certification and configuration directives set up in the Virtual Host for the port 443

Just ensure that the apache server is up
```
sudo service apache2 start
```

Looks good!

Adding some extra SSL related security

When I went through the steps to install the Nextcloud docker instance there was a point where I was told to activate the HTTP/2 and the HTST options from the SSL configuration in the Nginx Manager. Here I try to have a configuration that is as identical as possible, so there are some steps in the SSL Virtual Host:

As usual, SSH into dagobah
```
ssh xavier@dagobah`
```
Edit the configuration file for the Apache Virtual Hosts we have in the reverse proxy:
```
sudo nano /etc/apache2/sites-available/010-reverse-proxy.conf
```

Add the following lines to the *:443 Virtual Host:

Protocols h2 http/1.1
Header always set Strict-Transport-Security 'max-age=31536000; includeSubDomains'
RewriteEngine On
RewriteRule ^/\.well-known/carddav https://%{SERVER_NAME}/remote.php/dav/ [R=301,L]
RewriteRule ^/\.well-known/caldav https://%{SERVER_NAME}/remote.php/dav/ [R=301,L]

Save it with ctrl + o and exit with ctrl + x.
Restart the Apache server
```
sudo service apache2 restart
```

What we did here is:

Add the HTTP2 support with the Protocols directive
Ensure that the client is always using SSL with the Header directive
Add support for the CARDDAV and CALDAV routes for the related Nextcloud apps. They were part of a fix that we need to bring to the SSL here as well.

At this point we're almost there, but the Nextcloud Nginx is still willing to support the SSL, but he can't, we're not sending any anymore as dagobah takes care of it.

5. Stop the support for SSL in the Nexcloud's reverse proxy docker

Here this is simpler. We'll do it using the web app that the Nginx Manager brings to us.

Navigate the browser to http://tatooine:81. The web app from the Nginx Manager should appear and ask us for our credentials.
Type in the credentials
In main screen, click on the Proxy Hosts
In the list of Proxy Hosts, in the row of home.mydomain.com, click on the 3 dots at the extrem right side. Pops up a floating menu.
Click in Edit
Click in the tap SSL
Deactivate all options (Force SSL, HTTP/2 suport and HTST enabled)
Click Save

And at this point you should be able to navigate to http://home.mydomain.com and see how it redirects to HTTPS and the Nextcloud screen appears asking you for the login.

"F.ck Yeah"

Next steps

Even right now it just works, there are some things from this infra that does not sound right to me. For example, why do I have to maintain a docker container with a Nginx working as a reverse proxy that actually does nothing but bringing the traffic from the base host Tatooine to the internal Nextcloud container? Couldn't the Nextcloud container receive the traffic directly? This I will need to investigate...

Remember that corellia host? The idea is to be able to migrate that DigitalOcean dropplet to corellia and keep some money, which actually means to install a Mastodon into a Raspberry Pi 4 and wire it with SSL support through the reverse proxy, something that now it's just a matter of minutes 😁 Future infra

Wrapping up

Yes, that was a long article, but I really wanted to log why and what. It took me some time to end up with the infrastructure close to what I envisioned at the beginning, and it was the first time that I set up a reverse proxy. For any reason, that was something I never did in the past, and was kinda big in my head.

I am just happy to have learned that a reverse proxy has not too much secret, that it's easier when it handles the SSL and do not trying to forward the SSL request more internally. And now I feel unblocked for next projects 🚀

Previous Post Next Post