Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Wireguard 3 Ways: Cooking Up Security in a Surveillance State

This repository contains instructions and support files for the HOPE workshop.

Learning Objectives

We break learning objectives into Skills and Concepts.

Skills

By the end of this workshop, participants will be able to:

  • Create Wireguard peers and servers
  • Establish Wireguard networks across NAT boundaries

Concepts

By the end of this workshop, participants will understand:

  • Wireguard tunnel design
  • Wireguard asymmetric cryptography usage
  • Wireguard networking strategies
  • Basic mesh networking principals

Requirements

Technical Requirements

If you want to connect to the pre-built lab environment, you'll need to install Tailscale. We won't use the Tailscale service directly; instead, we will use the very open source tools discussed in the workshop to connect to the lab. So meta!

Instructions for connecting will be provided at the workshop.

This workshop uses Podman for its demonstrations. It can be installed on Linux, Windows or macOS. On Windows, Podman Desktop will require either WSL or Hyper-V to virtualize a small Linux kernel. We will also use Podman Compose, which is a package install for Linux, but Podman Desktop can support this function as well.

The workshop will also use Zellij for easy terminal control of the containers. If not participating with the provided resources, I recommend installing it.

The containers will have several terminal-based text editors installed (Vim, Nano, Helix) to suit your preference. But you will be editing text in the terminal.

Prerequisites

Familiarity with the Linux command line will be extremely helpful in this workshop. As will familiarity with basic networking concepts such as subnets and firewall rules.

Cloud VM

The Recipes provided in this repository are wholly self-contained. You can run them on any platform that supports Podman/Podman Compose. However, for full participation in the workshop, you will need two resources I can't provide for you:

  1. A registered domain name
  2. A cloud-hosted virtual machine

These are not zero-cost, zero-identity activities, and as such are optional. However, the workshop will involve demonstrating how to set up a real Wireguard network for use across the internet, which means public-facing assets. If you intend to fully participate in the workshop, these will be necessary.

I recommend Porkbun or Namecheap for a domain registrar, and Digital Ocean for a VPS provider.

Usage

Here are provided 3 "recipes" for creating Wireguard networks, in increasing complexity—and utility. Each is comprised of Podman containers, networked together. They can be run on any platform that Podman supports, but for the HOPE workshop, a cloud VM will be provided for participants.

Zellij

The recipes provded each have a startup script that performs some housekeeping on the Podman containers. It also launches a helper terminal multiplexer called Zellij. If you aren't using the provided resources, I recommend installing Zellij to make full use of the environment.

Zellij will create terminal windows for all necessary containers, as well as a separate window for interacting with the host itself. Zellij is mouse-sensitive and fairly intuitive—well, for a terminal tool.

The Recipes

I know it says "3 ways" on the box, but actually we need to start with a fundamental dish before moving on to the real recipes.

0. A Simple Mesh

In this recipe, all our peers live on the same network. That could make Wireguard redundant, but it will establish an additional layer of encryption on top of all peer-to-peer communications. This recipe is more about learning the concepts of Wireguard configurations. Each peer connects to every other peer via manual point-to-point connections. This works, but doesn't exactly scale.

1. The Lighthouse

If we want to connect multiple hosts across networks, but we don't want to open inbound ports, we need to use a public server that all our peers can see. This "lighthouse" server will be the one to which all the others connect.

2. The Lighthouse + Subnet Router

What if we want access to an entire home network without wanting to configure Wireguard on all of those devices? That's where "subnet routers" come in. This configuration allows a (Linux) Wireguard host to provide access to its neighbors through the Wireguard tunnel.

3. The Coordinated Mesh

One of the shortcomings of the "lighthouse" model is that the lighthouse becomes a termination point for the Wireguard encrypted tunnel. If we have doubts about our cloud service provider, then any otherwise unencrypted traffic may be sniffed with access to this server. But we need a central point that everyone can see to guarantee connection. How to square this circle?

Tailscale solves this problem by using multiple strategies to show peers how to connect, and then allows the peers to connect directly to each other, eliminating the machine-in-the-middle. And it's all built on top of Wireguard, so you get the same security with much less hassle.

But Tailscale is a cloud service. So suppose you didn't want to entrust your data there either? Luckily, the core of Tailscale is open source—both client daemon and server. Headscale is a way to self-host a Tailscale server and network. The setup for Headscale is considerably easier than raw Wireguard.

How to Use This Repo

The source of this repo is fully readable here, but the online "book" version for easy reading is online at https://book.wg3w.app.

Read along and follow the provide instructions to participate in the workshop!

Instructions

See the docs folder for the instructions. Or click next if you're reading the book!

Made with 💜 by The Taggart Institute

Why...reguard?

Why do this at all? Why bother setting up a custom private network when there are so many services that will do it for you? And why choose Wireguard when other turnkey technologies like OpenVPN have been around for so long?

Let's get into it.

Trust and Risk

If you're reading this, odds are you have a slightly different perspective on "the cloud" than most of the population. After all, it's just someone else's computer, right? Whenever you entrust your data to a third-party service, you are accepting some degree of risk to your data's confidentiality, integrity, and availability in that transaction. The service could go down due to demand, misconfiguration, or equipment failure (availability). Something could go catastrophically wrong with their infrastructure, leading to corrupting of data (integrity). Intentionally or otherwise, the service may enable others to view your data, even data intended to be secret (confidentiality).

These risks are hardly theoretical; we see the reality play out time and again.

Now imagine you're building a community that the government deems subversive. We don't even have to explore illegal activity, just activity that those in power don't like so much. Further imagine that you've entrusted your confidential data to a cloud service. In order to feel safe, you need the following guarantees:

  1. The service is doing everything securely on the technical level.
  2. There are no existing compromises to the service or technologies used by the service.
  3. The service will be less than eager to cooperate with a subpoena or a warrant.

Despite the marketing, the use of most commercial VPNs is an act of transferring risk not mitigating it. You've moved the need to trust your internet service provider to the need to trust your VPN—in some cases, due to common misconfigurations, data actually ends up going both to the ISP and the VPN, meaning you've now increased your risk profile.

But either way, you still have to trust those services. Some are probably trustworthy, but a system that relies on only a few good actors is a fragile system indeed. Instead, I'd like to build a private networking landscape in which many individuals can and do create the networks they require.

The Threat Model

What exactly are we defending against here? This isn't really about criminals stealing your identity, much less machine-in-the-middle attacks at the coffee shop. What we're talking about here concerns community safety. Maybe you're organizing for political action. Maybe you're sharing media that those in power would rather not be shared. Maybe you're trying to run an independent news organization. Many forms of community have good reason to be concerned about an overzealous state attempting to monitor their communications and data archives. Entrusting all of this data to third parties who have more interest in cooperating with the state than protecting your privacy is a dicey situation.

Wireguard

Wireguard offers the ability to create the kind of private network you need, not the one being sold to you. The tradeoff, at least in the more elementary setups, is knowing how the technology works. In this coming age, that knowledge is precious.

But what makes Wireguard so special as a technology? Four aspects make me very confident in networks built on Wireguard:

  1. Simplicity
  2. Ubiquity
  3. Performance
  4. Cryptography

Simplicity

Compared to other VPN technologies like OpenVPN or IPSec, Wireguard is extremely simple at its core. As this workshop demonstrates, a basic mesh is hardly more than a few lines of config—and that's before we get to the rich ecosystem of tools built on Wireguard, some of which we'll explore during this workshop.

Ubiquity

Wireguard is a Linux kernel module. That means every modern Linux machine has the capability of being a Wireguard node, and its residence in the kernel ensures a speedy pipeline to the networking hardware. But that's not all. Wireguard clients exist of Windows and macOS, as well as iOS and Android mobile operating systems. And because the Wireguard technology is licensed entirely under free and open source licenses, I don't expect it to disappear anytime soon.

Performance

Wireguard routinely outperforms other private networking technologies in throughput benchmarks. Anecdotally, I have experienced this reality. I have run full RDP sessions from conferences back to my lab at home over Wireguard, without a moment's hiccucp. If Wireguard can achieve line rate on a gigabit card, that means your VPN layer is not a bottleneck in the communication.

Cryptography

I encourage you to review the cryptographic details of the Wireguard protocol, if you're so inclined. I will never call myself a cryptography expert, but the ciphers in use and how they're used have been independently tested and attested. I'm confident in the cryptographic strength of the protocol.

Fun

This seems like it shouldn't matter, but it kind of does. If you've done any kind of network engineering, you know what an utter slog it can be. I shudder when I think of the nights I spent reconfiguring VLANs on Cisco switches a lifetime ago.

Wireguard doesn't feel like that. Setting up Wireguard networks is, dare I say it, pretty fun! Yes it's technical, but straightforward enough to make sense to most curious users. And because Wireguard itself is so simple, making it work often reveals other surprises in your network topology.

So that's why Wireguard.

Recipe 0: A Simple Mesh

We begin our Wireguard journey with the most elemental model: peers connecting to one another. This recipe sets up three hosts, with connections amongst them.

flowchart TD
    B["Peer 1"] <--> C["Peer 2"] & D["Peer 3"]
    C <--> D
    B@{ shape: rect}
    style B stroke:#D50000
    style C stroke:#00C853
    style D stroke:#2962FF

Wireguard Config

We need to get acquainted with the Wireguard config file. Take a look at peer.conf.

Interface

Each peer defines the details of its own Wireguard interface in the [Interface] section. Here the private portion of our keypair is defined. The public key derives from the private key, and for convenience I like to keep the public key in a comment alongide the private.

We also need to define the IP Address our peer will use on the Wireguard network. The ListenPort is optional, but it helps to define this explicitly so we can write firewall rules around known ports.

Peer

Peers sections are a little tricky. Depending on whether the Peer is connecting to us or we're connecting to it, we use the Peer section differently. When connecting to another Peer, we need an Endpoint directive that defines the non-Wireguard address and point where the Peer is located. We also use a PersistentKeepalive directive to keep the tunnel active with intermittent packets sent over the tunnel.

Now let's talk about AllowedIPs. This directive changes meaning depending on whether we're defining and incoming or outgoing connection. For incoming connections, AllowedIPs defines the IP addresses allowed in. Normally, this is a single IP address: the Wireguard address of the peer, written as in CIDR notation as a /32 subnet.

For outgoing connections, AllowedIPs defines what addresses get routed over the tunnel. Normally then, this will be the full subnet of the Wireguard network, like 172.16.100.0/24. Later, we'll learn some other tricks with AllowedIPs.

Setup

From this folder, run ./start.sh.

Usage

Writing the Configuration

In this recipe, we do everything the hard way so you know how it all works. We automate most of this in later recipes.

With all the hosts up, we need to build the configurations on each peer. This begins with creating a public and private key for Wireguard to use. Run these commands on each peer, once they're up.

privkey=$(wg genkey)
pubkey=$(echo $privkey | wg pubkey)
sed -i "s|<<PRIVATE_KEY>>|$privkey|" /etc/wireguard/recipe-0.conf
sed -i "s|<<PUBLIC_KEY>>|$pubkey|" /etc/wireguard/recipe-0.conf

This fills in the public/private key for each peer's configuration. But we're not finished. We also need to decide on IP addresses for each. I'd use the 172.16.100.0/24 subnet. So our IP addresses could be:

  • 172.16.100.1/24
  • 172.16.100.2/24
  • 172.16.100.3/24

Those go in the Address section of each peer config.

Now for those Peer entries. For each host, we need to create two [Peer] entries. Peer 1 needs 2 and 3; Peer 2 needs 1 and 3; and Peer 3 needs 1 and 2.

For each of these, we'll need the IP address in the non-Wireguard network for our Endpoint field. Use ip address show (or ip a s for short) to get those addresses.

The AllowedIPs for each Peer will be the /32 CIDR representation of their respective Wireguard IP addresses.

Finally, add the proper public key from each respective configuration to the appropriate Peer entry for all the configuration files. Save all these files and quit back to the containers' terminal prompts.

Bring up the Wireguard Network

On each host, run wg-quick up recipe-0 to activate the Wireguard tunnel.

You can then run wg show to show the status of the tunnel. If all has gone according to plan, you should have three hosts each showing two connections to other peers. You should also be able to ping those 172.16.100.0/24 addresses.

Congratulations! You brought up a Wireguard mesh the hard way. The subsequent recipes automate a lot of the busywork now that you've had exposure to the concepts.

Recipe 1: The Lighthouse

Our next Recipe simplifies our configuration by using a central "server" to which all other Peers connect. We call this server a "lighthouse" both because everyone needs to be able to see it, and because it will be hanging out there on rocky shores, also known as "the cloud."

In the Podman demo, the Lighthouse is just in another network segment. But in real-world application, you want the Lighthouse to be hosted someplace that's accessible anywhere in the world.

flowchart TD
    Lighthouse["Lighthouse🔦"] --- Peer["wg-home"] & n2["wg-roaming"]

    style Lighthouse stroke:#2962FF
    style Peer stroke:#00C853
    style n2 stroke:#D50000

Different Networks

Have a look at podman-compose.yml in this folder. You don't need to understand all the syntax here, but notice that each entry in the services section has a networks field. And there's a corresponding networks section at the bottom of the file. We have two networks defined: roaming, which is a stand-in for "the internet and everything not our home network," and home.

Imagine being on hotel Wi-Fi. You can't directly see a Wireguard peer that's back at home since it's behind a different router. We could open up ports on our home router, but that comes with pretty significant risks that we're trying to avoid here. Instead, we want a single point that all our Peers can see at once. That's our Lighthouse, and the easiest place to set one up is on a public cloud hosting provider. Our Podman deployment simulates that arrangement without the need for creating a cloud virtual machine.

There's one other important aspect of the setup visible in this file. Take a look at the lighthouse entry. You'll see a section that reads:

sysctls:
  net.ipv4.ip_forward: 1

This configures the lighthouse server to forward IP packets on to their intended destination using the system's routing. That means if it receives a packet intended for someone else—like say one Peer trying to reach another—it will forward the packet along however it knows how. This setting is how we turn a single Wireguard Peer into a lighthouse that allows many hosts to interconnect through one point. This trick is also why the lighthouse server is always a Linux/Unix-like device. IP forwarding is very simple to get going on these operating systems.

Recipe Setup

In the recipe-1 folder, run the start.sh script to bring up our peers.

Wireguard Config

This time, each host's Wireguard config (/etc/wireguard/recipe-1.conf) has a keypair configured. What remains is for us to connect the two peers to the lighthouse.

In this model, each non-lighthouse Peer adds the same information to their Peer entry: the lighthouse's public key and IP address, and setting AllowedIPs to the whole /24 subnet we're using for Wireguard (i.e. 172.16.100.0/24). That way, all traffic to any IP address in that subnet will be routed to the lighthouse.

For the lighthouse, a separate [Peer] entry is required for each Peer, but no Endpoint or PersistentKeepalive directives. That's because the lighthouse only receives tunnels; it does not initiate them. The AllowedIPs field for each peer will be the single IP address assigned to that Peer, written as a /32 CIDR.

Again, use wg-quick up recipe-1 to initiate the tunnels, and wg-show to view the tunnel status.

Usage

Try pinging one Wireguard address from the other! Also, you can try using netcat to simulate some network traffic. One one peer, run:

nc -nvlp 8000

This sets up a TCP listener on port 8000. Then, on the other peer, run:

nc <peer_wireguard_ip> 8000

You should see a TCP connection established!

Addendum: The Road Warrior

There's a common Wireguard setup known as the "Road Warrior" that we are conspicuously ignoring in this workshop. The Road Warrior looks like this:

flowchart TD
 subgraph s1["Home Network"]
        Home["Home Hosts"]
        n3["Wireguard Router"]
  end
    n3 --- Home
    Roaming["Roaming"] ---> s1

    style Roaming stroke:#2962FF
    style s1 fill:#BBDEFB

Instead of relying on a cloud-hosted server, the Road Warrior opens a port on one's home router, and connects back to that router and linked hosts via that open port. It's simpler, but depending on your threat model, a bit more dangerous. Personally, I consider my home router as a questionable ally at best, given the focus those devices receive from my adversaries. Entrusting a cloud hosting provider can also be quite risky, and there are some I wouldn't use for this purpose, but for me, I'd much rather move the risk to that third party than in my own house.

Recipe 2: The Lighthouse + Subnet Router

This one is a minor modification of the Lighthouse from the last recipe, but an important one. The lighthouse as we defined it works great to connect Wireguard clients together, but what about devices that can't run Wireguard? What if you just have to have access to that sweet, sweet Brother printer in your home office? Or more modernly, what if you want to access your local-only security cameras?

In this recipe, one of our Peers connecting to the lighthouse has a trick up its sleeve. Despite being an ordinary Linux box, it serves as a router, allowing Wireguard peers to access the networks it can access. This "subnet router" provides Wireguard clients access to an entire home network through a single node.

flowchart TD
 subgraph s1["Home network"]
        n2["Home Node"]
        n4["Webserver"]
  end
 subgraph s2["Roaming network"]
        n3["Roaming Node"]
  end
 subgraph s3["Internet"]
        n1["Lighthouse"]
  end
    n2 <---> s3
    s2 <---> s3
    n2 <---> n4

    n2@{ shape: rounded}
    n3@{ shape: rounded}
    n1@{ shape: rounded}
    style n2 stroke:#2962FF
    style n3 stroke:#AA00FF
    style n1 stroke:#D50000
    style s1 fill:#BBDEFB
    style s2 fill:#E1BEE7

Lab Setup

In the recipe-2 folder, run the start.sh script. As before, this will initialize the Podman containers. This lab starts 4 containers:

  1. The home router
  2. A non-Wireguard webserver on the home network
  3. A roaming Wireguard client on a separate network
  4. The lighthouse server

To save a little time, this lab setup automates the assignment of Wireguard IP addresses in the config files.

Router Config

To see how this works, take a look at home-router/home-router.conf. It's a Wireguard config, similar to the ones we've seen previously, but this one has a whole mess of PostUp and PostDown commands. Wireguard allows you to execute arbitrary commands upon startup aand shutdown. These commands set up firewalls rules to forward IP traffic from the Wireguard interface to the router's local interface, and "masquerade" the IP address—that is, pretend the packet is from the route rinternally, but translate it back to the original source once a response is received.

The commands here use nftables to create the firewall rules. NFTables is the successor to iptables, but the syntax is way more arcane. In your own configs, if you prefer iptables or firewalld commands, that's just fine.

This works in conjunction with the IP forwarding configuration on the lighthouse. Not much changes from the last configuration, save for the AllowedIPs field of the [Peer] config for our home router. While every other peer connecting to the lighthouse has a single IP in this field, the home router has a single IP, and the whole home subnet. That looks like:

AllowedIPs = 172.16.100.2/32, 192.168.99.0/24

When a request for a home network address comes over the lighthouse tunnel, the lighthouse will use its routing rules (created by Wireguard) to send the traffic down to the home router. The home router, in turn, handles the traffic with its firewall rules. And that's how we present an entire home network to Wireguard via one—well two—peers.

That same AllowedIPs pattern is necessary on all the clients we want to access the home network via the Wireguard tunnel. Remember that on clients, AllowedIPs amounts to a routing rule for the Wireguard tunnel. So our roaming client's Peer entry for the lighthouse, the AllowedIPs section looks like:

AllowedIPs = 172.16.100.0/24, 192.168.99.0/24

Lab Exercises

Tracepath

We can use tracepath to confirm that traffic to the 192.168.99.0/24 network is being routed over Wireguard. From the roaming client, run:

tracepath 192.168.99.10

The hops you see should be all 172.16.100.0/24 addresses until the very end.

cURL

We went to the trouble of setting up a webserver in the home network. Can you use cURL to access it from the roaming Wireguard client?

curl http://192.168.99.10

Yay! Web traffic!

The Dirty Secret

The lighthouse is doing its job, but there's a subtle flaw in this which, depending on your threat model, might be a dealbreaker.

Let's think about what happens with that HTTP request to our home webserver. The roaming client sends the HTTP request over the Wireguard tunnel, up to the lighthouse server, which forwards it on to the home router, which in turn forwards it to the webserver.

See the problem? Like I said, it's subtle.

The issue lies in the first forward. In order for the lighthouse server to determine how to route the incoming packets, the traffic must exit the Wireguard tunnel and be handled by the lighthouse's routing rules. And there's the rub: at that moment, as the traffic exits one Wireguard tunnel before being sent over another, the traffic is not encrypted by Wireguard. That means anyone who can listen to our lighthouse's traffic can read the raw traffic. And because our request was unencrypted HTTP, the request is fully visible.

We can demonstrate this by capturing packets on the lighthouse during the HTTP request. All the tools have been included in the containers to make this happen.

Start the capture on the lighthouse this way:

tcpdump -i recipe-2 tcp port 80 -w curl.pcapng

This will listen to the Wireguard interface for HTTP traffic. Then on the roaming client:

curl http://192.168.99.10?key=supersecretkey

Use Ctrl+C to stop the packet capture on the lighthouse. You can then use tshark (or termshark if you're feeling brave) to review the traffic. With tshark, the command would be:

tshark -r curl.pcapng

See anything interesting?

Why This Matters

We discussed in the rationale for Wireguard that many VPNs are just a shifting of risk from the ISP to the VPN. In this case, we have also introduced some potential risk. If someone besides us is snooping on the cloud virtual machine, we've lost our guarantee of confidentiality. Some cloud providers are more trustworthy than others. Personally, for the providers I use, I'm less concerned about that than I am with any potential compromises of home routers. Still, this is not a perfect end-to-end encryption solution. But don't worry—we can do better.

When you're done exploring, stop the containers with stop.sh in the recipe-2 folder.

Recipe 3: Coordinate Mesh with Headscale

We've learned that the lighthouse model, while convenient, has a potentially critical shortcoming: the relay exposes traffic as it crosses from one Wireguard tunnel to another. That means if the underlying protocol (HTTPS, SSH, etc.) is not encrypted, the traffic is perfectly readable by another with root access to the lighthouse. This isn't a problem in a direct mesh network, since the traffic is sent directly to a given peer. Unfortunately, we can't easily do that with just Wireguard when we're traversing multiple routers and the internet.

Ideally we'd want the lighthouse to negotiate some sort of pathfinding for peers to connect directly to each other, and let the peer-to-peer traffic remain encrypted.

That's exactly what Tailscale does.

Tailscale

Tailscale is a networking product built on Wireguard and designed for the exact use case we're describing: building custom secure networks across multiple NAT layers.

I'll let you dive into the details of how Tailscale operates, but essentially Tailscale adds a coordination component to the Wireguard network that allows peers connecting to the "coordination server" to discover the best way to directly connect to each other—even behind NAT. Tailscale adds a few other handy features, like mature authentication/authorization and access control.

Still, the Tailscale service is yet another service. Trustworthy? Probably, but I don't like needing to rely on third party services. Luckily, I don't have to, because Tailscale open sources almost all of its core code. The client and coordination server bases are available, and there is indeed an open source implementation of the coordination server known as Headscale

Headscale

Headscale allows us to spin up our own mini-Tailscale, using the Tailscale client for connection and authentication. The Tailscale client is also cross-platform (including mobile).

Headscale is shockingly easy to get going. The server installation documentation is extremely clear, as are the configuration options available.

This recipe shows Headscale in action in a safe, containerized environment. But if you're interested in "going live" in this workshop, it will be Headscale that we deploy to the cloud.

Lab Setup

Navigate to recipe-3 in your terminal.

Before starting the lab, you might want to review the config file at headscale/config/config.yaml. Here you can see how we configure Headscale for listening. What you won't see in this configuration is any concept of a user. For that, you might want to look at how we spin things up in start.sh. But you'll also have a chance to do this yourself in the lab.

As before, fire up the lab with ./start.sh. Your Zellij session will show three terminals, but only two containers: the home and roaming clients. What's up with that? The leftmost terminal is still on the host, because all of the commands for the Headscale server are run outside the terminal. That's just how Headscale decided to set it up. We'll work with it.

Lab Exercises

Run Headscale Commands

On the left-hand terminal, we can run commands against our Headscale server with podman container exec. Let's check the status of our nodes with:

podman container exec recipe-3_headscale headscale nodes list

We should see two online nodes.

Access the Home Webserver

We once again have a home network webserver set up to test route advertisement. From the roaming client, run:

tracepath 192.168.99.10

You should see hops that represent Tailscale nodes. That tells us that the 192.168.99.0/24 subnet is being properly routed across Tailscale.

You can of course then use curl to confirm it works.

Tailscale Hostnames

From either client, run tailscale status to see the hostnames Tailscale associates with the IP addresses. Try using ping or nc to establish network connections using hostnames.

Going Live: Leaving the Test Kitchen

Our recipes so far have demonstrated the concepts behind Wireguard networking in a safe, contained (and containerized) environment. But now it's time to start building our real private network. In this section, we walk through creating a real-deal Headscale server in the cloud.

"But Taggart, you can't trust the cloud!"

You can use a thing without trusting it. The value proposition of the cloud VM is its visibility from everywhere. But if we do this right (e.g. a proper Headscale setup), we won't be trusting it for too much. Its job will be to negotiate the initial connection between peers, and then let the Tailscale "magic" establish peer-to-peer connections, keeping the lighthouse out of the picture entirely.

Let's get going.

Required Materials

This is the part of the workshop that costs some money. To complete this setup as we've done it, you'll need:

  1. A registered domain name that you can control
  2. A cloud VM

The cloud VM doesn't have to be huge. Entry-level VMs on all major VPSes will be just fine.

Create the Cloud VM

It all starts with the cloud VM. I'm going to use Digital Ocean to demo the process, but if you're comfortabel with AWS|Azure|Vultr|Hetzner, go for it.

For commonality's sake, we'll use Ubuntu 24.04 LTS as our base. However, this choice means we'll need to manually install a newer version of Podman than is available directly from the package manager.

Different cloud providers have different methods of securing their VMs, but I strongly recommend the following best practices:

  • Use SSH Key authentication
  • Use firewall rules to limit SSH access to certain source IP addresses

Log in to the VM via SSH (or cloud console, if appropriate).

Housekeeping

We want to perform the following housekeeping measures, as needed:

  1. Create a new, non-root user
  2. Make sure our SSH Keys are present on the new user and that we can log in with it
  3. Disable root login
  4. Add firewall rules to allow HTTP/HTTPS traffic (80/tcp, 443/tcp)

Hey if you wanna get extra sneaky, change the ports you're using for SSH and HTTPS, but for now we'll leave these default.

Install Headscale

Head to the Headscale Releases and download the latest Headscale version.

For our server, we want the linux_amd64.deb version. Copy that link's URL and then use it like so:

wget https://github.com/juanfont/headscale/releases/download/v0.26.1/headscale_0.26.1_linux_amd64.deb
sudo dpkg -i headscale*.deb

If all goes well, you'll see some instructions to enable and start the headscale service. Let's do so.

sudo systemctl enable headscale
sudo systemctl start headscale

Now comes the configuration part. We'll start with the users.

Create Headscale users

For now, we'll just create one user, but for your real network, you can make as many as you need. Remember that machines are associated with users, so think about how you want to link those conceptually.

sudo headscale users create hope -d hope-demo

And right away we'll create a preauthkey for use when we're ready to connect a client.

sudo headscale preauthkeys create -u 1

Copy that and save it for later.

Configure Headscale Server

Now we need to edit the Headscale config to match our domain. Got the domain you intend to use? Great. We don't need to modify the DNS records yet, but it's coming up. For now, use Nano, Vim, or Helix (sudo snap install helix --classic). With any of those, open up /etc/headscale/config.yaml with sudo.

Change server_url to https://the-domain-you-chose.com:443

Change listen_addr to 0.0.0.0:443

Change acme_email to an email address you feel comfortable giving to LetsEncrypt.

Change tls_letsencrypt_hostname to the-domain-you-chose.com

And that's all we need to change for now. If you looked closely at the setup for Headscale in our recipes, you saw we configured a custom certificate for Headscale on startup. But now that we're on the internet proper, we can use LetsEncrypt to grab a certificate.

Save and quit.

Configure DNS

Time to head to your DNS registrar! Create a new A record for the domain name you've selected and point it at the IP address of your cloud VM.

Give it a few minutes for the record to propagate. You can test it with nslookup the-domain-you-chose.com.

Once it's ready, head back to your server and restart Headscale.

sudo systemctl daemon-reload
sudo systemctl restart headscale

Connect a Client Or Two

Now comes the fun part. Using the Tailscale Client, connect to the Headscale server. Here's how it looks on Linux:

sudo tailscale up --login-server=https://the-domain-you-chose.com --auth-key=the-authkey-you-created --accept-routes

Congratulations! You've created a Headscale network.

Going Further

Now that you have a way of connected trusts endpoints and networks together for secure communications, you can explore using some of these endpoints as exit nodes to alter where your internet traffic comes from. You can also explore DNS for your network to make it easier to reference endpoints.

You can also explore access control for more granular control of your hosts.

But for now, take a moment and congratulate yourself. You've just taken a big step toward owning your digital privacy. You're using the cloud, but on your terms.