This article is part of a series
As I mentioned yesterday, I have made further progress towards improving the quality of my website deployments. Specifically, my website is now running on a Flatcar VM that's running a pair of OCI containers. As I continue to improve the quality of my personal software and infrastructure, I also continue to document what I did, for reasons of education and thought leadership (a word which I absolutely hate, but alas, that is how we must speak for the business crowd).
Why containerise?
Well, a large part of it is CV-driven development, I'll not lie: I need to write thought leadership pieces demonstrating my skill with all aspects of technology and especially infrastructure and DevOps, and these days, that means you have to understand containers. As I've discovered in the course of this, container technology is really good for developer experience and makes a lot of tasks so much easier than doing them manually, so doing this was an excellent way to learn.
Aside from that, I worry a fair amount these days about a) the political status of my hosting and b) the ability to rapidly redeploy to a new platform if I get pushed off one. For that, containerisation is a wise thing to do: after all, OCI images are largely platform agnostic, which means that I can switch deployments from one cloud platform to another (or even to a local machine) that much faster.
I was also, in my first pass of this, building my website on the machine I was deploying it on. Not only does this pose security concerns, it takes time and creates a machine that's bulkier than it needs to be. While it's definitely possible to get better on these axes without containerising it, these days it's likely that you'd have to use a container for the build step anyway. Decoupling build and deploy steps also helps in the case that I need to redeploy rapidly: I only need to change my deploy step rather than my build.
Finally, containers are just a really fun technology to work with. There are so many cool things you can do with them, and getting the chance to learn the tech was something I couldn't pass up.
Step 1: building the container
The first step, naturally, was to actually build my website into a container. My first attempt to do this involved using the pack cli and Paketo buildpacks in order to avoid having to explicitly write a Dockerfile. Alas, making the cli work with Nuxt was a challenge: pack requires the build process to be run in development mode to access some cli utilities, while Nuxt's build utilities will not allow you to do that, even going to the extent of changing the NODE_ENV variable to production no matter what you initially set it to. This is probably resolvable in principle, but as it's a bit finicky to sort, I eventually decided just to write a Dockerfile, which ended up looking like this:
# Build
ARG NODE_VERSION=22.14.0
FROM node:${NODE_VERSION}-slim AS base
ARG PORT=3000
WORKDIR /src
FROM base AS build
COPY package.json package-lock.json .
RUN npm install
COPY . .
RUN npm run build
# Run
FROM base
ENV PORT=$PORT
ENV NODE_ENV=production
COPY --from=build /src/.output /src/.output
CMD [ "node", ".output/server/index.mjs" ]
We have here a two-stage build process: our first stage creates a lightweight Node container (based on Alpine, I believe), installs dependencies and runs the build script. Having run the build, we then copy the output over to the container that we want to actually run on, where we then set the port to listen on and the command to use to run the built bundle. The first container is then discarded, and we're left with a minimal container with only the built website in it.
By contrast with the build process described in Part 1, this is much lighter (and thus quicker to deploy), as well as being more secure thanks to the fact that neither the source code nor any of the dependencies are installed on the image (apart from the bundled dependencies in the output). This significantly reduces the size of the attack surface for the deployment compared to what I had earlier.
One could reasonably copy this bundle over to a VM and build an image using Packer, and this would probably result in a bundle size just as small. However, that would by nature have to be quite a bespoke process which takes quite a long time, and Gitlab CI/CD, which I'm currently using, makes it rather easier just to build a container than to work with images. Finally, I'm working with Hetzner, and I can attest that building and uploading an image is relatively annoying and slow (more on this later), so one should do it as little as possible.
While it's not strictly important in the same way that those other points are, the comparative portability of containers is always nice. If I have a container, I can elect to host it on pretty much anything and it'll run happily enough. I could even run it on a Kubernetes cluster if I wanted (though that'd be silly). What I would like to do, however, is be able to build the image every time I push code to my remote repository. Fortunately, Gitlab CI/CD makes this easy to do: all you need is a .gitlab-ci.yml file in your root directory. Mine looks like this:
build-image:
stage: build
image: quay.io/buildah/stable
variables:
STORAGE_DRIVER: vfs
BUILDAH_FORMAT: docker
FQ_IMAGE_NAME: "$CI_REGISTRY_IMAGE/stable"
before_script:
- echo "$CI_REGISTRY_PASSWORD" | buildah login -u "$CI_REGISTRY_USER" --password-stdin $CI_REGISTRY
script:
- buildah images
- buildah build -t $FQ_IMAGE_NAME
- buildah images
- buildah push $FQ_IMAGE_NAME
When you push a repository containing such a file to Gitlab, Gitlab will automatically execute the commands in the before_script block, then the script block. In this case, we only have one stage, which builds the image and pushes it to Gitlab's container registry for use during deployment.
There are only a few points to touch on beyond that. The first is that we have to authenticate to the container registry before we can push to it: fortunately Gitlab runners have the registry user and password defined as environment variables, so that's fairly easy to do. The second point is that I'm using Buildah to construct the image rather than Docker. While this supposedly has some security advantages over using Docker Build, I'm mostly just using it because it's a tool optimised for creating images at the expense of all else. That means that it's smaller, that the runner container that Gitlab CI/CD runs on is smaller, and thus that my builds run that much faster.
And with that, I have an image available in my image registry to host somewhere nice. The next step, then, is as follows: how do I sensibly host containers on Hetzner?
Step 2: Hosting the image
Obviously, Kubernetes isn't the correct answer here. While I'm relatively sure I could set up a cluster without too much trouble, the fact remains that the tool is just too complex and heavy to really be worth it: we need something simpler. While this could plausibly be Docker Swarm, realistically the best solution is just to run the containers on a minimal VM using a container engine of some kind. It does the job, we get a lot of the benefits of containerisation even when running a few containers on a server, and it's much, much easier to implement than anything else.
This then raises the question of what this image should look like. So far, I've been doing my hosting using Fedora servers. Now, while they're pretty good for the task (I've had no real issues all the time I've been working), it's a general-purpose operating system, which means that a) the image is bigger than it needs to be and b) it has a whole lot of packages that will never be used and that introduce weird security bugs. For that reason, I elected to use Flatcar Linux, which is a minimal Linux distro that's tailored specifically for running containers. The general idea of the thing is that it's a distro with nothing on it but the Docker Engine (not even a package manager), with secure-by-default settings and immutable operating system files. Not only does this reduce the attack service of the distribution considerably compared to Fedora, having an immutable OS filesystem outright denies a whole class of attacks, making for a much more secure deployment overall. The main issue with Flatcar, really, is that it's not a standard image on Hetzner, and as Hetzner doesn't directly allow you to upload your own images, this means we have to be a little creative.
Fortunately, we can work around this with Hashicorp's Packer tool for creating new VM images. To do this (once you've installed the software, of course), create a new directory (perhaps called packer), and create a packer template file in one of them, which should have the .packer.hcl extension. My packer template looks like this:
packer {
required_plugins {
hcloud = {
source = "github.com/hetznercloud/hcloud"
version = "~> 1.4.0"
}
}
}
variable "channel" {
type = string
default = "beta"
}
variable "hcloud_token" {
type = string
default = env("HCLOUD_TOKEN")
sensitive = true
}
source "hcloud" "flatcar" {
token = var.hcloud_token
image = "ubuntu-24.04"
location = "fsn1"
rescue = "linux64"
snapshot_labels = {
os = "flatcar"
channel = var.channel
}
ssh_username = "root"
}
build {
source "hcloud.flatcar" {
name = "x86"
server_type = "cx22"
snapshot_name = "flatcar-${var.channel}-x86"
}
provisioner "shell" {
inline = [
# Download script and dependencies
"apt-get -y install gawk",
"curl -fsSLO --retry-delay 1 --retry 60 --retry-connrefused --retry-max-time 60 --connect-timeout 20 https://raw.githubusercontent.com/flatcar/init/flatcar-master/bin/flatcar-install",
"chmod +x flatcar-install",
# Install flatcar
"./flatcar-install -s -o hetzner -C ${var.channel}",]
}
}
This looks fairly similar to Terraform or OpenTofu code, and it is in fact the same HCL (Hashicorp Configuration Language). The first few blocks are identical in function to Terraform blocks of the same type as well: we connect to the Hetzner plugin with our API key and set some variables.
After that we have a source block, which defines the source image for the custom image we're making. It doesn't much matter what this image is, as we're wiping it anyway: the example I found used Ubuntu, so we may as well go with that (this is the only appropriate use of Ubuntu). We have to label the source and set a few things, including, importantly, the rescue flag: we want the machine to boot into the 64 bit system rescue utility rather than the actual OS. Next we have a build step that installs gawk (a dependency of the Flatcar install script), then downloads the install script that Flatcar provides and runs it in Hetzner mode. Finally, it saves the fresh install as a snapshot (which in Hetzner is kinda the same thing as an image but not really). There's more you can do with Packer to customise the image, and I might do some of it in the future, but as Flatcar can apparently behave strangely when you boot an image that isn't pristine, I'm a little wary of it and elected not to do it just yet.
We can now access this image in our main.tf file with the following block:
data "hcloud_image" "flatcar-beta-x86" {
id = <img-id>
most_recent = true
}
Hetzner provides an image id for custom images which you can look up in Terraform, but honestly I was lazy and just grabbed it from the dashboard. With that, you can then use this data source to provision a server in the normal way. So, we have our image and we have our OS: now how do we actually deploy?
Step 3: deploying the image
Unlike Fedora, Flatcar doesn't support cloud-init: instead, it relies on an init system called Ignition: it's rather more powerful than cloud-init in some ways but also insists that your system configuration is entirely declarative: that is, it won't let you run random commands on startup. You don't usually write Ignition configs directly: rather, you write it in a YAML file and get a tool called Butane to turn it into an Ignition config. My Butane config looks like this:
variant: flatcar
version: 1.0.0
storage:
files:
- path: /etc/ssh/sshd_config.d/custom.conf
overwrite: true
contents:
inline: |
PermitRootLogin no
AllowUsers core
mode: 0600
- path: /opt/bin/docker-compose
contents:
source: https://github.com/docker/compose/releases/download/v2.11.2/docker-compose-linux-x86_64
verification:
hash: sha512-8ad55ea1234e206ecad3e8ecf30f93de5cc764a423bf8ff179b25320f148e475b569ab9ec68d90eacfe97269528ff8fef1746c05381c7d05edc85d2f2c245e69
mode: 0755
- path: /home/core/docker-compose.yml
contents:
local: compose.yml
mode: 0644
user:
name: core
group:
name: core
- path: /home/core/Caddyfile
contents:
local: Caddyfile
mode: 0644
user:
name: core
group:
name: core
- path: /home/core/.docker/config.json
contents:
local: registry_config.json
mode: 0644
user:
name: core
group:
name: core
- path: /etc/profile.env
mode: 0644
contents:
inline: |
export POSTGRES_URL=<postgres-url>
export POSTMARK_SERVER_TOKEN=<postmark-server-token>
systemd:
units:
- name: application.service
enabled: true
contents: |
[Unit]
Description=deadSimpleTech Website
[Service]
User=core
ExecStart=/opt/bin/docker-compose -f /home/core/docker-compose.yml up
[Install]
WantedBy=multi-user.target
There's a fair amount going on here, so I'll step through it a little bit. The first thing we do is write a bunch of files to the system. The first file prevents root login and password authentication by overwriting sshd_config: this is a standard security measure which we also did in the previous deployment. If we compare this to how we did it in the previous deployment, we can see that the first option is much simpler:
- sed -i '/PermitRootLogin/d' /etc/ssh/sshd_config
- echo "PermitRootLogin no" >> /etc/ssh/sshd_config
- sed -i '/Password Authentication/d' /etc/ssh/sshd_config
- echo "Password Authentication no" >> /etc/ssh/sshd_config
The next file is a Docker Compose executable that I'm downloading from Github and making executable as I'm using Docker Compose for some simple orchestration: it's worth noting here that in a production environment you should use the Docker Compose sysext for this, but I'm exhausted and burnt out and this worked, so I can't quite bring myself to try doing that yet. After that we write the Docker Compose file itself:
services:
deadsimpletech-website:
image: registry.gitlab.com/internal_websites/dead-simple-consulting/stable
restart: unless-stopped
ports:
- "3000:3000"
environment:
- POSTMARK_SERVER_TOKEN=<postmark-key>
- POSTGRES_URL=<postgres-url>
caddy:
image: caddy:2.10.0-alpine
restart: unless-stopped
ports:
- "80:80"
- "443:443"
- "443:443/udp"
volumes:
- ./Caddyfile:/etc/caddy/Caddyfile
- caddy_data:/data
- caddy_config:/config
volumes:
caddy_data:
caddy_config:
We note here a few things: firstly, we're reading this in from a local file, which cloud-init doesn't let you do. The file is injected into the Ignition config as a base64 string, and if your deployments mostly happen locally, this is a really useful tool to have on hand. Secondly, we have two images here: one is our website, and the other is a Caddy image that acts as a reverse proxy. Containerising Caddy in this way makes deploying my entire stack to any platform much easier and more repeatable than what I was doing previously and is honestly just a really clean and elegant way to do it. Finally, the eagle-eyed among you might have noticed that there are credentials in that file being assigned to environment variable. Yes, this is really bad practice: unfortunately, one of the things I really don't like about Flatcar is that it's hard to inject secrets into a Flatcar VM in a secure way and without SSH. Even if you're careful, the variables tend to end up in files on disk, and I've still not found a pattern for doing that that I actually like. This is likely because Flatcar is designed to be used with orchestration services like Kubernetes which has its own ways of dealing with this, but it's still a bit of a pain.
Having written that, we then write a Caddyfile (which is largely identical to the one in the last article) and the Docker configuration file, which lets Docker on Flatcar authenticate to my Gitlab container registry. This is another credential, which isn't ideal, but this one's somewhat less bad than the others as it's a heavily locked-down deploy token that can't do anything other than download the container image.
Finally, we create a systemd unit that runs Docker Compose with the configuration specified. It's important to specify that this is owned by the Core user otherwise we won't authenticate to the container registry correctly (and also because running this as root is a bad idea). Having constructed that config, we can then run
butane -d . init_butane.yml > init.json
in the directory with the config files in it to create our Ignition config. Now we can make some minor alterations to our main.tf file from last time:
resource "hcloud_server" "deadsimple_website_server" {
name = "deadsimple-website-server"
server_type = "ccx13"
location = "hel1"
image = data.hcloud_image.flatcar-beta-x86.id
firewall_ids = [hcloud_firewall.web_server_and_ssh.id]
network {
network_id = hcloud_network.deadsimple_network.id
}
user_data = file("${path.module}/config_files/init.json")
depends_on = [
hcloud_network_subnet.deadsimple_network_subnet
]
}
All we've done here is changed out the image we want to use for our custom Flatcar image and switched out what we're feeding to user_data for our new Ignition config. This being done, we can then run
tofu apply -replace="hcloud_server.deadsimple_website_server"
To recreate the server whenever we want to deploy. And there we have it: our deployment's containerised!
Reflections and next steps
So, how does it all work? The first thing to note is that working with containers feels idiomatically nicer than doing everything manually like I did last time. It's hard to say how meaningful that actually is, but the code feels better-ordered with fewer messy bits and less cruft. That's probably not enough to justify containerising on its own, but it is a significant consideration. Decoupling the build step from the deploy step and automating builds is another big thing: if I want to change where I deploy down the line, it makes it much easier to do that and it's just more aesthetically pleasing besides: building on the host was ugly. Finally, having images stored in a registry is excellent for distribution.
Deployment speed is a mixed bag. For some reason Hetzner takes considerably longer to create a VM from a custom image than it does from a standard one, so the OpenTofu run takes somewhat longer on average (two minutes as opposed to forty seconds or so). Against that, once the resource is created, startup times are much, much faster (basically immediate, whereas I'd have to wait two to three minutes for everything to build and start up after creation previously), so this is a significant improvement and will make it a lot easier to move to a zero-downtime deployment pattern down the line.
I can't really speak to security questions: I hate how I'm dealing with credentials at the moment, but the other options feel like they create further security issues so there are obvious tradeoffs to be made here. One of the biggest ones is that I'm going to struggle with a credential handling scheme that I don't actually have the memory or willpower to stick to or configure in the first place, so when I improve this, I need a sufficiently simple way of handling this. Overall though, containerisation and running on Flatcar is likely to have somewhat improved the security of my deployments, though by how much is anyone's guess.
Regarding where I want to go with this next, security improvements are an important point of attention, but shouldn't take too much work to implement (I hope). In terms of larger projects, I need to get my database off Neon, which means setting up a PostgreSQL cluster on Hetzner. This is probably going to be my next project. After that, I'd like to self-host a Gitlab or Gitea instance (haven't chosen which yet) and automate my deployments as well as my builds. Finally, I really need to sort out a testing strategy for this: I have some tests, but I've been fairly lax on them so far. Conveniently enough, learning how to do stuff with containers makes all of these things significantly easier.
Once I've done all of that, I think I'll have a website that I can be properly proud of.