Like what you read? Want to be updated whenever we post something new? Subscribe to our newsletter!

Migrating my website part 2 - Containerised Website Deployments

Author: Iris Meredith

Date published: 2025-05-09

This article is part of a series

Part 1

As I mentioned yesterday, I have made further progress towards improving the quality of my website deployments. Specifically, my website is now running on a Flatcar VM that's running a pair of OCI containers. As I continue to improve the quality of my personal software and infrastructure, I also continue to document what I did, for reasons of education and thought leadership (a word which I absolutely hate, but alas, that is how we must speak for the business crowd).

Why containerise?

Well, a large part of it is CV-driven development, I'll not lie: I need to write thought leadership pieces demonstrating my skill with all aspects of technology and especially infrastructure and DevOps, and these days, that means you have to understand containers. As I've discovered in the course of this, container technology is really good for developer experience and makes a lot of tasks so much easier than doing them manually, so doing this was an excellent way to learn.

Aside from that, I worry a fair amount these days about a) the political status of my hosting and b) the ability to rapidly redeploy to a new platform if I get pushed off one. For that, containerisation is a wise thing to do: after all, OCI images are largely platform agnostic, which means that I can switch deployments from one cloud platform to another (or even to a local machine) that much faster.

I was also, in my first pass of this, building my website on the machine I was deploying it on. Not only does this pose security concerns, it takes time and creates a machine that's bulkier than it needs to be. While it's definitely possible to get better on these axes without containerising it, these days it's likely that you'd have to use a container for the build step anyway. Decoupling build and deploy steps also helps in the case that I need to redeploy rapidly: I only need to change my deploy step rather than my build.

Finally, containers are just a really fun technology to work with. There are so many cool things you can do with them, and getting the chance to learn the tech was something I couldn't pass up.

Step 1: building the container

The first step, naturally, was to actually build my website into a container. My first attempt to do this involved using the pack cli and Paketo buildpacks in order to avoid having to explicitly write a Dockerfile. Alas, making the cli work with Nuxt was a challenge: pack requires the build process to be run in development mode to access some cli utilities, while Nuxt's build utilities will not allow you to do that, even going to the extent of changing the NODE_ENV variable to production no matter what you initially set it to. This is probably resolvable in principle, but as it's a bit finicky to sort, I eventually decided just to write a Dockerfile, which ended up looking like this:

# Build

ARG NODE_VERSION=22.14.0

FROM node:${NODE_VERSION}-slim AS base

ARG PORT=3000

WORKDIR /src

FROM base AS build

COPY package.json package-lock.json .

RUN npm install

COPY . .

RUN npm run build


# Run

FROM base

ENV PORT=$PORT

ENV NODE_ENV=production

COPY --from=build /src/.output /src/.output

CMD [ "node", ".output/server/index.mjs" ]

We have here a two-stage build process: our first stage creates a lightweight Node container (based on Alpine, I believe), installs dependencies and runs the build script. Having run the build, we then copy the output over to the container that we want to actually run on, where we then set the port to listen on and the command to use to run the built bundle. The first container is then discarded, and we're left with a minimal container with only the built website in it.

By contrast with the build process described in Part 1, this is much lighter (and thus quicker to deploy), as well as being more secure thanks to the fact that neither the source code nor any of the dependencies are installed on the image (apart from the bundled dependencies in the output). This significantly reduces the size of the attack surface for the deployment compared to what I had earlier.

One could reasonably copy this bundle over to a VM and build an image using Packer, and this would probably result in a bundle size just as small. However, that would by nature have to be quite a bespoke process which takes quite a long time, and Gitlab CI/CD, which I'm currently using, makes it rather easier just to build a container than to work with images. Finally, I'm working with Hetzner, and I can attest that building and uploading an image is relatively annoying and slow (more on this later), so one should do it as little as possible.

While it's not strictly important in the same way that those other points are, the comparative portability of containers is always nice. If I have a container, I can elect to host it on pretty much anything and it'll run happily enough. I could even run it on a Kubernetes cluster if I wanted (though that'd be silly). What I would like to do, however, is be able to build the image every time I push code to my remote repository. Fortunately, Gitlab CI/CD makes this easy to do: all you need is a .gitlab-ci.yml file in your root directory. Mine looks like this:

build-image:

stage: build

image: quay.io/buildah/stable

variables:

STORAGE_DRIVER: vfs

BUILDAH_FORMAT: docker

FQ_IMAGE_NAME: "$CI_REGISTRY_IMAGE/stable"

  

before_script:

- echo "$CI_REGISTRY_PASSWORD" | buildah login -u "$CI_REGISTRY_USER" --password-stdin $CI_REGISTRY

  

script:

- buildah images

- buildah build -t $FQ_IMAGE_NAME

- buildah images

- buildah push $FQ_IMAGE_NAME

When you push a repository containing such a file to Gitlab, Gitlab will automatically execute the commands in the before_script block, then the script block. In this case, we only have one stage, which builds the image and pushes it to Gitlab's container registry for use during deployment.

There are only a few points to touch on beyond that. The first is that we have to authenticate to the container registry before we can push to it: fortunately Gitlab runners have the registry user and password defined as environment variables, so that's fairly easy to do. The second point is that I'm using Buildah to construct the image rather than Docker. While this supposedly has some security advantages over using Docker Build, I'm mostly just using it because it's a tool optimised for creating images at the expense of all else. That means that it's smaller, that the runner container that Gitlab CI/CD runs on is smaller, and thus that my builds run that much faster.

And with that, I have an image available in my image registry to host somewhere nice. The next step, then, is as follows: how do I sensibly host containers on Hetzner?

Step 2: Hosting the image

Obviously, Kubernetes isn't the correct answer here. While I'm relatively sure I could set up a cluster without too much trouble, the fact remains that the tool is just too complex and heavy to really be worth it: we need something simpler. While this could plausibly be Docker Swarm, realistically the best solution is just to run the containers on a minimal VM using a container engine of some kind. It does the job, we get a lot of the benefits of containerisation even when running a few containers on a server, and it's much, much easier to implement than anything else.

This then raises the question of what this image should look like. So far, I've been doing my hosting using Fedora servers. Now, while they're pretty good for the task (I've had no real issues all the time I've been working), it's a general-purpose operating system, which means that a) the image is bigger than it needs to be and b) it has a whole lot of packages that will never be used and that introduce weird security bugs. For that reason, I elected to use Flatcar Linux, which is a minimal Linux distro that's tailored specifically for running containers. The general idea of the thing is that it's a distro with nothing on it but the Docker Engine (not even a package manager), with secure-by-default settings and immutable operating system files. Not only does this reduce the attack service of the distribution considerably compared to Fedora, having an immutable OS filesystem outright denies a whole class of attacks, making for a much more secure deployment overall. The main issue with Flatcar, really, is that it's not a standard image on Hetzner, and as Hetzner doesn't directly allow you to upload your own images, this means we have to be a little creative.

Fortunately, we can work around this with Hashicorp's Packer tool for creating new VM images. To do this (once you've installed the software, of course), create a new directory (perhaps called packer), and create a packer template file in one of them, which should have the .packer.hcl extension. My packer template looks like this:

packer {
    required_plugins {
        hcloud = {
        source = "github.com/hetznercloud/hcloud"
        version = "~> 1.4.0"
        }
    }
}

variable "channel" {
    type = string
    default = "beta"
}

variable "hcloud_token" {
    type = string
    default = env("HCLOUD_TOKEN")
    sensitive = true
}

source "hcloud" "flatcar" {
    token = var.hcloud_token
    image = "ubuntu-24.04"
    location = "fsn1"
    rescue = "linux64"

    snapshot_labels = {
        os = "flatcar"
        channel = var.channel
    }
    ssh_username = "root"
}

build {
    source "hcloud.flatcar" {
        name = "x86"
        server_type = "cx22"
        snapshot_name = "flatcar-${var.channel}-x86"
    }

    provisioner "shell" {

        inline = [
        # Download script and dependencies
        "apt-get -y install gawk",
        "curl -fsSLO --retry-delay 1 --retry 60 --retry-connrefused --retry-max-time 60 --connect-timeout 20 https://raw.githubusercontent.com/flatcar/init/flatcar-master/bin/flatcar-install",
        "chmod +x flatcar-install",
        # Install flatcar
        "./flatcar-install -s -o hetzner -C ${var.channel}",]
    }
}

This looks fairly similar to Terraform or OpenTofu code, and it is in fact the same HCL (Hashicorp Configuration Language). The first few blocks are identical in function to Terraform blocks of the same type as well: we connect to the Hetzner plugin with our API key and set some variables.

After that we have a source block, which defines the source image for the custom image we're making. It doesn't much matter what this image is, as we're wiping it anyway: the example I found used Ubuntu, so we may as well go with that (this is the only appropriate use of Ubuntu). We have to label the source and set a few things, including, importantly, the rescue flag: we want the machine to boot into the 64 bit system rescue utility rather than the actual OS. Next we have a build step that installs gawk (a dependency of the Flatcar install script), then downloads the install script that Flatcar provides and runs it in Hetzner mode. Finally, it saves the fresh install as a snapshot (which in Hetzner is kinda the same thing as an image but not really). There's more you can do with Packer to customise the image, and I might do some of it in the future, but as Flatcar can apparently behave strangely when you boot an image that isn't pristine, I'm a little wary of it and elected not to do it just yet.

We can now access this image in our main.tf file with the following block:

data "hcloud_image" "flatcar-beta-x86" {
    id = <img-id>
    most_recent = true
}

Hetzner provides an image id for custom images which you can look up in Terraform, but honestly I was lazy and just grabbed it from the dashboard. With that, you can then use this data source to provision a server in the normal way. So, we have our image and we have our OS: now how do we actually deploy?

First off, thank you to everyone who's chosen to support my writing through Patreon, Liberapay or Stripe, both recently and over the entire period of my writing. Seeing people support me helps me both financially, and through demonstrating that you value my writing enough to support me in writing it, it helps my mental health a lot and lets me maintain my human dignity. So, thank you!

Secondly, I'd ask that should you or anyone you know need a person with these skills (or my other ones), please get in touch with me. I gain most of my business through referrals, and a small contract or two would do me a lot of good at the moment. I know the self-promotion can be exhausting, but alas, this is the world we live in.

Finally, as always, you may subscribe to email notifications below:

Step 3: deploying the image

Unlike Fedora, Flatcar doesn't support cloud-init: instead, it relies on an init system called Ignition: it's rather more powerful than cloud-init in some ways but also insists that your system configuration is entirely declarative: that is, it won't let you run random commands on startup. You don't usually write Ignition configs directly: rather, you write it in a YAML file and get a tool called Butane to turn it into an Ignition config. My Butane config looks like this:

variant: flatcar
version: 1.0.0
storage:
  files:
    - path: /etc/ssh/sshd_config.d/custom.conf
      overwrite: true
      contents:
        inline: |
          PermitRootLogin no
          AllowUsers core
      mode: 0600
    - path: /opt/bin/docker-compose
      contents:
        source: https://github.com/docker/compose/releases/download/v2.11.2/docker-compose-linux-x86_64
        verification:
          hash: sha512-8ad55ea1234e206ecad3e8ecf30f93de5cc764a423bf8ff179b25320f148e475b569ab9ec68d90eacfe97269528ff8fef1746c05381c7d05edc85d2f2c245e69
      mode: 0755
    - path: /home/core/docker-compose.yml
      contents:
        local: compose.yml
      mode: 0644
      user:
        name: core
      group:
        name: core
    - path: /home/core/Caddyfile
      contents:
        local: Caddyfile
      mode: 0644
      user:
        name: core
      group:
        name: core
    - path: /home/core/.docker/config.json
      contents:
        local: registry_config.json
      mode: 0644
      user:
        name: core
      group:
        name: core
    - path: /etc/profile.env
      mode: 0644
      contents:
        inline: |
                    export POSTGRES_URL=<postgres-url>
                    export POSTMARK_SERVER_TOKEN=<postmark-server-token>


systemd:
  units:
    - name: application.service
      enabled: true
      contents: |
        [Unit]
        Description=deadSimpleTech Website
        [Service]
        User=core
        ExecStart=/opt/bin/docker-compose -f /home/core/docker-compose.yml up
        [Install]
        WantedBy=multi-user.target

There's a fair amount going on here, so I'll step through it a little bit. The first thing we do is write a bunch of files to the system. The first file prevents root login and password authentication by overwriting sshd_config: this is a standard security measure which we also did in the previous deployment. If we compare this to how we did it in the previous deployment, we can see that the first option is much simpler:

    - sed -i '/PermitRootLogin/d' /etc/ssh/sshd_config
    - echo "PermitRootLogin no" >> /etc/ssh/sshd_config
    - sed -i '/Password Authentication/d' /etc/ssh/sshd_config
    - echo "Password Authentication no" >> /etc/ssh/sshd_config

The next file is a Docker Compose executable that I'm downloading from Github and making executable as I'm using Docker Compose for some simple orchestration: it's worth noting here that in a production environment you should use the Docker Compose sysext for this, but I'm exhausted and burnt out and this worked, so I can't quite bring myself to try doing that yet. After that we write the Docker Compose file itself:

services:
  deadsimpletech-website:
    image: registry.gitlab.com/internal_websites/dead-simple-consulting/stable
    restart: unless-stopped
    ports:
      - "3000:3000"
    environment:
      - POSTMARK_SERVER_TOKEN=<postmark-key>
      - POSTGRES_URL=<postgres-url>
  caddy:
    image: caddy:2.10.0-alpine
    restart: unless-stopped
    ports:
      - "80:80"
      - "443:443"
      - "443:443/udp"
    volumes:
      - ./Caddyfile:/etc/caddy/Caddyfile
      - caddy_data:/data
      - caddy_config:/config
volumes:
  caddy_data:
  caddy_config:

We note here a few things: firstly, we're reading this in from a local file, which cloud-init doesn't let you do. The file is injected into the Ignition config as a base64 string, and if your deployments mostly happen locally, this is a really useful tool to have on hand. Secondly, we have two images here: one is our website, and the other is a Caddy image that acts as a reverse proxy. Containerising Caddy in this way makes deploying my entire stack to any platform much easier and more repeatable than what I was doing previously and is honestly just a really clean and elegant way to do it. Finally, the eagle-eyed among you might have noticed that there are credentials in that file being assigned to environment variable. Yes, this is really bad practice: unfortunately, one of the things I really don't like about Flatcar is that it's hard to inject secrets into a Flatcar VM in a secure way and without SSH. Even if you're careful, the variables tend to end up in files on disk, and I've still not found a pattern for doing that that I actually like. This is likely because Flatcar is designed to be used with orchestration services like Kubernetes which has its own ways of dealing with this, but it's still a bit of a pain.

Having written that, we then write a Caddyfile (which is largely identical to the one in the last article) and the Docker configuration file, which lets Docker on Flatcar authenticate to my Gitlab container registry. This is another credential, which isn't ideal, but this one's somewhat less bad than the others as it's a heavily locked-down deploy token that can't do anything other than download the container image.

Finally, we create a systemd unit that runs Docker Compose with the configuration specified. It's important to specify that this is owned by the Core user otherwise we won't authenticate to the container registry correctly (and also because running this as root is a bad idea). Having constructed that config, we can then run

butane -d . init_butane.yml > init.json

in the directory with the config files in it to create our Ignition config. Now we can make some minor alterations to our main.tf file from last time:

resource "hcloud_server" "deadsimple_website_server" {
  name = "deadsimple-website-server"
  server_type = "ccx13"
  location = "hel1"
  image = data.hcloud_image.flatcar-beta-x86.id
  firewall_ids = [hcloud_firewall.web_server_and_ssh.id]

  network {
    network_id = hcloud_network.deadsimple_network.id
  }

  user_data = file("${path.module}/config_files/init.json")

  depends_on = [
    hcloud_network_subnet.deadsimple_network_subnet
  ]

}

All we've done here is changed out the image we want to use for our custom Flatcar image and switched out what we're feeding to user_data for our new Ignition config. This being done, we can then run


 tofu apply -replace="hcloud_server.deadsimple_website_server"

To recreate the server whenever we want to deploy. And there we have it: our deployment's containerised!

Reflections and next steps

So, how does it all work? The first thing to note is that working with containers feels idiomatically nicer than doing everything manually like I did last time. It's hard to say how meaningful that actually is, but the code feels better-ordered with fewer messy bits and less cruft. That's probably not enough to justify containerising on its own, but it is a significant consideration. Decoupling the build step from the deploy step and automating builds is another big thing: if I want to change where I deploy down the line, it makes it much easier to do that and it's just more aesthetically pleasing besides: building on the host was ugly. Finally, having images stored in a registry is excellent for distribution.

Deployment speed is a mixed bag. For some reason Hetzner takes considerably longer to create a VM from a custom image than it does from a standard one, so the OpenTofu run takes somewhat longer on average (two minutes as opposed to forty seconds or so). Against that, once the resource is created, startup times are much, much faster (basically immediate, whereas I'd have to wait two to three minutes for everything to build and start up after creation previously), so this is a significant improvement and will make it a lot easier to move to a zero-downtime deployment pattern down the line.

I can't really speak to security questions: I hate how I'm dealing with credentials at the moment, but the other options feel like they create further security issues so there are obvious tradeoffs to be made here. One of the biggest ones is that I'm going to struggle with a credential handling scheme that I don't actually have the memory or willpower to stick to or configure in the first place, so when I improve this, I need a sufficiently simple way of handling this. Overall though, containerisation and running on Flatcar is likely to have somewhat improved the security of my deployments, though by how much is anyone's guess.

Regarding where I want to go with this next, security improvements are an important point of attention, but shouldn't take too much work to implement (I hope). In terms of larger projects, I need to get my database off Neon, which means setting up a PostgreSQL cluster on Hetzner. This is probably going to be my next project. After that, I'd like to self-host a Gitlab or Gitea instance (haven't chosen which yet) and automate my deployments as well as my builds. Finally, I really need to sort out a testing strategy for this: I have some tests, but I've been fairly lax on them so far. Conveniently enough, learning how to do stuff with containers makes all of these things significantly easier.

Once I've done all of that, I think I'll have a website that I can be properly proud of.

I'm currently open for contracts! If you need me to do work in any of the fields I've written about here, or know someone who does, please write to me at [email protected] and we can set up a conversation.

Otherwise, if you found this article interesting, insightful or inflammatory, please share it to social media using one of the links below or contribute to my Liberapay or Patreon (these help me even out my consulting income and let me keep writing). We also like getting and respond to reader mail: please direct that to [email protected] should you feel moved to write.

Share:

Follow me: Mastodon | Bluesky | LinkedIn | RSS | Email

Support my work: Liberapay | Patreon | Donate

This website is part of the Epesooj Webring

Previous website | Next website