Pipeliner - a CI/CD pipeline runner

Written by Michael Anckaert - Published on - Posted in general

This article is about a side project I started this weekend: a CI/CD pipeline runner based on Django, HTMX, Go and Docker. We had a "long weekend" in Belgium due to All Saints' Day, so it was the perfect opportunity to play around with an idea I had for a some time.

I love to build stuff and figure out how existing things work. Having used tools such as Jenkins, CircleCI or Github actions for many years, I always wanted to build my own version of it. So the idea is to build a CI/CD tool.

Sideproject Status

After 8 hours of hacking, I've reached a pretty impressive MVP! Currently I have a working CI/CD tool with the following features:

CI/CD Tool

The basic workings of such a tool can be described quite simple. But in the simplicity is a lot of hidden complexity. The basic gist is this: A Git commit triggers a new build that is registered in a controller. The build is then picked up by an agent that executes the build, and reports back to the controller.

On the agent, the build is typically executed in a closed environment. This avoids the issue where builds can "taint" the environment that runs the builds, such as by installing required software. A perfect technology to achieve this Docker (or similar OCI containers), which is also what we will use. Configuration of the build is done using a YAML config in the Git repository. A build configuration can range from very simple to very complex. Multi step, parallel builds are quite common in modern software builds.


The pipeliner architecture is pretty simple at the moment. I have chosen the technologies I'm most comfortable with, but also some I'm not extremely comfortable with because they're a good pick for what I want to achieve.

On the Controller side I've picked Django, my favourite python Web framework. It is the Controllers' responsibility to receive Webhook calls from Github (or other platforms), expose an API for the Agents to call and offer an interface to our users to display the builds and the build outputs. Django Rest Framework is used to expose a REST API.

I picked server side rendered HTML as the main technology for the user interface. Thanks to HTMX I was able to add some nice interactivity and UI / UX improvements without having to resort to a large scale framework such as Vue or React. Using HTMX makes me very productive on the frontend, which as a more backend leaning developer is a big bonus. Read more about HTMX and HTML over the wire concepts.

The Agent is written in Go. I wrote the first proof of concept in Bash and when validated I switched around to Go. The reasons I chose Go is the single binary deployment and easy concurrency. Thinking ahead (if this every becomes an actual service or product), offering a local agent to run on customer owned / controlled hardware should be quick and easy. A single binary to install on a Docker capable server is perfect for this use case.

Technical challenges

While the current MVP version is impressive, it lacks a lot of refinement. Besides refinement and polish, the largest challenge is to fix the log streaming between the build agent and the controller. Currently I capture the log output and only send it to the Controller when the current build step is complete. Ideally, I can capture the output and stream it to the controller to continually present it to the user.

Functionality to add