First of all, this is likely to be non-trivial to implement. I'd personally see this as a post-second release feature at best.

I've been running my build workloads from Gitea via Drone on my Kubernetes cluster for a while now, and I've grown to like the immutability of the surrounding environment for builds. On top of that, I think that Kubernetes is a great tool for managing these types of workloads that don't need to keep long term reserved resources.

Status Quo

Currently, Drydock relies on SSH nodes from Almanac to create resources from blueprints, and can't talk to kubeapi.
For people wishing to use Kubernetes for Harbormaster, that would mean keeping Pods around indefinitely in a cluster for Drydock and Harbormaster to use, which is wildly inefficient.

Proposal

Instead, Almanac would allow the storage of KubeAPI credentials (client certificate and key / token) so that other apps can connect to and interact with the cluster.

Drydock Host Blueprint would generate an ssh key, and upload the public part as a Secret with the key authorized_keys to the cluster. After that, it would create a Pod with a Secret mount to the /root/.ssh directory, followed by a Service that points to the Pod's port 22.
The image used by the Pod should be one that runs sshd and accepts ssh connections as root from /root/.ssh/authorized_keys and provides the relevant repository management tools or apk (the Alpine package manager), could be docker.io/linuxkit/sshd:v0.8 which supports non x86-64 architectures as well, but the name of which can be provided manually by the creator of the blueprint (along with the name of imagePullSecret in the cluster if necessary)

A Working Copy blueprint would then be able to use the standard SSH protocol to connect to the pod in one of two ways:

1. If Phorge is running in the same cluster, or is otherwise a part of the cluster's internal network (not to be confused with the LAN on which the cluster nodes are), in which case it can just SSH with the name of the Service as hostname
2. If Phorge is running outside of the cluster and does not have access to the cluster's internal network, in which case, kubectl would be used to forward a random local port to the cluster Service, and the SSH connection would be made to that port on localhost

Once the connection is established, Drydock would test for the presence of the relevant repository management tool, and install it via apk add if needed, before creating the working copy.

Once a Working Copy resource is ready, it can be used by Harbormaster for builds. The process for connecting to the SSH server would be similar to that for the Working Copy

Arc Lint + Arc Test builds will probably need to install the relevant toolsets on the pods as well

Up for debate

If this is to be implemented, the following details should be discussed more in depth

• Which container image and underlying distro to use? I picked the one mentioned because it supports multiple architectures, and not just amd64
• Should we/can we maintain our own image instead? the advantage here would be that we can skip the installation when creating Drydock resources and running Harbormaster builds, and instead require people who use their own images to provide at least the tools relevant to them
• Should we even have a default image? Similar to above, we could instead just ask the user to provide their own image built for the correct architecture, with the relevant tools available in the image, which would mean that the only additional steps for Working Copy and Builds would be the port forward (if running outside cluster)
• Should we use NodePort Services instead? - this would eliminate the need to use kubectl to forward a local port to the ssh service running on the cluster, but only if Phorge is on the same LAN as cluster nodes, or cluster nodes have public IPs
• Probably some other stuff that I (or someone else) didn't come up with yet.

Bonuses

The original implementation of Drydock and Harbormaster doesn't provide much in terms of isolation between builds and between build and the host machine. Implementing Kubernetes "hosts" would essentially give us this for free, provided the same Host isn't reused for multiple Working Copies of different repositories.