Running K3s on a Raspberry Pi sounds simple, until you try to make a rebuild repeatable: bootstrap the node, get Flux access to Git, and recreate credentials without committing plaintext secrets. I learned a bunch of things the hard way, and I hope writing them down saves the next person (or GenAI model) some of the same back-and-forth. The pattern that worked was a clean hand-off: Ansible prepares the host and the bootstrap prerequisites; Flux takes over once it can read the GitOps repo.
Ansible gets the first cluster running
I started with an Ansible inventory and a site playbook for the Pi. Ansible prepares the host, installs k3s, installs the Sealed Secrets controller, and installs Flux. The ordering matters: the Sealed Secrets controller has to exist before Flux can apply `SealedSecret` resources.
Flux has a bootstrap problem of its own. It cannot fetch the Git repository until it can authenticate to GitHub, so its first authentication Secret cannot come from that repository. I put the private deploy key and GitHub `known_hosts` value in Ansible Vault. The vault file is encrypted with `ansible-vault`, kept out of Git in this setup, and its password is stored separately. The deploy key itself is read-only and scoped to the GitOps repository.
On that first bootstrap, Ansible installs Flux, creates the ordinary Kubernetes Secret named `flux-system` from those Vault values, and only then applies the Flux `GitRepository` and sync objects.
Sealing credentials for Git
Once the controller was running, I could create a Kubernetes Secret on the server and pass its YAML through `kubeseal`. The output is a `SealedSecret`: encrypted for the controller, safe to commit to Git, and tied by default to its namespace and name. During normal operation, Flux reads that manifest from Git and applies it. The Sealed Secrets controller decrypts it, and Kubernetes gets the regular Secret the application needs.
The committed file contains encrypted values, not the original Secret data. This trimmed example shows the shape; the ciphertext is a placeholder, not something to apply:
```yaml
apiVersion: bitnami.com/v1alpha1
kind: SealedSecret
metadata:
name: some-credentials
namespace: some-namespace
spec:
encryptedData:
admin-password: "REDACTED_CIPHERTEXT_FROM_KUBESEAL"
template:
metadata:
name: some-credentials
namespace: some-namespace
type: Opaque
```
I used the same pattern for credentials I wanted represented in Git, including a sealed copy of the Flux `flux-system` credential. At first this sounds circular: Flux needs the Git deploy key to fetch the repository, but the sealed deploy key is stored in that repository. The way around that is for Ansible to apply the local copy of the SealedSecret directly to the cluster before it creates the Flux `GitRepository` and sync objects. Flux is not fetching its own bootstrap credential; Ansible already has the checked-out GitOps files and can reach the Kubernetes API. The Sealed Secrets controller decrypts the manifest and creates the ordinary `Secret/flux-system`; Ansible can then let Flux start pulling from Git.
The first setup needs a seed credential because there is no sealed deploy key in Git yet, so I used Ansible Vault. Once I had committed the SealedSecret, I changed Ansible's later-rebuild path: when both the local manifest and the matching controller-key backup are present, Ansible applies the manifest from its local Git checkout and waits for the controller to create `Secret/flux-system` before applying the Flux source. If either file is missing, Ansible falls back to Vault. Flux will never be in the akward situation of having to fetch the credential it needs in order to fetch Git.
```yaml
- name: Apply the sealed Flux Git credential
kubernetes.core.k8s:
state: present
src: "{{ flux_git_sealed_secret_file }}"
kubeconfig: /etc/rancher/k3s/k3s.yaml
no_log: true
when:
- flux_git_sealed_secret_file_status.stat.exists
- sealed_secrets_controller_key_file_status.stat.exists
```
I also mark the Vault-created Secret with `sealedsecrets.bitnami.com/managed: "true"`. That explicitly allows the controller to take it over and avoid the fight described next.
The fight over not managed secrets
While building out workloads, I sometimes created resources directly in the cluster and moved them into Git later. That led to a few reconciliation timeouts. `flux get kustomizations -n flux-system` showed the root reconciliation waiting on a SealedSecret that stayed `InProgress`; `kubectl -n some-namespace describe sealedsecret some-credentials` exposed the “not managed” conflict.
The cause was an ordinary Secret with the same name that already existed before the SealedSecret was applied. Since the controller will not automatically take over an existing Secret, it left the SealedSecret unreconciled and Flux's health check eventually timed out.
After I removed the conflicting Secret and reconciled the root Kustomization again, the controller created it from the sealed manifest. That was safe for this credential because it could be recreated.
The lesson: a Flux timeout is a symptom, not necessarily the cause. Inspect the resource Flux says is still progressing before increasing timeouts or deleting anything. The Envoy Helm hook was a different case, where the operation really did need more time.
A slow Helm install needs a longer timeout
Envoy Gateway also failed during its initial Helm install. I first suspected Docker Hub throttling, but there was no `429` or other evidence of registry throttling and the pod events showed the certgen image pull succeeded and took about 4 minutes and 55 seconds. The Helm operation timed out while the job was still in progress, then Kubernetes reported the job completed.
I raised the Envoy Gateway `HelmRelease` timeout to ten minutes. That gives the initial pull and certgen hook room to finish on the Pi.
The CRD has to exist before the custom resource
Once Flux was reconciling the networking manifests, I hit another ordering problem: `no matches for kind "GatewayClass"`. The `GatewayClass` resource and the Envoy Gateway HelmRelease were being applied as part of the same reconciliation. The Helm chart had not installed the Gateway API CRDs when Flux tried to build and apply the custom resource.
I split the work into two Flux Kustomizations. One reconciles the ingress path that installs Envoy Gateway and waits for it to become Ready. The other reconciles the network routes and declares `dependsOn: ingress`. That makes Flux wait for the ingress reconciliation, rather than relying on the order of files in a plain `kustomization.yaml`. That's a great tool to use when ordering does matter.
In simplified form, the two Flux resources look like this:
```yaml
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: ingress
namespace: flux-system
spec:
interval: 5m
wait: true
path: ./clusters/production/infrastructure/ingress
sourceRef:
kind: GitRepository
name: platform-gitops
---
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: network
namespace: flux-system
spec:
interval: 5m
wait: true
dependsOn:
- name: ingress
path: ./clusters/production/workloads/network
sourceRef:
kind: GitRepository
name: platform-gitops
```
Back up the key that does the decrypting
To rebuild the cluster without resealing every credential, I needed the Sealed Secrets controller's private key. I exported the controller Secret marked `sealedsecrets.bitnami.com/sealed-secrets-key=active` and stored the export on an offline USB backup, outside Git. If I lose that key, a fresh controller cannot decrypt the existing SealedSecret manifests. It is private key material.
My first Ansible ordering restored that key after installing the controller. That let the controller start up and generate a new key before the original one arrived, which meant an extra key and a controller restart. I changed the role to create the namespace, restore the backed-up key, and only then install the controller. On a rebuild, it starts with the key that can decrypt the existing Git-managed secrets.
Outro
Overall, it was a great weekend project: I dusted off my Kubernetes knowledge, learned Ansible, explored Gateway API and HTTPRoutes after reading about the Ingress NGINX retirement, and built a small observability stack with OpenTelemetry, Loki, Prometheus, and Grafana. GenAI helped me move quickly, but the most useful habit was treating its suggestions as hypotheses: check the manifests, inspect the live cluster, and ask “do I actually need this?” before adding another component. That back-and-forth helped me challenge my agents and end up with a solution that fits my needs.
No comments:
Post a Comment