GitOps Home Lab

A production-grade GitOps stack running on a single bare metal machine. Every component is declared in Git and reconciled by ArgoCD using the App of Apps pattern. Secrets flow through HashiCorp Vault to workloads via External Secrets Operator. TLS is automatic via cert-manager and Let's Encrypt with DNS-01 validation against Route 53. The stack is observable, policy-enforced, and reachable by hostname from any device on the network.

External traffic arrives at the IP assigned and announced by MetalLB via L2 ARP on the host network. ingress-nginx terminates TLS and routes requests to the appropriate service. ArgoCD watches the private GitHub repository and syncs changes to the cluster automatically. Vault runs as a systemd service on the host and unseals itself on startup via an AWS KMS customer-managed key.

Several Helm charts bundle their Custom Resource Definitions in a format that exceeds Kubernetes's 262KB annotation limit, causing installs to fail. The reliable fix is to install CRDs out-of-band using a CRDs-only manifest, then configure the chart to skip CRD management. This applied to kube-prometheus-stack, cert-manager, CloudNativePG, and Kyverno. Applying a full installation bundle instead of a CRDs-only manifest creates a second operator instance, which caused a separate crash loop described below.

The alertmanager and prometheus StatefulSets entered a persistent crash loop with pods cycling through init, completion, and termination thousands of times without stabilizing. The root cause was two prometheus-operator instances running simultaneously: one deployed by ArgoCD via Helm, and one created when the full prometheus-operator bundle was applied to work around the CRD size limit. Both operators were reconciling the same resources and overwriting each other's changes, including emptying generated secrets immediately after they were patched. Deleting the stray operator from the default namespace resolved the loop immediately.

K3s's built-in ServiceLB created iptables rules inside the container's network namespace rather than the host's, so external traffic never reached the ingress controller. Switching to MetalLB resolved the forwarding problem, but the initial IP assignment conflicted with the host's existing network interface, causing ARP responses to route through loopback. The fix required assigning MetalLB a dedicated IP outside the DHCP range, adding it as a secondary address on the host interface via Netplan, and specifying the physical interface explicitly in the L2Advertisement. A Kyverno-generated NetworkPolicy was also blocking all ingress to the ingress-nginx namespace, producing an identical connection refused error that required iptables tracing to isolate.

External Secrets Operator requires Vault to be unsealed before it can sync secrets, and Vault required manual unsealing after every restart. The seal was migrated from Shamir key shares to AWS KMS using a customer-managed key, with a dedicated IAM user scoped to the three KMS actions required for unseal on that specific key. Vault now unseals automatically on startup, eliminating the manual intervention that would otherwise block the rest of the stack from initializing.