GitLab on Kubernetes, staged and reproducible

A GitLab platform cluster that Flux brings up in order, every time, with no public origin. Plus the less glamorous half of Kubernetes: upgrading EKS from 1.33 to 1.35 without anyone noticing.

client
Anchore
role
Principal DevOps Engineer
when
2024 to present
drawn by
Jesse Tamburino
Git repodesired stateFluxGITLAB PLATFORM CLUSTERCRDsinstalled first01GitLab Operatorwaits on CRDs02Runner Operatorwaits on GitLab03GitLabweb + registryGitLab RunnersCI capacityeach stage applies only after the one above is readyCLOUDFLARE04TunnelGitLabTunnelregistryno public originSTATE, OUTSIDE PODSS3 objectsPostgresRedisEKS UPGRADES · SEVERAL ENVIRONMENTS051.331.341.35Bottlerocket node AMIsKarpenter node poolscore add-on version syncload-balancer-controller plan
Flux reconciles three stages in order; the edge is the only way in.Hover a step below to find it on the sheet. Scroll sideways on small screens.
3staged rollout phases
0public origins
1.35EKS, from 1.33

the problem

An internal GitLab platform has to come up the same way every time, and operators that depend on CRDs fall over if they arrive first. It also should not need a public origin to be reachable by the people who use it.

Constraints

  • Operators need their CRDs installed before they reconcile anything.
  • The runner operator needs a working GitLab to register against.
  • GitLab and its container registry both need to be reachable without opening inbound ports.
  • Stateful pieces should survive the cluster being rebuilt.

Stack

KubernetesFluxGitLab OperatorCloudflare TunnelEKSKarpenterBottlerocketS3PostgresRedis

how it is built

  1. 01

    CRDs first

    Flux installs the custom resource definitions as their own stage.

  2. 02

    GitLab Operator

    Applied only after the CRDs exist, then brings up GitLab and its registry.

  3. 03

    Runner Operator

    The last stage: GitLab Runner Operator, once there is a GitLab to register runners against.

  4. 04

    Cloudflare tunnels

    Tunnels for GitLab and its container registry. Users reach it through Cloudflare; the cluster has no public origin.

  5. 05

    EKS upgrades

    Separately, upgraded EKS from 1.33 through 1.35 across several internal environments: Bottlerocket node AMI updates, Karpenter node pool adjustments, core add-on version syncs, and load-balancer-controller planning.

calls I made

Why it looks like this.

01

Order is part of the config

Staging the rollout in Flux makes the dependency order explicit and repeatable, instead of a README that says "apply this one first".

02

Tunnels over load balancers

An outbound tunnel means nothing in the cluster listens on the internet. Access policy lives at the edge, next to the WAF and DNS I already manage.

03

Keep state out of the pods

Objects in S3, data in Postgres and Redis. The cluster becomes something you can rebuild from Git.

Want the longer version, with the parts that went wrong?

Ask me about it