Rethinking GitOps on Large Kubernetes Platforms

The Sweet Lie of GitOps

Once we crossed 50 clusters, problems started surfacing in our GitOps setup that I'd never seen before. The system had been designed around a belief: "Git is the single source of truth." At scale, that belief started creaking.

At first I chalked it up to tool configuration. I tuned polling intervals. Split repositories. Threw more resources at the controllers. Things would improve for a while, then the same symptoms came back as soon as we added more clusters. That's when it hit me. Maybe this isn't a tuning problem. Maybe the premise itself is wrong.

So here's what I want to work through: how much messy reality that tidy little slogan has been hiding all these years in the cloud native ecosystem, and why large environments need a different approach.

State, intent, reality — let's fix the vocabulary first

The word "state" shows up everywhere in Kubernetes discussions, and it's carrying three completely different concepts at once. Miss the distinction and the whole GitOps argument ends up circling the wrong drain.

Actual state — what's happening right now

The first is the cluster's actual runtime state. Which Pods are up at this exact second, which containers are healthy, what the IPs are, which nodes are getting hammered. This state never sits still. It changes by the second.

Last known state — operational memory

The second is the cluster's operational memory. In Kubernetes, that's what lives in etcd. The objects and metadata that controllers read and write constantly, holding the cluster's current understanding of the world. In practice, etcd is the cluster's real state store. Every debugging session I've ever run converged there eventually.

Desired state — our intent

The third is desired state — what we want the system to look like. YAML manifests, Helm charts, custom settings. These files don't describe what's happening. They describe what we'd like to happen.

Git enters here. It stores those manifests and version-controls the intent. The trouble starts the moment you lump all three together and declare "Git is the state."

Why Git isn't a real state store

A real state store continuously records and updates state. It handles frequent reads and writes, holds runtime information, and knows what the system looks like right now. Databases, distributed key-value stores, Kubernetes' etcd.

If Git were genuinely a state store, you'd expect controllers writing runtime information back to the repo nonstop — bidirectional GitOps, essentially. Replica count changed: commit. Pod created: commit. That Pod got IP 10.1.2.3: commit. That Pod fell into CrashLoopBackOff a minute later: commit.

None of that happens. Git doesn't record runtime changes. It has no idea whether a deployment failed or a Pod is dying. That's exactly why staring at Git history at 3am during an outage gets you nowhere.

Git is a system for storing static declarations written by humans or pipelines. It does not function as operational memory.

What Git actually does, and where it stops

Git's actual job is a lot humbler. It's a system of record for intent. Who changed what configuration, when, and why. That's genuinely valuable, but it's a different animal from storing operational state.

And that endlessly repeated line — "Git is the source of truth" — falls apart under mild inspection. In the common pattern, Git holds intent, while most of the dependencies that matter are only referenced indirectly. Helm charts reference other charts, charts reference container images. Picture a trivial manifest:

image: my-app:latest
# 또는
image: my-app:3.0.1

The manifest records the intent to deploy this application, but the actual artifact lives somewhere else, usually a container registry. Rebuild and push the image behind that tag and Git doesn't change by a single character, yet the cluster pulls something entirely different. Anyone who's wired a latest tag into production has been burned by this at least once.

The real artifact — the part that actually runs — lives outside Git. That fact alone undermines "Git = truth" from the start.

Where Git breaks at scale

Traditional GitOps works beautifully in small environments. A handful of clusters, a handful of apps, and nothing shows. The story changes when the platform grows.

Picture a typical platform setup. 50 clusters, 5 platform applications each, and you're running 250 GitOps-controlled deployments. Each GitOps controller checks its source for changes on an interval — 3 minutes by default. If the source is Git, that means every one of those controllers is in constant conversation with the repository.

A single poll drags several steps behind it. Authenticate, open a connection, sync the repository, compare the commit graph. All of it repeats every cycle even when nothing changed. The same work, over and over, just to establish that nothing happened.

Modern GitOps tools try to trim this overhead with shallow clones, sparse checkout, repository partitioning. Some teams experiment with webhooks to skip polling entirely. These help. What they can't change is that the Git protocol is chatty by nature.

Run hundreds of controllers at once and the Git server accumulates load it never needed to carry. Past a certain point that load stops looking like load and starts looking like a performance problem. It begins as the occasional slow sync. Later it spreads into deployment delays and timeouts.

OCI artifacts as the delivery layer

OCI registries were built to solve a completely different problem: distributing artifacts at scale. Container registries were designed from day one to serve thousands of nodes simultaneously. High concurrency and global distribution are their normal operating conditions, not an emergency.

When configuration is packaged as an OCI artifact, the interaction gets much simpler. The CI pipeline builds an artifact containing manifests and metadata and pushes it to an OCI registry. The GitOps controller checks whether that artifact's digest changed, and pulls and applies only when it did. The difference is that the controller is no longer negotiating repository history. It asks one question: "Is this digest still the same?" No commit graph comparison.

That simplicity makes OCI a much lighter delivery mechanism.

OCI has a clear boundary too. An OCI artifact is an immutable package. Perfect for delivery, wrong for the place where configuration gets authored and refined. Try to edit configuration there and it gets awkward fast. OCI is best understood as the delivery layer.

Why you need a configuration data layer

If Git stores intent and OCI delivers artifacts, one gap remains. Where do the actual configuration values live?

This is where a configuration management system like ConfigHub comes in. ConfigHub treats configuration as structured data instead of a pile of files scattered across repositories. That difference matters more than it sounds.

The approach deliberately embraces WET — Write Everything Twice. Not to duplicate human effort, but to produce explicit configuration a machine can read. Worship DRY long enough and you end up with abstractions stacked on abstractions until nobody can answer the simple question of what values actually land in this cluster. WET stands on the other side of that.

When configuration is written out explicitly, you can validate the final result before deploying, read how that configuration was assembled, and debug without heroics. It gets closer to a real WYSIWYG model: what you see is what gets deployed. If you've ever debugged by rendering a template in your head, you already know why this matters. Honestly, this is the point where I gave up on DRY as an article of faith.

ConfigHub pulls together Git repositories, secret stores like Vault, databases, and deployment metadata in one place, then converts those inputs into a validated configuration representation. It takes scattered truths and organizes them into one queryable form.

Regulation, the other kind of pressure

Scalability isn't the only reason platforms are revisiting their architecture. Regulation is becoming a serious driver.

The EU Cyber Resilience Act (CRA) brings strict security requirements to products containing digital components, and much of it becomes mandatory by 2027. If your organization sells software or SaaS into the EU, you'll have a hard time staying in the market without software supply chain transparency, vulnerability management, and a Software Bill of Materials (SBOM). CRA follows market principles, so any company selling software in the EU falls under it regardless of where headquarters sits. Being a Korean company doesn't get you out of it.

Git alone can't fully guarantee supply chain traceability. Merging commits or rewriting history quietly erases parts of the audit trail. When you have to prove your history is demonstrably safe, traceability that a single force push can shake is a risk in itself.

OCI artifacts, on the other hand, bundle deployment manifests, container images, SBOMs, and signatures into one immutable artifact. Bundled that way, each artifact becomes a verifiable snapshot of a deployment. Compliance stops being an operational burden and falls out of the architecture naturally.

A GitOps architecture that survives contact with reality

Don't misread this: the goal isn't to remove Git from the architecture. Git remains hard to replace for collaboration and version control. The point is putting it back in its proper role.

A GitOps architecture that holds up at scale typically splits responsibilities:

구성요소	역할	특징
Git	의도 기록	개발자와 플랫폼 팀의 변경사항 추적
OCI	아티팩트 배포	불변 배포 아티팩트, 대규모 확장 가능
ConfigHub	구성 집계	구성 구조에 대한 쿼리 가능한 진실의 원천
Kubernetes/etcd	운영 상태	클러스터의 실제 운영 상태 저장

Each component does only what it's best at. Don't dump operational state on Git, don't ask OCI to edit configuration, don't mistake etcd for a configuration management tool.

Once responsibilities separate like this, questions that a file-based repository could never answer suddenly become answerable.

• Which applications are running in staging across every cluster in Europe?
• Which workloads are missing a securityContext?
• Which deployments reference a vulnerable image?

These feel like questions you never need to ask — right up until an incident or a security event. In that moment, whether you can pin down the exact state of the system in a few minutes is what separates a good incident response from a bad one. You don't have time to grep across a dozen repositories.

Wrapping up

"Git is the single source of truth" was a useful mental model in the early GitOps days. It kept things simple in your head. As the platform grows, the complexity that slogan was papering over surfaces one piece at a time.

Git is a powerful tool. It just wasn't built to serve as your infrastructure's operational memory. Treat it like a state store and confusion and scaling problems follow. That's exactly what I hit somewhere around cluster 50. If I'd figured it out earlier I wouldn't have wasted all that time splitting repositories.

Change the scale and you change the rules. Assumptions that were correct in a small environment turn into traps in a large one, with the same tools. I've dropped the habit of using Git as a database, and I now treat configuration files as data that needs to be handled like data. Whether that's the answer that holds for the next decade, I honestly don't know yet.