Building an AI Agent to Run Kubernetes - A Practical 2026 Guide

Why operators aren't enough

3:17 a.m. One notification and my eyes are open. Laptop up, and within a minute I've got six panes on screen. Pod status, metrics, deploy history, logs. If you've run Kubernetes for a few years this scene needs no introduction. We bolted on all those Operators and Controllers, and a human still gets dragged out of bed.

The reason is simple. An operator handles the scenarios its author imagined in advance. Production breaks outside them.

For years, autonomous management of Kubernetes infrastructure has meant code — operators and controllers. Watch a resource type, compare current state to desired state, make a change that closes the gap. A well-built control loop is elegant. That elegance only holds inside the paths someone drew. Which is exactly why operators are both powerful and brittle. Step outside the imagined range and they can do nothing at all.

When a human debugs an incident, we're really doing one thing: keeping several screens open and cross-checking them in our head. Cross-checking is precisely what an operator can't do. It's looking at one resource.

As of 2026, AI agents with tool access to the Kubernetes API are shaking that loose. An agent isn't bound to preprogrammed scenarios. It reasons about a situation it hasn't seen, pulls on learned knowledge of Kubernetes operational patterns, queries several information sources at once, and proposes or executes a fitting action. It's a different kind of approach from an Operator. The research-prototype phase is over; this is coming down into production now.

Don't relax just because something "reasons," though. I'll come back to this, but reasoning is only safe on top of control.

Who this is for

Platform engineers and SREs who run Kubernetes clusters and want to automate operational work with AI agents. Some Kubernetes operations experience helps, plus a working understanding of Python and LLM APIs. No agent development experience required. When I first put this structure together, the agent side was close to a blank page for me.

Everything here is available today. I wrote it around implementations you can actually deploy. Diagrams that only look good on a slide are left out.

Five layers, and why the agent never touches kubectl

An AI-driven Kubernetes management agent has five layers. One design principle drives it: the AI agent never gets its hands on kubectl directly. It reaches the cluster only through a typed, audited tool interface — an MCP server.

Why lock it down that hard? Anyone who's run production knows. The danger isn't smart automation. It's uncontrolled automation.

Every tool call is logged. Every parameter is validated against defined constraints. Every write is stopped at an approval layer before execution. That controlled interface is what makes AI cluster management shippable to production. Put another way: handing an AI raw kubectl with no interface is a demo, not an operation.

The theme running through this whole post is security and control. Reasoning ability comes second.

Step 1: Set up the Kubernetes MCP server

A Kubernetes MCP server exposes cluster operations as typed tools the AI calls. As of 2026, the most mature implementations cover most of the kubectl API surface — reads (get, describe, logs), writes (apply, patch, delete), diagnostics (exec, port-forward, top).

One rule before anything else. The initial deployment is read-only. No exceptions.

# Kubernetes MCP 서버 설치
pip install kubernetes-mcp-server

# MCP 서버 서비스 계정에 대한 RBAC 구성
# 초기 배포 시 읽기 전용으로 시작
cat > k8s-mcp-rbac.yaml << 'EOF'
apiVersion: v1
kind: ServiceAccount
metadata:
  name: ai-agent-readonly
  namespace: platform-tools
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: ai-agent-readonly
rules:
- apiGroups: ['*']
  resources: ['*']
  verbs: ['get', 'list', 'watch']
- apiGroups: ['']
  resources: ['pods/log', 'pods/exec']
  verbs: ['get', 'create']
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: ai-agent-readonly
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: ai-agent-readonly
subjects:
- kind: ServiceAccount
  name: ai-agent-readonly
  namespace: platform-tools
EOF
kubectl apply -f k8s-mcp-rbac.yaml

That RBAC leaves pods/exec open. exec is useful for diagnostics, but it's also a door into the container. On a production cluster I sleep better leaving it out at first too.

Choosing which tools to expose

Turn on the smallest set of tools, validate how the agent behaves, then expand one at a time. Open everything at once and you'll never trace back which tool caused the problem.

# mcp-server-config.yaml
tools:
  # 읽기 도구 - 항상 사용 가능
  enabled:
    - kubernetes_get_pods
    - kubernetes_get_deployments
    - kubernetes_get_services
    - kubernetes_get_nodes
    - kubernetes_get_events
    - kubernetes_get_pod_logs
    - kubernetes_describe_resource
    - kubernetes_get_resource_usage
    - kubernetes_get_hpa_status
  
  # 쓰기 도구 - 승인 게이트 필요
  gated:
    - kubernetes_apply_manifest
    - kubernetes_patch_resource
    - kubernetes_rollout_restart
    - kubernetes_scale_deployment
    - kubernetes_delete_resource

  # 진단 도구 - 로깅과 함께 사용 가능
  diagnostic:
    - kubernetes_exec_command
    - kubernetes_port_forward

Splitting tools into enabled / gated / diagnostic is the whole trick. Reads run free, writes have to clear an approval gate, diagnostics always leave a log. Those three lines are basically the operational policy.

Step 2: Wire in observability context

An agent that only sees cluster state is working with half a brain. It can read Pod status, deployment specs and resource limits perfectly and still be unable to make an operational call. Same as a human looking at an incident. "The Pod died" on its own tells you nothing. You need metrics, logs and traces before why it died comes into view.

# Prometheus MCP 서버 구성
# 에이전트는 이제 Pod CPU/메모리, 서비스 오류율,
# HPA 메트릭, 사용자 지정 애플리케이션 메트릭을 쿼리할 수 있음
prometheus_mcp_config:
  endpoint: http://prometheus.monitoring.svc:9090
  allowed_queries:
    - 'rate(http_requests_total[5m])'
    - 'container_memory_working_set_bytes'
    - 'container_cpu_usage_seconds_total'
    - 'kube_pod_container_resource_limits'
    - 'kube_deployment_status_replicas_available'
  read_only: true
  max_query_range: 24h

The part worth noticing is allowed_queries — the query set is whitelisted. Let the agent throw arbitrary PromQL and cost leaks out the side, and one heavy query is enough to make Prometheus wobble.

# 로그 쿼리 MCP 구성
# 에이전트는 Pod를 직접 실행하지 않고도 구조화된 로그를 검색할 수 있음
loki_mcp_config:
  endpoint: http://loki.monitoring.svc:3100
  max_lines_per_query: 1000
  max_time_range: 6h
  allowed_label_selectors:
    - 'namespace'
    - 'app'
    - 'pod'
    - 'level'

With that in place, the agent digs through logs in a structured way instead of shelling into Pods. Limits like max_lines_per_query and max_time_range look trivial. Leave those guardrails out in production and one agent will wring your whole log backend dry.

Step 3: Build the actual agent

For a production Kubernetes management agent in 2026, Claude 3.7 Sonnet (Anthropic) and GPT-4o (OpenAI) are the most dependable picks for complex multi-step reasoning over tool use.

Three things matter in the model. A context window big enough — cluster state dumps run longer than you'd think. Reliable tool calling with parameter validation — calling a tool with wrong arguments is just an incident. And enough reasoning to read causality between cluster events. If it can't say "B died because of A," you don't have a diagnosis, you have a list of statuses.

Reliable tool calls beat impressive prose. In operations, a call that isn't wrong is worth more than reasoning that sounds clever.

from langchain_anthropic import ChatAnthropic
from langchain_mcp import MCPToolkit
from langgraph.prebuilt import create_react_agent
from langgraph.checkpoint.sqlite import SqliteSaver

# Initialize MCP toolkits
k8s_toolkit = MCPToolkit(server='kubernetes-mcp', config=k8s_config)
prom_toolkit = MCPToolkit(server='prometheus-mcp', config=prom_config)
log_toolkit = MCPToolkit(server='loki-mcp', config=loki_config)

# Combine all tools
tools = k8s_toolkit.get_tools() + prom_toolkit.get_tools() + log_toolkit.get_tools()

# System prompt defining the agent's role and constraints
SYSTEM_PROMPT = '''
You are a Kubernetes SRE agent with read-only access to cluster state, metrics, and logs.
Your role is to:
1. Investigate operational issues when given symptoms or alerts.
2. Diagnose root causes using the available tools.
3. Propose specific remediation actions with clear reasoning.
4. Never execute write operations without explicit human approval.

When investigating, always check in this order:
1. Pod status and recent events
2. Resource utilization (CPU, memory) vs limits
3. Recent deployments and configuration changes
4. Application logs for error patterns
5. Upstream service dependencies
'''

# Create the agent with persistent memory
# (SQLite for dev, PostgreSQL for production)
memory = SqliteSaver.from_conn_string(':memory:')
model = ChatAnthropic(model='claude-3-7-sonnet-20250219')
agent = create_react_agent(model, tools, checkpointer=memory,
                          state_modifier=SYSTEM_PROMPT)

Nailing the investigation order into the system prompt matters more than it looks. It's the same thing as onboarding a junior SRE with "events first, then metrics, then deploy history." You're handing the agent the same runbook. SQLite is fine for memory in development; in production, move it to PostgreSQL. Diagnosis history that lives only in memory and evaporates isn't much use.

Step 4: Implement the approval gate pattern

The approval gate is the line between "the AI proposes an action" and "the AI performs an action." Every write the agent puts forward has to cross it before executing. That one-line boundary decides whether the operator sleeps.

The most practical approval gate in 2026 is Slack as the approval UI. The on-call engineer approves or rejects the proposed action from their phone, in bed. Nothing to spin up, no new console. It rides on the channel where the alerts already land.

from slack_sdk import WebClient
import time, json

def approval_gate(action_type: str, action_params: dict,
                  agent_reasoning: str, channel: str) -> bool:
    '''
    Intercepts write operations and requires Slack approval.
    Returns True if approved, False if rejected or timed out.
    '''
    slack = WebClient(token=SLACK_BOT_TOKEN)

    message = slack.chat_postMessage(
        channel=channel,
        blocks=[
            {'type': 'section', 'text': {'type': 'mrkdwn',
             'text': f'*AI Agent Approval Request*\n`{action_type}`'}},
            {'type': 'section', 'text': {'type': 'mrkdwn',
             'text': f'*Parameters:*\n```{json.dumps(action_params, indent=2)}```'}},
            {'type': 'section', 'text': {'type': 'mrkdwn',
             'text': f'*Agent reasoning:*\n{agent_reasoning}'}},
            {'type': 'actions', 'elements': [
                {'type': 'button', 'text': {'type': 'plain_text', 'text': '✅ Approve'},
                 'style': 'primary', 'value': 'approve', 'action_id': 'approve'},
                {'type': 'button', 'text': {'type': 'plain_text', 'text': '❌ Reject'},
                 'style': 'danger', 'value': 'reject', 'action_id': 'reject'}
            ]}
        ]
    )

    # Wait for a response (30-minute timeout in production)
    return wait_for_slack_approval(message['ts'], timeout_seconds=1800)

The easy thing to miss is that the message carries the agent's reasoning along with it. Throw a bare approve button at someone and they'll press it with no context. That's not approval, that's a rubber stamp. People can only judge when they see what's about to happen and why. The timeout defaulting to reject (False) comes from the same instinct — if nobody answers, nothing happening is the safe outcome.

Step 5: Implement autonomous incident triage

Automated triage is where an AI-driven Kubernetes agent earns its keep. An alert fires, the agent investigates cluster state, gathers evidence, and produces a structured diagnosis. It's done before the on-call engineer has opened a laptop.

The worst part of a 3 a.m. page isn't the incident. It's the 5–10 minutes of half-asleep context rebuilding — six panes up, starting from zero. If the agent handles that initial collection, the human starts at judgment.

def handle_alert(alert: dict) -> dict:
    '''
    Triggered when a PagerDuty alert fires.
    Returns a structured diagnosis for the on-call engineer.
    '''
    service_name = alert['labels']['service']
    alert_type = alert['labels']['alertname']
    namespace = alert['labels']['namespace']

    # Build the investigation goal
    goal = f'''
    Alert: {alert_type} fired for service {service_name} in namespace {namespace}.
    Alert started at: {alert['startsAt']}

    Investigate this alert comprehensively:
    1. Check pod status and recent events for {service_name}
    2. Review metrics for the last 30 minutes (CPU, memory, error rate, latency)
    3. Check for recent deployments (last 2 hours)
    4. Analyze pod logs for error patterns
    5. Check downstream dependencies

    Provide:
    - Root cause hypothesis (confidence: high/medium/low)
    - Evidence from the tools you called
    - Recommended remediation steps
    - Whether the remediation is safe to execute autonomously
    '''

    # Run the agent investigation (read-only, no approval needed)
    result = agent.invoke(
        {'messages': [('user', goal)]},
        config={'configurable': {'thread_id': alert['fingerprint']}}
    )

    return parse_agent_diagnosis(result['messages'][-1].content)

Making the agent emit a confidence level (high/medium/low) alongside the diagnosis matters. It has to be able to say "I'm not sure about this one" so the human knows which part to look at. An agent that delivers every diagnosis with the same weight loses trust fast. Binding thread_id to the alert fingerprint is deliberate too — when the same incident recurs, the prior investigation context carries straight over.

Step 6: Everything else you can automate

Schedule the agent every 30 minutes to detect and report drift between what Terraform or Argo CD declared and what's actually running in the cluster. Drift is the classic operational debt: it piles up quietly, then one day it's an incident.

def drift_detection_scan():
    goal = '''
    Perform a comprehensive drift detection scan of the production cluster.

    For each namespace in the production cluster:
    1. List all Deployments, Services, ConfigMaps, and Secrets
    2. Compare resource limits and requests against the baseline in the knowledge base
    3. Check for resources missing required labels (team, env, cost-center)
    4. Identify pods running images with the 'latest' tag
    5. Check for ClusterRoleBindings with overly permissive rules

    Output a structured drift report containing:
    - Severity for each finding (critical/high/medium/low)
    - Affected resources and namespaces
    - The specific difference from expected state
    - Recommended remediation
    '''
    return agent.invoke({'messages': [('user', goal)]})

Catching latest tags and overly permissive ClusterRoleBindings is a realistic thing to include. Both are the classic debt you add "just for a second" and never remove. Last year I found a temporary ClusterRoleBinding I'd added three years earlier, still alive. I made it, and I'm the one who forgot it.

Weekly, run a resource utilization analysis to track cost. The agent queries Prometheus, looks at 7 days of CPU and memory usage patterns across every deployment, compares against current requests and limits, and produces a right-sizing report. Requests padded "just in case," quietly bleeding money — humans don't spot that with their eyes.

A nightly security posture scan is worth attaching too. Pods running as root, containers without a read-only root filesystem, services exposed on public load balancer IPs, RBAC bindings with cluster-admin, images from unapproved registries. Produce a prioritized findings report, and cut Kyverno policy violations into GitHub issues so remediation gets tracked. Security review is the thing humans postpone most, so it's better to let a machine grind through it every night.

Monthly capacity analysis has a different character. The agent queries current usage trends, HPA events and pod scheduling failures to project when capacity constraints turn into a bottleneck. Point out which node pool needs more capacity and which namespace is closing in on its quota, and you've bought yourself time to run the procurement cycle. Adding capacity after it blows up is too late.

Production deployment considerations

Some things can't be skipped when this goes to production. Skip one and you don't have a convenient tool, you have a new incident vector.

Bind the agent's service account to least privilege, scoped to specific namespaces. Log every MCP tool call to immutable, append-only storage, and gate every tier-1-and-above write behind Slack or PagerDuty approval. To prevent runaway loops, cap tool calls per minute and add a circuit breaker that suspends all writes when the error rate across the last 10 operations exceeds 20%. You also need a manual kill switch — one Slack command that instantly revokes the agent's write access.

On top of that: record a rollback command with every autonomous action, to run if it fails, and block the blast radius so a single unapproved action can't modify more than five resources. Use separate scoped credentials per MCP server, and ship agent decision logs, tool call logs and outcome logs to Grafana or Datadog.

Look at that list again and it's identical to the safety gear we've always bolted onto operational automation. Circuit breaker, kill switch, blast radius limits. Nothing here is special because it's AI. It's the same set of things you attach whenever you automate something dangerous.

The numbers

If you adopt it, measure it. These are the metrics worth watching in production.

메트릭	목표	측정 방법
Mean time to identify (MTTI)	30% reduction	PagerDuty alert → initial diagnosis complete
On-call interruptions	50% reduction	Overnight alerts requiring human intervention
Diagnostic accuracy	85%+	AI hypothesis vs actual root cause match rate
Autonomous resolution rate	60%+	Share of issues resolved fully automatically after approval
False positive rate	Under 5%	Share of unnecessary approval requests

The one I look at first is the false positive rate. No matter how high the diagnostic accuracy climbs, if unnecessary approval requests come often enough the on-call engineer sinks into alert fatigue and starts hitting approve without reading the screen. At that moment the approval gate is a formality and nothing more.

Traps to avoid

Trusting early results is the first trap. An agent that dazzled in the demo folds in front of a real production edge case. The hard part of operations isn't the ordinary 80% — it's the 20% that shows up rarely. Run it in advisory mode for at least 30 days before you enable autonomous writes. AI diagnoses, humans execute. Build a track record of diagnostic accuracy before you trust autonomous remediation. Trust accumulates; it isn't declared.

Opening RBAC scope too wide is the other common one. Handing the agent broad cluster-admin to "make it more capable" is the fastest route to an agent-caused outage. Start read-only, then add write permissions one at a time for specific, validated use cases. Permissions are easy to expand and hard to take back. Start small.

Thinking you'll add the audit trail later

Without a complete audit trail you can't diagnose why the agent took a wrong action. You can't verify it worked correctly, and you can't demonstrate compliance for automated production changes. In an environment like finance, where every change gets traced one by one, audit logging isn't a nice-to-have — it's a precondition you build on from day one. Try to retrofit it and you're already too late. I learned that in an audit response meeting.

Wrapping up

Building an AI agent that manages Kubernetes clusters is no longer a research project. It's an engineering project. Every part is on the market: Kubernetes MCP servers, LLM APIs with reliable tool use, orchestration frameworks like LangGraph, Slack approval gates. Assembly problem, not invention problem.

The proven patterns are clear too. Read-only investigation first, human approval on writes, confidence thresholds on autonomy, audit logging with no gaps. I've said this about four times in this post. There's nothing new about it. It's the same principle we've always attached to dangerous automation, carried over to AI.

Teams adopting this architecture in 2026 cut on-call cognitive load, improve MTTI, and catch operational issues earlier. The cost is a few weeks of platform engineering work: configuring MCP servers, implementing the approval gate, validating behavior on your own cluster. It isn't a grand R&D effort. It's an extension of work you already do.

What you get is an operational assistant that reads the cluster better than anyone and stands by at 3 a.m. without calling you. What makes that assistant safe, though, isn't how smart the model is. It's the control you build around the smart part. The market builds the clever agent. The operator designs the safe one.

I still haven't turned on autonomous writes. Day 45 in advisory mode, and I don't know when I'll flip it. Still don't.