kubernetes-observability-mcp-server

kubernetes-observability-mcp-server

A production-grade Model Context Protocol (MCP) server that gives AI assistants (Claude, Cursor, Windsurf, etc.) direct, structured access to your Kubernetes cluster for operations, network debugging, and Dynatrace observability correlation — with full OpenTelemetry self-monitoring that ships traces, metrics, and logs to Dynatrace automatically.

What This Does

Instead of copy-pasting kubectl output into a chat window, this MCP server lets your AI assistant:

• List pods, check logs, describe deployments — directly, without leaving the conversation
• Debug network policies, DNS, ingress routing, and service mesh configuration
• Pull live Dynatrace problems and correlate them with Kubernetes events
• View its own performance in Dynatrace via OTel self-monitoring

Architecture

┌─────────────────────────────────────────────────────────────────────┐
│                   AI Assistant (Claude Desktop / Cursor)            │
│                        MCP Client                                   │
└────────────────────────────┬────────────────────────────────────────┘
                             │ stdio (JSON-RPC)
                             ▼
┌─────────────────────────────────────────────────────────────────────┐
│              kubernetes-observability-mcp-server                    │
│                                                                     │
│  ┌──────────────────┐  ┌──────────────────┐  ┌──────────────────┐  │
│  │  K8s Management  │  │ Network Debugging│  │   Dynatrace      │  │
│  │                  │  │                  │  │   Correlation    │  │
│  │ • list_pods      │  │ • analyze_       │  │                  │  │
│  │ • get_pod_logs   │  │   network_policy │  │ • get_dynatrace_ │  │
│  │ • describe_pod   │  │ • check_service_ │  │   problems       │  │
│  │ • list_          │  │   endpoints      │  │ • correlate_     │  │
│  │   deployments    │  │ • find_orphaned_ │  │   alerts_with_   │  │
│  │ • describe_      │  │   services       │  │   k8s            │  │
│  │   deployment     │  │ • debug_dns_     │  │ • get_entity_    │  │
│  │ • get_namespace_ │  │   config         │  │   health         │  │
│  │   events         │  │ • coredns_health_│  │                  │  │
│  │ • get_namespace_ │  │   check          │  │                  │  │
│  │   health         │  │ • inspect_       │  │                  │  │
│  │                  │  │   ingress_routing│  │                  │  │
│  │                  │  │ • trace_service_ │  │                  │  │
│  │                  │  │   to_pods        │  │                  │  │
│  │                  │  │ • node_network_  │  │                  │  │
│  │                  │  │   status         │  │                  │  │
│  │                  │  │ • check_service_ │  │                  │  │
│  │                  │  │   mesh_config    │  │                  │  │
│  └────────┬─────────┘  └────────┬─────────┘  └────────┬─────────┘  │
│           │                     │                      │            │
│           ▼                     ▼                      ▼            │
│  ┌─────────────────────────────────────────────────────────────┐   │
│  │           withTelemetry() middleware (OTel wrapper)          │   │
│  │   span per call • invocation counter • latency histogram     │   │
│  └───────────────────────────┬─────────────────────────────────┘   │
└──────────────────────────────│──────────────────────────────────────┘
                               │ OTLP/HTTP
                               ▼
┌─────────────────────────────────────────────────────────────────────┐
│                         Dynatrace                                   │
│    Distributed Traces  •  Metrics Dashboard  •  Structured Logs    │
└─────────────────────────────────────────────────────────────────────┘
          ▲
          │ API v2 (problems, entities, metrics)
          │
┌─────────┴───────────────────────────────────────────────────────────┐
│              Kubernetes Cluster (@kubernetes/client-node)           │
│         ~/.kube/config or in-cluster ServiceAccount token           │
└─────────────────────────────────────────────────────────────────────┘

Prerequisites

• Node.js 20+ — node --version must show v20 or higher
• npm 9+ — comes with Node 20
• kubectl configured — run kubectl cluster-info and confirm it shows your cluster; the server reads ~/.kube/config by default
• Dynatrace account — free trial at dynatrace.com works; Dynatrace tools are optional if you only want K8s features

Quick Start

# 1. Clone the repository
git clone https://github.com/mrafay7655/kubernetes-observability-mcp-server.git
cd kubernetes-observability-mcp-server

# 2. Install dependencies
npm install

# 3. Configure environment
cp .env.example .env
# Open .env in your editor and fill in the values (see "Environment Variables" below)

# 4. Build
npm run build

# 5. Verify the server starts (Ctrl+C to exit)
npm start
# Expected output: [mcp] kubernetes-observability-mcp-server running on stdio

Claude Desktop Setup

Step 1 — Find your config file

OS	Path
macOS	~/Library/Application Support/Claude/claude_desktop_config.json
Windows	%APPDATA%\Claude\claude_desktop_config.json
Linux	~/.config/Claude/claude_desktop_config.json

Step 2 — Add the server entry

Open the config file and add (or merge) the mcpServers block:

{
  "mcpServers": {
    "kubernetes-observability": {
      "command": "node",
      "args": ["/ABSOLUTE/PATH/TO/kubernetes-observability-mcp-server/dist/index.js"],
      "env": {
        "DT_ENV_URL": "https://YOUR_ENV_ID.live.dynatrace.com",
        "DT_API_TOKEN": "dt0c01.YOUR_TOKEN_HERE",
        "OTEL_EXPORTER_OTLP_ENDPOINT": "https://YOUR_ENV_ID.live.dynatrace.com/api/v2/otlp",
        "OTEL_EXPORTER_OTLP_HEADERS": "Authorization=Api-Token dt0c01.YOUR_TOKEN_HERE",
        "OTEL_SERVICE_NAME": "kubernetes-observability-mcp-server"
      }
    }
  }
}

Replace /ABSOLUTE/PATH/TO/ with the full path to the cloned repository.

Windows path example:

"args": ["C:\\Users\\YourName\\Projects\\kubernetes-observability-mcp-server\\dist\\index.js"]

macOS/Linux path example:

"args": ["/Users/yourname/projects/kubernetes-observability-mcp-server/dist/index.js"]

Step 3 — Restart Claude Desktop

Fully quit Claude Desktop (system tray → Quit) and reopen it.

Step 4 — Verify tools are loaded

In a new conversation, click the + button (or the connectors icon) in the chat input bar. You should see kubernetes-observability listed under Integrations or Connected tools. If it shows a connection error, check that the path in args is correct and that npm run build has been run.

Step 5 — Test

Type: List pods in the default namespace

The assistant will call list_pods and return results from your cluster.

Claude Code (CLI) Setup

# Add to your user-level MCP config (persists across projects)
claude mcp add -s user kubernetes-observability \
  node /absolute/path/to/dist/index.js \
  -e DT_ENV_URL=https://YOUR_ENV_ID.live.dynatrace.com \
  -e DT_API_TOKEN=dt0c01.YOUR_TOKEN

# Verify it was added
claude mcp list

# Test a tool call
claude "list all pods in the kube-system namespace"

Environment Variables

Variable	Required	Description
DT_ENV_URL	Yes (for DT tools)	Your Dynatrace environment base URL, e.g. https://abc12345.live.dynatrace.com
DT_API_TOKEN	Yes (for DT tools)	API token — must start with dt0c01. (not dt0s personal token)
OTEL_EXPORTER_OTLP_ENDPOINT	No	OTLP endpoint for self-monitoring telemetry
OTEL_EXPORTER_OTLP_HEADERS	No	Auth header: Authorization=Api-Token dt0c01.YOUR_TOKEN
OTEL_SERVICE_NAME	No	Service name in Dynatrace (default: kubernetes-observability-mcp-server)
OTEL_SERVICE_VERSION	No	Service version tag (default: 1.0.0)
LOG_LEVEL	No	debug / info / warn / error (default: info)
KUBECONFIG	No	Override kubeconfig path (default: ~/.kube/config)

K8s-only mode (no Dynatrace): Leave DT_ENV_URL and DT_API_TOKEN unset. All 17 Kubernetes and network tools work without them; only the 3 Dynatrace tools will return an error.

Dynatrace Setup

Creating an API Token

1. Open your Dynatrace environment: https://YOUR_ENV_ID.live.dynatrace.com
2. Go to Settings → Integration → Dynatrace API → API tokens
3. Click Generate new token
4. Name it kubernetes-observability-mcp-server
5. Enable the following scopes:

Scope	Purpose
metrics.ingest	Send custom metrics via OTLP
logs.ingest	Send structured logs via OTLP
openTelemetryTrace.ingest	Send distributed traces via OTLP
problems.read	Read Davis AI problems
entities.read	Read entity health and metadata
metrics.read	Query service metrics (optional — not available on all plans)

Note: If your Dynatrace plan does not include metrics.read, skip it. The get_entity_health tool gracefully falls back and returns entity metadata without metrics.

6. Copy the generated token — it will not be shown again.

Verify Your Token Type

Your token must start with dt0c01. — this is an API token.
Tokens starting with dt0s are personal access tokens and will return 401.

OTLP Endpoint Format

https://YOUR_ENV_ID.live.dynatrace.com/api/v2/otlp

The server appends /v1/traces, /v1/metrics, and /v1/logs automatically.

Running Inside a Kubernetes Cluster

To run the server as a Pod inside the cluster (instead of from your laptop), create a ServiceAccount with appropriate RBAC:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: mcp-server
  namespace: default
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: mcp-server-reader
rules:
  - apiGroups: [""]
    resources: ["pods", "pods/log", "services", "endpoints", "events", "namespaces", "nodes", "configmaps"]
    verbs: ["get", "list"]
  - apiGroups: ["apps"]
    resources: ["deployments"]
    verbs: ["get", "list"]
  - apiGroups: ["networking.k8s.io"]
    resources: ["networkpolicies", "ingresses", "ingressclasses"]
    verbs: ["get", "list"]
  - apiGroups: ["discovery.k8s.io"]
    resources: ["endpointslices"]
    verbs: ["get", "list"]
  - apiGroups: ["networking.istio.io", "security.istio.io"]
    resources: ["virtualservices", "destinationrules", "peerauthentications"]
    verbs: ["get", "list"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: mcp-server-reader
subjects:
  - kind: ServiceAccount
    name: mcp-server
    namespace: default
roleRef:
  kind: ClusterRole
  name: mcp-server-reader
  apiGroup: rbac.authorization.k8s.io

Tools Reference

Tool	Description	Category	Required Permissions
list_pods	List pods in a namespace with status, readiness, and restart counts	Kubernetes	pods:list
get_pod_logs	Fetch logs from a pod (max 1,000 lines)	Kubernetes	pods/log:get
describe_pod	Full pod description: DNS policy, containers, IPs, conditions	Kubernetes	pods:get
list_deployments	List deployments with replica counts and conditions	Kubernetes	deployments:list
describe_deployment	Describe a deployment with full container specs	Kubernetes	deployments:get
get_namespace_events	K8s events filtered by type or reason (max 200)	Kubernetes	events:list
get_namespace_health	Health score rollup — pod counts, warnings, overall status	Kubernetes	pods:list, events:list
analyze_network_policy	Which NetworkPolicies apply to a pod and their risk	Network	pods:get, networkpolicies:list
check_service_endpoints	Detect services with no ready backends	Network	services:list, endpointslices:list
find_orphaned_services	Services whose selectors match no running pods	Network	services:list, endpointslices:list
debug_dns_config	CoreDNS health, Corefile config, pod DNS policy	Network	pods:get, configmaps:get, deployments:get
coredns_health_check	CoreDNS pod status and restart rate	Network	pods:list, deployments:get
inspect_ingress_routing	IngressClass validation and backend endpoints	Network	ingresses:list, ingressclasses:list, endpointslices:list
trace_service_to_pods	Full path: Service → EndpointSlices → Pods → NetworkPolicies	Network	services:get, endpointslices:list, pods:list, networkpolicies:list
node_network_status	Node NetworkUnavailable conditions, PodCIDR allocations	Network	nodes:list
check_service_mesh_config	Istio/Linkerd detection, VirtualServices, mTLS mode	Network	deployments:get, CRDs read
get_dynatrace_problems	Dynatrace Davis AI problems, filtered by namespace	Dynatrace	DT_API_TOKEN with problems.read
correlate_alerts_with_k8s	Correlate Dynatrace alerts with K8s events	Dynatrace	DT_API_TOKEN with problems.read
get_entity_health	Dynatrace entity metrics — response time, error rate	Dynatrace	DT_API_TOKEN with entities.read

Troubleshooting

Server not appearing in Claude Desktop

• Confirm the path in args is absolute and the file exists: node /path/to/dist/index.js
• Check for JSON syntax errors in claude_desktop_config.json
• Fully quit and restart Claude Desktop (menu bar → Quit, not just close the window)

CONNECT_TIMEOUT in Claude Code

The server may time out if the OTLP endpoint is unreachable on startup. Ensure OTEL_EXPORTER_OTLP_ENDPOINT points to a valid Dynatrace URL, or leave it unset to disable telemetry.

401 Unauthorized from Dynatrace tools

• Token must start with dt0c01. — personal access tokens (dt0s) are rejected
• Verify problems.read and entities.read scopes are enabled on the token

kubectl / kubeconfig errors

• Run kubectl get pods from the same terminal to confirm kubeconfig is valid
• Set KUBECONFIG=/path/to/config if your config is not at the default location

Tool returns empty results

• Kubernetes namespaces default to "default" — specify namespace if your resources are elsewhere
• Check kubectl get pods -n YOUR_NAMESPACE directly to confirm resources exist

Security

• No credentials in source code — all secrets are loaded from environment variables
• No exec or shell commands — uses the @kubernetes/client-node library API directly; no kubectl subprocess
• Read-only by default — all tools only read cluster state; no writes, no deletes
• Input validation — namespace names, event types, entity names, and entity types are validated before being used in API calls
• Log limits — get_pod_logs is capped at 1,000 lines regardless of the requested value
• Never commit .env — .gitignore excludes it; rotate any token that was committed

Contributing

Contributions are welcome! Areas for improvement:

• Additional Kubernetes resource support (StatefulSets, DaemonSets, Jobs)
• Dynatrace SLO querying tool
• Prometheus metrics integration
• Helm release inspection tool
• Kubernetes events streaming

Please open an issue first to discuss major changes.

License

MIT License — see LICENSE for details.