monitoring-capture-service

Monitoring the capture service with Grafana MCP

The capture service (rust/capture/) is PostHog's Rust HTTP ingestion endpoint. It receives events from SDKs, applies quota/rate limits, and produces to Kafka. Five deployments run the same binary with different CAPTURE_MODE configs, each in its own K8s namespace.

This skill teaches how to discover live metrics using the Grafana MCP tools rather than memorizing metric names that change as the code evolves.

Environment context

The Grafana MCP is connected to a single Grafana instance scoped to one environment. If the user hasn't specified, ask which environment they want to investigate:

• prod-us — US production (us-east-1)
• prod-eu — EU production (eu-central-1)

Most capture app metrics (e.g. capture_*, http_requests_*, envoy_cluster_*) are environment-specific by virtue of which Grafana you're connected to — they don't carry an environment label. MSK and CloudWatch metrics do carry environment labels but are still scoped to the connected Grafana's AWS account.

Cross-environment comparison requires switching Grafana instances (not possible in one session).

Observability landscape

Capture spans seven telemetry domains. Each has a Grafana datasource and a discovery entry point.

Domain	Datasource UID	Discovery tool	Scope filter
App metrics (VictoriaMetrics)	victoriametrics	list_prometheus_metric_names	regex: "capture_.*"
App metrics (realtime)	victoriametrics-realtime	same	same (lower retention, higher resolution)
Logs	P44D702D3E93867EC (Loki-logs)	list_loki_label_names	app=~"capture.*"
Profiling	pyroscope	list_pyroscope_profile_types	service_name="capture-analytics/capture-analytics"
Dashboards	n/a	search_dashboards	query "capture" or "ingestion"
CloudWatch (ElastiCache, MSK)	P034F075C744B399F	query_prometheus	environment="prod-us"
CloudWatch Root (prod-us only)	PAAE47F430CFD1449	same	root account AWS metrics (does NOT exist in prod-eu)

Stable waypoints

These facts change infrequently and are hard to discover dynamically.

Deployment roles

Each capture variant runs as a separate K8s deployment in its own namespace. The primary scope labels are namespace and container (not role — that label contains pod names and is not useful for filtering).

Deployment	Namespace	capture_mode	Pipeline	Notes
capture-analytics	capture-analytics	events	Main events	Highest volume; routes /e, /i/v0/e, etc.
capture-ai	capture-ai	events	AI/LLM events	Routes /i/v0/ai; OTel on port 4318
capture-replay	capture-replay	recordings	Session recordings	Routes /s/; CAPTURE_MODE=recordings
capture-mirrored	capture-mirrored	events	Mirror/canary	Not always running; same metrics as analytics
capture-logs	capture-logs	—	Log ingestion	OTel logs on port 4318

All variants share the same Rust binary (ghcr.io/posthog/posthog/capture).

Scope capture metrics with namespace=~"capture-.*" or container=~"capture-.*". For a single pipeline, scope by namespace (e.g., namespace="capture-analytics").

Envoy cluster naming

Envoy metrics use envoy_cluster_name to identify the upstream backend. Pattern: posthog_{deployment}_{port}.

Capture-related clusters: posthog_capture-analytics_3000, posthog_capture-ai_3000, posthog_capture-replay_3000, posthog_capture-mirrored_3000, posthog_capture-logs_4318, posthog_capture-logs-canary_4318.

capture-replay also has a proxy-as-a-service cluster used in KEDA autoscaling: proxy-as-a-service_capture-replay_3000.

Scope with: envoy_cluster_name=~"posthog_capture-.*".

Redis instance topology

Capture depends on up to three logical Redis instances, plus one external instance at the Envoy layer (not in the capture binary). None emit capture_redis_* metrics — Redis health is inferred from capture-side metrics and CloudWatch ElastiCache metrics.

1. Primary Redis (REDIS_URL env var)

• ElastiCache: posthog-solo (prod-us) or posthog-prod-redis-encripted (prod-eu; sic — typo in actual cluster name)
• Both envs use a read-only endpoint for the token cache (REDIS_READER_URL)
• Backs: billing/quota limits (CaptureQuotaLimiter), session replay overflow limiter
• Capture metrics: capture_billing_limits_loaded_tokens (by cache_key), capture_quota_limit_exceeded (by resource)
• Quota resources: events, exceptions, llm_events, recordings, survey_responses
• Cache keys: @posthog/quota-limits/{resource}, @posthog/capture-overflow/replay

2. Global Rate Limiter Redis (GLOBAL_RATE_LIMIT_REDIS_URL, optional)

• ElastiCache: capture-globalratelimit-{env}-redis (prod-us, prod-eu; not dev)
• Backs: per-(token, distinct_id) sliding-window rate limiter
• Falls back to primary Redis when URL is unset
• Optional read replica: GLOBAL_RATE_LIMIT_REDIS_READER_URL
• Toggle: GLOBAL_RATE_LIMIT_ENABLED (may be off in some envs during rollout)
• Metrics: global_rate_limiter_* (direct), capture_events_rerouted_overflow{reason="rate_limited"} (proxy signal)
• global_rate_limiter_window_seconds{scope} publishes each process's window. capture-analytics and capture-ai share the AI byte budget, so if they disagree on this gauge the shared budget has split
• CloudWatch cluster id: capture-globalratelimit-prod-redis

3. Event Restrictions Redis (EVENT_RESTRICTIONS_REDIS_URL)

• ElastiCache: ingestion-prod-redis (separate writable cluster in both envs)
• Stores Django-synced ingestion restriction configs
• Falls back to primary Redis when URL is unset
• Capture metrics: capture_event_restrictions_redis_fetch (labels: restriction_type, result in success/not_found/error/parse_error), capture_event_restrictions_stale, capture_event_restrictions_loaded_count

4. Contour Rate Limit Redis (ratelimit-{env}-redis) — NOT in capture binary

• Per-IP DoS protection at the Envoy ingress layer, in front of capture
• Metrics: ratelimit_service_* (label: domain="posthog")

Metric prefixes

Every prefix here can be discovered live with list_prometheus_metric_names using datasourceUid: "victoriametrics" and regex: "<prefix>.*".

Prefix	Domain	Scope label
capture_*	App metrics (~80 metrics)	namespace, container
http_requests_*	HTTP layer (shared)	namespace=~"capture-.*"
capture_kafka_*	Kafka producer (17 metrics)	namespace, container
capture_billing_*	Billing/quota tokens loaded	namespace, cache_key
capture_event_restrictions_*	Event restrictions (6 metrics)	namespace, restriction_type
capture_ai_otel_*	AI/OTel capture (12 metrics)	namespace="capture-ai"
envoy_cluster_*	L7 proxy	envoy_cluster_name=~"posthog_capture-.*"
aws_msk_*	MSK broker-side (JMX)	environment="prod-us" or "prod-eu"
ratelimit_service_*	Contour rate limit	domain="posthog"
overflow_redirect_*	Node.js ingestion overflow (downstream)	ingestion_pipeline
kube_* / container_*	K8s resources	namespace=~"capture-.*", pod=~"capture-.*"

Kafka topics

Topics capture produces to (discover live via topic label on capture_kafka_produce_avg_batch_size_bytes). Partition counts are encoded in topic names and differ by env (EU generally has fewer partitions).

Capture writes to two different Kafka backing systems depending on the pipeline:

• MSK ingestion cluster — analytics events (main, overflow, historical, turbo), heatmaps, error tracking, client warnings
• WarpStream — session replay (warpstream-replay-v2 VC), logs (warpstream-logs VC), traces (warpstream-traces VC)

Topic (prod-us / prod-eu)	Backing	Pipeline
ingestion-analytics-main-1024 / -512	MSK	Main events
ingestion-analytics-overflow-128	MSK	Overflow (rate-limited / high-volume tokens)
ingestion-analytics-historical-128	MSK	Historical backfill events
ingestion-analytics-turbo-1024	MSK	General turbo (prod-us only)
ingestion-heatmaps-main-128	MSK	Heatmaps
ingestion-errortracking-main-128	MSK	Error tracking
ingestion-errortracking-overflow-32	MSK	Error tracking overflow
ingestion-clientwarnings-main-16 / -32	MSK	Client warnings
ingestion-sessionreplay-main-512 / -256	WarpStream	Session replay
ingestion-sessionreplay-overflow-64 / -32	WarpStream	Session replay overflow
ingestion-logs	WarpStream	Log ingestion
ingestion-traces	WarpStream	Traces ingestion
ingestion-analytics-main-dlq (+ per-pipeline DLQ topics)	MSK	Dead letter queues

Pyroscope services

Service name	Deployment
capture-analytics/capture-analytics	Main capture
capture-ai/capture-ai	AI capture
capture-replay/capture-replay	Replay capture
capture-mirrored/capture-mirrored	Mirror/canary (when running)
capture-logs/capture-logs	Logs capture

Profile types: process_cpu:cpu:nanoseconds:cpu:nanoseconds, wall:wall:nanoseconds:wall:nanoseconds, memory:inuse_space:bytes:inuse_space:bytes, memory:inuse_objects:count:inuse_space:bytes.

Grafana dashboards

UID	Title	Use for
capture	Capture	Overview across capture deployments — start here
capture-v1-details	V1 Details	Per-panel detail for the v1 pipeline (/i/v1/analytics/events)
ddfkdj56ds11xce	Legacy Details	Per-panel detail for the legacy v0 pipeline (/e/, /batch/, /i/v0/e/)
ingestion-health	Ingestion - Health	Cross-service ingestion health
ingestion-analytics	Ingestion - Analytics	Downstream analytics pipeline, per lane
ingestion-pipelines	Ingestion - Pipelines	End-to-end pipeline throughput and lag
ingestion-reliability	Ingestion - Reliability	Ingested-event trends, processing-lag SLOs, pod restarts
ingestion-sessionreplay	Ingestion - Session Replay	Session replay ingestion
contour	Contour Ingress	Envoy L7 proxy (set envoy_cluster_name=posthog_capture-analytics_3000)
AWSRedis	AWS ElastiCache Redis	Per-cluster CloudWatch Redis metrics (filter cacheclusterId)

The two details boards live under Capture/Ingestion/ and cross-link each other plus the overview. Sections for machinery shared by both pipelines (Global Rate Limiter, Event restrictions, Hyper server, Contour, billing/quota) appear on both details boards, so use whichever one you are already on.

One board you will find via search_dashboards but should not rely on: capture-3000-envoy-codes is a personal, US-only Envoy investigation board (not synced to EU).

Discovery workflows

Prometheus / VictoriaMetrics

1. list_prometheus_metric_names — datasourceUid: "victoriametrics", regex: "capture_.*" to enumerate app metrics
2. Pick a metric, then list_prometheus_label_names scoped to it — see available dimensions
3. list_prometheus_label_values — discover actual values for a label (e.g. labelName: "cause" on capture_events_dropped_total)
4. query_prometheus with PromQL — always scope by namespace (or container) and set a time range

Loki (logs)

1. list_loki_label_names — datasourceUid: "P44D702D3E93867EC" (Loki-logs; do NOT use primary Loki P8E80F9AEF21F6940 which 502s intermittently)
2. list_loki_label_values for app or namespace — find capture containers
3. query_loki_logs — e.g. {app=~"capture.*"} |= "error"

Pyroscope (profiling)

1. list_pyroscope_profile_types — data_source_uid: "pyroscope"
2. fetch_pyroscope_profile — matchers: '{service_name="capture-analytics/capture-analytics"}', profile_type: "process_cpu:cpu:nanoseconds:cpu:nanoseconds"

Dashboards

1. search_dashboards — query "capture" or "ingestion"
2. get_dashboard_by_uid — use a known UID (e.g. "capture") to get panel details
3. get_dashboard_panel_queries — extract PromQL from existing panels

Redis / ElastiCache

• Capture-side: discover capture_billing_*, capture_event_restrictions_*, capture_quota_* metrics in VictoriaMetrics
• Infrastructure: CloudWatch datasource P034F075C744B399F for ElastiCache (CPU, memory, connections, latency). Cluster IDs: capture-globalratelimit-prod-redis, posthog-solo (prod-us primary)

Key metric domains

Categories of what to look for. Discover specific metrics live using the prefixes above.

HTTP layer — request rate, latency distribution (p50/p99), active connections, error rates by status code. Metrics: http_requests_* scoped by namespace, capture_active_connections.

Event lifecycle — the funnel from received to ingested to dropped/rerouted. capture_events_received_total -> capture_events_ingested_total -> capture_events_dropped_total. The cause label on drops has 20+ values (discover live). Key additions since the golden-chart migration: event_restriction_drop, event_too_big, otel_quota_drop, oversize_event, ai_opt_in, gathering, invalid_session, no_session_id, no_snapshot. Rerouting: capture_events_rerouted_overflow with reason label (rate_limited, force_limited, event_restriction). Also: capture_events_rerouted_custom_topic for topic-redirect restrictions.

Kafka producer — broker connectivity (capture_kafka_any_brokers_down, capture_kafka_broker_connected), queue saturation (_queue_depth / _queue_depth_limit), produce RTT (capture_kafka_produce_rtt_latency_us by quantile and broker), delivery errors (capture_kafka_produce_errors_total).

Billing and quota — capture_billing_limits_loaded_tokens by cache_key, capture_quota_limit_exceeded by resource (events, exceptions, llm_events, recordings, survey_responses).

Event restrictions — capture_event_restrictions_* for Redis fetch health, staleness, loaded count, applied restrictions by restriction_type (drop_event, force_overflow, redirect_to_topic, skip_person_processing).

Envoy proxy — upstream latency, response codes (2xx/4xx/5xx), connection health, circuit breakers (_open gauges), backend membership (healthy vs total), timeouts, retries. Always filter: envoy_cluster_name=~"posthog_capture-.*". For capture-replay, also check proxy-as-a-service_capture-replay_3000.

Contour rate limit — ratelimit_service_* for per-IP DoS protection. ratelimit_service_rate_limit_over_limit = actively rate-limited IPs.

MSK broker-side — aws_msk_* JMX metrics for capture-analytics, capture-ai, and other MSK-backed topics. Key signals: throttle time, network processor idle %, memory pool depletion, request queue size. Both envs have a dedicated ingestion MSK cluster separate from the events cluster (prod-us: c21; prod-eu: posthog-prod-eu-ingestion-2026-05-04).

WarpStream — warpstream_agent_* metrics for capture-replay, capture-logs, and traces. These pipelines produce to in-cluster WarpStream agents, not MSK. Key signals: warpstream_agent_control_plane_operation_counter (by operation), warpstream_agent_file_cache_client_fetch_local_or_remote_counter (cache hit/miss ratio). Dashboards: warpstream (Agent Overview), dbfj5c31spa1ogf (MSK vs WarpStream — Active Produce Topics). US-only personal dashboards (not synced to EU): ws-coarse-lag-explore (Coarse Lag), 8e93b023-… (CH Consumer Lag). Per-VC KMinion instances: kminion-warpstream-replay, kminion-warpstream-logs, kminion-warpstream-traces.

K8s resources — container_* and kube_* for CPU, memory, restarts, HPA state. Scope: namespace=~"capture-.*", pod=~"capture-.*".

Investigation playbooks

See references/investigation-playbooks.md for step-by-step workflows for common questions: health checks, event loss, latency, Kafka backpressure, rate limiting, Redis, and cross-env comparison.