exploring-mcp-tool-quality

Exploring MCP tool quality

Any MCP server instrumented with PostHog's MCP analytics SDK emits a $mcp_tool_call event on the shared events table every time an agent invokes a tool. There is no dedicated ClickHouse table — every field lives as a $mcp_* property on events, and every tool-quality metric (error rate, latency percentiles, reach) is an aggregation over this one event. This is the data behind the MCP analytics dashboard and tool-quality screens.

Governed metric first

For any MCP failure-rate headline, call posthog:metric-list before a typed tool or SQL recipe and look for mcp_tool_call_fail_pct. If it is approved and not drifted, run it with posthog:data-catalog-metric-run and use that result as the canonical headline. When the user also asks which tools drive failures, run the headline first, then use the workflows below for the breakdown and label that breakdown noncanonical. If no governed metric matches, state that the catalog has no match and label the derived rate noncanonical.

For a single tool, prefer the typed tools — posthog:query-mcp-tool-stats (calls, errors, p50/p95, users, sessions, intents), posthog:query-mcp-tool-failures (top error messages by harness), and posthog:query-mcp-tool-daily-stats (day-by-day trend). Each takes a toolName + dateRange, runs the same query runner as the tool-detail UI — no hand-written SQL needed.

HogQL via posthog:execute-sql is the path for cross-tool questions — the "which tool errors most" ranking below has no typed tool, so rank with SQL, then drill into the worst tool with posthog:query-mcp-tool-stats and posthog:query-mcp-tool-failures. The full property schema and the established query recipes live in the shared MCP data reference: products/posthog_ai/skills/querying-posthog-data/references/models-mcp.md. That reference is the single source of truth for the $mcp_* schema and the effective-tool-name idiom used below — this skill inlines only the noncanonical "which tool errors most" breakdown for convenience; pull the matrix, latency, and harness recipes from the reference rather than re-deriving them. Read it before writing queries.

The two rules that matter most

• Always use the effective tool name. New-SDK events wrap the real tool in a single-exec call, so grouping on raw $mcp_tool_name collapses everything under the wrapper. Use:

  coalesce(nullIf(toString(properties.$mcp_exec_tool_call_name), ''), toString(properties.$mcp_tool_name))
  

• Always read $mcp_is_error via toBool(...) and cast $mcp_duration_ms via toFloat(...). The properties are strings.

Always set a time range — these queries scan events otherwise.

Workflow: which tool has the highest error rate (noncanonical breakdown)

For the "which tool errors most" breakdown, rank tools by error rate, but guard against small-sample noise with a HAVING floor on call volume:

posthog:execute-sql
SELECT
    coalesce(nullIf(toString(properties.$mcp_exec_tool_call_name), ''), toString(properties.$mcp_tool_name)) AS tool,
    count() AS total_calls,
    countIf(toBool(properties.$mcp_is_error)) AS errors,
    round(countIf(toBool(properties.$mcp_is_error)) * 100.0 / count(), 1) AS error_rate_pct
FROM events
WHERE event = '$mcp_tool_call'
    AND coalesce(nullIf(toString(properties.$mcp_exec_tool_call_name), ''), toString(properties.$mcp_tool_name)) != ''
    AND timestamp >= now() - INTERVAL 30 DAY
GROUP BY tool
HAVING total_calls >= 20
ORDER BY error_rate_pct DESC, total_calls DESC
LIMIT 20

Report both rate and volume — a 100% error rate over 3 calls is rarely the real story; a 12% rate over 50,000 calls is. Offer to pull the top $mcp_error_message values for the worst tool (see below).

Workflow: tool-quality matrix

One row per tool with error rate, latency percentiles, and reach — mirrors the tool-quality screen. The ready-to-run query is in models-mcp.md under "Tool-quality matrix".

Workflow: why is a tool failing

For one tool's top failure buckets (grouped by harness), call posthog:query-mcp-tool-failures with the toolName — it's the typed equivalent of the query below. Failures come from the same source as the error rate: errored $mcp_tool_call events ($mcp_is_error), scoped by the effective tool name. Failures are grouped by $mcp_error_type (a semantic bucket: internal, validation, api_4xx, api_5xx, permission, timeout, rate_limited, missing_context) and the HTTP $mcp_error_status when present. To see individual errored calls inside a bucket — with the captured $mcp_error_message, session id, harness, and intent — pass the bucket's raw error_type/error_status to posthog:query-mcp-tool-failure-occurrences ($mcp_error_message is empty on events captured before message capture shipped):

posthog:execute-sql
SELECT
    concat(
        coalesce(nullIf(toString(properties.$mcp_error_type), ''), 'unknown'),
        if(empty(coalesce(toString(properties.$mcp_error_status), '')), '',
           concat(' (HTTP ', coalesce(toString(properties.$mcp_error_status), ''), ')'))
    ) AS failure,
    count() AS n
FROM events
WHERE event = '$mcp_tool_call'
    AND toBool(properties.$mcp_is_error)
    AND coalesce(nullIf(toString(properties.$mcp_exec_tool_call_name), ''), toString(properties.$mcp_tool_name)) = '<tool>'
    AND timestamp >= now() - INTERVAL 30 DAY
GROUP BY failure ORDER BY n DESC LIMIT 10

$mcp_error_type is only populated on newer SDK/server paths — a chunk of errored calls carry neither type nor status and fall into the unknown bucket.

Workflow: slowest tools

Swap the aggregate for latency percentiles (quantile(0.95)(toFloat(properties.$mcp_duration_ms))) and order by p95_ms. The matrix query already returns p50_ms / p95_ms.

Constructing UI links

• Dashboard: https://app.posthog.com/project/<project_id>/mcp-analytics/dashboard
• Tool quality: https://app.posthog.com/project/<project_id>/mcp-analytics/tool-quality

Always surface a UI link so the user can verify visually.

Tips

• Report error rate and call volume together; a HAVING total_calls >= N floor stops tools with very few calls from topping the list spuriously
• Exclude errored calls from latency percentiles only when asked — failed calls are often the slow ones, and dropping them hides the problem
• $mcp_client_name lets you cut quality by harness (Claude Code vs Cursor vs …); the canonical bucketing multiIf is in models-mcp.md
• Harness bucketing is resolved server-side by products/mcp_analytics/backend/mcp_harness.py — that's the source of truth, and posthog:query-mcp-harness-breakdown runs it. If your hand-written SQL disagrees with the screen, your bucketing has drifted from mcp_harness.py; prefer the typed tool over re-deriving it

Related skills

• exploring-mcp-sessions — drill into a single agent run and its tool sequence
• exploring-mcp-intent-clusters — group agent goals and see which intents drive the errors