telemetry-best-practices

Telemetry Best Practices

This extension uses @microsoft/vscode-azext-utils (callWithTelemetryAndErrorHandling, IActionContext.telemetry) to emit telemetry events. Telemetry is public — assume every property and measurement value is shipped to a backend and visible to whoever has access to the dashboards. Treat every new property as a privacy decision.

These rules apply to the entire codebase (extension host, webviews, language services, helper packages), not to any single feature area.

When to Use This Skill

Apply it whenever code:

• Calls callWithTelemetryAndErrorHandling(...), callWithTelemetryAndErrorHandlingSync(...), or registers a command (which wraps the same).
• Assigns to context.telemetry.properties.* or context.telemetry.measurements.*.
• Defines a helper (e.g. report*Stats, track*, *Telemetry) that mutates an IActionContext.
• Adds or renames a telemetry event string (the first argument to callWithTelemetryAndErrorHandling).

Hard Rules — Never Emit PII / EUII

The following must never appear in telemetry.properties or telemetry.measurements, and never as part of an event name:

• File names, base names, file paths, directory names, workspace names.
• User IDs, resource names (other than the Azure account name — see below), database names, container/collection names, connection strings, hostnames, endpoints, ports.
• Query text, document contents, schema/DDL contents, error messages containing any of the above, free-form user input, prompts, AI responses.
• Email addresses, user names, machine names, IP addresses.
• Deterministic identifiers derived from the above, including hashes, truncated hashes, and normalized hash values. Cryptographic hashes are not inherently reversible, but guessable inputs can be recovered by hashing candidate values, and stable outputs permit correlation across sessions.

Not PII (always allowed): the AI model identifiers modelId, modelFamily, modelVendor (vendor-published values from vscode.LanguageModelChat), bounded enums you control, durations, counts, and ratios.

Allowed exceptions: OII identifiers under predefined property names

A small set of Azure identifiers are OII (Organization Identifiable Information), not PII, and may be emitted as-is — but only under the predefined property names listed below. The telemetry pipeline (and @microsoft/vscode-azext-utils) recognizes these names and handles them accordingly (subscription scoping, sanitization of the resource path, organization-tier classification, etc.).

context.telemetry.properties.subscriptionId = subscriptionId;
context.telemetry.properties.tenantId = tenantId;
context.telemetry.properties.accountName = account.name;
context.telemetry.properties.resourceId = new vscode.TelemetryTrustedValue(resourceId.rawId);

Rules:

• Use exactly these key names. Logging the same value under any other key (subId, accountId, armId, cosmosAccount, …) bypasses the special handling and counts as PII.
• resourceId must be wrapped in new vscode.TelemetryTrustedValue(...).
• Do not decompose the resource id and emit its parts (resourceGroup, databaseName, containerName, …) under separate properties — only the four keys above are allowed; everything else is PII.
• tenantId is allowed OII, not personal data. Do not remove its predefined telemetry property or explicitly add tenant IDs to context.valuesToMask. The library's built-in error sanitization may still redact GUIDs.
• Still push the other listed values to context.valuesToMask so they are redacted from error messages emitted alongside the event.

Safe alternatives when you need to correlate or categorize

• A non-persistent, in-memory session id generated with crypto.randomUUID() per session/operation.
• A bounded enum (e.g. 'mongo' | 'postgres' | 'sqlserver'), never a free-form string.
• A boolean flag (e.g. hasCustomInstructions) instead of the actual value.

Keep hashes needed for local storage keys or comparisons local. For cohort selection, report only the boolean selection result, not the underlying machine-ID hash or its normalized numeric value. Reuse existing random session IDs rather than adding another identifier when they already provide the required operation correlation.

Red flags to look for in reviews

context.valuesToMask — Defense in Depth, Not a Substitute

IActionContext.valuesToMask supplies sensitive strings for sanitization by @microsoft/vscode-azext-utils. The library sanitizes string telemetry.properties before emission, using more aggressive masking for keys containing error or exception. Other string properties receive less aggressive masking but still use the registered values. Non-error TelemetryTrustedValue properties bypass this sanitization, and numeric telemetry.measurements are not masked. Registering a raw value does not mask its deterministic derivatives, such as hashes or normalized hash values.

Masks belong to the action context on which they are registered. Nested telemetry calls and separate tRPC invocations must register their own masks before operations that can fail; caller or webview masks are not automatically inherited.

For NoSQL connections, reuse maskConnectionTelemetry to register connection values and every configured credential's account key or managed-identity client ID, including fallback credentials. Entra tenant IDs are allowed OII and are not registered as masks. Add operation-specific masks (such as query text) separately. Explicit client-ID masks avoid relying solely on the sanitizer's GUID detection.

Use it as a safety net for sensitive values that your code touches and might end up in a thrown error or log line you don't fully control:

// Extension host: push directly to the action context
context.valuesToMask.push(connectionString);
context.valuesToMask.push(masterKey, endpoint, databaseId, containerId);
context.valuesToMask.push(account.subscription.subscriptionId);
context.valuesToMask.push(userProvidedName); // database/container/resource names entered in a wizard

// Webview-bridged events: register on the per-webview TelemetryContext instead
telemetryContext.addMaskedValue(connectionString);
telemetryContext.addMaskedValue([endpoint, databaseId, containerId]);

Rules

• Always push a value to valuesToMask as soon as you obtain it if it could end up in an error path. This includes:
  • Strict secrets that must never appear in telemetry: connection strings, keys, tokens, query text, document contents, partition keys, user-entered names (database, container, resource).
  • The OII identifiers subscriptionId, resourceId, and accountName — even though they are emitted as-is under their predefined keys, they should still be masked from error messages. Tenant IDs are exempt from explicit masking.
• This is defense in depth, not a license to put a strict secret into telemetry properties. Never do properties.connectionString = cs and rely on masking; emit only privacy-safe values by construction.
• Push non-empty strings only. Empty/whitespace values match everything and corrupt logs (the central Telemetry.ts filter already drops falsy values; do not bypass it).
• If a value has multiple equivalent forms a user might see (e.g. a partition key with and without a leading /), push all forms: context.valuesToMask.push(partitionKey, partitionKey.slice(1));
• Wizard prompt/validateInput steps that capture user input should push the captured value before the step returns. See CosmosDBContainerNameStep, CosmosDBConnectionStringStep, CosmosDBPartitionKeyStep for the pattern.
• Branch data providers and tree item factories should mask resource ids/names at construction time (see CosmosDBBranchDataProvider, AccountInfo).
• For values that should be masked across all events emitted by a single webview, register them once on that webview's TelemetryContext via addMaskedValue(value) (see src/Telemetry.ts); the mask list is then applied automatically to every reportWebviewEvent / reportWebviewError call from that webview.

Properties vs. Measurements

telemetry.properties are strings, telemetry.measurements are numbers. Use them correctly:

• String / enum / boolean → properties. Stringify booleans (String(value)), keep enums short and bounded.
• Counts, durations (ms), sizes (chars/bytes), ratios → measurements. Never put a number in properties just because it is convenient.
• Round ratios/floats to a sensible precision; avoid emitting NaN or Infinity — guard with a check before assignment.
• Do not emit null/undefined. Skip the assignment if the value is missing.

Naming Conventions

• Event names: cosmosDB.<area>[.<subarea>].<action> (two to four camelCase segments). Keep them static — no interpolated user data. Examples currently in use: cosmosDB.nosql.queryEditor.executeQuery, cosmosDB.migration.ddlExtractor.extract. Match an existing prefix if you are adding to an existing area instead of inventing a new top-level name.
• Property/measurement keys: camelCase, stable across versions. Renaming a key breaks dashboards and queries; prefer adding a new key over renaming.
• Reuse the same key name across events when the meaning is identical (e.g. sessionId, durationMs, errorCategory).

Standard Patterns

1. Best-effort sub-events: suppress display + don't rethrow

await callWithTelemetryAndErrorHandling('cosmosDB.<area>.<action>', async (ctx) => {
    ctx.errorHandling.suppressDisplay = true;
    ctx.errorHandling.rethrow = false;
    // ... record measurements/properties (sync or async) ...
});

The callback may be sync or async; keep it async whenever the work inside is async. Use this for fire-and-forget instrumentation that must never affect the user-visible flow.

2. Rollups via a second IActionContext

When per-call events would be too chatty, accumulate counters on a longer-lived context's measurements and emit a single summary event at the end. Pass that context as an extra parameter (commonly named phaseContext / rollupContext) to the reporting helper.

3. Centralize cross-cutting enrichment

When the same set of properties (session id, mode, source type, etc.) appears at many call sites, put it in a single helper (enrichWithMigrationContext, enrichWithQueryEditorContext, …; <Area> is a placeholder for the actual feature name) and call that at the top of every event. Extend the helper instead of duplicating assignments.

4. Errors

• Set properties.errorCategory to a bounded enum. Reuse values already used in the codebase ('ai', 'infrastructure' in migrationTelemetry.ts) before introducing new ones.
• Do not put raw error.message into properties — it will likely contain user values.
• Prefer error codes over messages. If a message must be retained, ensure it cannot contain user values (paths, names, queries).
• ctx.errorHandling.issueProperties is the data appended to the body of the GitHub issue created when the user clicks Report an Issue. It is not a sanctioned PII channel — the same PII rules apply. Use it for diagnostic context (model id, error category, sanitized codes) that helps maintainers debug a reported issue.

Review Checklist

When reviewing a diff that touches telemetry, confirm each item:

• No file names, paths, resource names, IDs, hostnames, queries, or free-form user input in any property/measurement. (Exceptions: OII identifiers subscriptionId, tenantId, accountName, and resourceId under those exact key names; resourceId wrapped in vscode.TelemetryTrustedValue.)
• Numbers go to measurements, strings/booleans/enums go to properties.
• Booleans are stringified (String(value)).
• Floats are rounded; no NaN/Infinity reach the assignment.
• Event name is static (no template interpolation of user values).
• Property/measurement keys use existing camelCase names where the meaning matches an existing key.
• Best-effort sub-events set suppressDisplay = true and rethrow = false.
• Errors are categorized via an enum, not raw messages.
• Any sensitive value the action touches (connection strings, keys, tokens, OII identifiers other than tenant IDs, resource/database/container names, endpoints, user-entered names, query text, partition keys) is pushed to context.valuesToMask as soon as it is obtained.
• JSDoc on any new report* / track* helper explicitly states "no file contents, paths, or names are emitted".