warehouse-source-new-version

Adding a new vendor API version to a warehouse source

Use this skill when a vendor has released a new API version and an existing source under products/warehouse_sources/backend/temporal/data_imports/sources/<dir>/ must support it while keeping every previously supported version functional.

How versioning works

• Every source class (subclass of _BaseSource in sources/common/base.py) declares:
  • supported_versions: tuple[str, ...] — opaque vendor labels, never parsed or ordered by the framework. Default ("v1",) (UNVERSIONED_API_VERSION) for vendors without meaningful versioning.
  • default_version: str — used when a source instance has no pin, and stamped onto newly created sources.
  • api_docs_url: str | None — the vendor's API docs/changelog page (where new versions are announced). Distinct from docsUrl (posthog.com).
  • deprecated_versions: tuple[VersionDeprecation, ...] — versions the vendor has deprecated (VersionDeprecation(version=..., sunset_at=date | None) from sources/common/base.py).
• Each ExternalDataSource row pins one version in its api_version column (NULL resolves to default_version). A schema may additionally carry a user-managed override in ExternalDataSchema.api_version (set from the schema's configuration page; not available for webhook-sync schemas) which wins over the source pin for that schema only. The sync pipeline resolves override → pin → default in workflow_activities/import_data_sync.py and hands the result to the source as SourceInputs.api_version — already resolved, never None there.
• A pinned source uses its version everywhere, not just at sync time. Every vendor-touching surface on the source classes takes an api_version: str | None = None parameter carrying the source instance's resolved pin (None → default_version): get_schemas, validate_credentials, get_endpoint_permissions, and the WebhookSource management methods (create_webhook, sync_webhook_events, webhook_inputs_updated, get_external_webhook_info, delete_webhook). Callers with a source row (creation, refresh_schemas, background sync_new_schemas, webhook endpoints, schema-scoped probes) pass the resolved pin; pre-creation flows (wizard database_schema, one-shot setup) omit it, which resolves to default_version — the version the new row is stamped with (get_endpoint_permissions currently has only the pre-creation caller, so its parameter is always None today). Base-path/URL/header construction from it happens inside each source. Deliberately NOT version-threaded (pure mappings or version-independent surfaces — thread them if a real vendor version ever diverges there): get_desired_webhook_events/webhook_resource_map (event-name mappings), get_connection_metadata, and GitHub's per-repo webhook helpers in github_warehouse_repos.py.
• These declarations are exposed publicly via GET /api/public_source_configs/ (versions, defaultVersion, apiDocsUrl, deprecatedVersions) and per-instance via the source API (api_version, api_version_deprecation). The api_version pin is queryable in HogQL via the data_warehouse_sources system table.
• Registry-wide invariants are enforced by sources/tests/test_source_versions.py: default in supported and always the last entry (declare supported_versions oldest→newest; flip the default in the same PR), deprecated ⊆ supported, default never deprecated, https api_docs_url.

First: does the version need to exist at all?

Spotting a new vendor label is not a reason to support it. Before touching any source file, diff the new version against the one it supersedes — the source's current default_version, not every entry in supported_versions — from the vendor's docs and changelog, area by area. Diff against the wire the client actually sends, not against the label: the UNVERSIONED_API_VERSION default (v1) does not mean the client targets the vendor's oldest API — a source built before it declared versions may already speak a modern wire, in which case adding that wire's real label is a declaration-only relabel-and-repin (below), not a new request path.

• authentication — credential fields, token/header scheme, scopes, permission probes
• base URL, version header, and the paths actually served per resource
• pagination — mechanism, params, cursor semantics, page limits
• the schema list — which endpoints/tables the source exposes
• schema formats — columns, types/formats, primary keys, incremental fields
• webhook payloads and subscription registration, for a WebhookSource
• rate limits, error signatures, and anything else the source's request layer touches

If none of that differs for what this source reads, don't add the version. Leave supported_versions and default_version untouched and close the task with the per-area, changelog-cited evidence that the new label is indistinguishable from the default here. An extra label buys nothing and costs: a pin users can select, a version the tests, API, and UI carry forever, and the implied claim that the framework dispatches on it.

Add it when any of these hold:

• any area above diverges from that baseline, however cosmetic it looks for our reads — then branch it (step 3). Divergence from an older still-supported label doesn't count: those pins keep serving their own request path either way (step 4), so a new label that matches the default is redundant no matter how far it sits from the legacy one;
• the vendor is retiring a version rows are still pinned to, so that label stops working — adopting the new one is the point even if the wire is identical, and the retired version moves to deprecated_versions in the same PR;
• the source must send the label to get the behavior it already wants (a required header or URL segment), i.e. the version is a request input, not just a name.

"Nothing changed" needs the same docs evidence as a divergence. An unread changelog is not a clean diff.

Adding a new version, step by step

1. Read the vendor's changelog (the source's api_docs_url) and list what changed between the currently supported version(s) and the new one: renamed/removed fields, changed pagination, new required headers, changed webhook payloads, or a field the source reads becoming opt-in behind a new query parameter (a field returned by default in the old version now empty unless requested — restore it by adding that parameter on the new version's request path). Verification is docs-only — there are no stored credentials and no live-sync harness, so the docs are the sole source of truth for what each version serves. This is also the evidence the gate above runs on.
2. Declare the version (only once the gate says the version has to exist): add the new label to supported_versions and flip default_version to it — new sources always start on the newest stable version. A pinned row's sync path is unaffected by a default flip (that is the point of pinning), but two things still follow the new default: discovery/get_schemas if the pin isn't threaded there (step 3), and any row whose api_version is NULL. Reference the request layer's version constants instead of duplicating string literals.
3. Dispatch on SourceInputs.api_version at the request layer:
   • Keep it minimal. If the version is just a header/URL segment and response shapes are compatible, thread the version string down to where the client/URL is built (see Stripe: StripeSource.source_for_pipeline passes self.resolve_api_version(inputs.api_version) → stripe_source(...) → StripeClient(stripe_version=...)). Resolve through resolve_api_version at the source class — never hardcode a fallback version in the request layer.
   • Only introduce per-version modules/branches where behavior genuinely diverges (different pagination, different field mapping). Keep all version branching inside the source's own directory — never in shared layers.
   • A source can front several vendor API families versioned on independent tracks, so a source-level bump may move only a subset of its endpoints (and change auth for just those) — dispatch path and auth per endpoint against the resolved pin, leave the rest on their existing wire, and keep the table set identical, rather than rewriting every endpoint to the new label.
   • When the new version renames endpoints, changes primary keys, or reshapes responses, the divergence must actually be branched — never leave the old single-version request path serving the new default. All the relevant surfaces can vary by version: get_rows receives the resolved pin in inputs.api_version; credential fields can key off default_version.
   • Conversely, don't add inert scaffolding: an api_version param no caller varies, or a version→URL map with identical values, is a review finding, not forward-compat. Declaration-only (supported_versions/default_version and nothing else) is the correct shape just when the gate above passed on a non-wire reason — the old label is being retired, the vendor switches behavior account-side rather than per request, or the source already reads the vendor's newest generation under the framework's legacy unversioned label (verify the request paths the source actually builds, not the label — a legacy label can already ride the new wire, so the new label just formalizes it for new rows and both resolve identically). If the gate passed on nothing at all, there is no PR. When no-header requests resolve to a version bound to the credential account-side (not a moving "latest"), threading a version header is not merely inert — it overrides the customer's chosen version, the silent move this framework exists to prevent — so stay declaration-only and don't send one.
   • Discovery and probe paths receive the pin — consume it. The framework passes the resolved pin as the api_version parameter of get_schemas, validate_credentials, get_endpoint_permissions, and the webhook management methods. A multi-version source MUST build its discovery/probe/webhook clients from that parameter, not from default_version or a hardcoded header — otherwise a pinned source discovers/reconciles under the wrong version and its tables can disappear, duplicate, or fail reconciliation. Resolve it with self.resolve_api_version(api_version) — callers with a row pass an already-resolved value (mirroring SourceInputs.api_version), so the source-side resolve only covers pre-creation calls that pass None. Ignoring the parameter is only correct when you can state why the version makes no difference to that path.
   • Watch for version-dependent column hints/schemas: e.g. Stripe's external_table_definitions were built for specific versions. When adding a version whose response shapes differ, gate the canonical column hints to the versions they were built for and let newer versions auto-infer the schema from the data (a set of hint-compatible versions checked where hints are applied). For has_managed_hogql_schema=True sources this includes the read path: hogql_definition's canonical column mapping is version-blind, so renamed columns need the canonical schema/descriptions updated too.
4. Keep old versions working: do not delete or alter the request path for previously supported versions. Removing a version is an explicit future decision, not part of a version-add PR.
5. Tests: extend the source's tests so both the old and new versions are exercised — at minimum that the version label reaches the client/request layer for each supported version (mock the boundary; parameterize over versions). The registry invariant test picks up declaration mistakes automatically. Don't re-test the base-class resolve_api_version contract (test_source_versions.py covers every source). When versions diverge, shape fixtures per version from the vendor docs — a v1-shaped mock under a v2 pin proves nothing.
6. One PR per source. Conventional title: feat(warehouse_sources): support <vendor> API version <label> — the scope is always warehouse_sources (the product), never the source dir/vendor name.

Deprecating a version

1. Implement the newer version first (steps above) if not already supported.
2. Add the old version to deprecated_versions with the vendor's announced sunset date (or sunset_at=None if none). Never deprecate default_version — flip the default to the new version in the same PR.
3. The in-product warning banner and API fields light up automatically from the metadata — zero per-source UI work.
4. Deprecated is not migrated. Existing pins move only when the vendor has announced the version will stop being served (a sunset/removal date). A deprecation without a sunset date is advisory: mark it, leave every existing pin on it fully supported, and write no migration — repinning working customers off a version the vendor still serves is exactly the silent version move the pinning framework exists to prevent. Two narrow cases still repin under an advisory (sunset_at=None) deprecation: the deprecated label resolves to a byte-identical request as the new default (a pure alias — no per-version dispatch — so the repin is not a move), or the vendor already errors on the old version (e.g. 410/406) so leaving pins is worse than moving them. A source that sends a per-version header/URL for a version the vendor still serves is neither — stay advisory.
5. Only for a sunsetting version: include a written-not-run migration script that repins affected ExternalDataSource rows (api_version column) from the deprecated version to the new one, plus any safe data/schema transforms. It must be idempotent and reviewable, and its reverse must be a no-op — repinned rows are indistinguishable from natively-created ones, so a blanket downgrade would clobber legitimate native pins. Repin only rows explicitly on the deprecated version: when the default is already the target (you are deprecating an old label without flipping the default in this PR), do not also repin NULL pins — they already resolve to the current default, so touching them is redundant churn — and leave other still-served deprecated versions alone. Where migration is lossy or unsafe — including when the new version needs credentials that can't be derived from the stored ones — do not script it: document the manual path in the PR. Do not execute migrations or backfills; humans review and run them.
6. Never touch ExternalDataSchema.api_version overrides in migration scripts — they are user-managed by design. The schema-level deprecation warning covers them; the user migrates them from the schema's configuration page.

Pinning semantics (do not break these)

• source.resolve_api_version(pinned) honors a present pin verbatim — even one no longer declared — because silently moving a customer to another version is the failure mode this framework prevents. Empty string / NULL fall back to the source class's own default_version.
• The API create path (_create_external_data_source in products/warehouse_sources/backend/presentation/views/external_data_source.py) stamps default_version, and migration 0075_backfill_externaldatasource_api_version backfilled pre-existing rows — so most rows carry a concrete pin. But api_version is nullable and direct-ORM creation paths that bypass the stamping (e.g. seed_engineering_analytics.py, and any future seeder/backfill/script) can leave it NULL, and a NULL pin resolves to default_version — so it follows a flip. Don't blanket-claim "every row is pinned, so a flip is safe"; verify the actual pin state for the source, and if a NULL cohort can exist, either back it out (written-not-run migration) or confirm the versions are request-identical.
• Repinning a customer = updating ExternalDataSource.api_version (support runbook: "Updating a warehouse source to a new vendor API version" in the PostHog/runbooks repo).

Common pitfalls

• Vendor version labels are opaque: "2026-02-25.clover", "v21.0", "2022-06-28". Copy them exactly; never normalize, sort, or parse.
• A source's per-endpoint URL versions (a hardcoded /v2/..., /v3/... in the endpoint config) are independent of the framework's source-level version label. A source may already call the vendor's newest per-resource routes while still carrying the UNVERSIONED_API_VERSION default — so a version-add can be correct as declaration-only even when the vendor's own version numbers look far apart. Diff what the source actually requests, not the vendor's headline version.
• A change the vendor calls "breaking" (e.g. resource ids migrating int→string) still needs no per-version branch when the source only passes the affected values through opaquely — a primary key whose column name is stable (type auto-inferred), cursors forwarded verbatim. The version still has to exist (the gate passed on a real divergence), but branch the request path only where the change hits a surface you hardcode: column hints, a parsed cursor, a typed primary key.
• A version bump often changes webhook payloads too — if the source is a WebhookSource, check whether webhook-created clients (created at source-setup time, not sync time) also need the version and whether existing webhook subscriptions must be updated.
• Credential-validation paths (validate_credentials, permission probes) run at creation time with no row pin; they may use the default/legacy version. Changing them is optional per version bump — verify the vendor accepts the validation calls under the new version before switching them.
• A passing credential probe is not evidence sync works — the probe hits one endpoint, get_rows hits the rest; when they diverge per version, the probe passes while every table 404s.
• Version → header/path maps must cover every supported label — a .get() fallthrough silently sends no version header (tracking "latest", the drift this framework prevents). Assert coverage or raise.
• First-time versioning of a source that sends no version selector today: keep the pre-existing label (the UNVERSIONED_API_VERSION default) sending nothing, and add the selector only for the new dated label. That preserves already-pinned rows byte-for-byte, and pinning the new default is the point — the no-selector path was tracking the vendor account's configured version, which is the drift. This is not the fallthrough bug above: the empty selector here is deliberate and belongs to one specific legacy label, not a .get() miss.
• Parallel version-bump PRs grab the same next migration number; the second to merge becomes a conflicting leaf and ci:preflight blocks it. Check max_migration.txt and renumber.
• Don't regenerate schemas for existing customers as part of a version add; schema changes only apply to rows repinned via the (human-run) migration.
• Discovery diffs under the SOURCE pin (sync_new_schemas, refresh_schemas, bulk sync-defaults). A schema-level api_version override on a version whose table set differs from the source's version can be disabled/soft-deleted by that diff — keep overrides to short verification windows, not as a long-term way to hold one table on another version.
• When a versioned source sends the pin on any vendor call — discovery or sync (a static endpoint catalog doesn't consume it at discovery, but get_rows still sends the version header) — add the vendor's version-rejection error signature (e.g. 406/410) to get_non_retryable_errors, otherwise a retired pin turns the retry cadence into a permanent error loop with no user-facing surface.
• A version bump can change the auth scheme, not just the wire format. Then the source config needs both credential shapes as optional fields, auth construction dispatches on the resolved version, and validate_credentials enforces the pair that version needs — form-level required can't express "depends on the pin".
• When the vendor renames a collection between versions, keep the schema/table name set identical across versions and put the rename in a per-version path on the endpoint config — otherwise discovery diffs orphan the table on repin.
• When the new version has no equivalent for an old endpoint (a dropped collection, not a rename), keep that table only on the versions that serve it and let the table set differ by version — never map it to a guessed path just to keep the sets equal, since docs are the sole source of truth and an unverified path makes a new source surface a table that 404s at sync time.
• A source with no dispatch may currently read its version from a constant on a shared model (e.g. the OAuth Integration model) that also drives version-independent flows like OAuth token minting. Repoint only the sync request path onto the resolved pin; leave that constant, since bumping it changes those other flows' version with a blast radius beyond this source.
• The opposite holds when the new version is a genuinely different vendor product rather than a renamed collection (the old and new labels serve different endpoints and tables): give the new version its own disjoint table set, don't force the old names onto it. That divergence is safe only because sources are never repinned — so a 2.5→3.0-style move is lossy, and its migration is documented-not-scripted, exactly like a rename that changes primary keys.
• The shared REST framework only binds a resolved parent field into a child resource's URL path, not its query string (_bind_path_params raises "Resolve query params not supported yet"). If a new version scopes a fan-out child by a query param where the old version used a path segment, the dependent-resource {"type": "resolve"} machinery can't express it — iterate the parent ids explicitly in the source and bake each id into the child's params (or POST body) instead.

Self-improvement

The default outcome of a PR is that this skill does not change. Edit it only for a learning that clears all three bars: it generalizes across sources, it would change what a future agent does, and it is not already stated or derivable from the sections above. Vendor changelog details, per-source dispatch chains or code paths, and test specifics never qualify — that context lives in your PR, not here.

When something clears the bar, fold it into the section where an agent would need it (the gate, a step, a pitfall) as one vendor-neutral line. Do not append a learnings list, changelog, or dated notes anywhere in this file.