django-migrations

Django migrations

Before you propose a change to the migration history or to how migrations run, check things already tried. It records the closed attempts at squashing the history and at the Person table cutover.

Read these files first, before writing or editing a migration:

• docs/published/handbook/engineering/developing-locally.md (## Django migrations, ### Non-blocking migrations, ### Resolving merge conflicts)
• docs/published/handbook/engineering/safe-django-migrations.md
• docs/published/handbook/engineering/databases/schema-changes.md
• products/README.md (## Adding or moving backend models and migrations) when working in products/*

If the task is a ClickHouse migration, use clickhouse-migrations instead.

Never delete a migration file

Adding migrations is fine. Deleting a historical one — any */migrations/NNNN_*.py already on master, even an app's 0001_initial.py, even to "undo" a schema change — is not. Deleting the file undoes nothing: the table and its constraints stay in every database where the migration ran, fresh databases never recreate them, and the "Migration Risk Analysis" CI job re-flags the file as a phantom new migration on every open PR that predates the deletion. The deleted-migration check in the repo-checks CI job (the hogli lint:migration-deletions command) blocks this. Genuinely intentional, reviewed deletions — a product/app move, a revert, a squash — are acknowledged in .github/scripts/migration-deletion-allowlist.txt, never by disabling the guard.

If a task asks you to delete a migration file, stop and flag it instead.

To retire a model/table:

1. Remove all usage and the model class. makemigrations, then wrap the generated DeleteModel in migrations.SeparateDatabaseAndState(state_operations=[...]) (state only, no DB change). KEEP this file. Keep the app in INSTALLED_APPS. If the model has a ForeignKey to posthog_team, posthog_user, posthog_organization or posthog_project, drop that constraint with DropForeignKey in database_operations in this same migration. This is required, not a cleanup. Django stops cascading into a table it can no longer see, so the child rows survive a parent delete. Those constraints are DEFERRABLE INITIALLY DEFERRED, so the parent delete runs its whole cascade and then fails at COMMIT, and team and organization deletion stay broken until someone drops the table.
2. Deploy, wait at least one full deploy cycle.
3. SafeDropTable later in a NEW migration — never by deleting old files. Treat this as owed work rather than an option whenever step 1 left a foreign key to a hot parent in place. DROP TABLE takes ACCESS EXCLUSIVE on every table its foreign keys reference, one relation at a time, so its lock order crosses that of any live multi-table read and the two deadlock. lock_timeout does not stop that, because the deadlock detector resolves the cycle first, and it kills whichever session waited longest: a live read (a 500) or the migration (a blocked deploy). SafeDropTable takes every lock up front under a budget derived from deadlock_timeout, so the migration loses the race instead. Never hand-write the RunSQL. See hot table hazard. python manage.py audit_orphan_hot_table_fks lists tables already in this state. Run it against a long-lived database: a squashed history has no CreateModel for a table that left Django's state before the squash, so a fresh database never creates it and the migration files hold no trace of it.

Full guide: safe-django-migrations.md (## Dropping Tables, ### Removing a whole product or app). Deleting a migration your branch added but never merged to master is allowed (regenerating).

Retire a column

Default: don't drop it. Take the field out of the ORM and leave the column. It keeps its data and needs no deploy coordination.

Deleting the field and running makemigrations is not that. Django names every concrete field in every SELECT and INSERT it writes, so the generated RemoveField drops the column in the same deploy that stops the code asking for it, and every pod still on the old release fails its queries. A # deprecated comment does not help, because the field is still on the model.

Two helpers in posthog.migration_helpers do it properly:

• deprecate_field(models.IntegerField(null=True)) wraps the field in place. No migration. Reads and writes warn, and the column leaves every query. The field must already be null=True, because nothing writes the column once it is hidden. Use raise_on_access=True to prove no caller is left.
• untrack_field("mymodel", "myfield") replaces the generated RemoveField with a state-only migration, so the field leaves the model class and the column stays.

A foreign key column must use untrack_field plus DropForeignKey in the same migration. Once Django cannot see the field, a parent delete stops cascading into it, and the deferred constraint fails the delete at COMMIT forever after. deprecate_field cannot be used on a foreign key at all, because it writes no migration and so has nowhere to put the constraint drop.

Only drop the column after one of those has been deployed for a full deploy cycle, and drop it with RunSQL ... DROP COLUMN IF EXISTS in a migration that follows the state removal. Coming from deprecate_field, delete the wrapped line and replace the generated RemoveField with untrack_field first; both migrations can go in one PR. Never ship the plain RemoveField that makemigrations writes, because it removes state and drops the column in one operation and the analyzer cannot tell it from an unstaged drop.

Check non-ORM readers first (nodejs/, rust/, Temporal workers, Metabase). Hiding a field from Django says nothing about them.

Full guide: safe-django-migrations.md (## Dropping Columns).

Retire dedicated migration tests

A data migration test protects the rollout, not the permanent behavior of the product. Remove the dedicated test after all supported environments have applied the migration, the rollback window has closed, and no supported upgrade still relies on the old data state.

Delete an expired test instead of marking it skipped. Keep the migration file. Fresh-schema CI still checks that the complete migration chain applies.

Do not apply this rule to migration tooling, migration safety checks, reusable backfill systems, or backfills that people can still run.

Growing enums need callable choices

A choices= list that grows over time generates an AlterField on every addition, on every model that uses the enum. Those migrations emit no SQL — choices is in Django's Field.non_db_attrs, so the schema editor skips them — but they still land in migration state, in CI's per-shard migration replay, and in max_migration.txt, where they collide with every other migration waiting in the merge queue.

Pass a callable instead. Django resolves it lazily, so the migration records the function reference once and never changes again:

def external_data_source_type_choices() -> list[tuple[str, str]]:
    return ExternalDataSourceType.choices


source_type = models.CharField(max_length=128, choices=external_data_source_type_choices)

Runtime behavior is unchanged: DRF still builds a ChoiceField, so the OpenAPI enum and the generated frontend types are intact; full_clean() still rejects unknown values; admin still renders a dropdown; get_FOO_display() still resolves the label. A real schema change (max_length, null, a new field) still generates a migration.

Reach for a callable when the enum is a registry that grows as the org adds things — sources, integrations, products, providers, model ids. Keep a plain list for a closed vocabulary intrinsic to the domain, like a status or a priority, where a new member is a genuine domain change worth recording.

Two things a callable does not do for you:

• Migrations reference it by import path forever, so renaming or moving it needs its own migration. Keep it next to the enum it wraps.
• It has to be a module-level named function. A lambda or a closure returned by a shared factory has no importable path, so Django cannot serialize it — one small function per enum is the intended shape, not duplication to factor out.
• It hides choice growth from migration state, not from the column. A new member longer than the field's max_length is still a real schema change, so check the headroom when the registry grows.

Workflow

1. Classify the change as additive (new nullable column, new table) or risky (drop/rename, NOT NULL, indexes, constraints, large data updates, model moves). A change is also risky if it touches a hot table, regardless of how additive it looks. See also the cross-language NOT NULL hazard below.
2. Generate: DEBUG=1 ./manage.py makemigrations [app_label]. For merge conflicts: python manage.py rebase_migration <app> && git add <app>/migrations (posthog or ee).
3. Apply safety rules from safe-django-migrations.md — the doc covers multi-phase rollouts, SeparateDatabaseAndState, concurrent operations, idempotency, and all risky patterns in detail.
4. Validate: ./manage.py sqlmigrate <app> <migration_number>, run tests, confirm linear migration sequence.

Use the migration helpers

posthog.migration_helpers has drop-in operations for the risky-but-common cases. Reach for these first; they track Django state, disable timeouts, and are idempotent under bin/migrate retries:

• Add/drop an index → SafeAddIndexConcurrently / SafeRemoveIndexConcurrently (model_name + models.Index). Never use Django's AddIndexConcurrently — CI blocks it.
• Add a CHECK constraint → AddConstraintNotValid then ValidateConstraint in a later migration (or same migration with atomic = False).
• Add a ForeignKey to a hot table → declare the FK with db_constraint=False on the model (so CreateModel / AddField emit no parent lock), then add the DB constraint back with AddForeignKeyNotValid and follow up with ValidateForeignKey in a later migration. See foreign keys to hot tables.
• Index expressed only as raw SQL (no Django Index) → CreateIndexConcurrently / DropIndexConcurrently wrapped in SeparateDatabaseAndState.
• Retire a column → deprecate_field on the model, or untrack_field in place of the generated RemoveField. See retire a column.
• Drop a retired table → SafeDropTable("posthog_oldfeature"). It locks the referenced parents up front under a budget below deadlock_timeout, so a lock cycle tends to end with the migration retried rather than a live read killed, and it is idempotent under a retry. It refuses a partitioned or inherited table, because its lock list does not cover the members of a hierarchy.
• Drop the check and unique rules on a retiring column → DropColumnConstraints(table, columns=[...]), before the release that stops writing the column. It finds every check, unique and exclusion constraint and every unique index on those columns in the catalog, including ones an earlier swap left behind that no migration names, and drops them under one bounded lock. Never type a Django-generated name into DROP CONSTRAINT IF EXISTS: a wrong name succeeds and drops nothing. The risk analyzer blocks a forward drop of a name with Django's hash suffix.
• Drop a foreign key left behind by a retirement → DropForeignKey(table, column=...) or DropForeignKey(table, to_table=...). It reads the name from pg_constraint, so nothing is hardcoded and a retry is a no-op. It locks every parent and then the child in one bounded phase, the same as SafeDropTable. The transaction holds those locks until COMMIT, so give one op every key on the table (column=["a_id", "b_id"]) and keep it alone in its migration, next to state-only operations at most. The risk analyzer blocks two drops, or a drop beside other database operations, in one atomic migration. For keys into several busy parents, use atomic = False with one op per key instead, so each drop commits alone, and acknowledge the migration in atomic_false_acknowledged_migrations.txt.

All concurrent-index ops require atomic = False.

Meta-principle when you hit a risky-but-common pattern with no helper: don't hand-roll the safe DDL from docs — ship a drop-in helper in posthog/migration_helpers and point the CI policy at it. A one-import helper beats a wall of caveated RunSQL every time.

Scripting against the risk analyzer

posthog/management/migration_analysis/models.py holds two risk dataclasses. OperationRisk scores one operation; MigrationRisk scores a whole migration file and exposes max_score, level, and category. Both answer .score, so on a MigrationRisk you can read either score or max_score — they return the same number.

Hot table hazard

posthog_team, posthog_user, posthog_organization, and posthog_project are read on virtually every request. Any ALTER TABLE on them — including a plain nullable AddField, which is "safe" everywhere else — needs an ACCESS EXCLUSIVE lock, and while that lock request waits behind in-flight queries, every later query on the table queues behind it. Even a metadata-only ADD COLUMN can stall site-wide traffic in waves (one per bin/migrate retry) until the ALTER wins the lock race. This has caused production 5xx incidents.

Before writing a migration that touches one of these models:

• For Team: put domain-specific fields on a Team extension model instead — posthog/models/team/README.md. That's a CREATE TABLE, no lock on posthog_team.
• CREATE INDEX CONCURRENTLY (via SafeAddIndexConcurrently) is fine — SHARE UPDATE EXCLUSIVE doesn't block reads or writes.
• If the field genuinely belongs on the hot table (core identity, cross-product settings, SDK config), the HotTableAlterPolicy analyzer blocks the migration in CI until <app_label>.<migration_name> is added to posthog/management/migration_analysis/hot_table_acknowledged_migrations.txt. That acknowledgment also means coordinating the deploy with infra for a low-traffic window.

Foreign keys to hot tables

A ForeignKey targeting a hot table is the same hazard from the other side, and it bites from any app — a plain product-app CreateModel or AddField with to="posthog.team" (or settings.AUTH_USER_MODEL, which is posthog_user). Creating the FK constraint takes a SHARE ROW EXCLUSIVE lock on the referenced parent, which conflicts with the ROW EXCLUSIVE every INSERT/UPDATE/DELETE on the parent holds; under write traffic the lock queues and lock_timeout cancels it on each bin/migrate retry. HotTableAlterPolicy now flags this case. Two ways out:

• db_constraint=False on the ForeignKey — emits no FK constraint and takes no lock on the parent at all (app-level enforcement only). This is the only truly lock-free path.
• A real DB constraint, two-phase — declare the FK db_constraint=False, then add it back as a DB constraint with AddForeignKeyNotValid, and ValidateForeignKey in a later migration. Be honest: ADD CONSTRAINT ... NOT VALID still takes a brief SHARE ROW EXCLUSIVE lock on the parent for the metadata add — it skips the row scan, so it shrinks the lock window but does not eliminate it. VALIDATE then runs lock-free on the parent.

Product database boundaries

Apps listed in products/db_routing.yaml migrate on their own database and nowhere else. A migration may only depend on migrations that apply to a database it applies to itself, so never add a dependency from another app onto one of those apps, and never the reverse. No foreign key or index can cross databases, so the edge buys nothing, and the CI schema restore relies on its absence: it forgets the routed apps' django_migrations rows so each job applies them under its own routing, and a dependant of a forgotten row makes Django refuse to migrate. posthog/test/repo_invariants/test_migration_dependencies_share_a_database.py blocks the edge.

Cross-language NOT NULL hazard

posthog_user, posthog_team, and other core tables in the main Postgres database are written by Django and by nodejs/ (plugin-server tests via insertRow), rust/ services, and Temporal workers. Those non-Django writers issue raw INSERTs that only list the columns they care about, so any new NOT NULL column without a Postgres-level DEFAULT will break them with null value in column "<col>" violates not-null constraint.

Django's default= alone does not create a Postgres-level default — by design, Django treats it as a Python-only attribute applied at Model.__init__:

• Callable defaults (default=list, default=dict, default=uuid.uuid4) are never emitted into SQL at all.
• Scalar defaults (default=False, default=0, default="") are emitted as ADD COLUMN ... DEFAULT X NOT NULL and then immediately dropped by a follow-up ALTER COLUMN ... DROP DEFAULT — verify with ./manage.py sqlmigrate.

Before merging, grep for external writers of the table:

rg -n "INSERT INTO <table>|insertRow\(.*'<table>'" nodejs rust products services

If any match, add both default= and db_default= to the model field. db_default= lands a real Postgres DEFAULT; default= keeps the Python-side value for ORM creates:

class User(models.Model):
    hide_mcp_hints = models.BooleanField(default=False, db_default=False, null=False)

makemigrations will emit a plain AddField(..., db_default=False, default=False, ...), and sqlmigrate shows just ADD COLUMN ... DEFAULT false NOT NULL — no DROP DEFAULT follow-up.

db_default= is also load-bearing for the nodejs / rust test suites. posthog/management/commands/setup_test_environment.py calls disable_migrations() and builds the test schema directly from model definitions, skipping the migration entirely. Plain default= is invisible to that path; db_default= is what Django bakes into the generated CREATE TABLE. Without it, the postgres-parity and Jest jobs in .github/workflows/ci-nodejs.yml will fail on raw INSERTs even though ./manage.py migrate looks correct in isolation.

For modifying the default on an existing column (no ADD COLUMN), use a plain RunSQL instead:

migrations.RunSQL(
    sql="ALTER TABLE <table> ALTER COLUMN <col> SET DEFAULT '[]'::jsonb;",
    reverse_sql="ALTER TABLE <table> ALTER COLUMN <col> DROP DEFAULT;",
)

Always verify with ./manage.py sqlmigrate <app> <number> that no stray DROP DEFAULT slipped through, and confirm ./manage.py makemigrations --dry-run reports no state drift.