databricks-migration

อัปเดตเช็ค — ครั้งละหนึ่งเซสชัน (บังคับ) เมื่อใช้สกิลนี้เป็นครั้งแรกในเซสชัน ให้รันสกิล check-updates ก่อนดำเนินการ

npx skills add https://github.com/microsoft/skills-for-fabric --skill databricks-migration

Telemetry — MANDATORY. Every api.fabric.microsoft.com call must carry x-ms-fabric-skill: databricks-migration (az rest: --headers "x-ms-fabric-skill=databricks-migration"), including every LRO poll, fabric_lro and retry. Snippets omit it — add it anyway.

CRITICAL NOTES

To find workspace details (including its ID) from a workspace name: list all workspaces, then use JMESPath filtering

To find item details (including its ID) from workspace ID, item type, and item name: list all items of that type in that workspace, then use JMESPath filtering

dbutils.widgets has no direct equivalent in Fabric — use notebook parameters (cell tag parameters); notebookutils.runtime.context is execution metadata, not parameter storage. If showing context fields, use documented names such as currentWorkspaceId, currentWorkspaceName, currentNotebookId, currentNotebookName, isForPipeline, and isForInteractive; activityId is the Livy job ID

dbutils.library (runtime library install) has no equivalent — use Fabric Environments for reproducible library management

Map each Unity Catalog catalog to a schema-enabled Lakehouse by default. This preserves the source schema.table hierarchy, with the Lakehouse representing the catalog; collisions arise only if multiple catalogs are intentionally consolidated into one Lakehouse

For an under-specified workspace-wide migration, ask focused questions about inventory, workload topology, security, data locations, and runtime constraints before recommending a Fabric topology

A completed Fabric migration must not retain executable dbutils.* calls in dual-runtime branches or try/except guards — replace the calls and Databricks paths outright

Databricks → Microsoft Fabric Migration

Prerequisite Knowledge

Read these companion documents before executing migration tasks:

COMMON-CORE.md — Fabric REST API patterns, authentication, token audiences, item discovery
COMMON-CLI.md — az rest, az login, token acquisition, Fabric REST via CLI
SPARK-AUTHORING-CORE.md — Notebook deployment, lakehouse creation, Spark job execution

For notebook and Lakehouse creation, see spark-cli. For Fabric Warehouse DDL/DML authoring, see sqldw-cli.

Topic	Reference
Migration Orchestrator	migration-orchestrator.md
Migration Workload Map	§ Migration Workload Map
Complete `dbutils` → `notebookutils` Mapping	dbutils-to-notebookutils.md
Unity Catalog → Fabric Lakehouse Schemas	catalog-migration.md
Before/After Code Patterns	code-patterns.md
Cluster Config → Fabric Spark Pools	§ Cluster Config → Fabric Spark Pools
Databricks Jobs → Spark Job Definitions	§ Databricks Jobs → Spark Job Definitions
Delta Sharing → Fabric External Data Sharing and OneLake Shortcuts	§ Delta Sharing → Fabric External Data Sharing and OneLake Shortcuts
MLflow → Fabric ML Experiments	§ MLflow → Fabric ML Experiments
Post-Migration Validation & Testing	validation-testing.md
Migration Gotchas & Troubleshooting	migration-gotchas.md
Multi-Notebook Migration Protocol	§ Multi-Notebook Migration Protocol
Failure Reporting	§ Failure Reporting
Must / Prefer / Avoid	§ Must / Prefer / Avoid
Authentication & Token Acquisition	COMMON-CORE.md § Authentication
Lakehouse Management	SPARK-AUTHORING-CORE.md § Lakehouse Management
Notebook Management	SPARK-AUTHORING-CORE.md § Notebook Management

Context Loading Guide

IMPORTANT — Load only what you need. Do NOT read all resource files upfront. Load the specific file for the phase you are executing:

When	Read This File
User asks to migrate a workspace (full orchestration)	migration-orchestrator.md
Applying code transforms (dbutils, namespaces, paths)	dbutils-to-notebookutils.md + code-patterns.md
Resolving Unity Catalog namespace collisions	catalog-migration.md
Post-migration verification	validation-testing.md
Troubleshooting failures or known issues	migration-gotchas.md

Migration Workload Map

Databricks Component	Fabric Target	Severity	Notes
All-purpose cluster (notebooks, REPL)	Fabric Notebook (Starter Pool or Custom Pool)	Info	No persistent cluster — Fabric provisions compute on session start
Job cluster (automated jobs)	Spark Job Definition (SJD)	Info	SJD maps one-to-one with Databricks Jobs on job clusters
Unity Catalog	Fabric Lakehouse (schema-enabled, one per catalog)	Info	Schema-enabled Lakehouse preserves `schema.table`; default one-Lakehouse-per-catalog has no collision — see catalog-migration.md
Databricks Repos (Git-backed notebooks)	Fabric Git Integration	Info	Connect workspace to Azure DevOps or GitHub; notebooks are synced
Delta Live Tables (DLT)	Fabric Notebooks + Data Pipelines	Blocker	No DLT equivalent — rewrite DLT datasets as parameterized notebook cells with pipeline orchestration
Databricks SQL Warehouses	Fabric Warehouse or Lakehouse SQL Endpoint	Info	SQL warehouse sessions → Warehouse (for write) or SQL Endpoint (for read-only)
MLflow Tracking	Fabric ML Experiments	Info	MLflow SDK is supported in Fabric — see § MLflow
Delta Sharing	OneLake Shortcuts + Fabric external data sharing	Warning	See § Delta Sharing → Fabric External Data Sharing and OneLake Shortcuts
Databricks Feature Store	Feature engineering on Lakehouse/Delta tables + MLflow	Warning	Fabric has no drop-in managed Feature Store; recreate feature tables as Delta tables in a Lakehouse and manage features via notebooks/MLflow. Verify current Fabric feature-store roadmap before committing an approach
dbutils (all sub-modules)	`notebookutils` (most sub-modules)	Info	See dbutils-to-notebookutils.md for full mapping
Scala notebooks	Fabric Notebook (Spark/Scala)	Warning	Scala is supported; swap cell magic `%scala` → `%%spark` and rewrite Databricks-specific APIs/libraries
R notebooks	Fabric Notebook (SparkR)	Warning	SparkR is supported; swap cell magic `%r` → `%%sparkr`, validate package availability, rewrite Databricks-specific APIs

Severity Definitions

Level	Meaning	Action
Blocker	Cannot run in Fabric without redesign or user decision	Stop — surface to user, require resolution
Warning	Migratable but requires validation or architectural decision	Migrate with review flag
Info	Direct substitution-level change	Auto-migrate

`dbutils` → `notebookutils` Quick Reference

The complete side-by-side API table is in dbutils-to-notebookutils.md. The key mappings are:

`dbutils` Call	`notebookutils` Equivalent	Compatibility Note
`dbutils.fs.ls(path)`	`notebookutils.fs.ls(path)`	Direct replacement
`dbutils.fs.cp(src, dest)`	`notebookutils.fs.cp(src, dest)`	Direct replacement
`dbutils.fs.mv(src, dest)`	`notebookutils.fs.mv(src, dest, create_path, overwrite=False)`	⚠️ Signature differs — see dbutils-to-notebookutils.md
`dbutils.fs.rm(path, recurse)`	`notebookutils.fs.rm(path, recurse)`	Direct replacement
`dbutils.fs.mkdirs(path)`	`notebookutils.fs.mkdirs(path)`	Direct replacement
`dbutils.fs.put(path, contents)`	`notebookutils.fs.put(path, contents)`	Direct replacement
`dbutils.fs.head(path, maxBytes)`	`notebookutils.fs.head(path, max_bytes)`	⚠️ Default differs — Python/Scala 100 KB, R 64 KB. See dbutils-to-notebookutils.md
`dbutils.fs.mount(...)`	`notebookutils.fs.mount(source, mountPoint, extraConfigs=None)`	✅ Supported — Microsoft Entra (default), `accountKey`, or `sasToken` auth. For cross-workspace / persistent sharing, prefer OneLake Shortcuts
`dbutils.secrets.get(scope, key)`	`notebookutils.credentials.getSecret(keyVaultUrl, secretName)`	Scope → Key Vault URL; key → secret name
`dbutils.notebook.run(path, timeout, args)`	`notebookutils.notebook.run(name, timeout, args)`	`path` → notebook `name` (relative to workspace)
`dbutils.notebook.exit(value)`	`notebookutils.notebook.exit(value)`	Direct replacement
`dbutils.widgets.get(name)`	See § Widgets Migration	No direct equivalent
`dbutils.library.install(...)`	Not available at runtime — use Fabric Environments	`dbutils.library.restartPython()` → `notebookutils.session.restartPython()`
`dbutils.data.summarize(df)`	`display(df.summary())`	Use `display()` or pandas `describe()`

Widgets Migration

dbutils.widgets has no direct equivalent in Fabric. Use these patterns instead:

Use Case	Fabric Pattern
Pass parameter from parent notebook	Mark a cell in the child notebook as a parameters cell (notebook UI: cell "..." menu → "Mark cell as parameters"). The parent calls `notebookutils.notebook.run("child", arguments={"param": "value"})` — at runtime the engine inserts a new cell beneath the parameters cell that overrides the defaults
Pipeline-driven parameterization	Same parameters-cell mechanism; the Fabric Pipeline notebook activity supplies override values via its Base parameters setting
Centralized cross-notebook config	Use `notebookutils.variableLibrary.getLibrary("<name>")` to read values from a Variable Library item (deployment pipelines activate the right value set per stage)
Interactive selection in notebook	Use `display()` with input cells, IPython widgets (Python only), or Fabric Data Activator

Note: notebookutils.runtime.context does not expose parameter values. It's for execution metadata (workspace/notebook/activity/user IDs, pipeline-vs-interactive flags, etc.). See dbutils-to-notebookutils.md § Runtime Context.

Cluster Config → Fabric Spark Pools

Databricks Cluster Concept	Fabric Spark Equivalent	Notes
All-purpose cluster (interactive)	Starter Pool	Auto-provisioned; no config; ideal for notebooks
Job cluster (single-use for jobs)	Custom Pool (or Starter Pool) attached to SJD	Configure node size, autoscale in Fabric capacity settings
Node type (e.g., `Standard_DS3_v2`)	Fabric node size (Small/Medium/Large/X-Large/XX-Large)	Map by vCore/memory ratio
Autoscale min/max workers	Custom Pool min/max node settings	Available in workspace Spark settings
`spark.conf` in cluster settings	Fabric Environment Spark properties	Move to Environment item; attach to workspace or notebook
`init_scripts` (cluster init)	Fabric Environment install script	Not fully equivalent — only library installs are supported
Databricks Runtime version	Fabric Runtime (1.1 = Spark 3.3, 1.2 = Spark 3.4, 1.3 = Spark 3.5)	Choose matching Spark version; test deprecated APIs
Photon accelerator	Fabric Native Execution Engine (NEE)	Enable in workspace Spark settings; vectorized execution similar to Photon

Databricks Jobs → Spark Job Definitions

Databricks Jobs Concept	Fabric SJD Equivalent	Notes
Job with single notebook task	SJD referencing a notebook	Attach a default Lakehouse; pass parameters via SJD args
Multi-task job (DAG of tasks)	Fabric Data Pipeline orchestrating multiple SJDs/notebooks	Pipeline activities map to job tasks; dependencies = activity dependencies
Job schedule (cron)	Pipeline schedule trigger	Cron expression → recurrence trigger in pipeline
Job parameters	SJD default arguments or notebook cell parameters	Parameters cell in notebook is injected at runtime
Job clusters per task	Pool attached to SJD	Each SJD can specify its Spark pool independently
Databricks Workflows	Fabric Data Pipelines	Full DAG orchestration with conditions, loops, and failure branches

Delegate to spark-cli for SJD creation and notebook deployment.

Delta Sharing → Fabric External Data Sharing and OneLake Shortcuts

Databricks Delta Sharing Pattern	Fabric Equivalent
Provider publishes a Delta share	Fabric external data sharing for cross-tenant Fabric data, or a OneLake Shortcut to ADLS Gen2 where the Delta data resides
Recipient reads shared data	Accept the external data share into a Lakehouse (Fabric creates a read-only OneLake Shortcut), or create a direct OneLake Shortcut to accessible ADLS Gen2 data
Cross-workspace table sharing within org	OneLake Shortcuts pointing to another workspace's Lakehouse tables — no data copy
Cross-tenant sharing	Fabric external data sharing — live, read-only, in-place access through a shortcut in the recipient tenant

When producing a migration workload map, include both paths: direct OneLake Shortcuts for accessible ADLS or same-tenant OneLake data, and Fabric external data sharing for native cross-tenant recipient sharing.

MLflow → Fabric ML Experiments

Fabric ML Experiments are built on the MLflow SDK — most code is directly portable:

Databricks MLflow Pattern	Fabric Equivalent	Migration Action
`mlflow.set_tracking_uri("databricks")`	Remove — Fabric tracking is automatic	Delete this line in Fabric notebooks
`mlflow.set_experiment("/path/exp")`	`mlflow.set_experiment("experiment_name")`	Use name only (not path); Fabric creates the Experiment item
`mlflow.log_metric(...)`	`mlflow.log_metric(...)` — identical	No change
`mlflow.log_artifact(...)`	`mlflow.log_artifact(...)` — identical	No change
`mlflow.autolog()`	`mlflow.autolog()` — identical	No change
`mlflow.register_model(...)`	`mlflow.register_model(...)` — identical	Model Registry is available in Fabric ML
Databricks Model Serving	Azure ML Online Endpoints or Fabric Data Activator	No direct Fabric model serving yet — use Azure ML

Must / Prefer / Avoid

MUST DO

Inventory before prescribing a workspace topology — for workspace-wide requests that omit workload inventory, dependencies, security requirements, data locations, or runtime constraints, ask focused clarifying questions and present conditional choices before selecting a Fabric design
Replace all dbutils.* calls using the mapping in dbutils-to-notebookutils.md — dbutils is not available in Fabric notebooks
Migrate dbutils.fs.mount() to notebookutils.fs.mount() (✅ supported — Microsoft Entra default, or accountKey / sasToken from Key Vault). For cross-workspace or persistent sharing, prefer OneLake Shortcuts instead. Always pair mount() with unmount() in try/finally — Fabric mounts are not released automatically on session end
Replace dbutils.secrets.get(scope, key) with notebookutils.credentials.getSecret(keyVaultUrl, secretName) — secret scopes map to Azure Key Vault URLs
Redesign widget-based parameter passing using notebook parameters cells (cell "..." menu → "Mark cell as parameters"); use notebookutils.variableLibrary for centralized cross-notebook config. notebookutils.runtime.context does not expose parameter values
Replace dbutils.library.install*() with Fabric Environments — runtime library installs are not supported in production. dbutils.library.restartPython() maps to notebookutils.session.restartPython() (Python / PySpark only)
Map Unity Catalog namespaces deliberately — default to one schema-enabled Lakehouse per catalog so schema.table is preserved; require a user-approved naming policy only when consolidating multiple catalogs into one Lakehouse. See catalog-migration.md
Map Databricks cluster init scripts to Fabric Environments — cluster-level library installs must move to Environment items

PREFER

Fabric Native Execution Engine (NEE) as the Photon equivalent — enable in workspace Spark settings for vectorized execution on Delta Lake
OneLake Shortcuts over data copy for Delta tables that already exist in ADLS Gen2 — point directly without re-ingesting
Fabric Git Integration as the replacement for Databricks Repos — connect workspace to ADO or GitHub for notebook version control
Fabric ML Experiments for direct MLflow continuity — tracking code requires minimal changes (remove set_tracking_uri)
Medallion architecture when restructuring migrated Databricks catalogs — align bronze, silver, gold Unity Catalog schemas to separate Fabric Lakehouses
Starter Pool for migrating interactive notebook workflows — eliminates cluster startup time that was a common pain point in Databricks job clusters

AVOID

Do not prescribe a one-size-fits-all workspace topology when the source inventory and migration constraints are missing
Do not import dbutils or attempt dbutils = ... assignments in Fabric notebooks — import attempts fail with ModuleNotFoundError, while unresolved dbutils references raise NameError; always use notebookutils
Do not retain dbutils.* calls behind runtime-detection guards (try/except, if IS_DATABRICKS) — replace the calls and Databricks paths outright with notebookutils and Fabric paths
Do not assume Unity Catalog governance policies transfer automatically — RBAC, row-level security, and column masking must be reconfigured in Fabric using workspace roles and Lakehouse permissions
Do not use %pip install in production Fabric notebooks at runtime — use Fabric Environments for stable, versioned library management
Do not attempt to port Delta Live Tables (DLT) pipelines verbatim — DLT has no Fabric equivalent; rewrite as parameterized notebooks orchestrated by Fabric Pipelines
Do not rely on Databricks-specific Spark configurations (e.g., spark.databricks.*) — these are proprietary and will be silently ignored or raise errors in Fabric
Do not use DBFS paths (dbfs:/...) — there is no DBFS in Fabric; all paths must use OneLake abfss:// or Lakehouse-relative paths

Multi-Notebook Migration Protocol

For workspaces with >3 notebooks or individual notebooks >5KB, process one notebook at a time (enumerate → export → transform → summarize → deploy → release) to avoid context overflow. Track each notebook through a status lifecycle (inventory → analyzed → converted → deployed → validated, or failed).

Full protocol, status definitions, and per-notebook summary schema: migration-orchestrator.md § Phase 2.

Failure Reporting

When migration cannot complete (permission failures, unresolvable Blockers such as DLT or OS-level init scripts, namespace collisions with no user-chosen policy, or repeated API failures), emit a structured failure report — do not abandon silently. The report captures phase_reached, blockers[] (item / pattern / reason / recommendation), partial_success counts, and next_steps.

Full report schema and stopping conditions: migration-orchestrator.md § Failure Reporting.

Examples

See dbutils-to-notebookutils.md and code-patterns.md for the full mapping. Key quick references:

dbutils.fs → notebookutils.fs

# Databricks
dbutils.fs.ls("/mnt/bronze/orders/")
dbutils.fs.cp("/mnt/raw/file.csv", "/mnt/archive/file.csv")

# Fabric (replace DBFS/mount paths with OneLake relative paths)
notebookutils.fs.ls("Files/bronze/orders/")
notebookutils.fs.cp("Files/raw/file.csv", "Files/archive/file.csv")

dbutils.secrets → notebookutils.credentials

# Databricks
pwd = dbutils.secrets.get(scope="prod", key="db-password")

# Fabric (scope → Key Vault URL, key → secret name)
pwd = notebookutils.credentials.getSecret("https://myvault.vault.azure.net/", "db-password")

Unity Catalog namespace → Lakehouse schema

# Databricks
df = spark.read.table("prod.silver.customers")

# Fabric (ProdLakehouse represents the source catalog and is attached as context)
df = spark.read.table("silver.customers")