A databricks migration project used to be a staffing event — a dozen contractors, a code freeze, a year of parallel runs. It isn’t that anymore. The mechanical part of the port is now largely machine work, and what’s left is a small group of people who understand your business logic well enough to decide what should survive the move. Get the headcount wrong in either direction and you pay for it: too many hands and you’re funding people to review generated SQL, too few and the modelling work stalls behind the platform work. Here is the shape of the team, the order you hire it in, and the two seats you should keep off the outsourcing list.
What is a Databricks migration project?
It’s the replatforming of an existing analytics estate — a warehouse, a set of ETL jobs, a semantic layer and a reporting surface — onto Unity Catalog, Delta tables and either Databricks SQL warehouses or jobs compute. Four workstreams, always:
- Platform: workspaces, Unity Catalog metastore, catalogs and schemas, identity federation, cluster policies, network, cost controls.
- Pipelines: the ingestion and transformation code — Informatica, SSIS, stored procedures, dbt, Airflow DAGs — converted, tested and cut over.
- Model: the dimensional layer, the grain decisions, the conformed dimensions, the metric definitions that downstream reports actually read.
- Consumption: BI connections, extract refreshes, row-level security, and the long tail of spreadsheets nobody documented.
The first two are increasingly tool-assisted. The third is not, and never will be. That asymmetry is the whole headcount argument.
How do you migrate to Databricks?
Assess, convert, validate, cut over, decommission. The useful change is how much of the convert step is now automated. Databricks’ own migration tooling automates up to 80% of migration tasks and roughly doubles implementation speed, and the conversion process saves developers over 80% of development time, shifting the human effort into review, adjustment and unit testing.
That reframes the sequence. You are not hiring a bench to retype SQL. You are hiring a small, stable group to judge output, fix the residual 20%, and rebuild the parts that were wrong before the migration started.
| Phase | Headcount | Duration | Engineer-months | What it buys |
|---|---|---|---|---|
| Assess and land the platform | 1 platform engineer | 6–8 weeks | ~2 | Unity Catalog design, cluster policies, cost guardrails, inventory of jobs and dependencies |
| Convert and review pipelines | 1 platform + 1–2 pipeline engineers | 10–14 weeks | ~7–10 | Automated port, residual hand-coding, unit tests, reconciliation harness |
| Remodel the dimensional layer | 1 modeller (+1 pipeline engineer part-time) | 10–16 weeks, overlapping | ~4–6 | Grain fixes, conformed dimensions, metric definitions with a named owner |
| Cut over and decommission | 1 platform + 1 pipeline engineer | 4–6 weeks | ~2–3 | Parallel run, BI repointing, source system shutdown |
Total: roughly 15–21 engineer-months over six to nine calendar months, delivered by three to four people who overlap rather than six to eight who hand off. The duration is set by the modelling and validation, not the conversion — which is why adding pipeline engineers past the second one rarely pulls the date in.
How do you migrate from Snowflake to Databricks?
Warehouse-to-warehouse is the easier variant and the one most likely to be over-staffed, because the SQL is close enough that teams assume the whole thing is mechanical. The structural work sits in four places:
Storage and table format
Snowflake’s micro-partitions have no direct analogue. You are choosing partition and clustering strategy on Delta, deciding where liquid clustering earns its keep, and setting file sizing for the query patterns you actually have. This is platform engineer work, not pipeline work.
Governance
Snowflake roles and masking policies map to Unity Catalog grants, row filters and column masks — but not one-to-one. Rebuilding the grant model is the single most common source of post-cutover incidents.
Warehouse-specific SQL
QUALIFY, semi-structured VARIANT access, TIME TRAVEL syntax, zero-copy clones, tasks and streams. Tooling converts most of it; the streams-and-tasks pattern usually needs a genuine redesign toward Structured Streaming or declarative pipelines.
Compute economics
Warehouse-sizing habits don’t transfer. Somebody has to own serverless versus classic, autoscaling bounds and job cluster reuse before the first invoice lands. This matters more than it used to: 57% of teams report rising warehouse and compute spend against just 36% reporting increased team budgets, so an uncontrolled migration shows up twice — once in headcount and again in run-rate.
What goes into a Databricks migration strategy?
Three decisions, made before anyone writes code.
First: how much do you lift and shift? Databricks’ own guidance is pragmatic — let tooling cover 60-80% of the initial conversion and reserve engineer time for the patterns you actually want to modernise. Pick the modernisation list up front and cap it. Every pipeline you reopen “while we’re in there” adds weeks to validation, not days.
Second: freeze or run in parallel? Parallel, almost always. A freeze looks cheap on the plan and is ruinous in practice, because the business keeps asking for things. The pressure is real: 97% of global enterprises are running active AI initiatives while only 5% say their data is ready to support them, which means your migration will be asked to deliver new surfaces mid-flight. Plan the team so one person can be pulled to delivery without the migration stopping.
Third: who owns the model afterwards? Ambiguous data ownership is now reported by 41% of teams as a persistent obstacle, while technical integration challenges have declined from 35% to 27% year-over-year. The integration problem is getting solved by tooling. The ownership problem is getting worse. A migration that ends without a named internal owner of the dimensional model has deferred its cost, not paid it.
The port is a tooling problem. The model is an ownership problem. Only one of them can be rented.
Who do you need on a Databricks migration team?
Three roles, four people at peak.
| Role | Count | Hire order | Core skills | Outsource? |
|---|---|---|---|---|
| Platform engineer | 1 | First — 4–6 weeks before anyone else | Unity Catalog, Terraform, identity federation, cluster policy, cost attribution | No |
| Pipeline engineer | 1–2 | Second, at convert phase | PySpark, Spark SQL, orchestration, CI/CD, reconciliation testing | Partly |
| Dimensional modeller | 1 | Third, overlapping convert | Kimball grain and conformance, metric definition, source-system fluency | No |
| BI/consumption analyst | 0.5 | Cutover only | Power BI or Tableau against Databricks SQL, RLS, extract strategy | Yes |
Why the platform engineer goes first
Because it is the hardest seat to fill. In the 2026 Stack Overflow developer survey, Databricks SQL was used by 3.7% of respondents with a 36.5% admiration score — high regard, thin population. People want to work on it; comparatively few have shipped on it. Start that search a month or two before you think you need to, and vet properly rather than on certification count alone — our 90-minute Databricks vetting structure and the skills matrix for hiring managers both exist because résumés in this space read better than they test.
Why one or two pipeline engineers, not four
With over 80% of conversion development time removed, pipeline engineers are reviewers and testers. Review work parallelises badly past two people — you get inconsistent conventions and a second round of rework. Two is the ceiling for most mid-size estates; one is enough below roughly 200 jobs.
Why the modeller is not optional
This is the person who decides that the three competing definitions of “active customer” become one, and that the fact table grain changes. It requires knowing your business, not just the platform. If you’re unsure whether you need a modeller or another engineer, the trade-off is covered in Scala data engineer or dimensional modeller.
Which roles should you never outsource?
The platform engineer and the modeller. Not because partners can’t do the work — many do it well — but because those two seats produce decisions rather than artefacts, and decisions walk out with the contract.
- Platform engineer: owns the Unity Catalog design, the grant model and the cost envelope. If that knowledge leaves, your next change request becomes an archaeology project.
- Dimensional modeller: owns semantics. Outsourced semantics is exactly the ambiguous-ownership failure mode that 41% of teams now report as a blocker.
- Pipeline conversion and review: safe to augment. It’s bounded, testable and reconcilable against the source warehouse. Contract capacity here is good value.
- BI repointing: outsource freely. It’s finite and verifiable.
The practical shape for most teams is a blended one — two internal seats held for the full six to nine months, contract capacity on the conversion peak, and nothing permanent beyond the cutover. That pattern, and when a systems integrator genuinely beats it, is argued out in GSI or in-house blended delivery and in Own your Databricks expertise.
What this costs you in practice
Fifteen to twenty-one engineer-months, concentrated in three or four people, is a meaningfully smaller bill than the twelve-person programme the same scope would have required before automated conversion — but only if you hold the line on headcount. The two failure modes are symmetrical and both expensive: staffing the convert phase as though it were still hand-written, or staffing the model phase as though it were a side task.
Time-box the whole thing. Six to nine months, named owners for platform and model, contract capacity sized to the conversion peak and released at cutover. If you need to see who’s actually available for those seats before you commit to a plan, the data engineering network is searchable without an account.