You can teach PySpark syntax. You cannot teach someone to care that a table is owned by a departed contractor’s personal identity, or that a job is burning all-purpose compute on a nightly batch that should run on job clusters. Those are the failures that show up ninety days after a Databricks hire starts, and they are exactly what a resume screen, a certification badge and an AI-assisted take-home will never surface. The fix is a short, paid, structured exercise — ninety minutes, three blocks, written pass/fail criteria — that makes a candidate set ownership, repair a bad Delta layout and defend what the cluster costs.
The two failure modes that actually cost you money
Almost every bad lakehouse hire fails in one of two directions, and neither is syntax.
The first is governance drift. Ambiguous data ownership is reported by 41% of data teams as a persistent obstacle — not a legacy issue, a current one — and it is the precise problem Unity Catalog grants, owners and lineage exist to solve. An engineer who treats the catalog as a place to park tables rather than a permissions and lineage surface will hand you a metastore nobody can audit. That matters more now that AI workloads sit downstream: Gartner expects organisations to abandon 60% of AI projects that are not supported by AI-ready data through 2026. Your governance hire and your AI roadmap are the same hire.
The second is cost blindness. 57% of teams report increased warehouse and compute spend against just 36% reporting increased team budgets. That gap is the whole argument for testing tuning in the interview rather than discovering it on the invoice. A Databricks engineer who cannot read a Spark UI, spot a shuffle spill or justify a cluster shape will quietly cost multiples of their day rate — and the overrun lands in the same budget line that was not going to grow anyway.
Hire for judgement about ownership and spend, because syntax is the cheapest thing on the list.
Why the take-home stopped working
The asynchronous take-home used to be a reasonable proxy. It is not anymore, because the thing it measured — can you produce working-looking code — is now free. The biggest single frustration developers report is AI solutions that are “almost right, but not quite”, cited by 66%, with debugging AI-generated code the second. Read that as a hiring signal rather than a developer complaint: nearly-correct code is the default artefact now, so an exercise that only checks whether the notebook runs tells you nothing. What you need to score is whether the candidate can spot the almost-right answer and say why it is wrong.
Certifications have the same problem in a different shape. They confirm familiarity with the platform’s vocabulary. They do not confirm that someone will set an owner, write a grant at group level, or push back when you ask for a 64-node cluster for a 400 GB job.
| What you are using now | What it actually measures | What it misses |
|---|---|---|
| CV and project list | Exposure to tools and scale | Whether they owned the cost or governance of any of it |
| Databricks certification | Platform vocabulary, API recall | Judgement under an ambiguous brief |
| Unpaid take-home | Ability to produce code that runs | Verification, trade-off reasoning, authorship |
| Three-round panel loop | Communication, culture read | Four weeks of calendar you do not have |
| 90-minute paid exercise | Ownership, layout repair, spend defence | Long-horizon architecture — interview that separately |
The 90-minute exercise, block by block
Give the candidate a disposable workspace, a small seeded dataset, read access to a Spark UI with a completed bad run, and the brief in writing twenty-four hours ahead. Tell them the tools they can use — including AI assistants, because banning them is unenforceable and uninformative. Tell them they will be asked to justify every line they did not write themselves. Pair with your senior engineer, screen shared, talking out loud.
Block 1 — Unity Catalog ownership and grants (0:00–0:25)
Seed the workspace with a deliberately broken catalog: a schema owned by an individual user account rather than a group, a table granted ALL PRIVILEGES to account users, one external location with no storage credential comment, and a view whose upstream table was renamed.
Ask: make this safe to hand to a new team on Monday.
You are watching for whether they reach for groups instead of users without being prompted, whether they explain the difference between a catalog-level and schema-level grant, and whether they check lineage before changing anything. The strongest candidates narrate the blast radius first — who loses access when I revoke this — before they touch a grant.
Block 2 — Repair a badly laid-out Delta table (0:25–1:00)
This is the heart of it. Hand them a table with small-file sprawl, partitioning on a high-cardinality column, no clustering, stale statistics, and a query that scans the lot. Ask them to make the query faster and explain the trade-off of each change.
Databricks claims its own optimised managed tables, using predictive optimisation and liquid clustering, deliver up to 20× faster queries and 50% lower storage costs than the unoptimised baseline. You are not asking a candidate to hit that number. You are asking whether they can explain where a number like that comes from — file sizes, data skipping, clustering keys chosen from actual predicates — and whether they can reproduce a meaningful fraction of it on your seeded table in thirty-five minutes.
The tell is sequence. A weak candidate runs OPTIMIZE immediately. A strong one looks at the query plan, counts files, checks the predicate columns, then decides whether the real fix is clustering, repartitioning the write, or deleting a partition scheme that should never have existed.
Block 3 — Justify the cluster (1:00–1:25)
Show them a cluster config and a Spark UI from a completed run: all-purpose compute, autoscaling 2–32, photon off, a stage with visible spill, and one task taking eleven minutes while the rest finish in forty seconds.
Ask two questions. What is wrong with this run, and what would you change about the cluster before the next one?
The skew diagnosis matters less than the reasoning chain. Does the candidate separate the data problem from the infrastructure problem? Do they know that adding nodes to a skewed job buys nothing? Can they say, in plain language, what the change saves per month and what it risks? An engineer who can hold a cost conversation with a finance stakeholder is worth more than one who can only hold it with a Spark UI.
Block 4 — The handover note (1:25–1:30)
Five minutes, written, unaided: what you changed, what you would do next, what you are unsure about. Short, and the single most diagnostic artefact in the whole exercise. Candidates who list what they are unsure about are the ones who will not silently ship an almost-right pipeline.
Pass/fail criteria
Score it live, on the same sheet, with the pairing engineer. Weight governance and cost equally against Spark mechanics — the split matters, because the cheap-to-hire engineer who cannot govern or budget is where the 60% AI project abandonment figure comes from.
| Block | Pass | Fail |
|---|---|---|
| Unity Catalog | Assigns group ownership, scopes grants to least privilege, checks lineage before revoking, names the blast radius | Grants to individual users, leaves account users access in place, edits objects without checking dependants |
| Delta layout | Inspects plan and file counts first, chooses clustering keys from real predicates, explains the storage/compute trade-off | Runs OPTIMIZE reflexively, keeps high-cardinality partitioning, cannot say why the query got faster |
| Cluster defence | Separates skew from capacity, moves batch off all-purpose compute, quantifies the monthly saving | Proposes more nodes for a skewed stage, cannot read the Spark UI, treats autoscale as the answer to everything |
| AI-assisted code | Verifies generated output, flags what they did not check | Cannot explain a line they submitted |
| Handover note | Honest unknowns, clear next step | “All done, no issues” |
Three passes out of four blocks is a hire. Two is a second conversation about scope, not a rejection — a strong governance candidate with weak tuning is a real archetype and sometimes exactly the archetype you need. A fail in the AI-assisted code row is a hard no regardless of everything else, because it means you cannot trust the rest of the score.
Run it paid, and run it once
Pay for the ninety minutes at the candidate’s rate. It changes who accepts — employed senior engineers do not do free homework — and it changes the standard you can hold them to, because you are buying work rather than asking a favour.
Then compress the loop. Employment of data specialists is projected to grow 35% from 2025 to 2035, far above the all-occupation average, which in practice means the engineer who clears your exercise on Tuesday has two other processes running. A three-round loop across a month is a loop that loses. One intro call, one paid exercise, one decision — inside a week.
If the shortlist itself is the bottleneck rather than the vetting, that is a sourcing problem and a different fix; how to hire Databricks developers covers that side, and lakehouse adoption stalls on skills rather than platform covers what happens when you skip the step entirely. When you need the shortlist to be anonymized, currently-employed specialists rather than whoever is on the bench, search the data engineering network.
Adapt the brief, keep the structure
The three blocks are the skeleton; the seeded workspace is the part you customise. If your estate is streaming-heavy, make block two a Structured Streaming checkpoint recovery instead of a static layout repair. If you are mid-migration, hand them a Hive metastore table and ask them to plan the Unity Catalog cutover, including what breaks. If the role is platform rather than pipeline, swap block three for a cluster policy design question — which policies, which instance pools, which guardrails you would set so that nobody else can spin up the cluster you just criticised.
What does not change: ownership before objects, evidence before optimisation, cost stated in money rather than in nodes, and a written note at the end. Those four habits predict the ninety-day outcome better than anything on a CV, and ninety minutes is enough to see all four.