Buying a lakehouse is the easy part. A procurement cycle, a workspace, a bill — you can have Databricks or Snowflake or Fabric standing up tables by Friday. What you cannot buy is the operational layer underneath it, and that layer is what a lakehouse data engineer owns: open table formats that behave under concurrent writes, incremental pipelines that do not reprocess the world every night, governance that survives an audit, and compute spend that does not quietly triple. Most teams are under-skilled and under-hired at exactly that layer, which is why the platform looks fine in the demo and wobbles in month four.
What is a data lakehouse?
A lakehouse is object storage holding open columnar files — Parquet, usually — with a transactional metadata layer on top that gives you ACID writes, schema evolution and time travel. Delta Lake, Apache Iceberg and Hudi are the three implementations that matter. The point is that one copy of the data serves BI, ad-hoc SQL, streaming and model training, instead of a lake feeding a warehouse feeding a feature store.
That is the architecture pitch, and it is broadly sound. The adoption data tells a different story about readiness. In 2026, only 9% of analytics engineering teams report running Iceberg in production, with 6% planning and 12% stuck in proof-of-concept. The format is a standard on paper and a specialist skill in practice — which means the pool of engineers who have actually operated an open table format at scale, with compaction schedules and catalog migrations and all the rest, is far smaller than the pool who can describe one.
The architecture is settled. The operating experience is not, and that is what you are hiring for.
What is data lakehouse architecture?
Four layers, and a candidate should be able to argue about all four.
Storage. S3, ADLS or GCS, partitioned deliberately rather than by whatever column the first pipeline happened to use. File sizing is a real engineering decision here: a table of four million 20KB files will bankrupt your query planner regardless of which engine reads it.
Table format. Delta or Iceberg, plus the catalog that tracks them — Unity Catalog, Polaris, Glue, Nessie. The catalog choice is the one with the longest shadow, because it determines which engines can read your tables in three years.
Compute. Spark, Photon, Snowflake virtual warehouses, DuckDB for the small stuff. Decoupled from storage, which is the whole promise and also the whole cost risk.
Orchestration and transformation. dbt, Airflow, Dagster, Dynamic Tables, Declarative Pipelines — whatever enforces dependency order and materialisation strategy.
The layer most job specs forget is the fifth one: lineage, data contracts, quality tests and access policy. That is not platform work you get for free with the licence. Someone writes it, and if nobody does, your lakehouse is a very expensive data lake with better marketing.
What is a data lakehouse vs warehouse?
The honest answer is that the two have converged enough that the architectural debate matters less than the skills debate. Warehouses added external tables and open formats; lakehouses added SQL ergonomics and fine-grained governance. What still differs is where the work lands.
| Dimension | Classic warehouse | Lakehouse |
|---|---|---|
| Storage | Proprietary, engine-managed | Open files in your own object store |
| Workloads | SQL and BI | SQL, BI, streaming, ML on one copy |
| Schema | Enforced on write | Enforced by table format and contracts |
| Cost model | Credits per warehouse | Compute per job, storage priced separately |
| Lock-in risk | Higher — migration is a rewrite | Lower — but you own the file layout problem |
| Engineering load | Modelling and SQL performance | Modelling plus file layout, compaction, catalog and spend |
| Failure mode | Slow dashboards | Silent correctness drift, runaway compute |
That last row is the one that should shape your hiring. A warehouse engineer who writes excellent dimensional models is not automatically someone who can diagnose why a merge is rewriting 900GB a night. If you are deciding which of those two profiles your roadmap actually needs, we broke the trade-off down in data lake vs warehouse engineer.
How do you build a data lakehouse?
In roughly this order, and the order is the advice.
- Pick the table format and catalog first. Everything downstream inherits that decision. Changing it later is a migration, not a config change.
- Land raw data immutably. Bronze, or whatever you call it. Replayability is the cheapest insurance you will ever buy.
- Build incrementally from day one. Full refreshes are fine at 50GB and ruinous at 5TB, and retrofitting incrementality into a year of full-refresh models is a quarter of work nobody budgeted.
- Write the tests with the models. Freshness, uniqueness, referential integrity, row-count deltas. Bolted on afterwards, they become a backlog item that never ships.
- Instrument cost before you need to. Tag jobs, attribute spend to teams, alert on cluster-hours per table.
- Then open it up. Self-serve access on an uninstrumented, untested lakehouse creates confident wrong answers at scale.
Step 3 is where specialists earn their rate visibly. Snowflake documented a customer who cut end-to-end latency from a 30-minute schedule to under a minute by moving a dbt batch job onto a Dynamic Tables pipeline. That is not a platform feature switching itself on — it is someone who understood the change-tracking semantics well enough to restructure the job around them.
What does a lakehouse data engineer do?
Five things, most of them unglamorous.
Owns the table format
Compaction and clustering schedules, file sizing, partition evolution, vacuum and retention policy, snapshot expiry. This is the work that determines whether query cost goes up or down as the table grows, and it is almost never in the job description.
Builds incremental pipelines
Merge strategies, late-arriving data, idempotent reruns, backfills that do not lock the table for six hours. Change data capture that handles deletes correctly, which is the one most implementations get wrong.
Keeps correctness visible
Tests, contracts, freshness SLAs, lineage that a stakeholder can read. Note where the industry’s attention is going instead: 72% of teams prioritise AI-assisted coding while only 24% prioritise AI-assisted pipeline management — testing, observability and quality controls. Generating more transformation code faster, with no matching investment in whether that code is right, is a reliability bill with a delay on it. A lakehouse data engineer is who you hire to pay it down.
Controls compute spend
Serverless made compute easier to start and much easier to waste. Databricks has said its own serverless compute usage grew more than 3x in a year. Whatever your platform, consumption trends one direction, and somebody has to own right-sizing, autoscaling bounds, job-level tagging and killing the 3am job nobody reads. Treat it as a named responsibility, not an annual panic.
Maintains, mostly
The single most useful number for workforce planning: 57% of data practitioners spend most of their workday maintaining or organising data sets rather than building new ones. If you staff a lakehouse programme as a build project and then hand the result to an existing team at flat headcount, you have budgeted for 43% of the actual job.
Why the role is hard to hire
The bottleneck is not conviction. When teams were asked what was holding lakehouse adoption back, the top barriers were knowledge gaps at 27% and unclear use cases at 27% — implementation complexity, not resistance to the architecture. Nobody is arguing against open formats. They are waiting for someone who has run them.
Supply is tight in the way specialist data roles generally are. US labour projections put data scientists at 33.5% employment growth from 2024 to 2034, the fourth-fastest growing occupation overall — a reasonable proxy for how the adjacent engineering market behaves. Add the 9% Iceberg-in-production figure and you get the real shape of the problem: a small cohort with genuine operating scars, mostly employed, mostly not reading job boards.
We wrote about the downstream effect of that gap in lakehouse adoption stalls on skills, not platform. The short version: platform spend and platform capability decouple, and the gap between them is measured in headcount you did not hire.
How to screen for it
Resumes list the stack. They do not reveal whether someone has debugged a small-files problem at 2am. Four questions that separate operators from readers:
| Ask this | You are listening for |
|---|---|
| Walk me through a merge that got slow. What did you change? | File sizing, partition pruning, Z-order or clustering — not “we scaled the cluster” |
| How do you handle a deleted upstream row in CDC? | Soft deletes, tombstones, and an answer about reconciliation |
| How would you cut this pipeline’s cost by 40%? | Scheduling, incrementality, job consolidation — before instance types |
| Which table format would you pick here, and what would make you regret it? | A named trade-off, not a vendor preference |
A candidate who can answer all four has run a lakehouse. One who answers two has used one. The distinction is worth about a quarter of delivery time. Our 90-minute Databricks vetting structure covers how to compress that into one call, and the data engineering network is where the anonymized, currently-employed profiles sit if you would rather start from people who already hold the scars.