Pull ten job ads with “data lake engineer” in the headline and you will find roughly one job described ten ways. The responsibilities underneath are almost always the same: Spark jobs reading and writing Parquet on object storage, a table format — Delta or Iceberg, often both — layered on top, a catalog wired in, and someone accountable for the cloud bill that all of it generates. The title is packaging. What you are actually buying is Spark-on-object-storage engineering with table-format fluency and cost discipline, and if you screen for the label instead of those three things, you will pay a premium for a résumé keyword and still miss on the skills that decide whether the platform works.
This is a mapping exercise. Here is what the title covers, what to test for, and what to budget.
The title is a label on a stack, not a discipline
The same scope ships under at least five headings depending on who wrote the req — platform team, analytics org, or a recruiter working from a hiring manager’s voice note.
| Title on the ad | What the scope usually is | Where it differs |
|---|---|---|
| Data lake engineer | Spark pipelines, Delta/Iceberg tables, object storage layout | Often owns storage cost and file layout directly |
| Lakehouse engineer | Same, plus a serving layer and BI-facing contracts | More governance and catalog work |
| Big data engineer | Same, legacy phrasing | Sometimes still Hive/Hadoop-era tooling in the estate |
| Platform data engineer | Same, plus infrastructure-as-code and cluster policy | Less modelling, more ops |
| Data engineer (lake) | Same | The honest version |
Treat these as one candidate pool with different emphases rather than five markets. If you want the sharper split — lake-side versus warehouse-side scope — we worked through it in data lake vs warehouse engineer. For screening purposes, the practical consequence is that you should be sourcing against the work, not the noun, or you will filter out people who have done exactly your job under a different banner.
Spark is the screen that matters most
If you only have time for one technical exercise, make it Spark. In the Iceberg community’s own ecosystem survey, 96.4% of respondents report using Spark with Iceberg — which means the engine is close to a constant across this entire category of work, whatever table format sits above it. A candidate who cannot reason about shuffle behaviour, partition pruning, small-file generation and join strategy is not a data lake engineer regardless of what the title on their last contract said.
What a useful Spark screen looks like, in order of signal:
- Read a physical plan. Hand them a query plan with an obvious broadcast-vs-sort-merge mistake and ask what they would change. This separates people who have tuned Spark from people who have written Spark.
- Diagnose skew from symptoms, not logs. “One task in the stage runs for forty minutes while 199 finish in thirty seconds.” Good answers reach for salting, AQE skew handling or a rethink of the join key — and say which they would try first and why.
- Explain a write path. How does their job produce files, how big are those files, and what happens to them a week later. This is where cost-awareness shows up or doesn’t.
Why typed output tells you less than it used to
Seventy-two percent of data practitioners now prioritize AI-assisted coding in their development process. A take-home that asks for working PySpark is therefore measuring a tool everyone has, and you learn almost nothing from a clean submission. Shift the weight to live reasoning: let them use whatever assistant they like, then ask them to defend the choices in the output. The valuable candidate is the one who can say which line of the generated job will blow up at 40× the data volume. Our 90-minute vetting structure is built around that inversion.
The code is cheap now. The judgement about which code survives next quarter’s volume is the thing you are hiring.
Screen for format interop, not format loyalty
The single most common screening error on this title is demanding exclusive Delta or exclusive Iceberg experience. The same Iceberg survey shows Delta Lake overlapping at 39.3% among Iceberg users — a sizeable share of practitioners are running both, which is exactly the estate most companies have after an acquisition, a platform migration, or two teams making independent choices in 2022.
So the question is not “do you know Iceberg.” It is:
- How do you expose one physical dataset to two engines without copying it?
- When does a conversion beat a read-only interop layer, and what breaks either way?
- Where does the catalog sit, and what happens to time travel and snapshot expiry when two writers disagree?
Candidates who have genuinely lived in a mixed estate answer these with scar tissue — stale snapshots, orphaned metadata, a vacuum that ate someone’s rollback window. Candidates who have only ever used one format answer them in the abstract.
Version currency is testable in one question
The release cadence across these projects is fast enough that “I know Spark” needs a year attached to it. Delta Lake 4.4.0 is built for Spark 4.2.0, 4.1.0 and 4.0.1, and Iceberg 1.12.0 removes Spark 3.4 and Flink 2.0 support — so a candidate recommending a format or an upgrade path should be able to say which runtime versions that choice actually supports. Ask it directly: “We’re on a 3.4-era runtime and want to move to Iceberg 1.12 — what’s the order of operations?” Someone current will immediately flag the runtime bump as a prerequisite rather than a detail. Someone coasting on 2023 knowledge will not.
This is also the cheapest possible test of whether they keep up. You are not checking memorised version numbers; you are checking whether they treat compatibility matrices as part of the job.
Cost control belongs in the job description
Compute spend is outrunning headcount. 57% of organizations report increased warehouse and compute spend, compared to just 36% reporting team budget growth — which means the person you hire will be expected to absorb more workload on infrastructure that is already a line item someone upstairs is annoyed about. For a lake engineer specifically, that pressure lands on file layout and maintenance, because that is where the waste is.
Things to probe, each of which maps to a real invoice:
| Area | The question | What a weak answer sounds like |
|---|---|---|
| Small files | Target file size and how you got to it | “We run OPTIMIZE nightly” with no sizing rationale |
| Compaction cost | When compaction costs more than the scans it saves | Treats compaction as free |
| Partitioning | Why this partition key, and what it costs at high cardinality | Partitions by ingest timestamp out of habit |
| Retention | Snapshot expiry and vacuum windows vs. rollback needs | Has never set one deliberately |
| Storage class | What is on hot storage that shouldn’t be | Never looked |
A candidate who can put approximate numbers on any one of these — “we cut scan time about 40% by moving to 512MB targets” — has operated a lake rather than built one. That distinction matters more than years of experience, and it is why the capacity story in data teams so rarely resolves with another generalist hire.
Budget against data engineer pay, not a lake premium
There is no separate market for “lake” engineers, and you should not price as if there were. The US median for the data engineer role this title sits under is $165,000. Anchor there, then adjust for the specific scarcity in front of you — deep Spark tuning, mixed Delta/Iceberg estates, or genuine cost-engineering experience all move the number, while the word “lake” on a CV does not.
Practical framing for a req:
- Anchor: data engineer median, $165,000 US, as the midpoint of the band rather than the ceiling.
- Uplift triggers: proven Spark performance work at scale, dual-format estate experience, ownership of a storage cost reduction with a number attached.
- Not uplift triggers: the title itself, a vendor certification, familiarity with a managed service’s UI.
Contract rates move with the same logic; we broke the day-rate side down in data warehouse engineer rates, and the lake-side bands track closely because the underlying pool is the same people.
A one-page screening rubric
If you take nothing else from this, take the order of operations. Run it in this sequence and you will know inside ninety minutes whether to proceed:
- Spark reasoning (35 min). Plan reading, skew diagnosis, write-path explanation. Non-negotiable.
- Format interop (20 min). One scenario with two engines and one dataset. Listen for catalog and snapshot awareness.
- Version currency (5 min). The upgrade-path question. Fast signal, almost free.
- Cost (20 min). File sizing, compaction trade-offs, retention. Ask for a number from their own history.
- Hand-off (10 min). How they document a table contract for the team downstream.
What you deliberately do not test: the title, the cloud provider, the specific managed runtime. Those are learnable in weeks. Spark judgement and cost instinct are not, and they are what you are short of. If you want to see who is available against that rubric rather than against a keyword, the data engineering network is searchable without an account.