A job spec that asks for “strong Scala and Spark internals, plus dimensional modelling and stakeholder-facing semantic layer ownership” is not one role. It is two roles fused into one line item, and the fusion is what makes it unfillable. One half of it lives in the JVM — serialization, partitioning, shuffle behaviour, custom sources. The other half lives in conversations about what a customer actually is and whose number wins when finance and product disagree. People who are genuinely senior in both exist. They are rare enough that writing the spec that way is a pricing decision, not a sourcing one.
So decide first. Most teams that write the combined spec do it defensively — they have been burned by a pipeline and burned by a bad model, and the requirement is a list of scars rather than a description of work. Separate the two and the hire gets dramatically easier, because one of those halves is the one you are actually short on.
The pools are not comparable sizes
Start with the arithmetic, because it sets the price of everything else in the spec. Just 2.2% of surveyed developers use Scala — 309 people out of 14,099 respondents. Python sits at 58.0%, roughly 26 times the Scala share.
That ratio compounds with everything else you bolt on. Scala and Delta internals and Kimball-literate modelling and your industry domain is a four-way intersection on top of a 2.2% base. Meanwhile demand for data-skilled staff is projected to grow 35% between 2025 and 2035, far faster than the labour market as a whole — so the narrow spec is competing for that intersection in a market that is already tight at the broad end.
| Requirement stacked | Effect on the pool | What it buys you |
|---|---|---|
| Python + SQL | 58.0% base — broad | Most pipeline and transformation work |
| Scala | 2.2% base — ~26x narrower | JVM-level Spark work, custom connectors, legacy codebases |
| + dimensional modelling | Narrower again | Grain discipline, conformed dimensions, a semantic layer that survives contact with finance |
| + your domain | Narrowest | Someone who knows what a “subscription” means in your business |
Every clause in a requirement is a multiplier on time-to-hire, and Scala is the most expensive multiplier in data engineering.
Scala is a codebase decision, not a seniority signal
There are real reasons to insist on Scala. You maintain a Spark codebase someone wrote in 2019 and nobody wants to rewrite it. You build custom sources, sinks or UDAFs where the JVM boundary actually costs you. You run Flink or Kafka Streams in anger. You need someone who can read a stack trace through Catalyst rather than around it. Those are legitimate and the premium is justified.
What Scala is not is a proxy for rigour. Teams use it as one — the reasoning goes that anyone who learned Scala must be serious — and that reasoning has aged badly. 85% of developers now regularly use AI tools for coding, with 62% relying on at least one assistant, agent or AI code editor. Raw language fluency is the part of the job that assistance has flattened hardest. Screening on it selects for the thing that has become cheapest to acquire while telling you nothing about whether the candidate knows why a fact table grain matters.
If the honest answer is “we have 40,000 lines of Scala and no one left who wrote it,” hire for Scala and say so plainly in the spec. If the answer is “we wanted someone rigorous,” you have written the wrong requirement and will pay a scarcity premium for a signal you could have tested directly.
The failures teams report are modelling and ownership failures
Ask teams what actually broke last quarter and the answers cluster somewhere else entirely. Ambiguous data ownership remains a challenge for 41% of respondents, effectively unchanged year over year — a number that stubborn is structural, not technical. Nobody fixes unclear ownership with a better serializer.
The second pattern is worse. 72% of data teams prioritise AI-assisted coding while only 24% prioritise AI-assisted pipeline management — testing, observability and quality controls. Teams are investing in producing more code three times as fast as they are investing in trusting it. The gap lands squarely on whoever owns the model: more tables, more transformations, more lineage, and no corresponding growth in the discipline that keeps the warehouse coherent.
That is the shortage. It is not a shortage of people who can write a Spark job. It is a shortage of people who will define a grain, hold it, name things consistently, write the tests, and tell a stakeholder no. If your last three incidents were a duplicated dimension, a metric that changed definition mid-quarter, or a dashboard nobody trusted — Scala in the spec is solving a problem you do not have.
Agents already write most of the pipeline code
The hand-written pipeline is becoming the exception. Over 70% of Spark Declarative Pipelines on Databricks are now created by agents rather than typed by engineers. Whatever you think of that, it changes what you are buying when you hire. The scarce contribution shifts from authoring transformations to specifying them correctly, reviewing what came back, and owning the contract the pipeline is supposed to honour.
And it compounds with the 72/24 split. More generated code, flat investment in validation, unchanged ownership ambiguity — that is a description of a warehouse accumulating entropy faster than anyone is clearing it. The person who clears it is a modeller with review discipline, not a language specialist.
This is also why the combined spec is the worst possible bet right now. You are paying a scarcity premium for the half of the work that has been most automated, in order to get the half that has not.
Two specs, written separately
Write them as two documents and the difference stops being subtle.
| Scala data engineer | Data modeller / analytics engineer | |
|---|---|---|
| Core problem | Throughput, cost, reliability of execution | Meaning, grain, trust, definitions |
| Typical artefacts | Custom sources and sinks, streaming jobs, library code | Conformed dimensions, semantic layer, tests, contracts |
| Fails when | Jobs are slow, expensive, flaky, or unmaintainable | Two dashboards disagree and nobody can say which is right |
| Interview should probe | Partitioning, shuffle, serialization, Delta internals, debugging under load | Grain selection, SCD trade-offs, how they’d model a messy real entity, how they say no |
| Pool | ~2.2% base, narrow | Broader, but senior judgement is the constraint |
| Hire it when | You own JVM code or streaming infrastructure | Definitions, ownership or trust are the pain |
Most teams reading that table already know which column describes their last six months. If you genuinely need both, hire them in sequence and let the first hire tell you how urgent the second is — or run a contractor for the narrow half while you hire the broad half permanently. The Scala work is usually bounded and project-shaped; the modelling work is permanent and relational. That alone suggests which one belongs on a payroll and which one belongs on a statement of work.
If the titles themselves are the confusion, the lake-versus-warehouse title question is worth settling before you write either spec, because it decides which of the two columns above is your centre of gravity.
Screen for the half you chose
For the Scala half, test the thing the agent cannot do: give them a slow job and a Spark UI, and watch them reason about skew, spill and join strategy. Do not test syntax. With 85% of developers using assistance by default, syntax is the least informative ninety minutes you will ever spend. A structured ninety-minute technical screen beats a take-home that an assistant finishes in twelve minutes.
For the modelling half, hand them a genuinely ugly entity from your own business — the one with three source systems and two competing definitions — and ask for a grain, a set of dimensions, and the tests they would write. Then ask what they would refuse to build and why. The 41% ownership number is the thing you are hiring against; the answer to “who owns this table and who gets told when it changes” tells you more than any code sample.
Map the skills explicitly before you interview, so the panel is scoring the same role — a skills matrix keeps the Scala interviewer from quietly failing a strong modeller. And when you brief the market, brief it on the half you actually need. A spec that asks for a 2.2% language and senior modelling judgement in the same bullet list does not get you both; it gets you a longer search and a worse shortlist. Describing the real problem rather than the fused requirement is what gets you a shortlist of people who have solved it.