Here’s the anatomy, bottom to top — because the way a good model gets built is almost always the reverse of how it gets pitched.

It starts from a decision with a price on it

A model that earns its keep is anchored to one specific, expensive decision — and someone can tell you roughly what that decision is worth. Not “understand our customers better,” but “this forecast changes how much stock we hold, and being wrong costs us about X a quarter.” That dollar figure is the whole business case. It’s what you’ll measure the model against, what justifies the maintenance cost, and what tells you, honestly, when the model isn’t pulling its weight and should be retired.

If nobody can price the decision, it’s usually a curiosity project. Nothing wrong with that — but fund it as a prototype, not as production.

The signal is real, invisible, and stable-ish

Three properties of the underlying pattern matter, and a worth-building model has all three:

The signal is real — there is genuinely more to know than the baseline already gives you. The embarrassing simple baseline (last value, the mean, the whiteboard rule) is beatable by a meaningful margin, not by a rounding error. If your fancy model beats the naïve forecast by a hair, it doesn’t clear the bar; the hair isn’t worth the machine.

The signal is invisible to a human. It lives in the interaction between dozens of features, or in a shape nobody could eyeball on a dashboard. This is the entire reason you reach for a model rather than a report — you’re finding structure that isn’t otherwise findable. If a domain expert could have written the rule, use their rule.

The signal is stable enough to survive shipping. Every model encodes an assumption about how the world works. When the world churns fast — a brand-new product with no history, a market mid-regime-change — that assumption rots before the model reaches production. A model earns its keep where the pattern has held long enough to learn and long enough to trust it next quarter.

The data is understood, not just available

This is where my first post connects. A model is only as durable as your understanding of the data feeding it. A model worth building sits on a dataset you’ve actually profiled — you know its gaps, its sentinel values, its lineage, and which parts you’d bet a decision on. Fragile models are almost always a symptom of fragile data understanding: you built intelligence on top of something you never really inspected, and the model inherits every hidden flaw. When the data is understood, you can tell the difference between the model being clever and the model memorising a bug.

There’s a human who acts differently because of it

A model earns its keep only if it changes what someone does. This is the most-skipped step and the biggest cause of abandoned ML. Before building, I want to see the shape of that change: where the output appears, in front of the person who acts, at the moment they’re about to act, in a form they trust enough to let it override their gut. A brilliant prediction buried in a report nobody opens is worth precisely nothing. The cheapest part of most successful model projects is the modelling; the expensive, decisive part is the last mile into a real workflow.

This is also why I care about explainability as a requirement, not a nicety. Not because stakeholders demand interpretability in the abstract, but because a person who acts on a model needs to sanity-check it. If the model says “deny this one” and can’t give a reason the operator respects, they’ll route around it — and now you’ve paid for a model you don’t actually use.

The maintenance is owned, from day one

Here’s the honesty test that separates teams that ship good models from teams with model graveyards: before you build, name who owns it in a year, and roughly what keeping it healthy costs. A model is a running commitment — pipelines that must keep feeding it, features that must stay available, drift that must be monitored, retraining that must happen when the world moves. If nobody’s budgeted for the boring forever-part, the model doesn’t earn its keep; it accrues a debt someone will pay later at high interest.

I like to write the model’s own “retirement criteria” down at the start: the conditions under which we’d knowingly switch it off. It forces the whole team to admit the model is a means to the priced decision from step one, not a thing that exists for its own sake.

What this looks like in the wild

Put together, a worthy model usually reads like: a specific, priced decision; a signal that’s real, hidden, and reasonably stable; data we actually understand; a named human whose behaviour changes when the output appears; and an owned plan to keep it alive. Miss any one of those and the model tends to be expensive theatre — accurate on the validation set, inert in the organisation.

When all of them line up, models are the best tool we have. They find what we can’t see, at a scale and speed no human matches, and they quietly compound value every day they run. That’s the version worth building. It’s just not the version most requests turn out to be — which is exactly the point of making them fight for their place.

Next, I want to zoom back out to the layer underneath all of this — making charts people actually trust: the small design and labeling choices that decide whether a non-specialist leans on a dashboard or quietly ignores it.

This is the filter I run before I let anyone (including me) greenlight modelling work. It’s three questions, and a model has to survive all three to be worth building.

Test 1 — Can you state the decision the model changes?

Every model exists to make a specific decision better, cheaper, or faster. If you can’t name that decision in one sentence — who acts on the output, what they do differently, and how you’d know the world got better — you don’t have a modelling problem yet. You have a reporting problem, or a “we’re curious” problem. Both are legitimate, but neither deserves the expense of a production model. Curiosity is fine; just call it what it is and prototype it, don’t operationalise it.

A huge share of “AI requests” I’ve seen die right here. Someone wants a prediction, but the moment you trace it forward, the real need is a number on a screen that a human already knows how to interpret. That’s a SQL query and a chart, not a model. Ship those, and you’ve solved it this week instead of this quarter.

Test 2 — Is the pattern already obvious to the people in the room?

A model earns its keep when the signal is invisible to a human and expensive to get wrong — that’s the whole justification. When there are hundreds of interacting features, or the relationship genuinely shifts over time, modelling finds structure nobody could see by eye.

When there are only a handful of conditions, and a domain expert could write the rule on a whiteboard in ten minutes, a model is a liability. You’ve taken something transparent and made it opaque, slower, and harder to defend. The business user now has a reason code they can’t explain, an auditor has a black box they can’t sign off, and the moment the model does something odd, nobody can tell you why. A simple rule you can read, argue with, and change on a Tuesday afternoon is worth more than an accurate model you have to retrain and re-litigate. “Would a spreadsheet answer this?” is a surprisingly sharp question. If yes, it usually should.

Test 3 — Would a human do it about as well, and the volume is low?

Even when a model could work, automation only pays off at volume. If a forecast is needed for thirty accounts and an analyst with a decent spreadsheet is 90% as good, the last 10% of accuracy is not worth the entire machine — the labelling, the pipeline, the monitoring, the drift. Automate what’s high-volume and repetitive. Leave the low-volume, high-judgment calls to humans and give them better tools and better data instead of a model.

The part nobody puts in the pitch deck

Models aren’t a build cost; they’re a maintenance cost that never stops. A model in production is an assumption about the world that quietly rots the day the world changes. Every one you ship is a recurring commitment: data pipelines that must keep feeding it, features that must stay available, thresholds that must be monitored, drift that must be detected and corrected, and a human who must own all of that when the original author has moved on. I’ve watched teams spend a quarter getting a model to work and then three years keeping it from falling over — for a problem a rules table would have covered on day one.

And there’s the trust cost. The moment a model gets a visible case wrong, the people you’re trying to help stop believing the whole thing, and you spend your credibility defending complexity instead of delivering clarity. A transparent rule that’s occasionally wrong in an explainable way holds trust far better than a slightly-more-accurate model that’s occasionally wrong in a mysterious way.

What “earning the model” actually looks like

I’m not anti-model. I build them. But I want them to survive scrutiny, so I give them a fair trial:

• A baseline that’s embarrassingly simple — often just the mean, or the rule the SME would write. If a gradient-boosted forest beats it by a rounding error, the model loses.
• A measure tied to the decision from Test 1, not an accuracy metric the business can’t feel. Precision on a chart is not the same as money saved or risk avoided.
• An honest total-cost estimate: build plus the maintenance forever, weighed against the value of that last sliver of improvement.

Run that gauntlet and you’ll find models are occasionally, genuinely the right call — and when they clear it, everyone trusts them precisely because they had to fight for their place.

The mindset I’m pushing for is simple: treat a model as an expensive answer, not a default one. Most of good analytics is getting clean, well-understood data in front of a thoughtful human with a transparent view — and then only reaching for the model where the invisible-and-expensive condition truly holds. Fewer models, better models.

Next time I’ll take the other side of the ledger: the anatomy of a model that actually earned its keep — what made it worth every bit of the maintenance, start to finish.

Here’s the order I actually work in. Not the textbook order — the one that survives contact with real data.

Day 1 — Don’t touch it. Just look.

The instinct is to open the file and start fixing. Resist it for a day. On a large, unfamiliar dataset, the single most damaging thing you can do is change a value before you understand what the value means. So day one is pure profiling. I’m answering three questions:

• Shape. How many rows, how many columns, and does the row count match what the business expects? A table that’s supposed to hold every customer and somehow has fewer rows than there are active accounts is telling you something already.
• Completeness. What share of each column is populated? Nulls aren’t uniformly bad — a column that’s 4% empty is a rounding error, a column that’s 60% empty is a story, and the story is almost never about the data.
• Distributions. For every column that matters, look at its range and its shape. Min, max, mean, median, the obvious outliers. If a “age” column tops out at 147, or a monetary column is full of exact repeats down to the cent, or a date column has records from 1970, you’ve just found the seams where the dataset was stitched together.

The output of day one isn’t a fix. It’s a list of surprises and a rough map of how bad it is.

Day 2 — Profile like a detective, not a statistician

The difference between an experienced data person and a script is that the script reports what is wrong; you need to work out why. Sentinel values masquerading as data are the classic example — a -1 or 9999 or a blank string that isn’t really null but definitely isn’t a real value. A GROUP BY on the “weird” columns and a quick eyeball of the top and bottom values surfaces these fast.

Duplicate keys are the second category worth real time. Is a duplicate a genuine data-quality bug (the same transaction captured twice) or a modelling question (the same customer legitimately placing two orders)? You cannot know that from the data alone, which is the segue into day three.

Day 3 — Follow the lineage before you follow your gut

Most “cleaning” mistakes are really lineage mistakes. Before I dedupe, impute, or standardise anything, I want to know where each field comes from. Which upstream system owns it. Whether it’s a daily snapshot or an append-only log. Who else downstream already depends on it.

This matters because the same field often means different things in different systems, and a value that looks broken in isolation is perfectly correct given how it was produced. A null that’s genuinely “missing” and a null that means “the user skipped an optional field” look identical in the file and require opposite treatments. You only tell them apart by walking the data back to its source and asking the people who own it a small number of specific questions.

The other reason to do lineage early: once you know what downstream systems read this data, you also know what you’re not allowed to change casually. That constraint shapes every decision after it.

Day 4 — Fix the unambiguous things, in the open

Now you’re allowed to fix something — but only the changes that are clearly right and cheap to reverse. Standardising formats (dates to one format, text to one case, units to one system). Collapsing obvious sentinel values into true nulls. Documenting every one of them in a single, boring, readable log that says what you changed, why, and how to undo it.

That log is the difference between a data professional and a person who quietly breaks a report. Nobody ever resents a well-documented change. They resent finding out three weeks later that the numbers moved and no one can say why.

Day 5 — Handle the judgment calls as decisions, not accidents

The genuinely hard problems don’t have a “correct” answer; they have a defensible one you chose on purpose. Do you impute the missing values, drop the rows, or leave the gap visible and let the consumer decide? Do you merge two records that are probably the same person, or keep them separate and accept the overcount? Each of these should be written down as a small, explicit decision with a rationale and a home (usually the same log), so that six months later you can explain the shape of the data to someone who wasn’t in the room.

A good rule I use: make the choice that a reasonable colleague, reading only the dashboard and not the code, would find least misleading. Data engineering optimises for correctness; decision support optimises for trust. On a messy dataset you’re doing both jobs at once.

The checklist I keep

Stripped down, the first-week sequence is:

1. Row counts and key uniqueness against what the business expects.
2. Null / empty rate for every column.
3. Range and distribution for every column that feeds a decision.
4. Hunt for sentinel values and impossible dates.
5. Write a “what I’d expect vs. what I see” list.
6. Map each field to its source system and its downstream users.
7. Fix only the reversible, unambiguous problems — and log every one.
8. Turn the remaining mess into explicit, documented decisions.

The point of the whole week isn’t to arrive at a clean dataset. It’s to arrive at a dataset you understand — one where you know exactly which parts are trustworthy, which parts are a judgment call, and which parts you’d never bet a decision on. That’s the honest version of “clean,” and it’s the only kind that actually holds up when someone asks you, a month later, “can we rely on this?”

Next time, I’ll go the other way on purpose: when NOT to build a model — the framework I use for spotting when a simple dashboard or a plain rule earns its keep better than machine learning ever could.

I’m Maximilian Jackson. I’ve spent the last twenty years working with data inside technology organisations — helping them take large, messy, complicated datasets and turn them into information people can actually make decisions with. My work has gradually moved from statistics and modelling into the broader world of data management, analytics, and AI leadership.

Right now I’m Director, Data & AI at B H S, doing some of the most hands-on work I’ve had in years. I’m organising the data, rebuilding the systems that sit on top of it, redesigning the processes and the jobs that depend on those systems, and training the people who use all of it every day. It’s equal parts setting strategy and getting things to actually work in practice.

At the core of what I do is a simple obsession: converting data into clarity. That spans the whole lifecycle — integration and management, then modelling with machine learning, advanced statistics, custom algorithms, network graphs, and operations research, and finally visualisation and dashboards that non-specialists can genuinely read and trust. I care just as much about the human side: translating between technical teams and business stakeholders, and building and leading teams that ship good work.

You can expect posts here about a few recurring themes:

• Data & analytics in practice — how you take a large, complicated dataset and shape it into something that actually improves a decision.
• AI and modern data systems — rebuilding data estates, and where machine learning and advanced modelling earn their keep (and where they don’t).
• Leadership and enablement — stakeholder management, team building, and training people to work confidently with data.

I started this blog as a place to think out loud and document lessons from the field — especially while I’m mid-rebuild at B H S. This is the messy, real version of the work that doesn’t fit neatly into a slide deck.

I don’t have a rigid schedule. The plan is to write when I have something worth sharing and keep it useful.

If you’d like to reach me or see more of my work, find my code on GitHub or connect with me on LinkedIn.

Thanks for reading — more soon.

(A note on dating: this is written up as it stood in early 2025. It reflects the tooling and the one-team context at GetGo at the time, and some of it — especially the platform speculation — has already moved on. Read it as field notes from a specific moment, not a current-state reference.)

The whole thing comes down to three sections: ETL, Delta Lake, and Orchestration.

The shape of the thing: medallion architecture

At a high level, we build the lakehouse on Delta Lake, and organise it into three “zones”:

• a raw zone for landing and processing incoming files,
• an enrichment zone that extracts transactional data, cleans it, and stores it as facts and dimensions,
• an aggregation zone that builds business-level summaries.

This is the medallion architecture — Bronze, Silver, Gold — named so the tiers are easy to refer to and reason about. Bronze is raw and trusting of nothing; Silver is clean, typed, and business-shaped; Gold is aggregated and ready for a dashboard.

The unified platform is what makes the one-person version viable. Out-of-the-box features — streaming ingestion, incremental change tracking, ACID tables, a built-in scheduler — amplify what a single data engineer can carry.

ETL

Data ingestion

The first step is ingestion: think of it as continually producing files of incremental data that need to be processed and stored in the warehouse. Those files get generated and dropped into cloud object storage, and from there the lakehouse takes over.

This is probably the hardest and most uncertain part of the whole lakehouse. Data comes from everywhere — email, relational databases, SFTP, documents, accounting software, CRMs, Excel sheets, customer chat platforms, REST APIs, server logs, social media, marketing platforms. The list genuinely goes on. Assuming each source maps to a validated business use case, our job as data engineers is to pipe it in, and the sky’s the limit on how.

Roughly, the toolbox splits into scripts and products:

Scripts

• Python on serverless (AWS Lambda, Azure Functions) or on servers — good for SFTP, social media, and REST APIs
• Google Apps Script — handy for email attachments when your mailbox is Gmail
• SQLAlchemy

Products

• Fivetran, Airbyte, Stitch, Power Automate, AWS Database Migration Service, Estuary, Informatica

The capability of each product, and the combinations of script-and-product you can pull off, are endless. In the end it reduces to a function of cost and the environment you operate in. If you’re not on Gmail, for example, you’ll need an email connector (Fivetran, Airbyte) or you’ll stand up AWS Simple Email Service to save attachments into S3 — and that S3 bucket should sit in a sanitisation zone so anti-virus can scan files before they ever reach the lakehouse.

For the mechanical SFTP→S3 problem, I have an older step-by-step that pairs the AWS CLI, a Makefile, and Python to move files into object storage. I wrote it in 2017 and it was still working in 2024 — a small testament to how boring and reliable the right ingestion plumbing should be.

Relational databases

For databases, I’ve used Fivetran and AWS DMS to perform Change Data Capture (CDC). They behave differently in a way that matters:

• Fivetran ships Teleport Sync, which pulls deltas without requiring you to enable binary logging on the source database.
• AWS DMS relies on the bin log to identify changes. The bin log degrades database performance from an IOPS perspective, but in our case the pros outweighed the cons.

A platform-specific wrinkle: Fivetran can push straight into your Bronze tables, whereas AWS DMS drops CSV/Parquet into a landing S3 bucket that then needs processing into Bronze. One integration point of friction, saved.

Databricks had also just acquired Arcion, a CDC specialist, and with Unity Catalog leaning on Lakehouse Federation, my then-hope was that native CDC replication into the lakehouse would become table stakes and pull ingestion inside the platform. Time will tell on that one.

Data processing

Bronze — the raw / ingestion zone

Files are now sitting in object storage. Autoloader does exactly what its name says: it automatically reads new files as they land, using Spark Streaming, and keeps a log of which files have already been ingested so the incremental pattern is reliable. New arrivals are picked up and loaded into a streaming DataFrame ready for processing.

For each source we keep two notebooks: one to initialise the Bronze table from everything currently in object storage, and one to process the incrementals (deltas). The init notebook just reads all the files that exist today and forms the initial raw table.

One deliberate choice when creating Bronze: cast every column to String. You’re not fighting messy type inference at ingestion; you’re just forming a structured table so cleaning can proceed. Real types come later, in Silver.

Once the Bronze table exists, it’s ready to take deltas via Autoloader. Reading a new batch into a Spark DataFrame:

bronze_booking_concluded_df = (
    spark.readStream
    .format('cloudFiles')
    .option('cloudFiles.format', 'csv')
    .option('header', 'true')
    .option('inferSchema', 'true')
    .option('cloudFiles.schemaLocation', schema_path)
    .option('rescuedDataColumn', '_rescue')
    .load(input_table_path)
)

New files are picked up at the next scheduled run. If a source adds new columns you didn’t plan for, Autoloader doesn’t blow up — the surprises land in a _rescue column as JSON, so the pipeline keeps flowing while you decide what the new field means. Then you append into the Bronze table:

(
    bronze_booking_concluded_df
    .writeStream
    .format('delta')
    .trigger(availableNow=True)
    .option('mergeSchema', 'true')
    .option('checkpointLocation', checkpoint_path)
    .outputMode('append')
    .start(output_table_path)
    .awaitTermination()
)

availableNow=True is the key operational trick: it processes all currently-available data and then exits, instead of running forever. That lets you schedule the Delta notebook as a batch job at fixed times. If you truly want continuous ingestion, you leave a cluster running 24/7 and drop the bounded trigger.

Silver — the enrichment zone

Silver tables are built from Bronze, and again each gets an init and a delta notebook. The init reads Bronze wholesale, but this time selects only the columns you want and casts them into proper types — strings become dates, integers, and so on — and removes duplicates. This is where raw text becomes trustworthy data.

The delta notebooks lean on Spark Streaming plus Delta Lake’s Change Data Feed to process only what’s changed since the last known point. (Practical note: don’t use Databricks’ own docs for CDF — go to the Delta Lake docs, they’re clearer and have better examples.)

Concretely, ingesting from a Bronze table created by AWS DMS CDC. upsertToDelta is the micro-batch function Spark Streaming calls to apply the deltas — “upsert” being Update/Insert/Merge in one, which is one of the genuinely nice Delta features. Without it you’d stage the incremental data in a temp table and then merge the staging table into the target by hand.

def upsertToDelta(microBatchOutputDF, batchId):
    (
        delta_table.alias("target")
        .merge(
            (
                microBatchOutputDF
                .filter(col('Op').isin(['I', 'U']))
                .withColumn(
                    'rnk',
                    rank().over(
                        Window.partitionBy(f'{join_key}')
                              .orderBy(col('transact_id').desc())
                    )
                )
                .filter(col('rnk') == 1)
                .drop('rnk')
                .dropDuplicates()
            ).alias('source'),
            f'source.{join_key} = target.{join_key}'
        )
        .whenMatchedUpdateAll(condition=update_condition)
        .whenNotMatchedInsertAll()
        .execute()
    )

(
    silver_booking_concluded_delta_df
    .writeStream
    .format('delta')
    .foreachBatch(upsertToDelta)
    .outputMode('update')
    .trigger(availableNow=True)
    .start()
    .awaitTermination()
)

Why all the Window / rank() ceremony? Because plain dropDuplicates() in Spark doesn’t keep the latest or earliest row — it drops duplicates arbitrarily. If a key arrives several times in a micro-batch, you want to deterministically keep the newest one by transact_id, so you rank within the partition and keep rank 1. It’s the difference between “mostly correct” and “defensible at 2am when someone asks why the numbers moved.”

If the data genuinely doesn’t care about row order, you can skip the windowing and keep it simple:

def upsertToDelta(microBatchOutputDF, batchId):
    (
        delta_table.alias('t')
        .merge(
            microBatchOutputDF.dropDuplicates([join_key]).alias('s'),
            f's.{join_key} = t.{join_key}'
        )
        .whenMatchedUpdateAll()
        .whenNotMatchedInsertAll()
        .execute()
    )

(
    fact_c_delta_df
    .writeStream
    .format('delta')
    .foreachBatch(upsertToDelta)
    .outputMode('update')
    .trigger(availableNow=True)
    .start()
    .awaitTermination()
)

Silver — fact and dimension tables

The whole point of the warehouse is to make information easy to analyse. Transactional databases aren’t built for that: they’re heavily normalised to preserve data integrity, which is great for OLTP and miserable for analytics. Fact and dimension tables are de-normalised — shaped for aggregation and, ultimately, dashboards that answer real business questions. This master data also preserves the business context that downstream teams rely on.

A few principles that keep the semantic layer humane:

• Enrich with data from outside the transactional system — weather, for instance — to widen the range of questions the model can answer.
• Replace numeric category references with the category names, so you shed unnecessary joins.
• Keep infrequently-used reference data separate and join it only when needed.

Each fact and dimension table gets its own init and delta notebooks, same as everything else. We also keep quarantine tables for records that fail quality or integrity checks. The value is that a workflow can continue running while bad rows are isolated and fixed before re-introduction — at the cost of an extra daily check that the quarantine batches are actually being worked.

The classic enrichment dimension is dim_dates. Here’s how I generate a calendar and shape it into a usable dimension (modified from a find on Stack Overflow, no shame):

begin_date = '2021-01-01'
end_date   = '2030-12-31'

(
    spark.sql(
        "select explode(sequence(to_date('%s'), to_date('%s'), interval 1 day)) "
        "as calendar_date" % (begin_date, end_date)
    )
    .createOrReplaceTempView('dates')
)

Then a SQL pass adds the calendar attributes:

create or replace temporary view temp_silver_dates as (
  select
    calendar_date,
    year(calendar_date)  as year,
    month(calendar_date) as month,
    lower(date_format(calendar_date, 'MMMM')) as calendar_month,
    day(calendar_date)   as day,
    lower(date_format(calendar_date, 'EEEE')) as calendar_day,
    weekday(calendar_date) + 1 as day_of_week,
    case when weekday(calendar_date) < 5 then 'Y' else 'N' end as is_week_day,
    case when calendar_date = last_day(calendar_date) then 'Y' else 'N' end as is_last_day_of_month,
    dayofmonth(calendar_date) as day_of_month,
    dayofyear(calendar_date)  as day_of_year,
    weekofyear(calendar_date) as week_of_year_iso,
    quarter(calendar_date)    as quarter_of_year
  from dates
)

Finally we materialise the Delta dimension, deriving week boundaries and left-joining a public-holiday table so analysts get is_public_holiday for free:

spark.sql(f'''
    create or replace table {target_catalog}.{silver_dim_date.schema}.{silver_dim_date.table}
    using delta
    select
      calendar_date, year, month, calendar_month, day, calendar_day, day_of_week,
      is_week_day, day_of_month, is_last_day_of_month, day_of_year,
      week_of_year_iso, quarter_of_year,
      date_sub(calendar_date, day_of_week - 1) as week_start,
      date_add(calendar_date, 7 - day_of_week) as week_end,
      if(h.holiday_date is null, 'n', 'y')     as is_public_holiday
    from temp_silver_dates d
    left join {target_catalog}.{silver_dim_public_holiday.schema}.{silver_dim_public_holiday.table} h
      on d.calendar_date = h.holiday_date
''')

Gold — aggregates

Gold is the summarisation layer: grouping individual quantitative records into categorical aggregates. “What’s the total sales of all retail stores in the North in December?” — that’s an aggregate function (sum, mean, average) applied over fact data and grouped by dimension data.

One design decision to make consciously: view or physical table? If the data is volatile or you want the number always current, a view that computes the summarisation on read can be right; if the aggregate is expensive or point-in-time stable, a physical table that receives appends is often cheaper to serve. The temporal nature of the data decides it.

Delta Lake

Delta Lake is the open-source storage layer the whole thing sits on — the best one-liner for it is parquet files on steroids. Three features do most of the heavy lifting here (and there are many more — the Delta docs have clearer, richer examples than any single post can).

Change Data Feed (CDF)

CDF is what lets each medallion tier consume incremental data efficiently off the tier below. Enable it deliberately, not by default — you’ll find tables where it isn’t needed, so turn it on per-table where you actually stream changes. Combined with Spark Streaming and Delta’s upsert/merge, CDF is the backbone of doing CDC entirely inside the lakehouse.

Time Travel

Time Travel lets you read a table as of an earlier version or timestamp, with roughly a year of history retained. This is unglamorous until it saves your week: we had production customer data accidentally deleted, and because the incrementals had also mirrored the deletion into the lakehouse, we recovered both the production system and the Delta table from a version before the delete. It’s a safety net you won’t appreciate until you need it.

Liquid Clustering

Liquid Clustering improves on manual partitioning and Z-ORDER by simplifying layout decisions to optimise query performance — and crucially it lets you redefine clustering columns without rewriting existing data, so layout evolves alongside analytic needs. In practice, it removes a whole class of “go optimise your tables” chores. We were still applying it to a few multi-terabyte tables at the time of writing.

Orchestration

The line that matters most: planning your update cadence is directly proportional to your spend on Databricks. Tight budget, low urgency? Run weekly. Fast decisions that need fresh numbers? Hourly, or every three or six hours. Cadence is a cost dial, and you should be turning it on purpose.

Databricks Workflows comes out of the box and is the de-facto scheduler on the platform — no separate orchestration infra to deploy or maintain. (You’ll know the alternatives: Apache Oozie, Airflow, Dagster.) The one honest caveat: orchestration reaches within the platform well, but ingestion still lives mostly outside it — a gap I hoped the Arcion acquisition would shrink.

Three things to get right when scheduling jobs:

Compute. Three types — Serverless, Job Compute, and All-purpose. Prefer a shared Job Cluster: it cuts overall cost and ships with Photon enabled by default, so you pay less for more. For irregular, infrequent jobs, try Serverless. For All-purpose clusters, actually look at the compute metrics to find the bottleneck and tune to it.

Number of jobs. As sources and tables multiply, job count grows, a single workflow stops being enough, and you start nesting workflows inside larger ones. That’s where cluster-driver problems surface. The easy path out is Serverless. The hands-on path is disciplined trial and error: grow the driver size, add worker nodes, or fall back from Spot to On-demand — but change one thing at a time so you learn which lever actually worked.

Dependency planning. Your warehouse design dictates which jobs depend on which. Build a core workflow that cannot be allowed to fail, and keep everything less critical in separate workflows so a noisy neighbour can never break essential loading.

Why this is enough

That’s the set — ingestion, the Bronze/Silver/Gold processing pipeline, the handful of Delta features that carry real weight, and a scheduling philosophy tied to cost. Individually none of them are exotic; together they’re more than one person needs to run a lakehouse that a whole business can depend on. The point was never to build less. It’s that with the right platform, the essentials are small enough to own — and once you’ve got them, “building a lakehouse” stops being daunting and starts being a thing you simply do.