Durable models on variable compute.

For ten years we optimized for the engine. We picked a warehouse, tuned it, re-platformed when the bill got embarrassing, and called the resulting churn "modernization." The next decade does not reward more of that.

The engine is now a commodity. Snowflake, BigQuery, Databricks, DuckDB, and whatever wins the next round all converge on the same set of capabilities and a familiar pricing curve. What survives the next migration is not the warehouse you chose. It is the model layer underneath it — the contracts, the lineage, the semantics — and how cleanly that layer is decoupled from the compute that happens to run it today.

The work that compounds is the model layer. Everything else is rented time on someone else's silicon.

I call the pattern durable models on variable compute. This essay is the field guide. It draws on a decade of platform work, most recently at GitLab, and on the public benchmarks behind talks like keeping AI inside the warehouse for a ninetieth-percentile cost reduction. It is opinionated. It is not vendor-neutral by accident — it is vendor-neutral on purpose, because that is the whole point.

Every team that has lived through a warehouse migration knows the real cost, and it is never the compute bill. It is the months spent re-deriving business logic that was encoded in SQL nobody owns, the dashboards that quietly changed meaning, the pipelines that ran green while reporting the wrong number. The migration was technically successful and semantically catastrophic.

That outcome is structural. When the model layer lives inside the engine — in stored procedures, in materialized views with implicit ordering, in warehouse-specific UDFs — the engine is the contract. Swap the engine and you have, by definition, broken the contract. You then spend the next two quarters convincing the business that nothing changed.

Three symptoms you have an engine-coupled model

• Your lineage graph stops at the warehouse boundary; what happens inside is opaque even to the team that owns it.
• Re-deriving a metric requires opening a SQL file in a vendor IDE, not a model definition in a repository.
• The cost of trying a second engine, even for a single workload, is a project, not an experiment.

None of these are bugs. They are the predictable consequence of putting semantics inside the engine. The fix is not a better engine. It is a model layer that outlives whichever engine you happen to be paying this quarter.

Durability, in this context, is not a vibe. It is a set of testable properties. A durable model layer is one where the contracts, lineage, and semantics can be expressed, versioned, tested, and consumed independently of the compute engine that materializes them.

If your model definition cannot survive a change of warehouse, it is not a model. It is a query.

The four properties

• Explicit contracts. Every table that crosses a team boundary has a typed, owned schema with a documented grain and a refresh SLA.
• Engine-neutral storage. The bytes live in an open table format — Apache Iceberg today — so any compatible engine can read them without an export step.
• Reproducible lineage. Every materialization is traceable to the model definition and the input snapshot that produced it.
• Portable semantics. Business logic is expressed in a layer that compiles down to engine-specific SQL, not written in engine-specific SQL by hand.

Most teams meet one or two of these. Hitting all four is the difference between a data platform and a vendor relationship.

The other half of the pattern is treating compute as a variable. Not as "we will rip out Snowflake next year" — that is just another lock-in measured in months instead of years. As "we can move this specific workload to a different engine without rewriting meaning, and we benchmark whether we should."

The case study I keep coming back to is the AI-inside-the-warehouse decision. Processing 75,000+ Zendesk tickets through Snowflake Cortex cost roughly $1,500 in a year, against more than $73,000 for a comparable external pipeline. Same model contract on either side. The only thing that changed was which engine touched the data. Because the contract was stable, the swap was a benchmark, not a project.

Variable compute is not "everything runs everywhere." It is "we choose where each thing runs, and the choice is reversible."

The bill of materials

• An open table format (Iceberg) as the single source of truth at rest.
• A neutral catalog that any engine can authenticate against and read.
• A model layer (dbt, SQLMesh, or equivalent) that compiles to engine-specific SQL from one definition.
• A governance layer — policy, lineage, snapshot lifecycle — that is engine-agnostic and lives next to the data, not inside the warehouse.

IceGov, the open-source project I maintain, is one attempt at that last piece. ObservaKit is the operational mirror — observability that does not assume the same vendor owns your storage, your compute, and your dashboards. Project Siphon is the production CDC backbone built on the same principles. The work is in /work.

The team-level consequences of taking this seriously are the part most architecture decks skip. If the model layer is the artifact that compounds, then the team's calendar should reflect that. The platform group owns the contracts, the catalog, and the governance. Domain teams own the model definitions. Compute is procurement.

Three habits that hold the pattern together

• Benchmark before you claim. Cost-per-event, end-to-end latency, and developer minutes are the three numbers that matter. Publish them.
• Promote staging-zone decoupling to a foundational move. The landing zone is not plumbing. It is the contract boundary between the world and your platform.
• Default to platform-agnostic. When two designs are roughly equivalent, pick the one that does not narrow your future engine choices.

You will be wrong about which engine wins. The model layer is how you stay right anyway.

The rest of this site is the working evidence. The numbered Chronicles develop the chapters in detail. The case studies on /work show the pattern in production. The talks at /speaking are the public benchmarks. If this is the way you think about platforms too, the newsletter is how you keep up.