Lakehouse: One Copy, Engines Chosen Separately

Lakehouse: One Copy, Engines Chosen Separately An architecture diagram generated by Archify. Source Systems · OLTP, SaaS, events · Architecture component Source Systems OLTP, SaaS, events Batch ELT · managed connectors, dbt · Architecture component Batch ELT managed connectors, dbt CDC Stream · Debezium into a durable log · Architecture component CDC Stream Debezium into a durable log Catalogue · discovery, permissions, lineage · Lakehouse: one physical copy · a separate decision Catalogue discovery, permissions, lineage a separate decision Table Format · Iceberg, Delta, Hudi · Lakehouse: one physical copy · ACID, time travel Table Format Iceberg, Delta, Hudi ACID, time travel Object Storage · S3, GCS, ADLS · Lakehouse: one physical copy Object Storage S3, GCS, ADLS BI Engine · Architecture component BI Engine ML and Training · Architecture component ML and Training Ad-hoc SQL · Trino, DuckDB · Architecture component Ad-hoc SQL Trino, DuckDB load, then transform upserts who may read what files Lakehouse: one physical copy Legend Frontend Backend Database Cloud Security Message bus External

One copy, many engines

  • • BI and training read the same tables
  • • Storage and compute scale on different curves
  • • An open format means an engine swap is not a data migration

Two decisions, not one

  • • Format is the file layout and transaction protocol
  • • Catalogue is discovery, permissions and lineage
  • • Choosing Iceberg does not choose Unity, Polaris, Nessie or Glue

The standing costs

  • • Frequent upserts produce small files; compaction is a job with an owner
  • • Merge-on-read or copy-on-write decides who pays, readers or writers
  • • Rehearse the backfill or replay before you need it