Table Format: Delta Lake vs Apache Iceberg#

Status: adopted as plan2 D13 (2026-10-30), resolving plan0 open question 9. The plan changes in §6 were approved the same day. Related:

  • plan2 D3 (conditional writes), D5 (Delta I/O manager), D6 (dlt), D10/D11 (governance, notebook access), D12 (lake convention)

  • docs/lake_convention.md §10

1. Why this needs a decision#

Delta was chosen implicitly: delta-rs and dagster-deltalake came with the original plan, and D3/D5/D6 verified them on SeaweedFS. Two findings while writing the lake convention (D12) forced the question:

  1. Athena cannot write Delta. Athena’s Delta Lake support is read-only: no INSERT, UPDATE, DELETE or MERGE, and no Delta time travel (Athena docs). Athena has full DML on Iceberg (AWS prescriptive guidance). As planned, dbt-athena could therefore only produce Iceberg or plain Parquet, never Delta.

  2. DuckDB writes Iceberg better than Delta. DuckDB’s Iceberg extension supports CREATE TABLE/INSERT (since 1.4.0) and UPDATE/DELETE (since 1.4.2) through a REST catalog (DuckDB blog, docs). Our test of DuckDB 2.0’s Delta writes failed: there is no COPY … (FORMAT delta), and ATTACH … (TYPE delta) rejected CTAS and named tables. dbt-duckdb’s delta plugin is read-only.

So the AWS side pulls toward Iceberg, and so does DuckDB’s write path. The question is whether that outweighs what Delta already gives us.

2. Comparison, for this platform#

Concern

Delta (delta-rs)

Iceberg (pyiceberg)

Needs a catalog service to commit?

No. Commits are atomic through S3 conditional writes, verified on SeaweedFS 4.48 under 16-way contention (D3). The table is self-describing in its directory.

Yes. Atomic commits go through a catalog: REST (Lakekeeper, Polaris, Nessie), SQL (Postgres) or Glue. On-prem this adds a service to Phase 2 and a hard runtime dependency.

Fits “the warehouse is a directory” (D12)

Literally: path = table, no pointer elsewhere

Only partly: the catalog holds the pointer to the current metadata

Dagster I/O manager

dagster-deltalake: official, released with Dagster, multi-type (pandas/Polars/PyArrow) verified (D5)

dagster-iceberg: community-maintained, “preview, APIs may change” (Dagster docs)

dlt

Verified on SeaweedFS, including merge and schema evolution (D6)

Supported via pyiceberg, table_format="iceberg", REST/SQL catalogs (dlt docs)

DuckDB read

delta_scan

iceberg_scan / attached catalog

DuckDB write

Poor (above)

Good, through a REST catalog

dbt on-prem (dbt-duckdb)

Needs a small plugin: the external materialization calls a plugin’s store(), which can call write_deltalake (extension point verified in dbt-duckdb 1.11.1)

Possible by attaching the REST catalog. To verify: dbt-duckdb materializing into attached Iceberg tables

Athena

Read only. Lake Formation table/column/row/cell filtering works on Delta reads (Athena engine 3) (Lake Formation docs)

Full DML; Lake Formation filtering

Glue Spark / EMR / Trino / Spark

Read and write

Read and write

AWS direction

Supported

AWS’s first-class format (S3 Tables, Glue Iceberg REST, Athena DML)

Interop bridges

UniForm makes Delta readable as Iceberg, but to verify: I found no evidence that delta-rs writes UniForm metadata

n/a

Maturity in Python

delta-rs: mature, widely used

pyiceberg 0.12 (2026-09): active and improving fast

3. Options#

  • (A) Delta everywhere, and Athena is a reader only.

    • AWS writes go through the same code as on-prem: delta-rs in Dagster runs (Fargate), Glue Spark (--datalake-formats delta), and dbt-duckdb with the Delta plugin running on AWS compute.

    • Athena reads with Lake Formation enforcement and serves analysts and Superset.

  • (B) Iceberg everywhere.

    • Add an on-prem REST catalog (Lakekeeper on Postgres is the strongest candidate: Apache-licensed, Rust, S3-compatible storage with vended credentials and remote signing (Lakekeeper)). On AWS, the Glue catalog or S3 Tables.

    • Replace D5’s I/O manager with dagster-iceberg, switch dlt to table_format="iceberg", and redo the D3/D5/D6 verification.

  • (C) Mixed: Delta for pipeline layers, Iceberg for marts. Two formats, two I/O managers, two sets of catalog rules. Rejected: it doubles the surface for little gain.

4. Recommendation: (A) Delta, with a documented Iceberg exit path#

Why Delta:

  1. No catalog in the commit path. This is the biggest operational difference. Delta’s correctness on SeaweedFS rests on a property we tested (D3). Iceberg’s would rest on a new service that has to be highly available, backed up and upgraded.

  2. What already works stays verified. D5 (multi-type I/O manager, official package) and D6 (dlt merge and schema evolution) were verified on Delta. The Iceberg Dagster integration is still a preview.

  3. Governance parity holds. Lake Formation enforces column and row filters on Athena reads of Delta. Writes don’t need Athena.

  4. Better parity, not worse. Running dbt-duckdb on both sides, with the same plugin and SQL, is a stronger “same code, either place” demonstration than dbt-duckdb on-prem vs dbt-athena on AWS. Those differ in SQL dialect (plan3 §4.8 already found GROUP BY ALL passing sqlglot but differing on Athena) and would differ in output format.

  5. It keeps the D12 thesis literal: the table is the directory.

What we give up, accepted:

  • Athena can’t be a dbt write target, and Athena time travel on Delta isn’t available. Engineers use DuckDB, Trino or delta-rs for time travel.

  • DuckDB can’t write Delta natively. We maintain a small dbt-duckdb plugin (one store() method around write_deltalake).

  • We are off AWS’s preferred format. The AWS demonstrations use Glue Spark and delta-rs for writes and Athena for reads.

The Iceberg exit path (what a switch would touch):

Component

Change

I/O manager (D5)

dagster-deltalake → dagster-iceberg (same Dagster TypeHandler pattern)

dlt (D6)

table_format="iceberg", plus catalog config

Catalog

Add Lakekeeper (on-prem) / Glue or S3 Tables (AWS). This also answers open question 5.

dbt

Drop the Delta plugin; attach the REST catalog in dbt-duckdb, or use dbt-athena on AWS

Lake convention (D12)

Unchanged: catalog lake.<layer>.<table> with table locations set to <bucket>/<layer>/<table>

Views / notebooks

delta_scan → attached Iceberg catalog

Governance (D10/D11)

Unchanged in shape; Lake Formation supports both formats

Revisit when any of these happens:

  • an AWS service we want to write with (Athena, Redshift, S3 Tables) is Iceberg-only and the workaround costs more than the switch;

  • dagster-iceberg leaves preview;

  • delta-rs development stalls, or the dbt Delta plugin becomes a burden;

  • we need catalog-level features such as multi-table transactions or branching.

Optional, cheap insurance (Phase 3 or later): an Iceberg spike that runs dlt → Iceberg through Lakekeeper on SeaweedFS and reads it with DuckDB and Trino. It proves the exit path without adopting it.

5. Things to verify#

  • Athena: whether CREATE VIEW over Delta tables works, which would let dbt-athena keep a views-only role. Also confirm the Glue registration form for native Delta (open question 10).

  • delta-rs: whether it writes UniForm/Iceberg metadata.

  • The dbt-duckdb Delta plugin: incremental models (merge) through write_deltalake/DeltaTable.merge.

  • SeaweedFS conditional writes with Iceberg catalogs, if the spike is run.

6. Changes to the plans (approved 2026-10-30)#

  1. plan0 capability matrix, “SQL modeling” row: AWS changes from dbt-athena to dbt-duckdb (same project and plugin) on AWS compute (a Dagster run on Fargate/ECS, writing Delta to S3). Athena becomes read-only: interactive SQL, Superset and Lake Formation enforcement.

  2. plan0 Phase 3: parity means the same dbt-duckdb project produces the same Delta tables on SeaweedFS and S3. dbt-athena is dropped, or kept only for views pending the §5 check.

  3. plan0 matrix “Table format: Delta on both”: unchanged, now justified by this note.

  4. New plan2 decision D13: Delta is the table format, with the exit path and triggers above.