Round-trip proof#

Proposed in plan2 D17; step 1 (generated data, the engines on-prem today) is built.

A round-trip proof takes a table through two engines and shows, step by step, that the data and the governance rules attached to it come out the way they went in. Anyone behind the install’s login can start one from the portal (/proof/), pick two engines, and watch it run. The copies live in a sandbox that is deleted as soon as the results are recorded.

It exists because “Delta is the contract” and “swap any layer” are claims. Tests in CI prove them to the platform’s developers. A proof you start yourself, on your own install, proves them to you. It’s also meant for teams on Databricks or Snowflake who want to check their own tables against an open platform: the fingerprint below is written so that they can compute it on their side and compare.

How a proof runs#

Generate a table built to break naive round trips: nulls next to NaN and infinities, 64-bit integer extremes, decimals, unicode and empty strings, dates before 1970, timestamps with and without a time zone, lists and structs. Its columns carry D14 tags: a confidential email, a quasi-identifier city.
Engine A writes it as Delta.
Engine B reads A’s table, fingerprints what it read and, if it can write Delta, writes it again.
Engine A reads B’s table back.
Every step is fingerprinted and compared with the generated data, and the read-back is compared row by row.
The governance checks run, the results are recorded in Dagster, and the sandbox is deleted.

The record is Dagster’s: the proof is the asset _system/roundtrip_proof, each check is an asset check on it, and a failed check fails the run, so it shows wherever Dagster’s asset health does.

Engine	Delta implementation	Writes
Polars, pandas, PyArrow	delta-rs	yes
DuckDB	delta-kernel	no (second engine only)
Spark	Delta Lake (JVM)	yes

A pair on the same implementation checks the engines more than the format. A pair across implementations, such as Polars and Spark, proves more.

The checks#

Check	Kind	Passes when
`rows_match`	data	every step has the generated row count
`schema_match`	data	both Delta tables have the generated names and types
`fingerprints_match`	data	every step’s fingerprint equals the generated data’s
`rows_identical`	data	A’s read of B’s table equals the generated data, row by row
`declared_schema`	governance	both tables match the declared (D14) column schema
`tags_carried`	governance	the D14 column tags and descriptions are inside the Delta files
`sensitivity_floor`	governance	no carried tag is weaker than declared
`profile_no_leak`	governance	profiles of confidential columns hold counts only
`delta_protocol`	governance	each engine read the other’s protocol and table features
`one_writer`	governance	every commit on each table is by the expected engine
`access_d11`	governance, warning	never passes yet: D11 enforcement on-prem is deferred, so it’s reported as a gap
`sandbox_discarded`	governance	the sandbox is empty afterwards

The ods fingerprint (version 1)#

One row of numbers that says two tables hold the same data, whatever the row order or file layout. It’s defined so that any SQL engine can compute it. The platform generates the query for DuckDB, Spark (and so Databricks) and Snowflake from a table’s schema: dagster_platform/proof/fingerprint.py, sql(table, schema, dialect).

Per table: rows. Per column:

Type	Figures
integer, decimal, string, boolean, date, timestamp	`nulls`, and `hash`: the sum over rows of the first 15 hex digits of `md5(canonical text)`, as an integer
float, double	`nulls`, `nan`, `posinf`, `neginf`, and over finite values only `min`, `max`, `sum` (sums compared with a relative tolerance of 1e-9: float sums depend on order)
list, struct, map	`nulls` only in version 1

Canonical text: integers and decimals cast to a string (decimals keep their scale, 12.30); booleans true / false; dates YYYY-MM-DD; timestamps as integer microseconds since 1970-01-01 UTC (a timestamp without time zone is read as UTC, so Spark sessions use spark.sql.session.timeZone=UTC).

The DuckDB and Spark versions are tested against each other on every proof that uses Spark: Spark computes the figures in its own SQL, the platform computes them with DuckDB, and they must match. The Snowflake version is written to the same spec but hasn’t been run against Snowflake yet.

What the proofs found, and what changed (2026-11-03)#

Polars, PyArrow and DuckDB agree on every check, in every pair tried.
pandas, through its default conversions, changed data: a NaN came back as a null, decimal(18,2) was re-inferred as decimal(8,2), and a nested 32-bit integer became a double. Fixed: the platform now loads pandas with Arrow-backed dtypes and casts pandas outputs to the declared column types, and pandas proofs pass.
Spark wrote Parquet timestamps as INT96 by default. delta-rs reads INT96 as nanoseconds, which only cover 1677 to 2262, so Spark’s tables with later timestamps couldn’t be read by delta-rs at all. Fixed: the platform’s Spark writes TIMESTAMP_MICROS; every Spark pair passes, and Spark’s fingerprint equals DuckDB’s.
Column tags didn’t travel in the files. PyArrow keeps Arrow field metadata in the Delta schema; Polars, pandas and Spark (from Parquet) drop it. Fixed: every platform write path now puts the D14 tags and column descriptions into the Delta schema, so anyone reading the files (a notebook, Databricks, Snowflake) sees them; tags_carried is now an error, not a warning.

Settings#

ODS_PROOF: generic (default: the generated data set), off, or all (later: actual data, with authority, after OIDC).
One proof runs at a time: the Dagster queue limits runs tagged ods/proof to one (onprem/core/dagster.yaml).
Who may start a proof is a placeholder (proof:run, granted to everyone behind the login) until roles are discussed (plan0 open question 25).