Feature Comparison: Databricks, Snowflake, dbt Labs#

Version

1.2

Date

2026-10-30

Baseline

Live release 0f8504f plus Phase 4 on-prem; decisions D1–D15 (plan2)

A high-level comparison of opendatastack’s coverage, built and scoped, against the three commercial platforms it is most often measured against. Vendor product names and packaging change often, so read them as orientation, not a spec. Revise this page when a phase lands or a gap is scoped, bump the version, and add a row to the revision history.

Key: ✅ built · 🟢 decided (D#) · 🟡 planned (phase) · ⚪ placeholder (open question) · ❌ not scoped

In short: built is ✅; planned is 🟢 or 🟡; missing is ⚪ or ❌. Rows with more than one mark are partly built.

Comparison#

Capability

opendatastack

Status

Databricks

Snowflake

dbt Labs

Storage + ACID table format, time travel

Delta on SeaweedFS / S3

✅ D3, D13

Delta, Unity Catalog managed tables

Native tables, Iceberg tables

(relies on the warehouse)

Orchestration

Dagster asset graph

✅

Lakeflow Jobs

Tasks

dbt Cloud jobs

Ingestion (batch connectors)

dlt, plus a sensor on landed files

✅ D6

Lakeflow Connect

Openflow, Snowpipe

(none)

Streaming / CDC

(none yet; placeholder: Kafka-compatible broker, Debezium for CDC, Amazon MSK on AWS)

⚪ open question 20

Structured Streaming, declarative pipelines

Snowpipe Streaming, Streams

(none)

Incremental / declarative transforms

dbt incremental + Dagster partitions and automation

✅ partitions · 🟡 dbt incremental

Declarative pipelines

Dynamic Tables

Incremental models

SQL modeling

dbt-duckdb on both sides, writing Delta

✅ on-prem (D13) · 🟡 AWS, Phase 3

SQL, notebooks

SQL, Snowpark

dbt core

Semantic layer / metrics

(none yet)

⚪ open question 19

Metric views

Semantic views

Semantic Layer (MetricFlow)

Interactive SQL engine

DuckDB / Trino / Athena

🟡 Phase 5

SQL warehouses (Photon)

Virtual warehouses

(none)

Large-scale compute

Spark via Pipes, Glue for burst

✅ on-prem Spark · 🟡 Glue (Phase 4, after Phase 1)

Spark

Snowpark

(none)

Notebooks

JupyterHub + DuckDB

🟢 D11 · 🟡 Phase 5

Notebooks

Notebooks

(none)

BI / dashboards

Superset, Dash

🟡 Phase 5, 7

AI/BI dashboards

Snowsight dashboards

(none)

Hosted apps

Dash + Flask stack

🟢 D7 · 🟡 Phase 7

Databricks Apps

Streamlit in Snowflake

(none)

Data quality

Asset checks, dbt tests, pandera, schema-drift check

✅ checks, dbt tests, schema drift (D14) · 🟡 pandera, Phase 7

Lakehouse monitoring, expectations

Data metric functions

dbt tests

Catalog / discovery

UC OSS / HMS, OpenMetadata ↔ Glue, DataZone

🟢 D10 · 🟡 Phase 6

Unity Catalog

Horizon

Catalog (Explorer)

Lineage

Dagster (built in) + our own OpenLineage emitter

✅ Dagster · 🟡 Phase 6

Unity Catalog lineage

Horizon lineage

Explorer lineage

Fine-grained access (column, row, tag)

Trino + OPA ↔ Lake Formation, tag-driven

🟢 D10, D11, D14 · 🟡 Phase 6

Unity Catalog ABAC

Masking and row access policies, tags

(none)

PII classification

Presidio ↔ Macie, plus D14 tags

🟡 Phase 6

Data classification

Sensitive-data classification

(none)

Audit

SeaweedFS audit log → Loki ↔ CloudTrail

🟡 Phase 6

System tables

Access history

(none)

Identity / SSO

Cognito; interim basic auth

🟡 Phase 1

Workspace SSO

SSO

SSO

CI/CD for data code

GitHub Actions, release deploys

✅ D9

Asset Bundles

(none)

dbt Cloud CI

Dev/prod environments, data cloning, branching

Code releases only; no separate data environments yet

⚪ open question 18

Workspaces, shallow clone

Zero-copy cloning

Environments, defer

Data sharing / marketplace

(none yet; Delta Sharing and Delta UniForm under evaluation)

⚪ open question 21

Delta Sharing, Marketplace

Secure sharing, Marketplace

(none)

Backup / disaster recovery

SeaweedFS → S3 → Glacier, plus a DR drill

✅ replica to S3 · 🟡 Glacier, DR drill (Phase 1, 7)

Cloud storage + replication

Fail-safe, replication

(none)

Observability / cost

Prometheus/Grafana/Loki, AWS Budgets

🟡 Phase 1, 7

System tables, budgets

Resource monitors

(none)

AI (LLMs in SQL, vector search, text-to-SQL, assistants)

Ollama/Bedrock through a gateway, Open WebUI

⚪ open question 14

Mosaic AI, Genie, Assistant

Cortex, Cortex Analyst

dbt Copilot

ML lifecycle / feature store

(none yet)

⚪ open question 22

MLflow, feature store

Snowflake ML

(none)

Cross-project modularity

(none yet)

⚪ open question 23

Multiple workspaces

Multiple accounts

dbt Mesh

Documentation

Sphinx (this site)

✅ D15 · ⚪ fold in the plans (open question 16)

Catalog docs

(none)

dbt docs

OLTP alongside analytics

Postgres exists, for Dagster only

❌ likely out of scope

Lakebase (Postgres)

Hybrid tables, Postgres

(none)

Runs on-prem and on a cloud, swappable

Yes, proven by parity tests

✅ against an S3 emulator · 🟡 real AWS, Phase 1

No (managed cloud only)

No

Partly (core is open source)

Reading the table#

  • Built: the on-prem lakehouse loop runs end to end: storage and table format, orchestration, batch and file ingestion, SQL modeling, data quality checks, lineage in the asset graph, and an off-site copy. Each piece is also run against an S3 API in the parity tests.

  • Scoped, not built: catalog, fine-grained access, PII classification and audit (governance phase); notebooks, interactive SQL and BI (self-service phase); Spark and burst compute.

  • Placeholders (plan0 open questions 18–23, added in 1.0) (plan0 open questions 18–23), roughly in order of importance:

    • dev/prod data environments

    • semantic layer

    • streaming / CDC

    • data sharing

    • ML lifecycle / feature store

    • cross-project modularity

  • Where opendatastack is structurally different:

    • no per-credit billing;

    • every layer can run on-prem or on AWS, with tests proving it;

    • Dagster’s asset graph and lineage are arguably stronger than the vendors’ built-in schedulers.

  • The cost: we operate it ourselves. There’s no serverless autoscaling, and capacity depends on the deployment’s hardware (plan0 open question 15).

Revision history#

Version

Date

Change

1.0

2026-10-30

First version. Gaps 1–6 recorded as plan0 open questions 18–23.

1.1

2026-10-30

Statuses updated for Phases 2–3 (on-prem): dlt and landing ingestion, dbt writing Delta, checks, replica, parity against an S3 emulator. Published on opendatastack.io (D15).

1.2

2026-10-30

Large-scale compute: Spark via Pipes built on-prem.