opendatastack#
An open data platform that runs on your own hardware by default, and uses a curated set of AWS services where they are clearly cheaper, better or less work. Not on-prem or cloud: both, with every swappable layer proven to run either way.
The idea#
Managed warehouses and lakehouses sell convenience, and charge for it per credit, per DBU or per seat, with a proprietary layer that makes leaving expensive. Running everything yourself in the cloud swaps license fees for always-on infrastructure. opendatastack takes a third path:
Open source components and open formats (Delta tables on S3-compatible storage), run on hardware you own. No platform subscription.
AWS where it earns its place: off-site backup and archive, identity, burst compute, notifications. Bulk data flows in; only small results flow back.
Every layer is a configuration choice, not a rewrite. The same asset code runs against either side, and parity tests check that the results match.
The mechanism is simple. In Dagster, every external system is a resource. Asset code calls the resource, and configuration decides whether it is on-prem or AWS.
Status#
The project is built in phases. This section separates what runs today from what is designed but not yet built. Last updated 2026-10-30.
Built and running
Core platform: Dagster on Docker Compose, a SeaweedFS S3-compatible lake, Postgres, TLS and authentication, and CI that tests every change and deploys it as a release.
Lake tables: Delta tables in a layered lake (
raw→staging→intermediate→marts), written and read from pandas, Polars or PyArrow through one I/O manager. Naming, ownership and sensitivity metadata are required and enforced by tests.Ingestion: dlt loads into
raw/with merges and daily partitions; a sensor turns files arriving inlanding/into loads.SQL models: dbt (DuckDB) models written back to the lake as Delta tables; dbt tests run as Dagster asset checks.
Spark: PySpark jobs written once with Dagster Pipes, run on-prem in their own container, reading and writing Delta.
Off-site copy: an incremental replica of the lake to any S3 bucket.
Parity: the whole demo graph (Python, dbt and Spark) runs unchanged against SeaweedFS and against an S3 API, with identical tables on both. The S3 side is an emulator today; the same suite against real AWS comes with the AWS foundation phase.
Designed, not yet built
AWS foundation: identity, budgets, the backup bucket and its archive tier.
AWS Glue for burst Spark, running the same job scripts.
Self-service: notebooks over the lake, interactive SQL, BI.
Governance on both sides: catalog, lineage, fine-grained access, PII classification, audit.
Hardening and an app layer; AI tooling as a later phase.
Explore#
Coverage against Databricks, Snowflake and dbt Labs: built, planned and missing.
Where the money goes in each approach, on a relative scale.
From an empty lake to a dashboard-ready mart. Coming soon.
Conventions, decisions and how the pieces work.