Spark Jobs#

Large computations run as PySpark scripts written once in spark_jobs/, launched from Dagster through Dagster Pipes. On-prem, each job runs in its own container from the Spark image. On AWS (after Phase 1), the same script runs as an AWS Glue job. The script reads and writes Delta tables in the lake and reports back to Dagster: logs, the materialization and its metadata.

How it works#

Piece	File	What it does
Job script	`spark_jobs/customer_ltv.py`	Reads its inputs, writes its output, reports to Dagster. Table URIs arrive as Pipes extras, so the script holds no paths or credentials.
Asset	`dagster_platform/assets/demo/spark.py`	`marts/customer_ltv`: declares its upstream tables as `deps`, passes their URIs, returns what the job reported.
Resource	`dagster_platform/resources/spark.py`	`SparkResource` starts the job with `PipesDockerClient` on the host’s Docker, passes the lake settings, and removes the container afterwards.
Image	`onprem/core/docker/spark/`	Python, Java 21, PySpark, and the Delta and S3A jars, installed at build time so runs download nothing.

Writing a job for both sides. Keep to APIs that Glue 5 also has (Spark 3.5, Python 3.11). Use s3:// URIs: the on-prem image maps them to S3A, as Glue maps them to its own S3 file system. The lake_session() helper in the script sets the S3 endpoint only when LAKE_ENDPOINT_URL is present, so on AWS nothing is overridden.

Reporting. The job calls pipes.report_asset_materialization() with the same runtime metadata as other lake assets (row count, URI, Delta version) plus the Spark version. On-prem, Pipes messages travel in the container’s log stream.

The Spark image#

Built from onprem/core/docker/spark/Dockerfile and tagged like the Dagster image (opendatastack/spark:<tag>), by build.sh and by CI. Compose has a build-only spark service: compose never starts it; Pipes starts one container per job on the ods-core network.

install_jars.py lets Spark’s own resolver fetch delta-spark and hadoop-aws with their dependencies, then keeps only the jars PySpark doesn’t already ship. Ivy also brings older copies of Hadoop, Jackson, Parquet and log4j, which would shadow Spark’s own.

The image is large (about 2.7 GB). Most of it is PySpark’s own jars and the AWS SDK v2 bundle that hadoop-aws requires. Layers are shared between releases, so each release adds little.

Security#

Run containers mount the Docker socket so they can start Spark containers, as the Dagster daemon already does. That gives a run root-equivalent access to the host. It is acceptable while every run executes this repository’s code on a single trusted host. A Docker socket proxy (Phase 7) is the planned hardening.

Adding a job#

Write spark_jobs/<name>.py with open_dagster_pipes(), reading its inputs from pipes.extras.
Add an asset that calls spark.run(context, "<name>.py", lake=lake, extras={...}) and returns .get_materialize_result(). Declare the inputs as deps and the output columns (D14).
Test it in tests/spark/ and add it to the parity suite.