Data engineering teams are under pressure to deliver high-quality data quickly, yet building and operating pipelines keeps getting harder. Databricks, which open-sourced the framework described here, says that after interviewing hundreds of data engineers and studying millions of real-world workloads, it found that engineers spend most of their time not writing transformation logic but absorbing the operational burden around it. Existing frameworks force teams to handle orchestration, incremental processing, data quality, and backfill by hand, and as data volumes and use cases grow, that burden turns data engineering into a bottleneck rather than an accelerator.
A familiar pattern: declarative wins
The industry has hit this wall before. Early data processing required writing a new program for every question, which did not scale. SQL changed that by making queries declarative: a user specifies what result is wanted, and the engine decides how to compute it. Spark Declarative Pipelines (SDP) applies the same idea one level up, extending the declarative model from individual queries to entire pipelines so that Apache Spark can plan and execute them end to end.
What Spark Declarative Pipelines does
Instead of manually moving data between steps, a developer declares which datasets should exist, and SDP is responsible for keeping them accurate over time. In a pipeline that calculates weekly sales, for example, SDP infers the dependencies between datasets, builds a single execution plan, and updates results in the correct order. It processes only new or changed data automatically, expresses data-quality rules inline, and handles backfill and late-arriving records without manual intervention. Because it understands query semantics, it can validate a pipeline before running it, execute steps safely in parallel, and recover correctly from failures, capabilities that require a pipeline-aware declarative API built directly into Spark.
How it compares in practice
The practical payoff is a large reduction in boilerplate. A weekly-sales pipeline that SDP expresses in roughly twenty lines would, in plain PySpark or dbt, require external tools and manual handling for the same guarantees. Several benefits stand out:
- Automated incremental processing. The framework tracks what has already been processed and reads only new or changed records, removing the need for manual MAX queries, checkpoint files, or conditional logic.
- Integrated data quality. A decorator such as
@dp.expect_or_dropautomatically filters out bad records. In PySpark, teams split good and bad records by hand; in dbt, this needs a separate model and manual handling. - Automatic dependency tracking. The framework detects that, for instance,
weekly_salesdepends onraw_salesand orders execution accordingly, with no explicit wiring. - Pre-run validation and easy backfill. A pipeline can be checked for syntax errors and schema mismatches without executing, and backfilling a table such as
raw_salesis a single command. - Built-in observability. Monitoring is available through a built-in interface rather than bolted-on external tooling.
Where it stands
SDP became a native capability in Apache Spark 4.1, contributed by Databricks from the framework previously known as Delta Live Tables. It offers Python and SQL APIs for defining datasets, supports both batch and streaming queries, tracks dependencies and runs efficient parallel updates, and ships with a command-line tool to prepare, verify, and run pipelines locally or in production. The roadmap is being developed openly with the Spark community, with planned support for continuous execution, more efficient incremental processing, and change data capture (CDC). A commercial version, Lakeflow Declarative Pipelines, is generally available and adds enterprise features and support on top of the open-source core.
Limitations and what to watch
The declarative model is a clear simplification, but it is not magic, and a few caveats are worth keeping in mind. Declarative frameworks trade fine-grained control for convenience: when a pipeline needs unusual logic the framework does not anticipate, working around its abstractions can be harder than writing imperative code directly. The headline productivity claims and the twenty-line comparison come from the vendor and assume a use case the framework handles natively, whereas real pipelines vary. As a newly open-sourced project, parts of the ecosystem, including tooling, connectors, and community knowledge, are still maturing, and some advanced capabilities remain on the roadmap rather than shipping today. Teams already standardised on dbt or hand-written Spark will also weigh migration cost against the benefit. Used where it fits, though, SDP meaningfully reduces the operational toil that has long made data engineering a bottleneck, which is why it pairs naturally with broader efforts to build AI-ready data pipelines on a unified data platform.
The takeaway
Spark Declarative Pipelines argues, in effect, that data engineering should follow the same path queries took decades ago: from telling the system how to do the work to declaring what should be true and letting the engine handle the rest. By building that model into Apache Spark itself, it aims to become a shared, extensible foundation for reliable batch and streaming pipelines, with the operational burden handled by the framework rather than by engineers.
