Declarative pipelines provide teams with an intent-driven way to create batch and streaming workflows. You define what needs to happen and let the system manage the execution. It reduces custom code and supports repeatable engineering patterns.
As organizations’ data usage increases, pipelines grow. Standards evolve, new sources are added, and more teams participate in development. Even small schema updates ripple across dozens of notebooks and configurations. Metadata-driven metaprogramming addresses these issues by moving pipeline logic into structured templates that are generated at runtime.
This approach keeps development consistent, reduces maintenance, and scales with limited engineering effort.
In this blog, you will learn how to build metadata-driven pipelines for Spark Declarative Pipelines using DLT-META, a project from Databricks Labs that implements metadata templates to automate pipeline creation.
As useful as declarative pipelines are, the work required to support them grows exponentially as teams add more sources and expand usage across the organization.
Why is it hard to maintain manual pipelines at scale?
Manual pipelines work on a smaller scale, but maintenance effort grows faster than the data. Each new source adds complexity, causing the logic to wander and be reworked. Teams patch pipelines instead of improving them. Data engineers constantly face these scaling challenges:
- Lots of artifacts per source: Each dataset requires new notebooks, configurations, and scripts. Operating overhead increases exponentially with each onboard feed.
- Argument updates are not propagated: Business rule changes fail to be applied to pipelines, resulting in configuration drift and inconsistent output across pipelines.
- Inconsistent quality and governance: Teams create custom assays and lineages, making it difficult to enforce organization-wide standards and making results highly variable.
- Limited secure contributions from domain teams: Analysts and business teams want to connect data; However, data engineering still requires reviewing or rewriting the logic, which slows down delivery.
- Maintenance increases manifold with each change: Simple schema changes or updates create a large backlog of manual work across all dependent pipelines, hindering the agility of the platform.
These issues show why a metadata-first approach matters. This reduces manual effort and keeps pipelines largely stable.
How DLT-META addresses scale and sustainability
DLT-META solves the problems of pipeline scale and stability. It is a metadata-driven metaprogramming framework for Spark declarative pipelines. Data teams use it to automate pipeline creation, standardize logic, and scale development with minimal code.
With metaprogramming, pipeline behavior is repeatedly derived from configuration rather than notebooks. This gives teams a clear advantage.
- Less code to write and maintain
- Faster onboarding of new data sources
- Production-ready pipelines from the ground up
- Consistent patterns across platforms
- Scalable best practices with lean teams
Spark declarative pipeline and DLT-Meta work together. Spark declarative pipelines define intent and manage execution. DLT-META adds a configuration layer that generates and scales pipeline logic. Combined, they replace manual coding with repeatable patterns that support governance, efficiency, and growth at scale.
How DLT-META addresses real data engineering needs
1. Centralized and templated configuration
DLT-META centralizes pipeline logic in shared templates to remove duplication and manual maintenance. Teams define ingestion, transformation, quality, and governance rules across shared metadata using JSON or YAML. Teams updates the configuration once when a new source is added or a rule changes. The logic automatically propagates across pipelines.
2. Instant Scalability and Fast Onboarding
Metadata driven updates make it easier to scale pipelines and incorporate new sources. Teams add sources or adjust business rules by editing metadata files. Changes are applied to all downstream workloads without any human intervention. New sources move to production in minutes instead of weeks.
3. Domain team contributing with applicable standards
DLT-META enables domain teams to contribute securely through configuration. Analysts and domain experts update metadata to accelerate delivery. Platform and engineering teams maintain control over validation, data quality, transformation, and compliance rules.
4. Enterprise-wide sustainability and governance
Organization-wide standards are automatically applied to all pipelines and consumers. The central configuration applies consistent logic to each new source. Built-in audit, lineage and data quality rules support regulatory and operational requirements at scale.
How teams use DLT-META in practice
Customers are using DLT-META to define ingestion and transformations once and apply them through configuration. This reduces custom code and onboarding speed.
Immediate impact was seen on cineplexes.
We use DLT-META to minimize custom code. Engineers no longer write pipelines differently for simple tasks. Onboarding JSON files enforces a consistent framework and handles the rest.-Aditya Singh, Data Engineer, Cineplex
PsiQuantum shows how efficiently small teams scale.
DLT-Meta helps us manage bronze and silver workloads with less maintenance. It supports large data volumes without duplicate notebooks or source code.-Arthur Valladares, Principal Data Engineer, PsyQuantum
Teams across all industries apply the same patterns.
- retail Stores and centralizes supply chain data from hundreds of sources
- logistics Standardizes batch and streaming ingestion for IoT and fleet data
- financial Services Enforces audits and compliance while onboarding feeds faster
- Health care Maintains quality and auditability across complex datasets
- Manufacturing and Telecommunications Large-scale ingestion using reusable, centrally governed metadata
This approach lets teams increase pipeline numbers without increasing complexity.
How to Get Started with DLT-META in 5 Simple Steps
You don’t need to redesign your platform to try DLT-META. start small. Use some sources. Let the metadata do the rest.
1. Get framework
Start by cloning the DLT- META repository. It gives you the templates, examples, and tooling you need to define pipelines using metadata.
2. Define your pipelines with metadata
Next, define what your pipelines should do. You do this by editing a small set of configuration files.
- Use conf/onboarding.json To describe raw input tables.
- Use conf/silver_transformations.json To define changes.
- Optionally, add conf/dq_rules.json If you want to enforce data quality rules.
At this point, you are describing the intention. You are not writing pipeline code.
3. Onboard metadata into the platform
Before running the pipeline, DLT-META must register your metadata. This onboarding step converts your configuration into dataflowspec delta tables that pipelines read at runtime.
You can run onboarding from a notebook, Lakeflow job, or the DLT-Meta CLI.
One. Manual onboarding via notebook Here
Use the provided onboarding notebook to process your metadata and organize your pipeline artifacts:
B. Automate onboarding through Lakeflow Jobs with Python Wheels.
The example below shows the Lakeflow Jobs UI for creating and automating DLT-META pipelines
C. Onboard using the DLT-META CLI commands shown in the repo: Here,
DLT-Meta CLI lets you onboard, run and deploy in an interactive Python terminal
4. Create a common pipeline
With metadata, you create a single common pipeline. This pipeline reads from dataflowspec tables and generates arguments dynamically.
Use pipeline/dlt_meta_pipeline.py as an entry point and configure it with reference to your bronze and silver specifications.
This pipeline remains unchanged as you add sources. Metadata controls behavior.
5. Trigger and run
Now you are ready to run the pipeline. Trigger it like any other Spark declarative pipeline.
DLT-META creates and executes pipeline logic at runtime.
The output is a production-ready bronze and silver table with consistent changes, quality rules and lineage applied automatically.
try it today
To start, we recommend starting a proof of concept using your existing Spark declarative pipelines with a few sources, moving the pipeline logic into metadata, and orchestrating the DLT-meta at scale. Start with a small proof of concept, and see how metadata-driven metaprogramming extends your data engineering capabilities further than you thought.
Databricks Resources
