Data engineering teams face a recurring problem: how to use AI to improve ETL and build reliable, production-grade pipelines without piling on new complexity. What they need is AI that streamlines workflows rather than scattering disconnected tools or stripping away context.
Databricks Lakeflow is built around that goal. It provides an integrated data engineering platform with embedded, governed AI that automates data processing end to end, surfaces more insight, and supports a wide range of business problems. Whether the task is generating pipeline code or running AI workloads, teams using Lakeflow can spend less time on manual glue work and more on the high-value patterns that move the business. This article explains, in plain terms, how organizations can productize and scale AI models by running them directly inside the data pipeline to unlock business insights automatically.
Bringing AI into the pipeline at scale
Data teams are increasingly buried in unstructured information: call transcripts, documents, images, and free-text fields that traditional ETL cannot easily interpret. Databricks addresses this with AI functions that can be called directly inside a pipeline. Several are task-specific and require no prompt engineering, including:
ai_extract: extracts specific entities from input text based on labels provided, such as person, place, or organization.ai_classify: classifies input text against labels provided, for example “urgent” versus “not urgent,” or subject categories.ai_translate: translates text into a specified target language.
A more recent addition, ai_parse_document, converts unstructured data into required structured formats. Built on a multimodal foundation model, ai_parse_document can parse text, extract tables, reason over figures, and turn images into AI-generated descriptions, opening up data that was previously difficult to analyze.
There is also a more general-purpose function, ai_query(), powered by serverless batch inference, which lets a pipeline call a chosen large language model on a column of data the same way it would call any SQL function. Rather than exporting data to a separate AI service and importing the results back, the model runs where the data already lives, inside a single governed platform. Common uses include:
- Sorting records into business-meaningful categories, running sentiment analysis across millions of multilingual reviews, or automating customer segmentation from natural-language signals at scale.
- Improving data quality through fuzzy matching and entity resolution to fix duplicates and inconsistencies.
The combination of Lakeflow and Agent Bricks lets an organization run its AI models on a single, unified, governed data platform, so the AI and the insights it produces carry the right business and enterprise context.
Two practical examples
Example 1: Turning raw call transcripts into business insight
Consider a sales team that needs to turn long, unstructured call transcripts into clear, actionable summaries. With hundreds of calls a day, many lasting 45 to 60 minutes, manual review quickly becomes impossible. Built-in AI functions can analyze each transcript, extract the key points, and structure the result. A function such as ai_query can summarize each call using a chosen model, producing consistent summaries that sales and account teams can scan at a glance, and the same workflow can generate personalized follow-ups. Those notes can flow directly into a CRM or sales tooling so teams know what to do immediately after a call, and can also be shared with a business-intelligence team to highlight gaps and improve the customer experience.
Example 2: Streamlining insurance claim processing
Consider a claims-processing pipeline for an insurance provider that needs faster, more consistent approvals. Claims often arrive by email with unstructured attachments, scanned documents, photos, and PDFs, that are hard to ingest and process at scale. With Agent Bricks AI functions, ai_parse_document can read those attachments, extract the relevant fields, and pass structured data downstream to the right team, depending on the use case. Below is an example of how such a workflow might look as a directed acyclic graph (DAG) in Lakeflow Jobs:
Combining Lakeflow and Agent Bricks makes it possible to take cluttered, unstructured inputs and turn them into structured, AI-enriched outputs within a single orchestrated pipeline.
Real-world applications
Many Databricks customers report using AI and Lakeflow to address business problems across pricing, customer success, and marketing. A New York-based fintech company has used Agent Bricks AI functions to build a scalable, accurate transaction-classification system that replaces manual, inconsistent legacy methods, allowing it to process very high transaction volumes, deliver personalized rewards, and generate richer insights. Banco Bradesco, one of Latin America’s largest banks, faced productivity barriers from lengthy coding, debugging, and documentation; by adopting the Databricks Assistant it reported cutting coding time substantially and enabling both technical and non-technical users to generate code and troubleshoot in natural language. A global omnichannel advertising platform used Lakeflow Jobs to orchestrate complex LLM training pipelines that its previous scheduler could not handle, reducing the operational burden enough that a single data scientist could build a generative-AI assistant that became a key product feature. Examples like these are part of a broader move from AI projects to operational efficiencies.
Limitations and what to watch
The approach is powerful, but several caveats apply. Calling large language models on every row of a large table has a real cost in both compute and latency, so teams should benchmark on a sample before running AI functions across millions of records. The outputs are probabilistic: an extraction or classification can be confidently wrong, which matters most in regulated workflows such as insurance claims or financial transactions, where human review and validation rules remain essential. Results also depend heavily on prompt design and the chosen model, and they can drift as models are updated, so versioning and monitoring belong in the pipeline from the start. Finally, the customer outcomes above are vendor-reported and reflect organizations with significant data maturity; they illustrate what is possible rather than guaranteeing similar results. Used with these guardrails, embedding AI functions directly in a unified data platform is a genuine simplification over stitching together separate services.
The takeaway
Lakeflow’s core idea is to bring AI to the data rather than move data to the AI. By exposing models as SQL-callable functions inside a governed pipeline, and pairing them with Agent Bricks, Databricks lets data teams integrate AI into existing ETL with less glue code and more context, provided the cost, accuracy, and governance trade-offs are managed deliberately.
