In July 2025, the US FDA publicly released an initial batch of more than 200 complete response letters (CRLs) — the decision letters that explain why drug and biologic applications were not approved on first review. The release marked a major transparency shift: for the first time, sponsors, physicians, and data teams can study the agency’s own language about clinical, CMC, safety, labeling, and bioequivalence deficiencies in a centralized, downloadable set of PDFs. This walkthrough, based on a Databricks engineering blog post, shows how Databricks Agent Bricks turns those letters into structured, analyzable data.
The plain-language version: regulatory intelligence is locked in PDFs that are hard to process at scale. Teams that can extract it systematically can de-risk their own submissions, spot common pitfalls, and reach market faster.
Why AI extraction projects stall
Technical depth often pushes engineers to build in silos, opening a gap between what gets built and what the business needs. By the time a subject matter expert (SME) sees results, they are frequently not what was needed, the feedback loop is slow, and momentum dies. A second failure mode is the missing baseline: without ground truth, teams spend months relying on subjective “vibe checks” instead of empirical evidence.
The workflow described here rests on three principles: SMEs and technical leads collaborating in a single interface rather than email loops; business-defined ground-truth labels built directly into the workflow for formal scoring; and a full-platform approach with automated pipelines, LLM-as-judge evaluation, managed GPU throughput, and Unity Catalog governance end to end.
Step 1: Parse PDFs with ai_parse_document()
CRLs have complex layouts — headers, footers, tables, charts, multiple columns — where naive text extraction fails. The ai_parse_document() SQL function uses multimodal AI to understand document structure: it extracts text in reading order, preserves irregular table hierarchies, and generates captions for figures, with OCR and vision-language models handling historically difficult PDFs. Databricks says the approach scales to enterprise volumes at a claimed 3-5x lower cost than leading competitors, under compute-based rather than per-license pricing.
Operationally, a Unity Catalog volume is pointed at the cloud storage containing the PDFs, and a single SQL command processes them into a structured table of parsed content — with the GPU and model backend fully hosted, so no infrastructure or external LLM calls need configuring.
Step 2: Collaborative extraction with Agent Bricks
With clean text available, the goal becomes extracting specific structured fields — the application ID, what was deficient, the rejection citation. The Agent Bricks Information Extraction interface provides a real-time collaborative UI where the SME specifies the fields to capture, the agent turns those requirements into editable extraction guidelines, and both SME and engineer see the resulting JSON output instantly against the source document. If a field extracts incorrectly, either party can refine the prompt, add a field, or adjust instructions and see results in seconds.
Step 3: Evaluate formally, not by vibes
An agent that looks correct during development still needs a quality gate: a prompt change that fixes one document may break ten others. Agent Bricks builds in two evaluation paths. With ground-truth labels, an SME-provided “golden set” of manually validated extractions lets the platform score every field, compare agent output side by side with correct answers, and flag regressions immediately after any change. Without labels — the common cold-start problem — an LLM-as-judge workflow has an evaluation model generate expected answers from the source text and score the agent’s output against them, providing scalable quality measurement that can also run in production.
Step 4: Deploy into the pipeline with ai_query()
Once accuracy is verified, the agent deploys as a serverless model endpoint in one click, and the ai_query() SQL function applies it inside an ETL pipeline as new documents arrive. With Lakeflow Jobs orchestrating, raw PDFs landing in cloud storage are parsed, run through the extraction agent, and written to tables ready for analysis, reporting, or downstream retrieval applications — a fully automated, production-grade pipeline.
Limitations and what to watch
Several caveats are worth noting. The cost and accuracy claims (“3-5x lower cost,” “human-level accuracy”) come from Databricks’ own marketing and should be validated against a team’s actual documents before committing. LLM-as-judge evaluation is convenient but imperfect — judge models share failure modes with the models they evaluate, so a small human-labeled set remains the stronger anchor. Extraction quality on regulatory documents also has compliance implications: outputs feeding real regulatory decisions need human review regardless of benchmark scores. Finally, this is Part 1 of a series covering extraction only; the retrieval and copilot layers build on top. Related Databricks platform news is covered in this post on Iceberg support in Delta Sharing.
