Databricks operates at a scale where Databricks’ internal logs and datasets are constantly changing – schemas evolve, new columns appear, and data semantics drift. This blog discusses How Databricks uses Databricks internally so that, as the platform continues to change, PII and other sensitive data can be labeled correctly.
To do this, Databricks built LogSentinel, an LLM-powered data classification system on Databricks, which tracks schema evolution, detects labeling drift, and feeds high-quality labels into its governance and security controls. The team uses MLflow to track experiments and monitor performance over time, and the company is integrating the best ideas from LogSentinel back into databricks data classification products so that customers can benefit from the same approach.
Why does this system matter
This system is designed to move three concrete business levers to the platform, data and security teams:
- short compliance cycle: Recurring review tasks that previously took weeks of analysts’ time are now completed in hours, because columns are pre-labeled and pre-tried before a human can even see them.
- low operational risk: The system continuously detects labeling drift and schema changes, so sensitive areas are less likely to slip silently with incorrect or missing tags.
- strong policy enforcement: Trusted labels now directly drive masking, access control, retention, and residency rules, turning “best-effort governance” into an executable policy.
In practice, teams can plug new tables into a standard pipeline, monitor drift metrics and exceptions, and trust the system to enforce PII and residency constraints without having to build a custom classifier for each domain.
System Architecture at a Glance
Databricks built an LLM-powered column classification system that continuously annotates tables using internal data classification, detects labeling drift, and opens remediation tickets when something looks wrong. The various components involved in the system are outlined below (tracked and evaluated using MLFlow):
- Data Ingestion: Ingesting various data sources (including Unity Catalog column data, label classification data, and ground truth data)
- Data Augmentation: Augmenting Data Using Databricks Vector Search AI comment generation
- LLM Orchestra
- Level labeling system
- Model versioning: running multiple models in parallel
- Label prediction: predicting final labels using mixture of experts (MOE) approach
- Ticket creation: Finding violations and creating JIRA tickets
The end-to-end workflow is shown in the figure below
data ingestion
for each log type or dataset to be annotated, the system randomly samples values from each column and send the following metadata to the system: table name, column name, type, existing comment, and a small sample of values. To reduce LLM costs and improve throughput, multiple columns from the same table are batched together in a single request.
The taxonomy is defined using protocol buffers and currently includes over 100 hierarchical data labels, with room for custom extensions when teams need additional categories. This gives governance and platform stakeholders a shared agreement on the meaning of “PII” and “sensitive” beyond a handful of regexes.
data enrichment
Two enhancement strategies significantly improve classification quality:
- AI column comment generation: used when comments are missing Databricks AI-Generated Comments To synthesize concise, human-readable descriptions that help both LLMs and future table consumers.
- Few-shot example generation: the system maintains a ground truth dataset and use both static examples and dynamic examples obtained through vector search; For each column, the system creates an embedding from name, type, comment, and context, then retrieve columns with top-like labels to include in the prompt.
During early stages or when labeled data is limited, static notation is best, providing consistency and reproducibility. Dynamic prompting is more effective in mature systems, using vector search to pull similar instances and adapt to new schemas and data domains in large, diverse datasets.
LLM Orchestra
At the core of the system is a lightweight orchestration layer that manages LLM calls at production scale.
Key capabilities include:
- Multi-model routing in internally hosted LLMs (for example, Llama, Claude, and GPT-based models) with automatic fallback if a model is unavailable.
- Re-examine the logic for rate limits with transient failures and exponential backoff.
- Validation hooks that detect empty, invalid, or confused labels and re-run those cases with the backup model.
- Batch processing that annotates multiple columns at once to optimize token usage without losing context.
Level labeling system
The system predicts three types of labels per column:
- Granular labels are built from a set of 100+ fine granularity options that power masking, reduction, and strict access control.
- Hierarchical labels, which aggregate related granular labels into broad categories suitable for monitoring and reporting.
- Residency labels, which indicate whether data must remain in the region or can be transferred cross-region, directly feed data movement policies.
to keep predictions consistent and minimize hallucinations, the system uses a two-step flow: a broad classification step specifies a high-level category, then a refinement step chooses accurate labels within that category. This demonstrates how a human reviewer would first decide “this is workspace data” and then choose the specific workspace identifier label.
Model version and label prediction
Instead of relying on a single “best” configuration, each model setup is treated as an expert competing to label a column.
Multiple model versions run in parallel with differences in the following:
- Primary and fallback LLM options.
- Use of generated observations versus raw metadata.
- Signaling strategy (static vs. dynamic few-shots).
- Label granularity and taxonomy subsets
Each expert produces a label and a confidence score between 0 and 100. The system then selects the label from the expert with the highest confidence, a mixture-expert style approach that improves accuracy and reduces the impact of sometimes poor predictions from any one configuration.
This design makes it safe to experiment: new models or accelerated strategies can be introduced, run alongside existing models, and evaluated on both metrics and downstream ticket volume before becoming the default.
Ticket generation
The pipeline continuously compares the current schema annotations with the LLM predictions to uncover meaningful deviations.
Typical cases include:
- New columns added without any annotation.
- Existing annotations that no longer match the contents of the column.
- Columns with sensitive values that are labeled as eligible for cross-region movement.
When the system detects a violation, it creates a policy entry and files a JIRA ticket to the owner team with the table, column, proposed label, and confidence reference. This turns data classification issues into an ongoing workflow that teams can track and resolve in the same way they track other production events.
Impact and evaluation
The system was evaluated on 2,258 labeled samples, of which 1,010 contained PII and 1,248 were non-PII. On this dataset, it reached 92% accuracy and 95% recall for PII detection.
More importantly for stakeholders, the deployment produced the operational results that were required:
- Manual review effort for each large-scale audit cycle reduced from weeks to hours as reviewers start with high-quality suggested labels rather than raw schema.
- Instead of being discovered during an annual review, labeling drift is now detected continuously as the schema is developed.
- Alerts about sensitive data mislabeled as secure are more targeted, so security teams can take immediate action instead of testing noisy rules-based scanners.
- Masking and residency policies are largely enforced using the same label classification that powers the analysis and reporting.
Precision and recall act as guardrails, but the system is tailored to outcomes such as review time, latency to drift detection, and volume of actionable tickets produced per week.
conclusion
By combining taxonomy-driven labeling and an MoE-style evaluation framework, the design enabled existing engineering and governance workflows in Databricks to manage experiments and deployments using MLflow. This keeps labels fresh when the schema changes, makes compliance reviews faster and more focused, and provides the enforcement hooks needed to consistently enforce masking and residency rules across the platform.
A central goal of the project is integrating these internal learnings directly into the data classification product. As the approaches are piloted and validated inside LogSentinel, they are incorporated into Databricks data classification.
The same patterns – ingesting metadata and samples, enhancing context, orchestrating multiple LLMs, and feeding predictions into policy and ticketing systems – can be reused wherever reliable, evolving understanding of the data is needed. By incorporating these insights into the core product, Databricks aims to let organizations apply the same data-intelligence approach to compliance and governance that it uses internally.
Limitations and what to watch
LogSentinel is described here from an internal engineering account, so its reported benefits reflect Databricks’ own environment rather than independent evaluation. Because the system relies on large language models to classify columns, it can still mislabel sensitive data, miss novel patterns, or produce inconsistent results, which is why its design emphasizes ensemble voting, validation hooks, and human review through ticketing rather than fully automated enforcement. Effectiveness depends heavily on the quality of ground-truth labels, taxonomy design, and ongoing monitoring of drift. Organizations applying similar patterns should treat LLM classifications as high-quality suggestions that feed governance workflows, keep humans in the loop for residency and compliance decisions, and validate accuracy on their own data before relying on automated labels.
