Dashboard performance issues rarely originate from a single location. They usually have combined effects dashboard design, warehouse concurrency and caching And Data layout in your lakehouse. If you optimize just one layer—SQL, or compute sizing, or table layout—you’ll often see a partial win, but under actual use the dashboard may still feel slow or unpredictable.
In this post, we take a holistic approach Databricks for AI/BI performance. We’ll follow dashboard interactions end-to-end: from the browser and AI/BI orchestration layer, through Databricks SQL ingress and caching behavior, to file scanning and data skipping in Lakehouse. Plus, we’ll shed light on patterns that often lead to latency spikes, queuing, and massive costs — especially when multiple users interact with the same dashboard simultaneously.
Anatomy of an AI/BI Dashboard Refresh
To optimize performance, you must first understand the journey through the stack from a click. When a user opens a dashboard or changes a filter, a chain reaction occurs across multiple layers. If a layer is configured incorrectly, the user feels lag.
- Browser (client-side): This is the first line of defense. For datasets with fewer than 100,000 rows and less than 100 MB, the browser acts as a local engine, handling field filters and cross-chart interactions immediately in memory. If your data exceeds this limit, each interaction will have to be returned to the warehouse.
- Dashboard Design (Orchestrator): The AI/BI service determines which queries need to be fired. A “single-page” design sends each widget’s query simultaneously, creating a huge concurrency spike. A “multi-page” design requests data only for the visible tabs, effectively reducing the size of the demand on your compute.
- Databricks SQL (Engine): Your SQL warehouse (ideally serverless) receives the burst. it checks Cache – which consists of several layers – To see if the work has already been completed. If not, Intelligent Workload Management (IWM) intercepts the query, autoscaling the cluster in seconds to handle the load without queuing.
- Lakehouse (storage): Finally, the engine hits the data. It scans delta files cloud object storage. Here, liquid clustering and data types determine I/O efficiency. The goal is to leave as much data as possible using file-level statistics and metadata to feed the result set back into the chain.
By optimizing each of these four touchpoints, you move away from brute-force calculations and toward a streamlined architecture that suits your users.
BET – Understand your data and your dashboard
Before optimizing anything, you must first define what you are optimizing for. Dashboard performance is not a single concept, and improvements are only meaningful when tied to a clear goal. Common goals include reducing time to first view, improving interaction latency, keeping performance stable under concurrency, or reducing cost per dashboard view.
Once the goal is clear, you need to understand the parameters that shape it. These include the size and growth of the data, the number of users and their access patterns, and how the queries behave in practice – how many fire on page load, how much data they scan, and whether the results are reused or constantly recalculated. Without this context, optimization becomes guesswork and often shifts cost or latency from one layer to another.
Effective dashboard optimization, therefore, is intentional: choose a measurable goal, understand the data and usage patterns that impact it, and only then apply technical optimizations.
Optimization #1: Organize the dashboard into pages (tabs)
Every visible tile is a potential trigger: It runs on first load and can be run again when filters/parameters change, refresh, and when the user returns to a page. Tabs limit those re-executions to the active page, reducing bursts and head-of-line blocking.
AI/BI Dashboard Lets you create multi-page reports. Group visuals into pages aligned with user intent (Overview → Investigate → Deep Dive), so only the current page executes. This reduces head-of-line blocking, shapes concurrency into smaller bursts, and increases cache hit rates for repeated deterministic queries.
Recommended page types:
- Overview: Fast counters and trendlines to paint first, keep heavy additions/windows away from the landing page.
- Check: Entity-centric exploration (customer/product/region) with filters that push predicates to SQL (parameters) when pre-aggregation reduction is required.
- Deep dive: Expensive aggregations driven by scheduled refreshes or materialized/metric views (you can export a dashboard dataset to a materialized view).
Prioritize deterministic tiles (avoid() for now) to maximize result cache hit, monitor top queued questions And if constant > 0 then increase cluster size or max clusters.
The drill-through feature in AI/BI dashboards enables navigation from high-level views to detailed pages while moving the selected context. A useful strategy to implement page-based design is to improve first-paint performance and reduce unnecessary concurrency spikes by postponing expensive queries until user intent is clear.
Callout – Why it helps on any type of warehouse: Small, predictable bursts make Serverless IWM respond faster and avoid over-scaling, and they prevent Pro/Classic from saturating cluster slots during page loads.
For more information visit: https://www.databricks.com/blog/whats-new-in-aibi-dashboards-fall24
Customization #2: Customize “First Paint” with Smart Defaults
The first impression of a dashboard is defined by its first paint
