Apache Spark Structured Streaming powers large-scale, long-running data pipelines across streaming ETL, analytics, and machine learning. But as operational use cases matured, teams increasingly needed something more demanding: sub-second latency for applications such as fraud detection, personalization, anomaly detection, and real-time alerting. Meeting those requirements historically meant bolting a specialised engine onto Spark — and Real-Time Mode (RTM) for Spark Structured Streaming is designed to remove that tradeoff.
Why a second engine used to be necessary
For most of its history, Spark’s micro-batch model was excellent for high-throughput work but not built for millisecond-level latency. To serve latency-sensitive applications, many teams ran a separate system such as Apache Flink or Kafka Streams alongside Spark. That fragmentation carried real costs: duplicate codebases, separate governance models, and the need for specialists to tune and maintain engine-specific infrastructure.
Real-Time Mode, introduced in public preview in August 2025 and later released to general availability as part of Spark 4.1 in December 2025, evolves the Spark execution engine so that a second system is no longer required. Engineers can address the full spectrum of workloads — from high-throughput ETL to low-latency applications — through the same Spark API.
The performance claim, in context
RTM uses a new execution engine built around continuous data flow, pipeline scheduling, and streaming shuffle to deliver consistent sub-second latency, reportedly as low as single-digit milliseconds for time-critical workloads. In Databricks’ own benchmarking, Spark with RTM ran up to 92% faster than Flink. That figure comes from the vendor and a specific test setup, so it is best read as an indication that a separate engine is no longer automatically required, rather than as a universal result.
What it changes for real-time machine learning
The architecture’s clearest benefit shows up in real-time ML. Fraud detection and similar workloads need a smooth handoff between high-throughput batch processing for model training and low-latency streaming for live inference. When training happens in Spark and inference is moved to Flink, the same business logic ends up implemented twice, which is error-prone and invites “logic drift” — a model trained on one representation of the data but scoring on another. Keeping transformation code identical across training and inference in Spark reduces that risk and can produce features faster and more consistently. RTM also allows latency to be adjusted with minimal code changes, so freshness can be tuned to business needs rather than fixed by the engine.
How organizations are using it
Databricks reports several production users of RTM. A recommendation use case computes real-time intent features from a user’s current session so suggestions update as the person interacts with a product. A transportation and logistics company uses live telemetry to detect anomalies and shift from reactive to proactive decisions within milliseconds. The sports-betting and fantasy-sports company DraftKings has described using RTM, together with the TransformWithState API, to power feature calculations for its fraud-detection models, reporting improvements in both latency and pipeline design and, for the first time, unified feature pipelines for ML training and online inference. Databricks also lists Coinbase and MakeMyTrip among early adopters.
Limitations and what to watch
The headline comparison with Flink is a vendor benchmark and will not generalise to every workload; teams with heavy investment in Flink or specific operator semantics should test against their own pipelines. Achievable latency depends on data volumes, cluster sizing, and the specific operations used, so single-digit-millisecond figures represent best cases rather than guarantees. RTM is also relatively new, and behaviour, supported operators, and tuning guidance continue to evolve, so any production design should be validated against current Databricks and Apache Spark documentation. The most durable takeaway is architectural: consolidating batch and real-time work on one API removes duplication and a class of logic-drift errors, independent of any single benchmark number.
Further detail is available in the Databricks posts Real-Time Mode: ultra-low latency streaming on Spark APIs and the general availability announcement. For related reading on this site, see coverage of building applications on Databricks.
