As organizations look to scale real-time operational intelligence, traditional streaming architectures have become costly bottlenecks. Managing message buses like Kafka and handling schema registries and connector frameworks creates a significant “complexity tax” that diverts high-value engineering resources from strategic business initiatives. Meanwhile, duplicate storage drives up cloud bills and multi-hop architecture delays critical insights. Finally, data in transit often resides outside centralized governance frameworks, creating compliance risks and lineage blind spots.
Introducing Zerobus Ingest: Closer to real-time streaming in the lakehouse
Today, we are excited to announce the general availability of Zerobus IngestPart of Lakeflow Connect. Zerobus Ingest is a fully managed, serverless service that streams data directly into controlled Delta tables, removing intermediate layers to provide a simplified, high-performance architecture.
By enabling data flow directly from producers to lakehouses, Zerobus Ingest reduces costs and eliminates equipment sprawl. It also delivers high performance at scale, supporting thousands of concurrent connections and achieving over 10GB/sec overall throughput across a table in less than 5 seconds.
Single-Sync Benefits: Simplified architecture for major cost reduction
Traditional message buses like Kafka were designed as multi-sink architectures: universal hubs that deliver data to dozens of independent consumers. However, this flexibility can come at a high cost when your only destination is a lakehouse. Zerobus Ingest uses a fundamentally different approach, with a single-sink architecture optimized for a single task: pushing data directly into Lakehouse.
This architectural choice eliminates complexity and reduces costs significantly:
- no broker Scaling up as your data volume grows
- no division to tune for optimal performance
- No consumer group To monitor and debug
- no cluster upgrade planning and executing
- no special expertiseLike Kafka, essential in your team
With Zerobus Ingest, there is a single, managed Databricks endpoint. Create your table in Unity Catalog, start writing data with the API or SDK, and you’re done. That’s it, nothing else to setup. The serverless architecture automatically scales to support gigabyte-per-second ingestion without any configuration changes.
Zerobus Ingest data allows producers to bypass the message bus and push events directly into managed Delta tables in your Lakehouse.
Zerobus Ingest simplifies the traditional streaming architecture from five managed systems to two components, eliminating multiple failure points, reducing operational overhead and removing the need for specialized expertise.
- Traditional Architecture: Source System → Message Bus (Kafka Cluster) with Schema Registry → Connectors → Lakehouse
- Zerobus Ingest Architecture: Source Systems → Zerobus Ingest → Lakehouse
By eliminating the intermediate message bus, you remove two major cost centers: compute and storage for the bus, and the dedicated engineering time required to manage it. Zerobus Ingest provides ingest at a fraction of the cost per gigabyte compared to running and maintaining a self-managed Kafka cluster.
Zerobus Ingest provides ingest at a fraction of the cost per gigabyte compared to running and maintaining a self-managed Kafka cluster.
Learn more about how Zerobus works in Deep Dive Databricks Community Blog or in documentation.
Supported interfaces and native integration
Developers can integrate through gRPC and REST APIs, or use language-specific SDKs. Zerobus Ingest provides a comprehensive set of push-based interfaces for industry-specific integrations, making it a flexible, single tool that simplifies ingestion.
- gRPC API: Recommended for high-performance applications requiring minimal latency and highest throughput.
- Rest API (Beta): Ideal for webhooks, serverless functions, and languages where gRPC support may be limited.
- SDK: Production-ready libraries for Python, Java, Rust, Go, and TypeScript simplify authentication and batching logic using gRPC.
- Open Telemetry (Beta): Bring your operational logs, metrics, and traces into Lakehouse for long-term historical analysis with only configuration changes. Learn more about the Open Telemetry ecosystem Here.
Learn more about Difference between REST and GRPC.
Plus, since every write is governed by the Unity Catalog, you get automatic lineage tracking and fine-grained access control from the moment the data is created – ensuring that your streaming data has unified governance with the rest of your Lakehouse.
Driving customer successes: faster insights faster at scale
Real Time Manufacturing Monitoring for Toyota Motor Corporation
Toyota sought a unified solution to instantly process telemetry from thousands of factory devices, without the latency and complexity of traditional IoT architectures.
“Zerobus Ingest allows us to detect overheating factory conditions in minutes instead of hours, which directly supports our carbon-neutrality strategy and operational efficiency. But it’s not just about tracking temperature telemetry; Zerobus Ingest as an additional data ingest option gives us the ability to collect diverse factory data in real-time and trigger immediate countermeasures that have transformed our operations.” -Kento Izumi, General Manager, Digital Transformation Promotion Division, Toyota Motor Corporation
Instead of linking multiple cloud services together, Toyota uses Zerobus Ingest, which integrates with global IoT connectivity psoracomTo reduce the high maintenance costs of real-time operations, transform its manufacturing operations, and support its sustainability goals.
IoT Data Pipeline Architecture: From Edge to Analytics Platform with Zerobus Ingest and Soracom Beam
Izumi also explained that they have been able to accelerate their operational efficiency, “Combined with ‘Vista,’ our integrated data and AI platform powered by Databricks, we are not only collecting data faster; we are optimizing our data operations.”
Joby Aviation: Accelerating flight performance analysis from days to minutes
An early adopter of Zerobus Ingest, Joby Aviation streams gigabytes of aircraft telemetry every minute directly to Lakehouse, enabling their engineering teams to analyze flight performance in real-time. Read the Joby Aviation case study.
“Zerobus Ingest reduced our telemetry resolution latency from days to minutes. This allows our engineering teams to analyze flight performance in real-time and accelerate our mission to transform transportation.” -Dominic Muller, Factory Systems Lead, Joby Aviation
power industry use cases
Traditional infrastructure slows down real-time operations. By removing the complexity of intermediate message buses, Zerobus Ingest creates a direct, sub-5-second path to value across industries.
Accelerate your digital transformation by delivering data from any source across industries directly to your lakehouse.
Manufacturing: Maximize factory floor efficiency. Use the Zerobus Ingest SDK to create custom forwarding agents that stream mass sensor volumes to Lakehouse. This optimizes machine performance by eliminating heavy network infrastructure overhead.
Telecommunications and IoT: Monitor global networks at large scale. Deployed at the edge, Zerobus Ingest pipes telemetry from your network to Lakehouse to track your network load in real time. Our partnership with Soracom expands integration with secure, reliable global IoT data ingestion via cellular, satellite and LPWAN networks.
IT and cyber security: Identify threats without ETL delays. Bypass complex pipelines by streaming logs and behavioral events directly to Lakehouse. It enables threat detection within seconds, adaptive model re-training and rapid incident response.
Commerce and clickstream: Personalize experiences in real time. Capture high-volume clickstream data from apps and devices with minimal infrastructure overhead. This power personalization engine enables instant data availability for A/B testing and conversion optimization.
Availability
Zerobus Ingest is now generally available Support on AWS, Microsoft Azure, and Google Cloud platforms is coming soon. Pricing is volume-based under lakeflow jobs serverless sku.
As part of the GA launch, we are introducing a 6 month promotional pricing period. Learn more on the Lakeflow Connect pricing page.
Getting Started with Zerobus Ingest
Ready to eliminate the complexity of streaming infrastructure? With just a few lines of code, you can start streaming data directly to your Unity Catalog-governed tables, ensuring that your data is ready as it arrives to help deliver insights.
Check out the following Zerobus Ingest resources to get started today:
- Try Zerobus Ingest now: access to Documentation and quick start guides.
- Take a Product Tour: Navigate through Zerobus Ingest and learn how to start ingesting data.
- Build an end-to-end application: A real-time sailing simulator tracks a fleet of sailboats using the Python SDK and REST API with Databricks Apps and Databricks Asset Bundles. read blog.
- Create a Digital Twins Solution: Learn how to maximize operational efficiency, gain real-time insight, and accelerate predictive maintenance with Databricks Apps and Lakebase. Read the blog.
