What’s New in Apache Kafka 4.0
Written by
Sugam Sharma
.
Co-Founder & CIO
Published on
Mar 10, 2025
Apache Kafka 4.0 is a major milestone in the evolution of the platform, bringing significant architectural changes, performance improvements, and new features.
Below is a detailed breakdown of the key updates in Kafka 4.0:
ZooKeeper is Removed – KRaft is Now Default

What changed?
Kafka 4.0 completely removes ZooKeeper and adopts KRaft (Kafka Raft Metadata Mode) as the default and only metadata management system.
Why it matters
KRaft simplifies deployment, reduces operational overhead, and enhances scalability. It eliminates the need for a separate ZooKeeper ensemble, making Kafka clusters easier to manage.
Impact
Existing ZooKeeper-based clusters must migrate to KRaft before upgrading to Kafka 4.0
Performance Enhancements
Controller Performance
Kafka 4.0 improves controller performance by optimizing metadata replication and reducing failover times.
Consumer Rebalance (KIP-848)
A new consumer group protocol significantly improves rebalance performance, reducing downtime and improving responsiveness in large-scale deployments.
Transaction Improvements (KIP-890)
Kafka 4.0 enhances transaction handling, reducing the risk of "zombie transactions" during producer failures.
New Queues for Kafka (KIP-932)
What changed?
Kafka introduces share groups, enabling traditional queue semantics for point-to-point messaging.
Why it matters
This allows Kafka to be used for queue-based workloads, making it more versatile for applications that require message ordering and load balancing.
Java and API Updates
Java 17 Requirement
Kafka Brokers, Connect, and Tools now require Java 17 (Kafka Clients and Streams require Java 11).
Deprecated APIs Removed (KIP-896)
Kafka 4.0 removes APIs deprecated for over 12 months, simplifying the platform and encouraging adoption of newer features.
Log4j2 Migration (KIP-653)
Kafka transitions from Log4j1 to Log4j2, improving security and logging capabilities.
Security and Resilience
Pre-Vote Mechanism (KIP-996)
Kafka introduces a pre-vote mechanism to reduce unnecessary leader elections, improving cluster stability.
Eligible Leader Replicas (KIP-966)
Kafka 4.0 introduces Eligible Leader Replicas (ELR), ensuring safer leader elections and preventing data loss.
Observability and Metrics
Client Metrics (KIP-1076)
Kafka now collects client metrics directly from brokers, improving observability for client behavior.
KRaft Performance Metrics (KIP-938)
New metrics help measure KRaft-mode performance, including ActiveControllersCount
and TimedOutBrokerHeartbeatCount
Kafka Streams Enhancements
Foreign Key Extraction (KIP-1104)
Kafka Streams now allows foreign keys to be extracted from both record keys and values, simplifying joins.
Custom Processor Wrapping (KIP-1112)
Developers can now wrap processors to apply cross-cutting logic, reducing redundancy in stream processing applications.
Kafka Connect Improvements
Jakarta EE Upgrade (KIP-1032)
Kafka Connect now uses Jakarta EE and JavaEE 10 APIs, aligning with modern standards.
Replication of Internal Topics (KIP-1074)
Kafka Connect now allows replication of user-defined internal topics, improving flexibility in mirroring setups.
Migration Path
For ZooKeeper Users
Upgrade to Kafka 3.9 (which still supports ZooKeeper)
Migrate to KRaft mode using tools provided in Kafka 3.x
Finally, upgrade to Kafka 4.0
For Older Clusters
Additional steps may be required, such as upgrading ZooKeeper to a compatible version before migrating
Apache Kafka 4.0 introduces long-anticipated architectural simplifications, performance optimizations, and security enhancements. While these changes mark a significant evolution for the Kafka ecosystem, Condense has not only incorporated these capabilities early but has extended them with domain-specific intelligence to deliver immediate, real-world value across industries.
Condense has been built on a modern streaming-first architecture from day one. Kafka 4.0’s capabilities are already available and production-hardened within the Condense platform. More importantly, Condense wraps core Kafka capabilities with a vertical-aware layer that includes event modeling, transformation logic, anomaly detection, and automation hooks.
Kafka 4.0.0 Compatibility | Condense Implementation |
---|---|
KRaft architecture | Fully adopted; no external ZooKeeper dependency since inception |
Controller scalability | Enhanced with context-aware broker routing for vertical-specific topologies |
Share groups | Unified with stream transformations, allowing per-device, per-entity message targeting |
TLS, authentication | Enforced at both stream and tenant levels; integrates with enterprise IAM |
Metrics and observability | Augmented with custom pipeline dashboards, anomaly detection, and latency tracing |
Stream joins and transformations | Delivered through Condense low-code transformation, In Built IDE, with domain-specific templates |
Industry-Centric Enhancements: Vertical Intelligence
Kafka offers scalable data transport. Condense goes beyond by embedding vertical intelligence into the stream pipeline. This allows customers to extract business value without building custom logic for every industry.
Key Enhancements Include
Domain-specific event definitions (e.g., asset idling, route deviation, process state changes)
Context-aware transformation engines
Geospatial intelligence & mapping optimizations
Condition-based triggers and workflows
Integrated schema registry with vertical models
Real-World Example: Enhanced Kafka for Mobility

Kafka 4.0 introduces support for share groups, enabling point-to-point communication — useful for mobility use cases such as per-vehicle command delivery. Condense takes this further by integrating location intelligence, fleet rules, and localized behavior into the pipeline itself.
How Condense Adds Value
Feature | Kafka 4.0 | Condense Intelligence Layer |
---|---|---|
Share groups | Yes | Used to route events per vehicle, enriched with vehicle context |
Reverse geocoding | N/A | Optimized using Plus Code caching to reduce API calls |
Geofencing | External logic needed | Built-in support for polygonal geofences with real-time alerts |
Localization | Out of scope | Auto-translated ETAs and alerts based on recipient region |
Rule engine | External | Integrated stream rule engine with UI-driven workflows |
These enhancements mean mobility operators using Condense can automate rerouting, trigger alerts on abnormal behavior, and localize communication — all in real time, and without building these features from scratch.
Applicable Across Verticals

Condense enhanced Kafka core and intelligent data streaming are applicable across sectors
Manufacturing
Detect anomalies in equipment behavior, trigger predictive maintenance.
Logistics
Track multi-modal shipments, adjust ETAs based on real-time constraints.
Energy
Stream grid telemetry, initiate automated controls during fluctuations.
Finance
Detect fraud patterns, ensure compliance through event auditing.
In every case, Kafka’s backbone is leveraged — but Condense provides the intelligence layer that makes it actionable.
Apache Kafka 4.0 represents a maturing of the open-source stream processing ecosystem. But organizations adopting Kafka still face a substantial integration and development burden to achieve production-ready, industry-specific solutions.
Condense eliminates this gap. Kafka 4.0’s features are not only available in Condense — they’re operationalized, extended, and contextualized for vertical use cases.
Whether you’re modernizing a fleet operation, digitizing a factory floor, or streamlining supply chains, Condense offers a real-time data platform that is Kafka-native and industry-smart.
Ready to Move Faster with Kafka-Grade Intelligence?
Learn how Condense can accelerate your real-time strategy — Contact us for a live demo or consultation.