TL;DR
Over the past few years, Kubernetes has become the default platform for deploying modern applications. Microservices, APIs, AI workloads, CI/CD pipelines, and supporting infrastructure are increasingly managed through Kubernetes, giving platform teams a consistent way to deploy, scale, monitor, and automate workloads across cloud environments. As organizations invest heavily in Kubernetes, it's only natural to ask the next question: Should Kafka run there too?
On paper, the answer seems obvious. Running Kafka on Kubernetes promises infrastructure consistency, automated deployments, declarative configuration, rolling upgrades, and a single operational model for the entire application stack. For engineering teams already operating Kubernetes at scale, consolidating Kafka onto the same platform appears to reduce complexity rather than introduce it.
However, Kafka is fundamentally different from the stateless applications Kubernetes was designed to orchestrate. Every broker maintains persistent state, relies on stable storage, participates in replica synchronization, and forms part of a distributed consensus system where broker identity and network stability directly influence cluster health. Kubernetes, by contrast, assumes workloads are disposable. Pods are expected to restart, move between nodes, and recover automatically without preserving local state. While Kubernetes provides mechanisms such as StatefulSets and Persistent Volumes to support stateful applications, these abstractions do not eliminate the operational challenges associated with running distributed storage systems like Kafka.
This doesn't mean Kafka should never run on Kubernetes. Many organizations successfully operate Kafka on Kubernetes in development environments, internal platforms, and even production. The challenge is understanding where Kubernetes aligns with Kafka's architecture, where it introduces operational friction, and whether the trade-offs are justified for your workload.
In this article, we'll examine the architectural realities of running Kafka on Kubernetes, explore the operational challenges that emerge as clusters scale, and discuss when a Bring Your Own Cloud (BYOC) managed Kafka platform such as Condense offers a more practical alternative by delivering cloud-native deployment without requiring platform teams to manage Kafka as another stateful Kubernetes workload.
Why Kafka on Kubernetes Seems Appealing
The growing adoption of Kubernetes has encouraged organizations to standardize their entire technology stack around a single orchestration platform. Once stateless applications, APIs, databases, and supporting services are running successfully on Kubernetes, extending the same operational model to Kafka appears to be a logical next step. From an infrastructure perspective, it promises consistency, automation, and a familiar deployment experience for platform teams.
One of the biggest advantages is operational consistency. Instead of maintaining separate deployment pipelines and infrastructure management practices for Kafka, teams can provision brokers using the same Infrastructure as Code (IaC) workflows, GitOps pipelines, and Kubernetes manifests that already manage the rest of their applications. Monitoring, logging, RBAC, secrets management, and policy enforcement can also be integrated into the existing Kubernetes ecosystem, reducing the number of operational tools engineers need to manage.
Organizations that have already invested heavily in Kubernetes often see additional benefits. Existing cluster capacity can be utilized more efficiently, infrastructure teams require fewer platform-specific skills, and automation tools such as Helm, Argo CD, or Flux can be reused to manage Kafka deployments alongside application workloads. This creates a consistent operational model that simplifies provisioning and standardizes deployment practices across development, staging, and production environments.
Kubernetes also provides capabilities that are attractive for running distributed systems, including self-healing, declarative configuration, rolling updates, resource scheduling, and automated recovery from infrastructure failures. Combined with Kafka Operators such as Strimzi, these features make it possible to deploy and manage Kafka clusters with significantly less manual effort than traditional VM-based installations.
At first glance, this appears to solve many of the operational challenges associated with running Kafka. However, these advantages primarily address how Kafka is deployed, not how Kafka behaves as a distributed stateful system. As clusters grow and workloads become more demanding, the architectural differences between Kubernetes and Kafka begin to surface, introducing complexities that aren't immediately obvious during initial deployments.
Understanding these differences is essential before deciding whether Kubernetes is the right platform for production Kafka workloads.
The Stateful Workload Problem: What Makes Kafka Different from Stateless Applications
To understand why Kafka becomes challenging to operate on Kubernetes, it's important to understand how Kafka differs from the majority of applications Kubernetes was originally designed to orchestrate.
A stateless application treats every instance as interchangeable. If a pod crashes, Kubernetes simply schedules another one, attaches it to a Service, and traffic resumes with little or no impact. Since application state is typically stored in an external database or cache, replacing a pod does not affect the application's consistency or availability.
Kafka operates very differently.
Each Kafka broker is responsible for storing partitions on local disk, replicating data to peer brokers, participating in leader elections, maintaining replica synchronization, and serving client requests. Every broker has a unique identity within the cluster, and each partition replica is tightly coupled to the broker that hosts it. While client applications connect through bootstrap servers, the Kafka cluster internally relies on stable broker identities and predictable network communication to coordinate replication and leadership across brokers.
This architectural difference means that replacing a Kafka broker is fundamentally different from replacing a stateless application pod. When a broker becomes unavailable, partition leaders may be re-elected, followers must catch up with the latest offsets, clients reconnect to new leaders, and the cluster begins working to restore replication health. Depending on the workload and replication strategy, this process can temporarily increase network traffic, disk I/O, and request latency until the cluster stabilizes.
Kubernetes provides StatefulSets, Persistent Volumes, stable DNS names, and ordered pod management to support stateful applications. These capabilities significantly improve Kafka deployments compared to standard Deployments, but they don't eliminate the operational characteristics of a distributed log system. Broker identity, storage performance, replica synchronization, and network stability remain critical factors that Kubernetes cannot abstract away.
This distinction is where many production deployments encounter unexpected complexity. Kubernetes is highly effective at orchestrating containers, but Kafka requires careful coordination of storage, networking, replication, and broker lifecycle events that extend beyond container orchestration. As clusters grow and workloads become more demanding, these differences become increasingly significant and influence everything from storage performance to broker recovery, partition rebalancing, and maintenance operations.
PVCs and Storage: Broker Disk I/O on Kubernetes vs. Bare Metal
Storage is the foundation of Kafka's architecture. Unlike traditional messaging systems that primarily rely on memory, Kafka persists every message to disk and serves reads directly from its commit log. Producer throughput, consumer performance, replica synchronization, and broker recovery all depend heavily on storage latency and sustained disk throughput. As a result, the underlying storage subsystem has a direct impact on the stability and performance of the entire Kafka cluster.
On bare metal or dedicated virtual machines, Kafka brokers typically have exclusive access to local SSDs or high-performance block storage. This provides predictable latency, consistent I/O performance, and direct control over disk utilization. Because the storage remains attached to the broker throughout its lifecycle, replication, log compaction, segment flushing, and recovery operations can execute without additional storage orchestration.
Running Kafka on Kubernetes introduces another layer between the broker and its storage. Each broker typically relies on a Persistent Volume Claim (PVC) backed by a Persistent Volume provisioned through a Kubernetes StorageClass. While this abstraction enables persistent storage across pod restarts, the actual performance characteristics depend entirely on the underlying storage implementation. Network-attached block storage, shared storage systems, or dynamically provisioned volumes may introduce higher latency, variable throughput, or IOPS limits that are not immediately apparent during development or testing.
Storage also influences broker recovery. When a broker is rescheduled, Kubernetes ensures that the associated Persistent Volume is reattached before the broker starts serving traffic. Depending on the storage backend and cloud provider, this attachment process can take time, extending broker recovery and delaying replica synchronization. During this period, partition leaders may shift to other brokers, followers continue catching up, and the cluster remains in a partially degraded state until replication is fully restored.
This does not mean Kubernetes storage is unsuitable for Kafka. High-performance CSI drivers, local Persistent Volumes, and carefully selected storage classes can deliver excellent results. However, achieving predictable performance requires a deep understanding of both Kafka's storage access patterns and the capabilities of the underlying storage platform. Simply provisioning a Persistent Volume does not guarantee the low-latency, high-throughput disk performance that production Kafka clusters demand.
For organizations operating large-scale streaming workloads, storage planning should be treated as a core architectural decision rather than an infrastructure detail. Selecting the appropriate storage backend, validating I/O performance under production workloads, and understanding recovery characteristics are just as important as sizing brokers or configuring replication factors.
Pod Disruption Budgets: What Really Happens During Node Drains
Routine Kubernetes maintenance is built around the assumption that workloads can be safely moved between nodes. Whether it's a Kubernetes version upgrade, security patch, infrastructure maintenance, or cluster autoscaling event, nodes are regularly drained to allow pods to be rescheduled elsewhere. For stateless applications, this process is usually uneventful. Pods restart on another node, reconnect to the Service, and continue serving traffic.
Kafka brokers, however, respond very differently.
When a node hosting a Kafka broker is drained, Kubernetes gracefully terminates the broker pod and schedules a replacement. Although StatefulSets preserve the broker's identity and the associated Persistent Volume is reattached to the new pod, the broker is still unavailable until it restarts, recovers its log segments, rejoins the cluster, and synchronizes with its replicas.
During this period, Kafka begins reconfiguring itself. Partition leadership may move to other brokers, followers continue replicating data, and clients reconnect to new leaders. If multiple brokers are restarted simultaneously or maintenance is not carefully coordinated, the cluster may experience under-replicated partitions, increased replication traffic, higher request latency, and, in severe cases, temporary unavailability for partitions that cannot maintain quorum.
This is where Pod Disruption Budgets (PDBs) become critical. A PDB limits the number of broker pods that Kubernetes can voluntarily disrupt at the same time, helping ensure that sufficient brokers remain available during maintenance operations. While a properly configured PDB reduces the risk of multiple broker disruptions, it does not eliminate the need for careful operational planning. Kubernetes enforces the disruption policy, but Kafka is still responsible for leader elections, replica recovery, and restoring cluster health after each broker restart.
Production Kafka clusters therefore require maintenance procedures that account for both Kubernetes and Kafka. Node drains should be performed incrementally, cluster health should be monitored throughout the process, and maintenance should pause until partition leadership stabilizes and under-replicated partitions return to normal. Treating broker maintenance as a standard Kubernetes rolling update can introduce unnecessary operational risk, particularly in clusters handling high-throughput or latency-sensitive workloads.
The key takeaway is that Kubernetes manages the lifecycle of the broker pod, while Kafka manages the lifecycle of the cluster. Understanding the difference is essential for performing safe maintenance without compromising availability.
Broker Rebalancing Triggered by Kubernetes Rescheduling
Every Kafka broker contributes compute, storage, and network capacity to the cluster. When a broker becomes unavailable, Kafka immediately begins adapting to maintain availability. Partition leaders hosted on the unavailable broker are elected on in-sync replicas (ISRs), producers and consumers reconnect to the new leaders, and replication traffic increases as the cluster works to maintain redundancy.
In a Kubernetes environment, broker rescheduling can occur for reasons beyond broker failures. Node upgrades, autoscaling events, infrastructure maintenance, or pod evictions may temporarily remove brokers from the cluster even though the underlying infrastructure remains healthy. While Kubernetes successfully restarts the broker, Kafka still needs time to restore the cluster to a fully healthy state.
When the broker comes back online, it doesn't instantly resume normal operation. Replica partitions must catch up with the latest log offsets before they can rejoin the ISR. Until synchronization is complete, the cluster may continue operating with under-replicated partitions, elevated replication traffic, and reduced fault tolerance. During this recovery period, client requests may experience higher latency as brokers simultaneously serve application traffic and replicate data.
Over time, repeated broker movements can also create an uneven distribution of partition leadership and workload across the cluster. Some brokers may end up serving significantly more leader partitions or handling disproportionate client traffic than others. While Kafka continues to function, this imbalance can reduce overall throughput and lead to uneven resource utilization.
Correcting this imbalance often requires explicit administrative operations such as preferred leader election, partition reassignment, or automated balancing tools like Cruise Control. These activities consume additional network bandwidth and disk I/O, making them operational tasks that must be planned rather than performed continuously.
Kubernetes successfully restores the broker pod, but it has no awareness of Kafka's internal partition placement, replica synchronization, or leadership distribution. These responsibilities remain entirely within Kafka itself. As cluster size and event throughput increase, coordinating Kubernetes operations with Kafka's recovery mechanisms becomes increasingly important to maintain predictable performance and high availability.
Networking Complexity: Advertised Listeners and Pod IP Changes
Networking is one of the most overlooked aspects of running Kafka on Kubernetes. Unlike many distributed applications that communicate through a single service endpoint, Kafka clients connect to a bootstrap server only for initial cluster discovery. Once connected, the broker returns metadata containing the address of every broker in the cluster through its configured advertised.listeners. Producers and consumers then establish direct connections to the appropriate broker responsible for the partition they need to access.
This architecture works well when broker addresses remain stable. In traditional VM or bare metal deployments, brokers typically have fixed hostnames or static IP addresses that rarely change, making client connectivity predictable throughout the broker's lifecycle.
Kubernetes introduces a different networking model. Pods are ephemeral, and their IP addresses change whenever they are recreated or rescheduled. While StatefulSets provide stable pod names and Kubernetes offers stable DNS entries for each broker, Kafka must still advertise addresses that are reachable by every client connecting to the cluster. This becomes increasingly complex when applications connect from outside the Kubernetes cluster, across multiple Kubernetes clusters, or from hybrid cloud and on-premises environments.
Misconfigured advertised.listeners are among the most common causes of Kafka connectivity issues on Kubernetes. A client may successfully connect to the bootstrap service, only to receive broker addresses that are inaccessible from its network. The result is often confusing connection failures where the cluster appears healthy, but producers and consumers repeatedly fail to establish connections with individual brokers.
Supporting multiple client networks further increases the complexity. Internal applications may require one set of broker addresses, while external clients need another. Kubernetes Services, Ingress controllers, LoadBalancers, DNS configuration, and TLS certificates must all be coordinated with Kafka's listener configuration to ensure that every client receives an address it can actually reach.
Kafka Operators such as Strimzi simplify much of this configuration by automatically managing listeners, services, certificates, and broker discovery. However, operators cannot eliminate the underlying networking requirements. Engineering teams must still design a networking architecture that accounts for client locations, DNS resolution, load balancing, security policies, and cross-network connectivity.
For organizations operating Kafka across multiple environments or serving large numbers of distributed clients, networking often becomes one of the most operationally demanding aspects of running Kafka on Kubernetes. It requires expertise in both Kubernetes networking and Kafka's broker discovery model, making it far more complex than deploying stateless services behind a single Kubernetes Service.
The Operator Problem: Strimzi, KRaft, and the Limits of Automation
One of the strongest arguments for running Kafka on Kubernetes is the availability of Kubernetes Operators. Projects such as Strimzi automate many operational tasks, including broker provisioning, rolling upgrades, certificate management, listener configuration, and cluster lifecycle management. Instead of manually configuring brokers or maintaining deployment scripts, platform teams can define the desired cluster state declaratively, allowing the operator to reconcile the running cluster automatically.
This level of automation significantly improves the deployment experience, but it is important to understand what an operator actually does. A Kafka Operator automates cluster management, not distributed systems management. It can create brokers, restart pods, rotate certificates, and apply configuration changes, but it cannot eliminate the operational characteristics of Kafka itself.
For example, if a broker becomes unavailable, the operator can restart the pod, but it cannot instantly restore partition replicas, rebalance leadership, reduce replication traffic, or recover client throughput. Those responsibilities remain part of Kafka's own replication and consensus mechanisms. Similarly, if storage performance degrades, partitions become imbalanced, or network latency increases, the operator can observe these conditions but cannot solve the underlying architectural limitations.
The introduction of KRaft (Kafka Raft Metadata Mode) simplifies Kafka's internal architecture by removing the dependency on ZooKeeper and replacing it with a built-in metadata quorum. This reduces operational overhead and streamlines cluster management, particularly for new deployments. However, KRaft does not change how Kafka interacts with Kubernetes. Brokers still require stable storage, predictable networking, replica synchronization, and carefully coordinated maintenance. The operational challenges associated with running a stateful distributed log on Kubernetes remain largely unchanged.
Some organizations build custom operators to automate workflows specific to their environment, integrating Kafka management with internal tooling, GitOps pipelines, or platform engineering frameworks. While this provides greater flexibility, it also shifts responsibility for maintaining and evolving the operator to the engineering team. Every Kubernetes upgrade, Kafka release, API deprecation, or infrastructure change may require updates to the operator itself, introducing another operational component that must be maintained alongside the Kafka cluster.
Operators are therefore best viewed as an automation layer rather than a complete operational solution. They simplify deployment, configuration, and lifecycle management, but they do not remove the need to understand Kafka's storage model, replication behavior, networking requirements, or performance characteristics. As production clusters grow, successful operations depend as much on architectural decisions as they do on automation.
The question is no longer whether Kafka can run on Kubernetes. It clearly can. The more important question is whether your engineering team wants to operate Kafka as another complex Kubernetes workload or consume it as a managed streaming platform that abstracts away much of this operational burden.
When Kafka on Kubernetes Actually Makes Sense
Despite the operational challenges discussed so far, Kafka on Kubernetes is far from a bad architectural choice. In fact, many organizations successfully run Kafka on Kubernetes every day. The key is understanding the scale, operational maturity, and workload characteristics for which Kubernetes is best suited.
Kafka on Kubernetes is often an excellent fit for development, testing, and staging environments. Engineering teams can quickly provision clusters, validate application changes, test client integrations, and tear down environments using the same Kubernetes workflows already adopted across the organization. The ability to automate deployments through GitOps pipelines, Helm charts, or Operators significantly improves developer productivity while maintaining consistency across environments.
It is also well suited for small production clusters with predictable workloads and experienced platform teams. Organizations that already operate Kubernetes as their primary infrastructure platform and have established practices for storage management, networking, monitoring, and cluster maintenance can successfully manage Kafka alongside their application workloads. In these cases, the operational overhead remains manageable because the platform team already possesses the expertise required to support stateful services on Kubernetes.
The equation begins to change as workloads become larger and more business critical. Clusters processing billions of events per day, supporting hundreds of client applications, or requiring strict availability objectives demand continuous attention to broker health, storage performance, partition leadership, capacity planning, replication, and recovery operations. At this scale, engineering teams often find themselves spending more time operating Kafka than building the streaming applications that depend on it.
The decision, therefore, isn't whether Kafka can run on Kubernetes. It is whether Kubernetes should become the operational layer for one of the most critical components in the data platform. For some organizations, the answer is yes. For others, particularly those prioritizing operational simplicity, predictable scalability, and faster delivery, separating Kafka operations from Kubernetes operations can provide a more sustainable long-term approach.
The Alternative: Bring Your Own Cloud (BYOC) Managed Kafka
For many organizations, the objective isn't to run Kafka on Kubernetes. The objective is to build a reliable, scalable, cloud-native streaming platform with minimal operational overhead. Kubernetes is often chosen because it provides automation, consistency, and infrastructure standardization, but those benefits don't necessarily require operating Kafka as another stateful Kubernetes workload.
A Bring Your Own Cloud (BYOC) Managed Kafka approach offers a different operating model. Instead of deploying Kafka into a vendor-managed environment, the platform is deployed directly within the customer's own AWS, Microsoft Azure, or Google Cloud account. This allows organizations to retain complete ownership of their infrastructure, networking, security policies, and data while offloading the operational complexity of managing Kafka clusters.
This model enables platform teams to preserve many of the benefits that originally attracted them to Kubernetes, including cloud-native deployment, infrastructure control, compliance, and data sovereignty, without taking responsibility for broker lifecycle management, storage optimization, cluster upgrades, partition balancing, or day-to-day operational maintenance.
Condense follows this architecture through its Fully Managed BYOC deployment model. Kafka clusters are provisioned and managed within the customer's cloud environment, while the platform automates cluster provisioning, upgrades, monitoring, scaling, and operational management. Beyond managed Kafka, Condense provides an integrated streaming platform with built-in observability, pipeline orchestration, prebuilt connectors, and a Custom Transform Framework for building real-time data pipelines without assembling multiple independent tools.
This approach allows engineering teams to continue building cloud-native applications while reducing the operational burden of running Kafka as a stateful Kubernetes workload. Instead of spending engineering effort maintaining brokers, troubleshooting storage issues, coordinating upgrades, or managing cluster operations, teams can focus on developing event-driven applications and delivering business value through real-time data.
For organizations that have already standardized on Kubernetes, this doesn't mean replacing Kubernetes. Stateless applications, APIs, and microservices can continue to run on Kubernetes, while Kafka is consumed as a fully managed streaming platform deployed within the same cloud environment. The result is a simpler operational model that preserves cloud-native architecture without inheriting the full complexity of operating Kafka on Kubernetes.
Related Reading:
Why Managed Kafka Is Not Enough for a Complete Streaming Data Platform
What Is Bring Your Own Cloud (BYOC) and Why Is It Important?
Building Cloud-Native, BYOC-Compatible Real-Time Data Pipelines in Minutes with Condense
Conclusion
Running Kafka on Kubernetes is neither inherently right nor inherently wrong. Kubernetes has matured into an exceptional platform for orchestrating containerized applications, and with StatefulSets, Persistent Volumes, and Operators, it provides the building blocks required to deploy stateful systems like Kafka. Many organizations successfully run Kafka on Kubernetes, particularly in development environments, internal platforms, and smaller production deployments.
The challenge emerges as operational complexity grows. Storage performance, broker recovery, networking, partition leadership, maintenance operations, and cluster balancing remain Kafka-specific concerns that Kubernetes cannot abstract away. While Kubernetes automates container orchestration, engineering teams are still responsible for operating a distributed event streaming platform whose availability depends on careful coordination of storage, networking, and replication.
Choosing the right deployment model ultimately comes down to where your engineering effort creates the most value. If operating Kafka is a core platform competency and your team is equipped to manage its day-to-day lifecycle, Kubernetes can be an effective deployment platform. However, if the priority is building real-time applications rather than managing broker operations, a Fully Managed BYOC platform offers a simpler path by preserving cloud-native deployment while significantly reducing operational overhead.
Condense is built around this philosophy. By delivering Fully Managed Kafka within the customer's own cloud environment, combined with built-in observability, pipeline orchestration, prebuilt connectors, and enterprise-grade operational management, it enables organizations to focus on developing event-driven applications instead of operating Kafka infrastructure.
Kafka on Kubernetes is ultimately an architectural decision, not a trend to follow. The best choice is the one that aligns with your operational maturity, scalability requirements, and long-term platform strategy.
