Kafka

Enabling everyone to run Apache Kafka on Kubernetes is an important part of our mission to put a streaming platform at the heart of every company. This is why we look forward to releasing an implementation of the Kubernetes Operator API for automated provisioning, management, and operations of Kafka on Kubernetes.

Kafka for Kubernetes

This community seeks to provide:

• Production-worthy Kafka setup for persistent (domain- and ops-) data at small scale.
• Operational knowledge, biased towards resilience over throughput, as Kubernetes manifest.
• A platform for event-driven (streaming!) microservices design using Kubernetes.

Pre Requisites:

• Kubernetes 1.3 with alpha APIs enabled and support for storage classes
• PV support on underlying infrastructure
• Requires at least v2.0.0-beta.1 version of the helm to support dependency management with requirements.yaml

Chart Details

• This chart will do the following:
• Implement a dynamically scalable Kafka cluster using Kubernetes StatefulSets
• Implement a dynamically scalable zookeeper cluster as another Kubernetes StatefulSet required for the Kafka cluster above
• Expose Kafka protocol endpoints via NodePort services (optional)

Extensions

Kafka has a rich ecosystem, with lots of tools. This section is intended to compile all of those tools for which a corresponding Helm chart has already been created.

Schema-registry - A confluent project that provides a serving layer for your metadata. It provides a RESTful interface for storing and retrieving Avro schemas.

Known Limitations

• Only supports storage options that have backends for persistent volume claims (tested mostly on AWS)
• KAFKA_PORT will be created as an envvar and brokers will fail to start when there is a service named Kafka in the same namespace. We work around this be unsetting that envvar unset KAFKA_PORT.