Author : HASSAN MD TAREQ | Updated : 2021/05/11

Creating Kafka Topic

Method-1: using yaml manifest file

kafka-topic.yaml

apiVersion: kafka.strimzi.io/v1beta1
kind: KafkaTopic
metadata:
  name: my-topic
  namespace: kafka
  labels:
    strimzi.io/cluster: my-cluster
spec:
  replicas: 3
  partitions: 3
#    retention.ms: 7200000
#    segment.bytes: 1073741824

Method-2: Strimzi Github Repo

kubectl --namespace kafka apply -f https://raw.githubusercontent.com/strimzi/strimzi-kafka-operator/main/examples/topic/kafka-topic.yaml

You can also try following (topic will be created by connecting to kafka leader cluster and executing command directly):

kubectl exec -it my-cluster-kafka-0 -c kafka -- bin/kafka-topics.sh --bootstrap-server my-cluster-kafka-bootstrap:9092 --topic my-topic --create --partitions 3 --replication-factor 3

Listing Kafka Topic

List of topics

# Using 'kubectl get ...'
kubectl get kafkatopics # kubectl get kafkatopics.kafka.strimzi.io
kubectl get kafkatopic foo-topic
kubectl get kt foo-topic # kt -> kafkatopic
kubectl get kt foo-topic -o wide
kubectl get kt foo-topic -o yaml


# Using 'kubectl exec ...'
kubectl exec -it my-cluster-kafka-0 -c kafka -- bin/kafka-topics.sh --bootstrap-server my-cluster-kafka-bootstrap:9092 --list

Get details of a topic

# Using 'kubectl get ...'
kubectl get kafkatopic my-topic
kubectl get kt my-topic

# Using 'kubectl exec ...'
kubectl exec -it my-cluster-kafka-0 -c kafka -- bin/kafka-topics.sh --bootstrap-server my-cluster-kafka-bootstrap:9092 --topic my-topic --describe

Deleteing Kafka Topic

Deleteing single user

kubectl delete $(kubectl get kafkatopic -o name)

Deleteing all users

kubectl delete $(kubectl get kafkatopics -o name)

Creating Kafka User

If you are using Azure cloudshell: code foo-user.yaml

foo-user.yaml

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaUser
metadata:
  name: foo-user
  namespace: kafka
  labels:
    strimzi.io/cluster: my-cluster
spec:
  authentication:
    type: tls
  authorization:
    type: simple
    acls:
      - resource:
          type: topic
          name: my-topic
          patternType: literal
        operation: All
      - resource:
          type: topic
          name: test-topic
          patternType: literal
        operation: read

Command: kubectl apply -f foo-user.yaml --namespace kafka

Listing Kafka Users

All users

kubectl get kafkausers
kubectl get kafkausers -o name # name only
kubectl get kafkausers -o wide
kubectl get kafkausers -o yaml # output as yaml
kubectl get kafkausers -o json # output as json

Single user

kubectl get kafkauser foo-user 
kubectl get ku foo-user # kafkauser -> ku
kubectl get ku foo-user -o <output-flag>

Deleting Kafka User

Deleteing single user

kubectl delete $(kubectl get ku foo-user -o name)

Deleteing all users

kubectl delete $(kubectl get kafkausers -o name)

Consumer Group

Create Consumer Group

kafka-console-consumer.bat --bootstrap-server <LOADBALANCER_PUBLIC_IP>:9094 --topic my-topic --consumer.config client-ssl.properties --from-beginning --group cg-xyz

List consumer group

kafka-consumer-groups.bat --bootstrap-server <LOADBALANCER_PUBLIC_IP>:9094 -list --command-config client-ssl.properties
kafka-consumer-groups.bat --bootstrap-server 20.194.x.y:9094 --list --command-config client-ssl.properties

# in local -> ./bin/kafka-consumer-groups.sh --list --bootstrap-server localhost:9092

List consumer group members

kafka-consumer-groups.bat --bootstrap-server 20.194.x.y:9094 --command-config client-ssl.properties --describe --group new-user --members

# local -> ./bin/kafka-consumer-groups.sh --describe --group new-user --members --bootstrap-server localhost:9092

Details: