Author : MD TAREQ HASSAN | Updated : 2021/06/29
Prerequisites
In case you want to delete already running kafka and start fresh:
Max Message Size
Broker level or Topic level (do either one, topic level is preferred)
- Broker level:
message.max.bytes- to increase the message size on broker level
- When running in AKS:
config:section - When running in local:
config/server.properties
- Topic level:
message.max.bytes- If only one topic should be able to accept lager files
- The broker configuration must not be changed
- If topic level setting does not work, try using
replica.fetch.max.bytessetting in broker also with topic level setting
Broker configuration: strimzi-kafka-deployment.yaml
# ... ... ...
config:
message.max.bytes: n # i.e. 1024 * 1024 * 10 (10 MB), message size allowed by broker
replica.fetch.max.bytes: n # i.e. 1024 * 1024 * 10 (10 MB), for replicatioin inside cluster
# ... ... ...
Topic configuration: my-topic.yaml
apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaTopic
metadata:
name: my-topic
namespace: kafka
labels:
strimzi.io/cluster: my-cluster
spec:
partitions: 2
replicas: 2
max.message.bytes: n # i.e. 1024 * 1024 * 10 (10 MB)
Producer configuration
- Configuration file
- Kafka console producer:
producer.properties - Kafka C# cleint:
Program.cs
- Kafka console producer:
max.request.size:- The maximum size of a request in bytes: (number of messages in a request) * (size of a single message)
- https://kafka.apache.org/documentation/#producerconfigs_max.request.size
Program.cs
// ... ... ...
const string KEY_REQUEST_SIZE = "max.request.size";
var configProperties = new Dictionary<string, string>
{
[KEY_REQUEST_SIZE] = 1024 * 1024 * 10 * n, // n messages per request where each message is 10 MB
};
var producerConfig = new ProducerConfig(configProperties);
// ... ... ...
Consumer configuration
- Kafka console consumer:
consumer.properties - Kafka C# cleint:
Program.cs
Program.cs
// ... ... ...
var brokerList = "10.10.0.75"; // external bootstrap (Strimzi kafka running in AKS)
var consumerConfig = new ConsumerConfig
{
BootstrapServers = brokerList,
GroupId = "my-topic-group",
MaxPartitionFetchBytes = 1024 * 1024 * 10,
FetchMaxBytes = 1024 * 1024 * 10, // * n (if required)
// ... ... ...
};
// ... ... ...
See: https://stackoverflow.com/a/67721976/4802664
Network buffer size
# ... ... ...
config:
replica.socket.receive.buffer.bytes=65536
socket.request.max.bytes=104857600
# default values of the buffers for sending and receiving messages might be too small for the required throughput
socket.send.buffer.bytes=1048576
socket.receive.buffer.bytes=1048576
# ... ... ...
https://strimzi.io/blog/2021/06/08/broker-tuning/