提问说明

kafka生产消息到消费整个流程如下：

1.生产者生产消息到缓冲区时间：1572228409758 ms
2.从Event事件回调得到生产者消息成功发送到broker的整个网络时延：2ms

LOG-7-RECV: [thrd:IP:9292/bootstrap]: IP:9292/1: Received ProduceResponse (v7, 53 bytes, CorrId 2, rtt 2.48ms)

3.kafka消费者消费消息时间：1572228410256 ms
整个过程中总耗时： 498ms

kafka集群配置如下:

broker.id=1
port=9292
host.name=xxx
listeners=PLAINTEXT://xxx:9292
advertised.listeners=PLAINTEXT://xxx:9292

auto.create.topics.enable = false
auto.leader.rebalance.enable=true

# Replication configurations
num.replica.fetchers=4
replica.fetch.max.bytes=1048576
replica.fetch.wait.max.ms=500
replica.high.watermark.checkpoint.interval.ms=5000
replica.socket.timeout.ms=30000
replica.socket.receive.buffer.bytes=65536
replica.lag.time.max.ms=10000

# Log configuration
num.partitions=8
num.recovery.threads.per.data.dir=1
message.max.bytes=1000000
log.index.interval.bytes=4096
log.index.size.max.bytes=10485760
log.retention.hours=24
log.retention.bytes=5000000000
log.retention.check.interval.ms=300000
log.flush.interval.ms=10000
log.flush.interval.messages=20000
log.flush.scheduler.interval.ms=2000
log.roll.hours=6
log.segment.bytes=2000000000
log.dirs=/data/data1/svckafka/logdata,/data/data2/svckafka/logdata,/data/data3/svckafka/logdata,/data/data4/svckafka/logdata,/data/data5/svckafka/logdata,/data/data6/svckafka/logdata


# ZK configuration
zookeeper.connect=xxx:2181,xxx:2181,xxx:2181,xxx:2181,xxx:2181
zookeeper.connection.timeout.ms=24000
zookeeper.sync.time.ms=2000

# Socket server configuration
num.io.threads=12
num.network.threads=20

socket.receive.buffer.bytes=1048576
socket.send.buffer.bytes=1048576

kafka生产者配置如下：

statistics.interval.ms ：5000
queue.buffering.max.ms ：500
debug : msg,protocol
request.required.acks : 1

kafka消费者配置如下:

statistics.interval.ms ： 5000
debug : consumer,security,broker
enable.partition.eof : true
max.partition.fetch.bytes : 10240000
auto.offset.reset : latest

预览