返回到文章

采纳

编辑于 2年前

kafka __consumer_offsets数据被删除如何解决?

kafka

咨询下,kafka broker宕机时间比较久,恢复后为了快速同步清理了该节点的data数据,__consumer_offsets数据也被删掉了,应用侧倒是消费都正常,日志也没什么报错,但感觉是有隐患的。

所有broker data下都没有__consumer_offsets数据了。

zk里/brokers/topics/__consumer_offsets节点是有的下边就没有子节点了。

auto.create.topics.enable是关闭的,所以没有再次创建__consumer_offsets

消费offset topic描述:

$ bin/kafka-topics.sh --bootstrap-server localhost:9092 --topic __consumer_offsets --describe

Topic:__consumer_offsets    PartitionCount:50    ReplicationFactor:3    Configs:segment.bytes=104857600,cleanup.policy=compact,compression.type=producer    MarkedForDeletion:true
Topic:__consumer_offsets    Partition: 0    Leader: none    Replicas: 7,2,3    Isr:
Topic:__consumer_offsets    Partition: 1    Leader: none    Replicas: 8,3,4    Isr:

重新检查了下11个节点data下都没有__consumer_offsets了,这个集群没怎么做过优化没改过特别的配置:

log.dirs=***
num.network.threads=24
num.io.threads=48
socket.send.buffer.bytes=10485760
socket.receive.buffer.bytes=10485760
socket.request.max.bytes=104857600
num.partitions=1
log.retention.hours=240
log.segment.bytes=1073741824
log.retention.check.interval.ms=60000
log.cleaner.enable=false
message.max.bytes =104857600
num.replica.fetchers=8
auto.create.topics.enable=false
zookeeper.session.timeout.ms=120000
auto.leader.rebalance.enable=false
replica.fetch.max.bytes=104857600
replica.fetch.min.bytes=1
controlled.shutdown.enable=true
controlled.shutdown.max.retries=3
controlled.shutdown.retry.backoff.ms=5000
zookeeper.connect=***
zookeeper.connection.timeout.ms=60000
start_log_dir=***
num.recovery.threads.per.data.dir=16
replica.lag.time.max.ms=60000
delete.topic.enable=true

zk数据:

/brokers/topics/__consumer_offsets
{"version":1,"partitions":{"45":[8,7,9],"34":[8,6,7],"12":[8,4,5],"8":[4,10,11],"19":[4,11,1],"23":[8,5,6],"4":[11,6,7],"40":[3,1,2],"15":[11,7,8],"11":[7,3,4],"9":[5,11,1],"44":[7,6,8],"33":[7,5,6],"22":[7,4,5],"26":[11,8,9],"37":[11,9,10],"13":[9,5,6],"46":[9,8,10],"24":[9,6,7],"35":[9,7,8],"16":[1,8,9],"5":[1,7,8],"10":[6,1,2],"48":[11,10,1],"21":[6,2,3],"43":[6,4,5],"32":[6,3,4],"49":[1,11,2],"6":[2,8,9],"36":[10,8,9],"1":[8,3,4],"39":[2,11,1],"17":[2,9,10],"25":[10,7,8],"14":[10,6,7],"47":[10,9,11],"31":[5,2,3],"42":[5,3,4],"0":[7,2,3],"20":[5,1,2],"27":[1,9,10],"2":[9,4,5],"38":[1,10,11],"18":[3,10,11],"30":[4,1,2],"7":[3,9,10],"29":[3,11,1],"41":[4,2,3],"3":[10,5,6],"28":[2,10,11]}}

这种情况应该怎么处理下呢,现在的思路是停生产消费应用,开启自动创建,重启kafka,重置消费组偏移量到最新日期避免消费之前的数据,大神帮看看这样有啥问题没?