1、在线上业务spark-streaming中,出现了OffsetOutOfRangeException这个错误,经初步分析是由于zk中保存的offset小于broker中的offset。这个问题,版主有什么高见?
2、在验证上面的问题中,有几个困惑。在kafka0.10中,kafka自己保存consumergroup的信息,图片传不上来,直接粘贴吧
[root@kvdb08 lmy]# kafka-topics --zookeeper kvdb08:2181/kafka --list
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/KAFKA-2.1.0-1.2.1.0.p0.115/lib/kafka/libs/slf4j-log4j12-1.7.21.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/KAFKA-2.1.0-1.2.1.0.p0.115/lib/kafka/libs/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See https://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
__consumer_offsets
上面__consumer_offsets应该是保存consumergroup的信息吧?还是跟以前版本一样,consumergroup信息保存在zk中?但是我用命令
kafka-run-class kafka.tools.ConsumerOffsetChecker --broker-info kvdb08:9092 --zookeeper kvdb08:2181/kafka --topic consumer-count --group consumer-count
查看的结果是可以查看的
Group           Topic                          Pid Offset          logSize         Lag             Owner
consumer-count  consumer-count                 0   9614            11216           1602            none
consumer-count  consumer-count                 1   9846            11487           1641            none
consumer-count  consumer-count                 2   10030           11707           1677            none
consumer-count  consumer-count                 3   9778            11408           1630            none
consumer-count  consumer-count                 4   10330           12054           1724            none
consumer-count  consumer-count                 5   10334           12061           1727            none
但在zookeeper-client中没有找到consumer-count这个消费组的信息
[zk: localhost:2181(CONNECTED) 0] ls /kafka/consumers
[console-consumer-17136, console-consumer-7906, console-consumer-36419, console-consumer-60409, console-consumer-36808, schema-registry-kvdb08-8081, console-consumer-73372, console-consumer-86852]
所以consumerGroup还是存在于kafka-broker中?
 
        
kafka-consumer-group.sh --new-consumer --bootstrap-server .... --list
你的答案