kafka手动提交offest异常问题修订版本明细

源

采纳

编辑于 4年前

小白

kafka手动提交offest异常问题

kafka

flume定义了kafkasource消费数据, 手动提交偏移量的方式 , kafka版本 2.1.0-cdh6.2.1 topic的分区数是1 topic的数据量增量一天千万左右, 为了保证顺序所以用了单分区

数据量较大的情况下出现

2020-12-24 17:13:38,725 (PollableSourceRunner-DirectKafkaSouce-source1) [ERROR - org.apache.kafka.clients.consumer.internals.ConsumerCoordinator$OffsetCommitResponseHandler.handle(ConsumerCoordinator.java:812)] [Consumer clientId=consumer-1, groupId=group61] Offset commit failed on partition A_PREPAY_FLOW-0 at offset 431122116: The request timed out.
2020-12-24 17:13:38,727 (PollableSourceRunner-DirectKafkaSouce-source1) [INFO - org.apache.kafka.clients.consumer.internals.AbstractCoordinator.markCoordinatorUnknown(AbstractCoordinator.java:706)] [Consumer clientId=consumer-1, groupId=group61] Group coordinator hnbigdata006:9092 (id: 2147483323 rack: null) is unavailable or invalid, will attempt rediscovery

2020-12-24 17:13:38,830 (PollableSourceRunner-DirectKafkaSouce-source1) [INFO - org.apache.kafka.clients.consumer.internals.AbstractCoordinator$FindCoordinatorResponseHandler.onSuccess(AbstractCoordinator.java:654)] [Consumer clientId=consumer-1, groupId=group61] Discovered group coordinator hnbigdata006:9092 (id: 2147483323 rack: null)
2020-12-24 17:13:38,830 (PollableSourceRunner-DirectKafkaSouce-source1) [INFO - org.apache.kafka.clients.consumer.internals.AbstractCoordinator.markCoordinatorUnknown(AbstractCoordinator.java:706)] [Consumer clientId=consumer-1, groupId=group61] Group coordinator hnbigdata006:9092 (id: 2147483323 rack: null) is unavailable or invalid, will attempt rediscovery
2020-12-24 17:13:38,932 (PollableSourceRunner-DirectKafkaSouce-source1) [INFO - org.apache.kafka.clients.consumer.internals.AbstractCoordinator$FindCoordinatorResponseHandler.onSuccess(AbstractCoordinator.java:654)] [Consumer clientId=consumer-1, groupId=group61] Discovered group coordinator hnbigdata006:9092 (id: 2147483323 rack: null)

请大佬帮忙指导下

预览