Monday, 30 September 2019

kafka client is sending request to partition where broker which went down

I am using kafka-node module to send message to kafka. In a clustered environment where I have a topic with 3 partitions and replication factor as 3.

Topic describe is -

Topic:clusterTopic      PartitionCount:3        ReplicationFactor:3    Configs:min.insync.replicas=2,segment.bytes=1073741824
        Topic: clusterTopic     Partition: 0    Leader: 1       Replicas: 1,2,3 Isr: 1,2,3
        Topic: clusterTopic     Partition: 1    Leader: 2       Replicas: 2,3,1 Isr: 1,2,3
        Topic: clusterTopic     Partition: 2    Leader: 3       Replicas: 3,1,2 Isr: 1,2,3

Producer config -

        "requireAcks": 1,
        "attributes": 2,
        "partitionerType": 2,
        "retries": 2

When I send data it follows partition-type as cyclic(2) like round-robin fashion

when I follow below steps

  • Get a HighLevelProducer instance connected to kafka:9092,kafka:9093
  • send a message
  • stop the kafka-server:9092 manually
  • try and send another message with the HighLevelProducer and send() will trigger the callback with error: TimeoutError: Request timed out after 30000ms

What I am expecting is if a partition is not accessible (as a broker is down) producer should automatically send data to next available partition but I am losing message because of exception



from kafka client is sending request to partition where broker which went down

No comments:

Post a Comment