Monday 8 March 2021

Failing to connect to Kafka on MSK

I have created a Kafka cluster on MSK and now I'm trying to connect to the cluster with python.

I wrote this shortcode:

from kafka import KafkaProducer
import json

producer = KafkaProducer(
    bootstrap_servers=['host1:9092', 'host2:9092'],
    value_serializer=lambda x: json.dumps(x).encode('utf-8'),
    api_version=(2, 4, 1)
)

producer.send('test', value={'hello':'world'})

The problem is that every time I running it I'm getting this error:

KafkaTimeoutError: Failed to update metadata after 60.0 secs.

I thought it might be related to Kafka creating topics so I added this line to the configuration.

auto.create.topics.enable=true

But I'm still getting the same error.

This is my full configuration file:

default.replication.factor=3
min.insync.replicas=2
num.io.threads=8
num.network.threads=5
num.partitions=1
num.replica.fetchers=2
socket.request.max.bytes=104857600
unclean.leader.election.enable=true
auto.create.topics.enable=true
zookeeper.connection.timeout.ms=5000

What am I missing here? I read somewhere that is may relate to SSL authentication but at any step, there wasn't any .pem file, .ca file, or anything like that.



from Failing to connect to Kafka on MSK

No comments:

Post a Comment