Kafka POCs

Kafka POC projects

How to configure Kafka consumers to seek offsets by timestamp
- Normally, we consume Kafka messages from the beginning/end of a topic, or the last committed offsets. For backfilling or troubleshooting however, we occasionally need to consume messages from a certain timestamp. The Kafka consumer class of the kafka-python package has a method to seek a particular offset for a topic partition. Therefore, if we know which topic partition to choose – such as by assigning a topic partition – we can easily override the fetch offset. When we deploy multiple consumer instances together however, we make them subscribe to a topic, and topic partitions are dynamically assigned, which means we do not know which topic partition will be assigned to a consumer instance in advance. In this post, we will discuss how to configure the Kafka consumer to seek offsets by timestamp where topic partitions are dynamically assigned by subscription.
Simplify Streaming Ingestion on AWS – Part 1 MSK and Redshift
- Apache Kafka is a popular distributed event store and stream processing platform. Previously loading data from Kafka into Redshift and Athena usually required Kafka connectors (e.g. Amazon Redshift Sink Connector and Amazon S3 Sink Connector). Recently these AWS services provide features to ingest data from Kafka directly, which facilitates a simpler architecture that achieves low-latency and high-speed ingestion of streaming data. In part 1 of the simplify streaming ingestion on AWS series, we discuss how to develop an end-to-end streaming ingestion solution using EventBridge, Lambda, MSK and Redshift Serverless on AWS.
Simplify Streaming Ingestion on AWS – Part 2 MSK and Athena
- In Part 1, we discussed a streaming ingestion solution using EventBridge, Lambda, MSK and Redshift Serverless. Athena provides the MSK connector to enable SQL queries on Apache Kafka topics directly and it can also facilitate the extraction of insights without setting up an additional pipeline to store data into S3. In this post, we discuss how to update the streaming ingestion solution so that data in the Kafka topic can be queried by Athena instead of Redshift.
Integrate Glue Schema Registry With Your Python Kafka App
- Glue Schema Registry provides a centralized repository for managing and validating schemas for topic message data. Its features can be utilized by many AWS services when building data streaming applications. In this post, we will discuss how to integrate Python Kafka producer and consumer apps in AWS Lambda with the Glue Schema Registry.
Kafka Development with Docker
- Apache Kafka is one of the key technologies for modern data streaming architectures on AWS. Developing and testing Kafka-related applications can be easier using Docker and Docker Compose. In this series of posts, I will demonstrate reference implementations of those applications in Dockerized environments.

Consumer MSK connectivity

Hello,

When I try to use consumer.py against an MSK cluster I get:

INFO:botocore.credentials:Found credentials in environment variables.
INFO:kafka.conn:<BrokerConnection node_id=bootstrap-1 host=[REDACTED]:9198 <connecting> [IPv4 ([REDACTED], 9198)]>: connecting to [REDACTED]:9198
INFO:kafka.conn:Probing node bootstrap-1 broker version
INFO:kafka.conn:<BrokerConnection node_id=bootstrap-1 host=[REDACTED]:9198 <connecting> [IPv4 ([REDACTED], 9198)]>: Connection complete.
ERROR:kafka.conn:<BrokerConnection node_id=bootstrap-1 host=[REDACTED]:9198 <connected> [IPv4 ([REDACTED], 9198)]>: socket disconnected
INFO:kafka.conn:<BrokerConnection node_id=bootstrap-1 host=[REDACTED]:9198 <connected> [IPv4 ([REDACTED], 9198)]>: Closing connection. KafkaConnectionError: socket disconnected
INFO:kafka.conn:Broker is not v(0, 10) -- it did not recognize ApiVersionRequest_v0
INFO:kafka.conn:<BrokerConnection node_id=bootstrap-1 host=[REDACTED]:9198 <connecting> [IPv4 ([REDACTED], 9198)]>: connecting to [REDACTED]:9198
INFO:kafka.conn:<BrokerConnection node_id=bootstrap-1 host=[REDACTED]:9198 <connecting> [IPv4 ([REDACTED], 9198)]>: Connection complete.
ERROR:kafka.conn:<BrokerConnection node_id=bootstrap-1 host=[REDACTED]:9198 <connected> [IPv4 ([REDACTED], 9198)]>: socket disconnected
INFO:kafka.conn:<BrokerConnection node_id=bootstrap-1 host=[REDACTED]:9198 <connected> [IPv4 ([REDACTED], 9198)]>: Closing connection. KafkaConnectionError: socket disconnected
INFO:kafka.conn:Broker is not v(0, 9) -- it did not recognize ListGroupsRequest_v0
INFO:kafka.conn:<BrokerConnection node_id=bootstrap-1 host=[REDACTED]:9198 <connecting> [IPv4 ([REDACTED], 9198)]>: connecting to [REDACTED]:9198
INFO:kafka.conn:<BrokerConnection node_id=bootstrap-1 host=[REDACTED]:9198 <connecting> [IPv4 ([REDACTED], 9198)]>: Connection complete.
ERROR:kafka.conn:<BrokerConnection node_id=bootstrap-1 host=[REDACTED]:9198 <connected> [IPv4 ([REDACTED], 9198)]>: socket disconnected
INFO:kafka.conn:<BrokerConnection node_id=bootstrap-1 host=[REDACTED]:9198 <connected> [IPv4 ([REDACTED], 9198)]>: Closing connection. KafkaConnectionError: socket disconnected
INFO:kafka.conn:Broker is not v(0, 8, 2) -- it did not recognize GroupCoordinatorRequest_v0
INFO:kafka.conn:<BrokerConnection node_id=bootstrap-1 host=[REDACTED]:9198 <connecting> [IPv4 ([REDACTED], 9198)]>: connecting to [REDACTED]:9198
INFO:kafka.conn:<BrokerConnection node_id=bootstrap-1 host=[REDACTED]:9198 <connecting> [IPv4 ([REDACTED], 9198)]>: Connection complete.
ERROR:kafka.conn:<BrokerConnection node_id=bootstrap-1 host=[REDACTED]:9198 <connected> [IPv4 ([REDACTED], 9198)]>: socket disconnected
INFO:kafka.conn:<BrokerConnection node_id=bootstrap-1 host=[REDACTED]:9198 <connected> [IPv4 ([REDACTED], 9198)]>: Closing connection. KafkaConnectionError: socket disconnected
INFO:kafka.conn:Broker is not v(0, 8, 1) -- it did not recognize OffsetFetchRequest_v0
INFO:kafka.conn:<BrokerConnection node_id=bootstrap-1 host=[REDACTED]:9198 <connecting> [IPv4 ([REDACTED], 9198)]>: connecting to [REDACTED]:9198
INFO:kafka.conn:<BrokerConnection node_id=bootstrap-1 host=[REDACTED]:9198 <connecting> [IPv4 ([REDACTED], 9198)]>: Connection complete.
ERROR:kafka.conn:<BrokerConnection node_id=bootstrap-1 host=[REDACTED]:9198 <connected> [IPv4 ([REDACTED], 9198)]>: socket disconnected
INFO:kafka.conn:<BrokerConnection node_id=bootstrap-1 host=[REDACTED]:9198 <connected> [IPv4 ([REDACTED], 9198)]>: Closing connection. KafkaConnectionError: socket disconnected
INFO:kafka.conn:Broker is not v(0, 8, 0) -- it did not recognize MetadataRequest_v0
Traceback (most recent call last):
  File "/home/appuser/aws-glue-schema-registry/serializer-deserializer-msk-iam/target/test.py", line 62, in <module>
    consumer = Consumer(
  File "/home/appuser/aws-glue-schema-registry/serializer-deserializer-msk-iam/target/test.py", line 23, in __init__
    self.consumer = self.create()
  File "/home/appuser/aws-glue-schema-registry/serializer-deserializer-msk-iam/target/test.py", line 31, in create
    return KafkaConsumer(
  File "/usr/local/lib/python3.9/site-packages/kafka/consumer/group.py", line 356, in __init__
    self._client = KafkaClient(metrics=self._metrics, **self.config)
  File "/usr/local/lib/python3.9/site-packages/kafka/client_async.py", line 244, in __init__
    self.config['api_version'] = self.check_version(timeout=check_timeout)
  File "/usr/local/lib/python3.9/site-packages/kafka/client_async.py", line 909, in check_version
    version = conn.check_version(timeout=remaining, strict=strict, topics=list(self.config['bootstrap_topics_filter']))
  File "/usr/local/lib/python3.9/site-packages/kafka/conn.py", line 1297, in check_version
    raise Errors.UnrecognizedBrokerVersion()
kafka.errors.UnrecognizedBrokerVersion: UnrecognizedBrokerVersion

All ENV vars are exported properly.
Netcat works from the same environment against those brokers.

In the same fashion, I experienced the same issue with kafka-connect image, and I resolved it by introducing the security.protocol=SASL_SSL.

Not sure what's the case here, any insights?

Thank you so much!

jaehyeon-kim / kafka-pocs Goto Github PK

kafka-pocs's Introduction

Kafka POCs

Kafka POC projects

kafka-pocs's People

Contributors

Stargazers

Watchers

Forkers

kafka-pocs's Issues

Consumer MSK connectivity

Recommend Projects

React

Vue.js

Typescript

TensorFlow

Django

Laravel

D3

Recommend Topics

javascript

web

server

Machine learning

Visualization

Game

Recommend Org

Facebook

Microsoft

Google

Alibaba

D3

Tencent