Questions tagged [kafka-consumer-api]

Use for questions related to the Apache Kafka consumer API

0
votes
1answer
16 views

What is the difference between implementing Deserializer and Serde in Kafka Consumer API?

I try to simulate Gwen (Chen) Shapira's kafka-clickstream-enrich kafka-stream project on GitHub (https://github.com/onurtokat/kafka-clickstream-enrich). When I consume a topic using consumer class ...
0
votes
0answers
30 views

Kafka Consumer integration in unity3d C# [duplicate]

I have this code snippet in C# windows application. It is connecting and working fine in winform environment. Desktop APP Snippet static void Main(string[] args) { string bootstrapServers = "...
0
votes
0answers
15 views

consumer consuming the same message twice at the starting only

At the very first consuming, my consumer is consuming the same message twice, this only happens at the first consuming, after that it only consumes once. Attaching the consumer conf code ...
0
votes
0answers
24 views

Kafka Consumer Hangs Indefinitely after Rebalancing

I am trying to utilize a kafka consumer library that is prewritten in my organization. It takes JSON data from a Kafka topic and stores it in a Mongo database. While I cannot post this code, it is a ...
0
votes
0answers
17 views

Can we use multiple queues for all the topics, like one dedicated queue to each topic in Apache Kafka?

I want to make a dedicated queue to for each topic in Kafka. Is that possible or feasible?
0
votes
0answers
10 views

How to convert conflunet Kafka AVRO Messages to Spark dataframe in 2.0.2 Version

Unable to save the Avro messages to spark/scala dataframe in spark 2.0.2 version I am using the golden gate and confluent Kafka producer to get the changes in Oracle database. I am getting Kafka ...
-1
votes
0answers
19 views

KafkaAvroDeserializer is not an instance of Deserializer

The code below in java throws exceptions as shown at the bottom. Environment : Tomcat, Java8 KafkaConsumer <String, Map<ConfidentialityLevel, CommonCoreEvent>> consumer; Properties props ...
1
vote
1answer
18 views

In the new Kafka Consumer API versions >0.9 what property replaces consumer.timeout.ms?

I cannot figure out from the documentation which property now replaces consumer.timeout.ms =-1 (Default) which was available in Kafka Consumer API versions before 0.9. Does anyone have the link to the ...
0
votes
2answers
47 views

java kafka get message offset in partition by unique key

If all my kafka messages contains a unique key, what will be the fastest way to query the offset of a message with its unique key within a partition in the broker? Assume I already know this message ...
0
votes
0answers
17 views

How to connect Kafka consumer with SSL setup with any of Node JS modules

I want to connect external kafka topic provided by vendor; as we are already developed service on top of Node JS. So I am looking for NodeJS kafka consumer and with SSL setup; as the kafka-...
0
votes
1answer
25 views

My kafka error handler is not getting invoked, should I invoke explicitly?

I am developing a kafka consumer API which consumes my messages from a topic. When it consumes an incorrect message (say not well formed JSON message), I am expecting my error handler should be called ...
0
votes
1answer
47 views

Exception in thread “main” org.apache.kafka.common.KafkaException: Failed to construct kafka consumer

When I trying to connect the confluent Kafka cluster with simple consumer code using Scala. But it is failed to connect and throws the exception " Exception in thread "main" org.apache.kafka.common....
5
votes
1answer
111 views

Consumer and producer failing with error: “Connection to 0 was disconnected before the response was read”

I have a cluster of 3 Kafka brokers with replication factor of 3 for all topics. Since last few days I am facing this issue, suddenly(few times in a day) consumers and producers are stuck while ...
1
vote
1answer
20 views

Avro schema Java deepcopy issue with field order

I am currently looking into solutions for unexpected behaviour when dealing with particular AVRO schema evolution scenarios when using Java and doing a deepcopy in a consumer to parse the ...
1
vote
1answer
39 views

InvalidGroupIdException: in Kafka scala consumer program even after setting group.id

I am facing the org.apache.kafka.common.errors.InvalidGroupIdException: To use the group management or offset commit APIs, you must provide a valid group.id in the consumer configuration. ...
0
votes
1answer
24 views

Alternative to Spring-Kafka (Kafka libraries/implementation)

I want suggestions of alternatives to Spring-Kafka. I have used Kafka integrated in Spring in my application. I want to explore any alternative libraries available. Would be good if comparative ...
1
vote
1answer
40 views

Kafka Consumer only reads after 'enough' data has been produced

I am implementing an endpoint in spring-boot that, when invoked, will do a dump of all the messages sitting in a kafka topic (for testing). The behaviour I am expecting is that when a producer ...
0
votes
1answer
20 views

Kafka message loss because of later message

So I got some annoying offset commiting case with my kafka consumers. I use 'kafka-node' for my project. I created a topic. Created 2 consumers within a consumer-group over 2 servers. Auto-commit set ...
1
vote
1answer
29 views

How to scale Kafka stream processing dynamically?

I have a fixed number of partitions of a topic. Producers produce data at varying rate in different hours of the day. I want to add consumers dynamically based on hours of the day for the processing ...
0
votes
1answer
20 views

Consumer Id and Group Id in Kafka: what makes two consumers the same

I've been using Kafka for some months now, and I realized that some of the core concepts are not so clear for me yet. My doubt is related to the relation between consumerId, groupId and offsets. In ...
0
votes
0answers
25 views

Offset out of range resetting offset 2nd consumer group

Offset is getting reset when single topic is consumed by two different consumer groups. I am using Kafka version 0.10.1.0 and Spring-Kafka version 2.2.4 Release. I produce a message in topic "topic_X"...
0
votes
0answers
25 views

Kafka consumer groups suddenly stopped balancing messages among instances

We have a microservice architecture communicated by Kafka on Confluent where each service is set in its own consumer group in order to balance message delivery between the multiple instances. For ...
0
votes
0answers
40 views

ERROR:kafka.client:Unable to bootstrap from kafka

Here is my consumer config. I am using SSL for authentication. consumer = KafkaConsumer(config['kafka_topic'], bootstrap_servers=config['kafka_brokers'], ...
0
votes
1answer
35 views

Kafka consumer crashing after processing but before committing offset?

I'm doing some POC work with Kafka and am researching the exactly once features using the transactional API and am still stuck on a few questions. What happens if a consumer crashes after processing ...
0
votes
1answer
23 views

How to run shell script through “out_exec” plugin?

There a log file which delivers the message from FluentD to Kafka (FluentD --> Kafka). Fluentd configurations Settings are as below. <source> @type tail path /home/ubuntu/demo/files/test ...
0
votes
1answer
16 views

consume min of n records and max n records but return records after waiting for m seconds

I am having a use case that consumer has to consume minimum of n records to start the next processing, but after waiting for m seconds, must return p records where p< n. Is there a way to achieve ...
1
vote
2answers
76 views

Avoid Data Loss While Processing Messages from Kafka

Looking out for best approach for designing my Kafka Consumer. Basically I would like to see what is the best way to avoid data loss in case there are any exception/errors during processing the ...
0
votes
1answer
33 views

Kafka Consumer is not able to deserialize timewindowed key which has start and end time

I have one Kafka Streams app (V 2.1.1) which produces record and put into output topic in key value format. key is windowed time serde where I am expecting key and handle to window start/end time. ...
2
votes
1answer
35 views

How to get Consumer Lag for a consumer group in kafka in java

I want to know the consumer lag for a consumer group using java. I have tried using kafka-consumer-groups --describe --bootstrap-server localhost:9092 --group MyGroupName and the lag is visible. ...
0
votes
0answers
16 views

Console Kafka Consumer does not receive any value from Producer?

Going through this website i created Kafka Producer and Kafka Consumer https://www.tutorialkart.com/apache-kafka/kafka-console-producer-and-consumer-example/ but my consumer is not taking any value ...
-1
votes
0answers
44 views

Kafka Consumer without Threads in C# Unity

I am trying to write a Kafka Consumer in Unity3d Environment. As unity is not thread safe so I am trying to write a thread less kafka consumer builder example. Initally i get coding idea from this ...
1
vote
1answer
33 views

Kafka Streams: configuring `AdminClientConfig` or `ConsumerConfig` without overriding the values both

Working with kafka version: 2.0.1 & kafka-streams-scala version 2.0.1 Log DEBUG messages such as: DEBUG 2019-05-08 09:57:53,322 [he.kafka.clients.NetworkClient] [ ] [ ]: [...
0
votes
0answers
31 views

Kafka consumer is very slow to consume data and only consuming first 500 records

I am trying to integrate MongoDB and Storm-Kafka, Kafka Producer produces data from MongoDB but it fails to fetch all records from Consumer side. It only consuming 500-600 records out of 1 million ...
-2
votes
2answers
34 views

How to increase KAFKA consumer on demand in single Group

Lets say we have one topic "topic-1" in kafka with partition 5. Consumer Group-A with 5 consumer attached to "topic-1" each partition. Due to large workload large number of message get publish. Now ...
0
votes
0answers
8 views

Multiple consumers simultaneously consume messages from different partitions under the same topic

My cluster has built 3 nodes.I created a topic named test Topic:test PartitionCount:3 ReplicationFactor:3 Configs: Topic: test Partition: 0 Leader: 2 Replicas: 0,1,2 Isr: 2,1,0 Topic:...
1
vote
1answer
29 views

Is there a way to specify the isolation level in Kafka-Python consumer API?

I am trying to write a consumer using Kafka-Python ensuring the exactly once semantics. The messages in the partitions are produced using a transaction aware producer. I read from the Kafka docs that ...
0
votes
0answers
13 views

Kinesis Retries vs. Kafka Retries

I was reading up on how Uber implements retry logic in Kafka: https://eng.uber.com/reliable-reprocessing/ Uber uses X number of retry topics / streams to retry X number of times. The reason for ...
0
votes
0answers
14 views

kafkaspout is not fetching data from topic

I have 2 kafkastorm topologies in which kafka-producer topology reads data from mongodb oplog and append on to kafka topic and good thing is it is working absolutely fine ! While kafka-consumer ...
0
votes
1answer
30 views

Kafka Consumer - Random access fetching of partition range

The question: How can I randomly fetch an old chunk of messages with a given range definition of [partition, start offset, end offset]. Hopefully ranges from multiple partitions at once (one range for ...
2
votes
1answer
64 views

Kafka consumer.poll returns no records

The first N calls to poll return nothing when I register a consumer with a new group id. I want to test that when I call a service, a Kafka event is published. The issue is that whenever I change the ...
1
vote
0answers
15 views

Apache Kafka spout is not working on Consumer Side

I am trying to integrate MongoDB and Storm-Kafka, Kafka Producer produces data from MongoDB but it fails to fetch from Consumer side. Kafka version :0.10.* Storm version :1.2.1 Do i need to add any ...
1
vote
1answer
33 views

Kafka balancing load between multiple tenants

I'm considering Kafka as one of several technologies to serve as a message broker for worker nodes that will eventually send push notifications to users. An important constraint is that I don't want ...
1
vote
2answers
67 views

Apache Kafka get list of consumers on a specific topic

As it can be guest from the title, is there a way to get the consumer list on a specific topic in java? Untill now Im able to get the list of topics like this final ListTopicsResult ...
1
vote
2answers
39 views

Is it possible to reset offsets to a topic for a kafka consumer group in a kafka connector?

My kafka sink connector reads from multiple topics (configured with 10 tasks) and processes upwards of 300 records from all topics. Based on the information held in each record, the connector may ...
3
votes
1answer
58 views

Kafka consumer reconnection after getting disconnected

I have my project set up using Spring Boot and Spring Kafka, and there are three consumers. Checking the logs, I can see that from time to time the consumers get disconnected: catalina.out:2019-04-...
0
votes
0answers
34 views

Get timestamp when payload published to topic

I have kafka-streams application where payload goes through multiple topics, say it starts from topic A->B->c->.. -> L, and ends at L. I have a requirement to find out how much time it took for data ...
0
votes
1answer
29 views

Unable to consume messages from kafka topic (using Python) even after making successful connections and getting messages in a kafka consumer console [duplicate]

I have a setup of Debezium which uses Kafka. I am able to consume messages from kafka console as described in the doc. However, when I create a kafka consumer using Python on my local, I am unable to ...
0
votes
0answers
46 views

How to fix: java.lang.OutOfMemoryError: Direct buffer memory in flink kafka consumer

We Are running a 5 node flink cluster (1.6.3) over kubernetes, with a 5 partitions Kafka topic source. 5 jobs are reading from that topic (with different consumer group), each with parallelism = 5. ...
0
votes
1answer
45 views

kafka consumer group id does not work as expected

I am new people on apache Kafka. When I go through quick start instruction via http://kafka.apache.org/quickstart with latest version kafka_2.12-2.2.0. I got a problem and can't figure it out by ...
1
vote
1answer
47 views

What is the difference between pulsar and kafka in regards to consumption?

In order to consume data from Kafka, we can have multiple consumers on a topic, totally decoupled. Then, what is meant by no shared consumption on the page(https://streaml.io/blog/pulsar-streaming-...