Results for query "challenges while processing Kafka"

what is best practice to consume messages from multiple kafka topics?

I need to consumer messages from different kafka topics,


Should i create different consumer instance per topic and then start a new processing thread as per the number of partition.or
I should subscribe all topics from a single consumer instance and the should start different processing threads


Thanks & regards,
Megha

This really depends on logic of your application - does it need to see all messages together in one place, or not.

Kafka Message at-least-once mode at multi-consumer

Kafka messaging use at-least-once message delivery to ensure every message to be processed, and uses a message offset to indicates which message is to deliver next.When there are multiple consumers, if some deadly message cause a consumer crash during message processing, will this message be redelivered to other consumers and spread the death?

kafka streams simulate join within the stream

Need suggestions on best approach to solve this problem -

I am developing a kafka stream processing application where the source log stream contains two types of messages - employeeInfo and departmentInfo.I do not have any control over this log stream, so I cannot change the schema or how it is written.

MS SQL CDC with Kafka Connect and Apache Kafka

if there is surge in MS SQL records Spark processing takes more time than batch interval and spark ends up sending duplicate records to Kafka.As an alternate to this I am thinking of using Kafka Connect to read the messages from MS SQL and send records to Kafka topic and maintain the MS SQL CDC in Kafka.

Understanding Debezium

Provided a use case:

A stream processing architecture;
Events go into Kafka then get processed by a job with a MongoDB sink.Database name: myWebsite
Collection: users

and the job sinks user records in the users collection.

How do I get from “Big Data” to a webpage?

And while I understand what it is that tools like Hadoop/Cassandra/Kafka etc do, no one seems to explain how the data gets from these large processing tools to rendering something on a client/webpage.explain how the data gets from these large processing tools to rendering something on a client/webpage.

Kafka streams application design principals

I want to dive into stream processing with Kafka and I need some help to get my head around some design principals which are currently not very clear to me.would you make one topic "price" keyed (and therefore partitioned) by the stock symbol?