java – Spring @KafkaListener and concurrency
java – Spring @KafkaListener and concurrency
Kafka doesnt work that way; you need at least as many partitions as consumers (controlled by concurrency in the spring container).
Also, only one consumer (in a group) can consume from a partition at a time so, even if you increase the partitions, records in the same partition behind the stuck consumer will not be received by other consumers.
If you want to have failover Kafka, you must spin up more instances of your application.
Example: you have a topic named test with 1 partition, you will create 2 instances of your app with the same Kafka group. One instance will process your data, the other will wait and start processing messages in case the first instance crashes. Same if you have N partitions with N + 1 or 2 or 3 instances of your application. Also, every instance will only have one consumer thread.
For more info about it search on Google: Kafka Consumer Groups.