Featured Post

Event Sourcing Video from Michael Ploed

Event Sourcing I want to share a great video I found few days ago that describes very well what Event Sourcing is.

Thursday, February 1, 2018

KAFKA PARTITIONS

TOPIC

A Topic log consists of many partitions which can be spread on multiple Kafka nodes. Kafka distributes topic partitions on different nodes in a cluster for high performance with horizontal scalability.

Partitions

A topic partition is the unit of parallelism in Kafka.

Producers

  • Producers writes to different partitions can be done fully in parallel and dont' wait for acknowledgements from Kafka and sends messages as fast as Kafka can handle. 
  • Kafka appends records to the end of a topic log. Producer chooses partition to send a record via Round-Robin or based on the Record’s Key


Consumers

  • Consumers can use Consumer Groups 
  • A Consumer Group consists of many Consumer Thread that always receives from a single partition.
  • So the degree of parallelism in the consumer within a consumer group is bounded by the number of partitions.
  • Therefore, an higher number of partitions increases the throughput to achieve.


References

No comments :

Post a Comment