Apache Kafka: All about Kafka Broker (2024)

Apache Kafka: All about Kafka Broker (1)

Report this article

Saikrishna Cheruvu

BigData🔸PySpark🔸Hadoop🔸SQL🔸AWS🔸GCP🔸AZURE🔸Snowflake🔸DWH🔸Power BI🔸DBT

Published Oct 14, 2022

+ Follow

Kafka brokers

A Kafka cluster is a group of multiple Kafka brokers.
A Kafka broker is a server in the cluster this will receive and send the data.
Each Kafka broker is identified with an ID (integer).
Each broker will have certain topic partitions.
All the topic partitions data is Distributed across all brokers(load balanced).
After connecting to any broker (bootstrap broker) you can have connectivity to the entire cluster.
A good number to get started is 3 brokers, but some big clusters have 100+ brokers.
You can create any number of brokers you want no limit to that.

How brokers and topics are related?

Consider a scenario where Topic-A has three partitions and Topic B has two partitions. Brokers 101, 102, and 103 are the final three Kafka brokers. Broker 101 will therefore have Topic-A, Partition 0, while Broker 102 will have Topic-A, Partition 2. This is not an error. Broker 103 is then discussing Topic-A, Partition 1. Therefore, as we can see, the subject divisions will be distributed among all brokers in any sequence. We also have Topic-B, Partition 1 on Broker 101, and Topic-B, Partition 0 on Broker 102 for this topic. Thus, in this instance, We can see that the data is spread, and since the two partitions have already been added to our Kafka broker, it is expected that Broker 103 does not have any Topic-B data partitions. And this is Kafka's power. As you can see from the example, the data and your partitions will be distributed throughout all brokers. This is how Kafka scales, and it is what is referred to as horizontal scaling. The more partitions and brokers we add, the more evenly the data will be dispersed throughout our whole cluster. We also take note of the fact that the brokers only have the data that they ought to have—not all of it.

Apache Kafka: All about Kafka Broker (3)

Kafka broker discovery

Every Kafka broker is called a ‘bootstrap server’
That means that you only need to connect to one broker, and the Kafka clients will know how to be connected to the entire cluster.
‘One bootstrap broker connectivity helps to understand the other clusters partitions’
Each broker knows about all brokers, topics, and partitions.

Apache Kafka: All about Kafka Broker (4)

Thank you!

Comment

3 Comments

Insights from the community

Data Engineering What are the most important considerations for extracting data with Apache Beam?
Data Engineering What are the best practices for troubleshooting Kafka errors and exceptions?
Data Architecture What are the benefits of using Apache Flink for data management?
Data Science How can you ensure the reliability of Apache Airflow?
Data Management How can you ensure scalability in Apache Spark jobs?
PostGIS How do you migrate and backup materialized views in PostGIS?
Data Warehousing What are the most effective techniques for optimizing ETL performance in Apache Kafka?
Database Administration How can you fine-tune performance in CouchDB?

Others also viewed

Step-by-Step Guide: How to Set Up a Kafka Cluster for High-Performance Distributed Data Processing Vijay Reddy Tirupari 1y
Quick Jump Start on Apache Kafka – CLI: Baladhandapani Nagarajan 3y
Kafka - A long Term Storage Engine Arun Muthuramalingam 1y
When should you use Kafka, and when is it not necessary? Mihai Munteanu 3mo
Deep Dive into Kafka Brokers Syed Ashfaq Hussain 9mo
The best Apache Kafka training part 3 Mohammad Hossein Abdipour 1y
Debezium - Turn your database to event streams Rajaneesh K 4y
What is a Kafka Topic? Maria Hatfield, PhD 4y
THE SECRET OF APACHE KAFKA Jatin Trikha 3y
🎁 A collection of Kafka-related talks 💝 Robin Moffatt 3y

Explore topics

Sales
Marketing
Business Administration
HR Management
Content Management
Engineering
Soft Skills
See All