Data & Intelligence

How to Produce and Customize a Kafka Message

The Kafka became more and more popular as a famous real-time message delivering and communication system for a period of years. It was born in LinkedIn and contributed to Apache in 2011. Then it grew to be one of the top Apache projects. For now, it has been accepted by thousands of enterprises as the basic part of the system. In this blog, I will introduce producing and consuming of the Kafka message.

1.Prerequisite

First, you need to install a single node Kafka system or a Kafka Cluster.  Then you need to know a little about the Kafka Topic: A topic is a category or feed name to which records are published. Topics in Kafka are always multi-subscriber; that is, a topic can have zero, one, or many consumers that subscribe to the data written to it [1]. The topic is the key for application consuming. You can create one as follows:

a.Using command:                                         How to Produce and Customize a Kafka Message

b.Using program:

How to Produce and Customize a Kafka Message

Data Intelligence - The Future of Big Data
The Future of Big Data

With some guidance, you can craft a data platform that is right for your organization’s needs and gets the most return from your data capital.

Get the Guide

2.Kafka message producer

Then we can start our first program to populate the message. Before we start programming, we need to add the reference to the necessary Kafka packages:                                                                                                                        How to Produce and Customize a Kafka Message

It is time to create our first Kafka message producer. You need to instantiate a KafkaProducer<TKey,TValue> and set the necessary configuration like Kafka cluster broker list, message serialize helper class and so on.How to Produce and Customize a Kafka Message

After instantiation, you can send the message as below:How to Produce and Customize a Kafka MessageHow to Produce and Customize a Kafka Message

Kafka message consumer:

Like the producer, we need to add the reference to the necessary Kafka packages. Then we can create the Kafka message consumer. You need to instantiate a KafkaConsumer<TKey,TValue>and set the necessary configuration like Kafka cluster broker list, message deserialize helper class and so on.How to Produce and Customize a Kafka Message

After instantiation, you can receive the message as below:How to Produce and Customize a Kafka Message

How to Produce and Customize a Kafka Message

 

[1]From Apache Kafka, Official Site

About the Author

More from this Author

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Subscribe to the Weekly Blog Digest:

Sign Up