Important
- Install Confluent Kafka
- Confluent Kafka Documentation
- Confluent Kafka Windows Download
- Confluent Kafka Download
- Confluent Kafka Version
This feature is available as a preview feature. A preview feature is a component of Confluent Platform that is beingintroduced to gain early feedback from developers. This feature can be used for evaluation andnon-production testing purposes or to provide feedback to Confluent.
This is a hands-on demo of Cluster Linking and its capabilities in the context of common use cases. The scripts and README are available on GitHubat demo-scene/cluster-linking
What the Demo Covers¶
Get hands-on with tutorials based on tested, executable code to help you learn Apache Kafka, Kafka Streams, and ksqlDB. Check out examples showcasing end-to-end solutions and Confluent’s event streaming platform, built by the original creators of Apache Kafka. Native for Windows, Mac & Linux. Easy as 1.2.3 ️. Works on any Apache Kafka cluster. Confluent Cloud, Aiven, Amazon MSK Schema Registry, Kafka Connect PLAINTEXT, SSL, SASL. Conduktor loves all Kafka clusters ?. Download & Generate schemas on the fly Analyze versions of your schema.
By the end of this demo, you will have configured two clusters and successfullyused Cluster Linking to migrate data across the clusters for a variety of use cases.
In the following steps, you will:
- Create a topic link
- Learn how to describe cluster links
- Run through a basic topic mirroring example to share topic data
- Open up two windows on a command terminal to produce messages on the source cluster and consume from a mirrored topic on the destination cluster
- Migrate a consumer from west cluster to east cluster, and learn how to monitor it
- Stop linking and perform demo teardown
Setup and Prerequisites¶
The demo includes a Docker Compose file that pulls Docker images and sets up two Kafka clusters, each with its own ZooKeeper:
- ZooKeeper
- Kafka
- Confluent Server
Several scripts are included, which are used to set configurations, run commands, and demo the use cases.
- Docker
- Docker version 1.11 or later isinstalled and running.
- Docker Compose is installed. Docker Compose is installed by default with Dockerfor Mac.
- Docker memory is allocated minimally at 8 GB. When using Docker Desktop for Mac, the default Docker memoryallocation is 2 GB. You can change the default allocation to 8 GB in Docker. Navigate to Preferences > Resources > Advanced.
- Internet connectivity
- Operating System currently supported by Confluent Platform
- Networking and Kafka on Docker
- Configure your hosts and ports to allow both internal and external components to the Docker network to communicate. For more details, see this article.
Install Confluent Kafka
Start the services¶
Clone the Confluent demo-scenerepository from GitHub and work in the
cluster-linking/
subdirectory, whichprovides the sample code you will compile and run in this tutorial.Tip
The following
gitclone
example uses SSH, if your Git configuration is set for HTTPS, use gitclonehttps://github.com/confluentinc/demo-scene.git
instead.Share a mirrored topic¶
- Start Docker Compose.This pulls the latest Docker images and starts the containers.
- Create a topic, cluster link, and mirrored topic.Run the following script to create a topic in the west cluster, link the west cluster to east cluster, and mirror the topic data on the source topic in the destination topic.You should see output similar to the following.To accomplish this, the create-links-topics script runs:
kafka-cluster-links
on the source to create a cluster link calledwest-cluster-link
, pass in group filters (in a JSON file) that definethe consumer group to mirror from consumer offsets (all are replicated), identify the bootstrap server of the cluster to link to (broker-west
),enable consumer offset syncs, and sync up offsets every 10 seconds.kafka-cluster-links--list
to view the links createdkafka-topicscreate
to create a mirror of thewest-trades
topic on the destination cluster (broker-east
)
TipTopic renaming is not supported for this preview. The original topic and mirrored topic must have the same name. - Run the list-links-and-lag script which executes the kafka-configs command to describe the link and topic.You should see output similar to the following.TipThe demo is not set up to generate link metrics. To learn more about how to collect and view metrics on the cluster link, see Monitoring Cluster Metrics and Optimizing Links and Cluster Linking Tutorial.
- Open up three command window sessions, one for the west cluster, one for the east cluster, and one to monitor lag between the two. (Make sure all sessions are in the same repository and directory you’ve been working in,
demo-scene/cluster-linking
.)- In the window for the east cluster (destination, with the mirrored topic), execute the run-consumer script to consume the messages from the mirrored topic
west-trades
.When the consumer starts, the following message is shown. - In the window for the west cluster (source, with the original topic), execute the run-producer scriptto produce 100 messages to the topic
west-trades
on the west cluster.When the producer runs, the following message is shown. (This producer auto-produces the messages, then shuts down and returns you to the prompt.) - Look back at your consumer command window for the east cluster. You should see the following output as your consumer (on the destination) reads from the mirrored topic.Type Ctl-C to shut down this consumer and get your prompt back.
Set up a consumer group to read from a topic on the source¶
- Open a new command window, create a consumer group in the west cluster that consumes from
west-trades
.The set-up-consumer script does the following:- configures the group with a property to automatically commit offsets
- names the group
someGroup
- sets up a consumer to read from the
west-trades
topic on the west cluster (source) and runs it
You should see output similar to the following as the consumer group reads messages 1-100 from the mirrored topic. - In your “lag” window, run the
kafka-consumer-groups
command to validate offsets on both the source and destination cluster. (These should match.)- To get the offsets for the west cluster (source), provide the following command.The output for the source cluster should resemble the following.
- To get the offsets for the east cluster (destination), provide the following command.The output for the destination cluster should resemble the following.
TipIt can take up to 10 seconds for offsets to be migrated.
Migrate a group from source to destination¶
- Update the offset migration to stop migrating consumer offsets from the west cluster to the east cluster.The migrate-one-cg scriptupdates the offset migration and cluster link to accomplish the migration, produces more messages to the
west-trades
topic, andthen consumes them from the consumer group in the east.Watch the output messages and note that the script accomplishes the following tasks.- Updates
consumer.offset.group.filters
to set an exclusion filter forsomeGroup
which excludes it from consumer offsets - Uses the
kafka-configs
command to update the cluster link from the east to exclude migration ofsomeGroup
consumer offsets - Produces another 100 messages in the west cluster
- Consumes the new consumer in the east cluster
- Monitors the consumer offsets that have been migrated from one side to the other
- Updates
- In your “lag” window, rerun the
kafka-consumer-groups
commands on both clusters to verify that the migrated consumer group onbroker-east
is fully caught up to offset 200.- First, on the west cluster. (This is the cluster you produced to, so just verify that the producer worked.)The output for the source cluster should resemble the following.
- Now, verify that
broker-east
is caught up.These should match, with no lag because we are not producing more messages at this point.
Change a topic from a mirrored topic to a writable topic¶
Confluent Kafka Documentation
Run the stop-link script to change a topic from a mirrored topic to writable topic.
The script uses
kafka-replica-status
to show the mirrored topic, kafka-topics--alter--mirror-actionstop
to stop the link, and kafka-replica-status
again to monitor the changes.Confluent Kafka Windows Download
You should see output similar to the following as the link stops, and the mirrored topic changes to a writable topic.
Teardown¶
- Shut down any running producers or consumers cleanly with Ctl-C in their respective command windows. (If you forget, the shutdown script also will stop these for you.)
- Run the shutdown script to stop and remove Docker containers.Your output should resemble:
Confluent Kafka Download
For more information, refer to the official Docker documentation.