Docker compose 快速部署Kafka集群

这里使用docker compose快速部署Kafka集群(3台Broker节点，一台zk)

确定docker-compose环境

[root@tmp-01 kafka]# docker compose version
Docker Compose version v2.18.1

#docker版本
[root@tmp-01 kafka]# docker info
Client: Docker Engine - Community
 Version:    24.0.2

接下来编写Kafka docker-compose.yaml文件

自行修改192.168.31.70 为宿主机IP，采用nat端口映射方式访问

version: '3.8'
services:
  zookeeper:
    image: wurstmeister/zookeeper
    container_name: zookeeper
    ports:
      - "2181:2181"
    restart: always
  kafka1:
    image: wurstmeister/kafka
    depends_on: [ zookeeper ]
    container_name: kafka1
    ports:
      - "9091:9091"
    environment:
      HOSTNAME: kafka1
      KAFKA_BROKER_ID: 0
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka1:9091
      KAFKA_LISTENERS: PLAINTEXT://0.0.0.0:9091
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181/kafka
    extra_hosts:
      kafka1: 192.168.31.70
  kafka2:
    image: wurstmeister/kafka
    depends_on: [ zookeeper ]
    container_name: kafka2
    ports:
      - "9092:9092"
    environment:
      HOSTNAME: kafka2
      KAFKA_BROKER_ID: 1
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka2:9092
      KAFKA_LISTENERS: PLAINTEXT://0.0.0.0:9092
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181/kafka
    extra_hosts:
      kafka2: 192.168.31.70
  kafka3:
    image: wurstmeister/kafka
    depends_on: [ zookeeper ]
    container_name: kafka3
    ports:
      - "9093:9093"
    environment:
      HOSTNAME: kafka3
      KAFKA_BROKER_ID: 2
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka3:9093
      KAFKA_LISTENERS: PLAINTEXT://0.0.0.0:9093
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181/kafka
    extra_hosts:
      kafka3: 192.168.31.70

启动docker-compose

[root@tmp-01 kafka]# docker compose up -d
[+] Running 27/4
 ✔ kafka3 Pulled                                                          71.9s
 ✔ zookeeper 16 layers [⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿]      0B/0B      Pulled          45.9s
 ✔ kafka2 7 layers [⣿⣿⣿⣿⣿⣿⣿]      0B/0B      Pulled                       71.9s
 ✔ kafka1 Pulled                                                          71.9s
[+] Building 0.0s (0/0)
[+] Running 5/5
 ✔ Network kafka_default  Created                                          0.0s
 ✔ Container zookeeper    Started                                          1.8s
 ✔ Container kafka3       Started                                          0.8s
 ✔ Container kafka2       Started                                          0.8s
 ✔ Container kafka1       Started                                          0.8s
[root@tmp-01 kafka]#

检查kafka服务

[root@tmp-01 kafka]# docker compose ps
NAME                IMAGE                    COMMAND                  SERVICE             CREATED             STATUS              PORTS
kafka1              wurstmeister/kafka       "start-kafka.sh"         kafka1              5 minutes ago       Up 5 minutes        0.0.0.0:9091->9091/tcp, :::9091->9091/tcp
kafka2              wurstmeister/kafka       "start-kafka.sh"         kafka2              5 minutes ago       Up 5 minutes        0.0.0.0:9092->9092/tcp, :::9092->9092/tcp
kafka3              wurstmeister/kafka       "start-kafka.sh"         kafka3              5 minutes ago       Up 5 minutes        0.0.0.0:9093->9093/tcp, :::9093->9093/tcp
zookeeper           wurstmeister/zookeeper   "/bin/sh -c '/usr/sb…"   zookeeper           5 minutes ago       Up 5 minutes        22/tcp, 2888/tcp, 3888/tcp, 0.0.0.0:2181->2181/tcp, :::2181->2181/tcp

进入zk，查看Broker id

[root@tmp-01 kafka]# docker exec -it zookeeper bash

root@aeb2ed05799a:/opt/zookeeper-3.4.13# ./bin/zkCli.sh
[zk: localhost:2181(CONNECTED) 3] ls /kafka/brokers/ids
[2, 1, 0]

#这里表示3个节点服务都正常

接下来到Kafka 容器，创建topic检查

[root@tmp-01 kafka]# docker exec -it kafka1 bash

#这里创建一个2个分区，名称为Abcdocker的topic
root@fa12e66b1540:/# kafka-topics.sh --create --zookeeper 192.168.31.70:2181/kafka --replication-factor 1 --partitions 2 --topic abcdocker
Created topic abcdocker.

模拟生产者

root@fa12e66b1540:/# kafka-console-producer.sh --topic=abcdocker --broker-list kafka1:9091,kafka2:9092,kafka3:9093
>I am abcdocker
>site:i4t.com
>2023-08-22
>ok
>END
>^Croot@fa12e66b1540:/#

模拟消费者

消费成功

root@fa12e66b1540:/# kafka-console-consumer.sh --bootstrap-server kafka1:9091,kafka2:9092,kafka3:9093 --from-beginning --topic abcdocker
site:i4t.com
END
I am abcdocker
2023-08-22
ok

这里我们没有将Kafka和zk持久化数据，仅供测试使用。生产请注意数据持久化！