Skip to content

Commit e9604ba

Browse files
author
rokh-conduktor
authored
Use docker volumes to persist states (#96)
* add docker volumes * update all recipes ♻️ * fix typo * add doc 📝
1 parent 3674553 commit e9604ba

6 files changed

+194
-11
lines changed

‎README.md

+30-1
Original file line numberDiff line numberDiff line change
@@ -62,6 +62,12 @@ docker-compose -f zk-single-kafka-single.yml up
6262
docker-compose -f zk-single-kafka-single.yml down
6363
```
6464

65+
Clean state (zookeeper & kafka) with :
66+
```
67+
docker-compose -f zk-single-kafka-single.yml down
68+
docker-compose -f zk-single-kafka-single.yml rm
69+
```
70+
6571
## Single Zookeeper / Multiple Kafka
6672

6773
If you want to have three brokers and experiment with kafka replication / fault-tolerance.
@@ -76,6 +82,12 @@ docker-compose -f zk-single-kafka-multiple.yml up
7682
docker-compose -f zk-single-kafka-multiple.yml down
7783
```
7884

85+
Clean state (zookeeper & kafka) with :
86+
```
87+
docker-compose -f zk-single-kafka-multiple.yml down
88+
docker-compose -f zk-single-kafka-multiple.yml rm
89+
```
90+
7991
## Multiple Zookeeper / Single Kafka
8092

8193
If you want to have three zookeeper nodes and experiment with zookeeper fault-tolerance.
@@ -90,6 +102,11 @@ docker-compose -f zk-multiple-kafka-single.yml up
90102
docker-compose -f zk-multiple-kafka-single.yml down
91103
```
92104

105+
Clean state (zookeeper & kafka) with :
106+
```
107+
docker-compose -f zk-multiple-kafka-single.yml down
108+
docker-compose -f zk-multiple-kafka-single.yml rm
109+
```
93110

94111
## Multiple Zookeeper / Multiple Kafka
95112

@@ -104,6 +121,12 @@ docker-compose -f zk-multiple-kafka-multiple.yml up
104121
docker-compose -f zk-multiple-kafka-multiple.yml down
105122
```
106123

124+
Clean state (zookeeper & kafka) with :
125+
```
126+
docker-compose -f zk-multiple-kafka-multiple.yml down
127+
docker-compose -f zk-multiple-kafka-multiple.yml rm
128+
```
129+
107130
## Full stack
108131

109132
Need a UI? We recommend using [Conduktor](https://conduktor.io) as your tool to bring a unified UI to all these components
@@ -123,6 +146,12 @@ Need a UI? We recommend using [Conduktor](https://conduktor.io) as your tool to
123146
docker-compose -f full-stack.yml down
124147
```
125148

149+
Clean state (zookeeper & kafka) with :
150+
```
151+
docker-compose -f full-stack.yml down
152+
docker-compose -f full-stack.yml rm
153+
```
154+
126155
# FAQ
127156

128157
## Kafka
@@ -133,7 +162,7 @@ A: Add the following line to your docker-compose environment variables: `KAFKA_L
133162

134163
**Q: How do I delete data to start fresh?**
135164

136-
A: Your data is persisted from within the docker compose folder, so if you want for example to reset the data in the full-stack docker compose, do a `docker-compose -f full-stack.yml down`.
165+
A: Your data is persisted from within docker volumes , so if you want for example to reset the data in the full-stack docker compose, do a `docker-compose -f full-stack.yml rm`.
137166

138167
**Q: Can I change the zookeeper ports?**
139168

‎zk-multiple-kafka-multiple-schema-registry.yml

+44-3
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,14 @@
1-
version: '2.1'
2-
1+
version: '3.8'
2+
volumes:
3+
zkmultiple-kmultiple-sr_kafka1-data:
4+
zkmultiple-kmultiple-sr_kafka2-data:
5+
zkmultiple-kmultiple-sr_kafka3-data:
6+
zkmultiple-kmultiple-sr_zoo1-data:
7+
zkmultiple-kmultiple-sr_zoo2-data:
8+
zkmultiple-kmultiple-sr_zoo3-data:
9+
zkmultiple-kmultiple-sr_zoo1-log:
10+
zkmultiple-kmultiple-sr_zoo2-log:
11+
zkmultiple-kmultiple-sr_zoo3-log:
312
services:
413
zoo1:
514
image: confluentinc/cp-zookeeper:6.2.1
@@ -11,6 +20,13 @@ services:
1120
ZOOKEEPER_CLIENT_PORT: 2181
1221
ZOOKEEPER_SERVER_ID: 1
1322
ZOOKEEPER_SERVERS: zoo1:2888:3888;zoo2:2888:3888;zoo3:2888:3888
23+
volumes:
24+
- type: volume
25+
source: zkmultiple-kmultiple-sr_zoo1-data
26+
target: /data
27+
- type: volume
28+
source: zkmultiple-kmultiple-sr_zoo1-log
29+
target: /datalog
1430

1531
zoo2:
1632
image: confluentinc/cp-zookeeper:6.2.1
@@ -22,6 +38,13 @@ services:
2238
ZOOKEEPER_CLIENT_PORT: 2182
2339
ZOOKEEPER_SERVER_ID: 2
2440
ZOOKEEPER_SERVERS: zoo1:2888:3888;zoo2:2888:3888;zoo3:2888:3888
41+
volumes:
42+
- type: volume
43+
source: zkmultiple-kmultiple-sr_zoo2-data
44+
target: /data
45+
- type: volume
46+
source: zkmultiple-kmultiple-sr_zoo2-log
47+
target: /datalog
2548

2649
zoo3:
2750
image: confluentinc/cp-zookeeper:6.2.1
@@ -33,7 +56,13 @@ services:
3356
ZOOKEEPER_CLIENT_PORT: 2183
3457
ZOOKEEPER_SERVER_ID: 3
3558
ZOOKEEPER_SERVERS: zoo1:2888:3888;zoo2:2888:3888;zoo3:2888:3888
36-
59+
volumes:
60+
- type: volume
61+
source: zkmultiple-kmultiple-sr_zoo3-data
62+
target: /data
63+
- type: volume
64+
source: zkmultiple-kmultiple-sr_zoo3-log
65+
target: /datalog
3766

3867
kafka1:
3968
image: confluentinc/cp-kafka:6.2.1
@@ -47,6 +76,10 @@ services:
4776
KAFKA_ZOOKEEPER_CONNECT: "zoo1:2181,zoo2:2182,zoo3:2183"
4877
KAFKA_BROKER_ID: 1
4978
KAFKA_LOG4J_LOGGERS: "kafka.controller=INFO,kafka.producer.async.DefaultEventHandler=INFO,state.change.logger=INFO"
79+
volumes:
80+
- type: volume
81+
source: zkmultiple-kmultiple-sr_kafka1-data
82+
target: /var/lib/kafka/data
5083
depends_on:
5184
- zoo1
5285
- zoo2
@@ -64,6 +97,10 @@ services:
6497
KAFKA_ZOOKEEPER_CONNECT: "zoo1:2181,zoo2:2182,zoo3:2183"
6598
KAFKA_BROKER_ID: 2
6699
KAFKA_LOG4J_LOGGERS: "kafka.controller=INFO,kafka.producer.async.DefaultEventHandler=INFO,state.change.logger=INFO"
100+
volumes:
101+
- type: volume
102+
source: zkmultiple-kmultiple-sr_kafka2-data
103+
target: /var/lib/kafka/data
67104
depends_on:
68105
- zoo1
69106
- zoo2
@@ -81,6 +118,10 @@ services:
81118
KAFKA_ZOOKEEPER_CONNECT: "zoo1:2181,zoo2:2182,zoo3:2183"
82119
KAFKA_BROKER_ID: 3
83120
KAFKA_LOG4J_LOGGERS: "kafka.controller=INFO,kafka.producer.async.DefaultEventHandler=INFO,state.change.logger=INFO"
121+
volumes:
122+
- type: volume
123+
source: zkmultiple-kmultiple-sr_kafka3-data
124+
target: /var/lib/kafka/data
84125
depends_on:
85126
- zoo1
86127
- zoo2

‎zk-multiple-kafka-multiple.yml

+44-4
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,14 @@
1-
version: '2.1'
2-
1+
version: '3.8'
2+
volumes:
3+
zkmultiple-kmultiple_kafka1-data:
4+
zkmultiple-kmultiple_kafka2-data:
5+
zkmultiple-kmultiple_kafka3-data:
6+
zkmultiple-kmultiple_zoo1-data:
7+
zkmultiple-kmultiple_zoo2-data:
8+
zkmultiple-kmultiple_zoo3-data:
9+
zkmultiple-kmultiple_zoo1-log:
10+
zkmultiple-kmultiple_zoo2-log:
11+
zkmultiple-kmultiple_zoo3-log:
312
services:
413
zoo1:
514
image: confluentinc/cp-zookeeper:6.2.1
@@ -11,7 +20,13 @@ services:
1120
ZOOKEEPER_CLIENT_PORT: 2181
1221
ZOOKEEPER_SERVER_ID: 1
1322
ZOOKEEPER_SERVERS: zoo1:2888:3888;zoo2:2888:3888;zoo3:2888:3888
14-
23+
volumes:
24+
- type: volume
25+
source: zkmultiple-kmultiple_zoo1-data
26+
target: /data
27+
- type: volume
28+
source: zkmultiple-kmultiple_zoo1-log
29+
target: /datalog
1530
zoo2:
1631
image: confluentinc/cp-zookeeper:6.2.1
1732
hostname: zoo2
@@ -22,6 +37,13 @@ services:
2237
ZOOKEEPER_CLIENT_PORT: 2182
2338
ZOOKEEPER_SERVER_ID: 2
2439
ZOOKEEPER_SERVERS: zoo1:2888:3888;zoo2:2888:3888;zoo3:2888:3888
40+
volumes:
41+
- type: volume
42+
source: zkmultiple-kmultiple_zoo2-data
43+
target: /data
44+
- type: volume
45+
source: zkmultiple-kmultiple_zoo2-log
46+
target: /datalog
2547

2648
zoo3:
2749
image: confluentinc/cp-zookeeper:6.2.1
@@ -33,7 +55,13 @@ services:
3355
ZOOKEEPER_CLIENT_PORT: 2183
3456
ZOOKEEPER_SERVER_ID: 3
3557
ZOOKEEPER_SERVERS: zoo1:2888:3888;zoo2:2888:3888;zoo3:2888:3888
36-
58+
volumes:
59+
- type: volume
60+
source: zkmultiple-kmultiple_zoo3-data
61+
target: /data
62+
- type: volume
63+
source: zkmultiple-kmultiple_zoo3-log
64+
target: /datalog
3765

3866

3967
kafka1:
@@ -48,6 +76,10 @@ services:
4876
KAFKA_ZOOKEEPER_CONNECT: "zoo1:2181,zoo2:2182,zoo3:2183"
4977
KAFKA_BROKER_ID: 1
5078
KAFKA_LOG4J_LOGGERS: "kafka.controller=INFO,kafka.producer.async.DefaultEventHandler=INFO,state.change.logger=INFO"
79+
volumes:
80+
- type: volume
81+
source: zkmultiple-kmultiple_kafka1-data
82+
target: /var/lib/kafka/data
5183
depends_on:
5284
- zoo1
5385
- zoo2
@@ -65,6 +97,10 @@ services:
6597
KAFKA_ZOOKEEPER_CONNECT: "zoo1:2181,zoo2:2182,zoo3:2183"
6698
KAFKA_BROKER_ID: 2
6799
KAFKA_LOG4J_LOGGERS: "kafka.controller=INFO,kafka.producer.async.DefaultEventHandler=INFO,state.change.logger=INFO"
100+
volumes:
101+
- type: volume
102+
source: zkmultiple-kmultiple_kafka2-data
103+
target: /var/lib/kafka/data
68104
depends_on:
69105
- zoo1
70106
- zoo2
@@ -82,6 +118,10 @@ services:
82118
KAFKA_ZOOKEEPER_CONNECT: "zoo1:2181,zoo2:2182,zoo3:2183"
83119
KAFKA_BROKER_ID: 3
84120
KAFKA_LOG4J_LOGGERS: "kafka.controller=INFO,kafka.producer.async.DefaultEventHandler=INFO,state.change.logger=INFO"
121+
volumes:
122+
- type: volume
123+
source: zkmultiple-kmultiple_kafka3-data
124+
target: /var/lib/kafka/data
85125
depends_on:
86126
- zoo1
87127
- zoo2

‎zk-multiple-kafka-single.yml

+34-1
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,12 @@
1-
version: '2.1'
1+
version: '3.8'
2+
volumes:
3+
zkmultiple-ksingle_kafka1-data:
4+
zkmultiple-ksingle_zoo1-data:
5+
zkmultiple-ksingle_zoo2-data:
6+
zkmultiple-ksingle_zoo3-data:
7+
zkmultiple-ksingle_zoo1-log:
8+
zkmultiple-ksingle_zoo2-log:
9+
zkmultiple-ksingle_zoo3-log:
210

311
services:
412
zoo1:
@@ -11,6 +19,13 @@ services:
1119
ZOOKEEPER_CLIENT_PORT: 2181
1220
ZOOKEEPER_SERVER_ID: 1
1321
ZOOKEEPER_SERVERS: zoo1:2888:3888;zoo2:2888:3888;zoo3:2888:3888
22+
volumes:
23+
- type: volume
24+
source: zkmultiple-ksingle_zoo1-data
25+
target: /data
26+
- type: volume
27+
source: zkmultiple-ksingle_zoo1-log
28+
target: /datalog
1429

1530
zoo2:
1631
image: confluentinc/cp-zookeeper:6.2.1
@@ -22,6 +37,13 @@ services:
2237
ZOOKEEPER_CLIENT_PORT: 2182
2338
ZOOKEEPER_SERVER_ID: 2
2439
ZOOKEEPER_SERVERS: zoo1:2888:3888;zoo2:2888:3888;zoo3:2888:3888
40+
volumes:
41+
- type: volume
42+
source: zkmultiple-ksingle_zoo2-data
43+
target: /data
44+
- type: volume
45+
source: zkmultiple-ksingle_zoo2-log
46+
target: /datalog
2547

2648
zoo3:
2749
image: confluentinc/cp-zookeeper:6.2.1
@@ -33,6 +55,13 @@ services:
3355
ZOOKEEPER_CLIENT_PORT: 2183
3456
ZOOKEEPER_SERVER_ID: 3
3557
ZOOKEEPER_SERVERS: zoo1:2888:3888;zoo2:2888:3888;zoo3:2888:3888
58+
volumes:
59+
- type: volume
60+
source: zkmultiple-ksingle_zoo3-data
61+
target: /data
62+
- type: volume
63+
source: zkmultiple-ksingle_zoo3-log
64+
target: /datalog
3665

3766

3867
kafka1:
@@ -53,6 +82,10 @@ services:
5382
KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
5483
KAFKA_JMX_PORT: 9999
5584
KAFKA_JMX_HOSTNAME: ${DOCKER_HOST_IP:-127.0.0.1}
85+
volumes:
86+
- type: volume
87+
source: zkmultiple-ksingle_kafka1-data
88+
target: /var/lib/kafka/data
5689
depends_on:
5790
- zoo1
5891
- zoo2

‎zk-single-kafka-multiple.yml

+26-1
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,10 @@
1-
version: '2.1'
1+
version: '3.8'
2+
volumes:
3+
sksingle-kmultiple_kafka1-data:
4+
sksingle-kmultiple_kafka2-data:
5+
sksingle-kmultiple_kafka3-data:
6+
sksingle-kmultiple_zoo1-data:
7+
sksingle-kmultiple_zoo1-log:
28

39
services:
410
zoo1:
@@ -11,6 +17,13 @@ services:
1117
ZOOKEEPER_CLIENT_PORT: 2181
1218
ZOOKEEPER_SERVER_ID: 1
1319
ZOOKEEPER_SERVERS: zoo1:2888:3888
20+
volumes:
21+
- type: volume
22+
source: sksingle-kmultiple_zoo1-data
23+
target: /data
24+
- type: volume
25+
source: sksingle-kmultiple_zoo1-log
26+
target: /datalog
1427

1528

1629
kafka1:
@@ -25,6 +38,10 @@ services:
2538
KAFKA_ZOOKEEPER_CONNECT: "zoo1:2181"
2639
KAFKA_BROKER_ID: 1
2740
KAFKA_LOG4J_LOGGERS: "kafka.controller=INFO,kafka.producer.async.DefaultEventHandler=INFO,state.change.logger=INFO"
41+
volumes:
42+
- type: volume
43+
source: sksingle-kmultiple_kafka1-data
44+
target: /var/lib/kafka/data
2845
depends_on:
2946
- zoo1
3047

@@ -40,6 +57,10 @@ services:
4057
KAFKA_ZOOKEEPER_CONNECT: "zoo1:2181"
4158
KAFKA_BROKER_ID: 2
4259
KAFKA_LOG4J_LOGGERS: "kafka.controller=INFO,kafka.producer.async.DefaultEventHandler=INFO,state.change.logger=INFO"
60+
volumes:
61+
- type: volume
62+
source: sksingle-kmultiple_kafka2-data
63+
target: /var/lib/kafka/data
4364
depends_on:
4465
- zoo1
4566

@@ -56,5 +77,9 @@ services:
5677
KAFKA_ZOOKEEPER_CONNECT: "zoo1:2181"
5778
KAFKA_BROKER_ID: 3
5879
KAFKA_LOG4J_LOGGERS: "kafka.controller=INFO,kafka.producer.async.DefaultEventHandler=INFO,state.change.logger=INFO"
80+
volumes:
81+
- type: volume
82+
source: sksingle-kmultiple_kafka3-data
83+
target: /var/lib/kafka/data
5984
depends_on:
6085
- zoo1

0 commit comments

Comments
 (0)