4.0: A question about quorum queue disc space footprint

[alexff91]

Describe the bug

During the migration from version 3.13.7 to 4.0.5, we faced an issue that our queues with classic mirroring were no longer supported, so we recreated queues as quorum queues.

Once we made this switch, our previous cluster setup, consisting of 3 nodes with 4 GB of disk space in K8s, started experiencing problems with the constant growth of the disk space used.

At first, we increased it to 6 GB, but we see that it continues to constantly grow. What we've noticed is that the segment's folders are quite strange and they contain old data.

This could be related to the discussion #12147

Current cluster state:
RabbitMQ 4.0.5

Erlang 26.2.5

Reproduction steps

1. Run cluster with 3 nodes docker.io/bitnami/rabbitmq:3.13.7-debian-12-r0 and classic mirroring queue
2. Recreate queues to quorum queues
3. Update the docker image to the latest RabbitMQ version image: docker.io/bitnami/rabbitmq:4.0.5-debian-12-r0
4. Produce/ consume messages in quorum queues.
5. Disk space continues to grow even if the messages are acknowledged.

Example of the folder with segment file:
2F_SCD7P1LOMCM21CN.zip

Expected behavior

Disk space is used normally and after message acknowledgment, the messages are deleted from segments/ old segments are deleted/ truncated.

Additional context

Our current filesystem looks like this:
$ ls -la
total 11312
drwxrwsr-x 33 1001 1001 4096 Jan 21 11:40 .
drwxrwsr-x 3 1001 1001 4096 Mar 25 2022 ..
-rw-r--r-- 1 1001 1001 11415458 Jan 21 11:58 00000093.wal
drwxrwsr-x 4 1001 1001 4096 Jan 18 00:04 2F_ELIB5TTE5O455YO
drwxrwsr-x 4 1001 1001 4096 Jan 18 00:01 2F_ELIKG4EUNZTCOVJ
drwxrwsr-x 4 1001 1001 4096 Dec 30 16:54 2F_EUVK6VVMPWC74YW
drwxrwsr-x 4 1001 1001 4096 Dec 30 16:54 2F_PROJ5RUYKCS3H0Q
drwxrwsr-x 4 1001 1001 4096 Jan 21 04:43 2F_PROQ23BGRRNRM3S
drwxrwsr-x 4 1001 1001 4096 Jan 21 11:55 2F_PUB3OPTTYUBLXGG
drwxrwsr-x 4 1001 1001 4096 Jan 16 08:55 2F_PUB7VTUEUDLK4K8
drwxrwsr-x 4 1001 1001 4096 Dec 30 16:54 2F_PUBD54PTDKDSQZ7
drwxrwsr-x 4 1001 1001 4096 Jan 21 04:35 2F_PUBFMDOYF2TQ8U9
drwxrwsr-x 4 1001 1001 4096 Jan 21 07:00 2F_PUBITK4WBSHBRZV
drwxrwsr-x 4 1001 1001 4096 Jan 21 07:00 2F_PUBJMGUJAUCWO6U
drwxrwsr-x 4 1001 1001 4096 Jan 21 10:55 2F_PUBKQ1QLGBRW6I2
drwxrwsr-x 4 1001 1001 4096 Dec 30 16:54 2F_PUBKYEXQU3IAYBE
drwxrwsr-x 4 1001 1001 4096 Jan 21 04:50 2F_PUBV1EWQ5B3EXGN
drwxrwsr-x 4 1001 1001 4096 Jan 21 05:05 2F_PUBVGPDJ04QOENZ
drwxrwsr-x 4 1001 1001 4096 Dec 30 16:54 2F_PUBXQQD181VE2CQ
drwxrwsr-x 4 1001 1001 4096 Dec 30 16:54 2F_PUBZRRWG0G3CRWH
drwxrwsr-x 4 1001 1001 4096 Dec 30 16:54 2F_SCD1RA99OPWGYUT
drwxrwsr-x 4 1001 1001 4096 Jan 20 23:35 2F_SCD2SKORYN2RCR0
drwxrwsr-x 4 1001 1001 4096 Dec 30 16:54 2F_SCD3TDF5R38Y0SC
drwxrwsr-x 4 1001 1001 4096 Jan 21 04:40 2F_SCD7P1LOMCM21CN
drwxrwsr-x 4 1001 1001 4096 Jan 21 07:00 2F_SCDE8TOB00R5U4X
drwxrwsr-x 4 1001 1001 4096 Jan 16 02:15 2F_SCDFZ5DGPYVPOUS
drwxrwsr-x 4 1001 1001 4096 Dec 30 16:54 2F_SCDG4K4E6JV55JY
drwxrwsr-x 4 1001 1001 4096 Dec 30 16:54 2F_SCDZNH852K5Y5EO
drwxr-sr-x 4 1001 1001 4096 Jan 20 23:30 2F_TES8OZX9NGLSYJ2
drwxrwsr-x 4 1001 1001 4096 Dec 30 16:54 2F_WHO6Y5R4GR7OO6Z
drwxrwsr-x 4 1001 1001 4096 Dec 30 16:54 2F_WIW0DE8WSG36F92
drwxrwsr-x 4 1001 1001 4096 Jan 16 02:15 2F_WIW9NQ5RYPA36JU
drwxrwsr-x 4 1001 1001 4096 Dec 30 16:54 2F_WIWIK6RZ4OGUYYS
drwxrwsr-x 4 1001 1001 4096 Jan 16 08:55 2F_WIWJ9NUGNNR2IBH
-rw-rw-r-- 1 1001 1001 13547 Jan 21 11:58 meta.dets
-rw-rw-r-- 1 1001 1001 9524 Jan 20 22:19 names.dets

we've noticed that config is not changing

$ ls -la 2F_PUB7VTUEUDLK4K8
total 8800180
drwxrwsr-x 4 1001 1001 4096 Jan 16 08:55 .
drwxrwsr-x 33 1001 1001 4096 Jan 21 11:40 ..
-rw-rw-r-- 1 1001 1001 1488441126 Jan 6 23:00 00000002.segment
-rw-rw-r-- 1 1001 1001 4460219750 Jan 16 08:55 00000003.segment
-rw-r--r-- 1 1001 1001 3062721156 Jan 21 11:40 00000004.segment
drwxrwsr-x 2 1001 1001 4096 Dec 30 15:51 checkpoints
-rw-rw-r-- 1 1001 1001 2666 Dec 30 15:51 config
drwxrwsr-x 3 1001 1001 4096 Dec 30 16:54 snapshots
$ ls -la 2F_PUBKQ1QLGBRW6I2
total 32
drwxrwsr-x 4 1001 1001 4096 Jan 21 10:55 .
drwxrwsr-x 33 1001 1001 4096 Jan 21 11:40 ..
-rw-r--r-- 1 1001 1001 135828 Jan 21 11:40 00000009.segment
drwxrwsr-x 2 1001 1001 4096 Jan 21 10:55 checkpoints
-rw-rw-r-- 1 1001 1001 3208 Dec 30 14:14 config
drwxrwsr-x 3 1001 1001 4096 Jan 21 10:55 snapshots
$

RabbitMQ config:

rabbitmq.conf: |-
## Clustering
cluster_formation.peer_discovery_backend = rabbit_peer_discovery_k8s
cluster_formation.k8s.host = kubernetes.default.svc.cluster.local
cluster_formation.node_cleanup.interval = 10
cluster_formation.node_cleanup.only_log_warning = true
cluster_partition_handling = autoheal
# queue master locator
queue_master_locator=min-masters
# enable guest user
loopback_users.guest = false
vm_memory_high_watermark.absolute = 1500MiB
management.tcp.port = 15672
management.tcp.ip = 0.0.0.0

[kjnilsson]

What you are describing seems completely normal. QQ log truncation isn't something that happens all the time, it may leave data behind for quite some time before it is deleted. 6GB is much to small a disk for modern RabbitMQ with quorum queues. I would at least 10x that.

See the documentation for more details: https://www.rabbitmq.com/docs/quorum-queue

[michaelklishin]

@alexff91 in the quorum queue guide, there is a dedicated section that demonstrates exactly the same "seesaw pattern" and explains why specifically the QQ on disk footprint follows it.

Furthermore, the Deployment Guidelines mentions that you probably want to use disk several times larger than the amount of memory a node is allowed to use. In your case, the ratio is close to 1:1.

#13103 demonstrates something completely different, per reporter's own words the leader node behaves exactly as expected, their question is about why the nodes hosting the followers have a different behavior.

#12147 can be relevant or not, the solution suggested there (which is also documented) won't help if you barely have enough disk space for a couple of segment files with all defaults.

[alexff91]

hello @michaelklishin ,
Thank you for your reply.

What is bothering us - is that the segment has not been reclaimed for 15 days, is it normal and expected behavior?

[michaelklishin]

@alexff91 if some of your quorum queues sit there without any consumers or any activity at all, their segment files won't change. Unacknowledged deliveries is another common cause for this behavior but modern versions have a delivery acknowledgement timeout as a protection mechanism.

I know nothing about the workload in question and we do not guess in this community.