3.13.6 Node memory spike when polling for 10K messages over the HTTP API

[radeth]

Describe the bug

Hi RabbitMQ Team,
I'm experiencing an issue with RabbitMQ deployed using the Cluster Operator version v0.15.0 on a Kubernetes cluster running in GCP. After upgrading RabbitMQ from version 3.10.2 to 3.13.6, the system started crashing under certain conditions, and I’m seeking your guidance to resolve this.
The error in the logs indicates:
[os_mon] memory supervisor port (memsup): Erlang has closed.
The crashes occur when I attempt to fetch more than 10,000 messages with the automatic ack option enabled, particularly when the queue size exceeds 100MB. However, the issue also happens sporadically without a clearly identifiable trigger. Interestingly, monitoring the RAM usage does not show any significant increase prior to the crash. When fetching fewer messages (below the threshold that causes a crash), the memory usage appears normal and aligns with expected behavior.
This problem was not present in the previous RabbitMQ version 3.10.2 and only started occurring after the upgrade to 3.13.6. My deployment uses the RabbitMQ Cluster Operator v0.15.0, and this setup has been stable prior to the upgrade.
Is this a known issue or compatibility problem between RabbitMQ 3.13.6 and the Cluster Operator v0.15.0? Could the error be related to the Erlang runtime, and are there any specific configurations that might prevent the memsup crash? If additional logs or diagnostic details would help, I am happy to provide them.
Thank you for your time and support in resolving this issue.
Best regards,

Reproduction steps

1. Upgrade rabbit in GKE to version 3.13.6
2. Read large amount of lagre messages, more than 100mb with automatic ack mode

Expected behavior

Rabbit should not crash

Additional context

No response

[mkuratczyk]

First of all:

• cluster operator 0.15.0 is 4.5 years old alpha version of the operator. There are no "known issues" with this version because no-one is using it and no-one is testing it. I'd recommend upgrading ASAP.
• RabbitMQ 3.13.6 is out of community support now (upgrade to 4.0 or purchase long term support for 3.13). Even if this discussion leads to a bug being identified and fixed, that fix will only be included in 4.0 (and perhaps back-ported to the 3.13 LTS version)

There's not enough details to suggest much. Most likely this particular operation uses a bit more memory than it used to but you didn't even mention what type of queue that is.

If you want further assistance, provide clear reproduction steps and your deployment YAML, so we can see your memory configuration. You can also play around with https://perftest.rabbitmq.com/ to reproduce and test different settings. If you end up using perf-test to reproduce, the commands will give us all the details we need (queue type and arguments, message size, prefetch size and so on and so forth).

[michaelklishin]

If "crash under certain conditions" means a node (pod) termination, the first suspect is usually an OOM killer, so start with Reasoning About Memory Use.

If it means a logged exception (a "crash" in Erlang terms), then we cannot suggest anything without seeing the logs (on 4.0.5) with at few hours of log entries before and after the event in question.

[radeth]

A few things I wanted to clarify:

1. We are using operator version 2.10.0.
2. Queue type: classic, but I also tested on quorum, durable: true, storage_version: 1.
3. On the RAM usage chart, when I use a lot of RAM but not enough to cause a crash, I can see the memory usage, but not before the crash.

[radeth]

The value on charts is on bytes. Should i do this test on local machine or on GKE ?

Here is my yaml confing for gke

apiVersion: rabbitmq.com/v1beta1
kind: RabbitmqCluster
metadata:
  name: dev-rabbitmqcluster
  namespace: rabbitmq-system
spec:
  replicas: 1
  resources:
    requests:
      cpu: 1
      memory: 4Gi
    limits:
      cpu: 1
      memory: 4Gi
  persistence:
    storageClassName: rabbitmq-sc-cmek 
  override:
    statefulSet:
      spec:
        template:
          spec:
            containers:
            - name: rabbitmq  
              image: rabbitmq:4.0.5-management
              securityContext:
                privileged: true
            initContainers:
            - name: setup-container
              image: rabbitmq:4.0.5-management

  rabbitmq:
          additionalConfig: |
                  vm_memory_high_watermark.absolute = 2460MiB
                  default_user = user 
                  default_pass = pass
                  default_user_tags.administrator = true
          additionalPlugins:
            - rabbitmq_shovel_management
  service:
    annotations: 
      cloud.google.com/load-balancer-type: "Internal"
    type: LoadBalancer            

[mkuratczyk]

As long as you can trigger a significant memory spike, it doesn't really matter where you try it.

[radeth]

I used pref test on my local kubernetes and i restore this crash on my machine. I have used this command
kubectl -n rabbitmq-system run perf-test --image=pivotalrabbitmq/perf-test -- --uri amqp://user:pass@dev-rabbitmqcluster:5672 -y 0 --queue preftest --json-body --size 25600 -C 10000 --rate 100
To produce messages, next i use read messesges with automatic ack option on menagment gui.

[mkuratczyk]

Ok, I can reproduce. For reference, here's an easy way to trigger this locally:

# publish
perf-test -y 0 --json-body --size 25600 -C 10000 -c 100 --queue test
# consume
curl -u guest:guest 'localhost:15672/api/queues/%2F/test/get' -X POST -d '{"count":10000,"ackmode":"ack_requeue_false","encoding":"auto"}'

The direct culprit is the auto encoding. If you use base64, there's only a reasonable spike to serve the request. Alterntively, adding "truncate": 1234 also solves the problem. In other words, it's a combination of message payload (valid UTF8), auto encoding, the size of messages and the number of messages requested.

To be honest, my first instinct is simply to refuse to serve more than 100 messages or so, through the /get endpoint. I don't think there are valid reasons to need more than that. The HTTP API is not meant to be used as the primary message publishing/consumption mechanism, but mostly for some debugging and occasional usage.

What's your use case for GETting 10000 messages like that?

[michaelklishin]

@radeth there is no justification for using a polling consumer over the HTTP API. The docs are flashing warnings in red recommending against the use of such feature.

rabbitmqadmin v2 explicitly warns against that in red, this is a feature that's ONLY mean to be used in development and integration tests, no exceptions.

The core team should not spend any more time on this. Use a regular consumer, a small standalone tool or from a REPL (e.g. with Python, JS, Java) or something like that.

[radeth]

I think I'm dealing with two problems here that I mistakenly identified as one. Downloading messages via the API is not recommended and thanks for explaining this problem, but I found something else. After the update I thought rabbitmq was resetting itself, but after analyzing I managed to understand what was happening. GKE wants to reset rabbitmq statefulset
GKE StatefulSet update
test-rabbitmqcluster-server in app-platform-01 rabbitmq-system | Feb 7, 06:48
then what happens in the logs happens
reversed_text_payloads (1).txt

Then this happens
GKE StatefulSet update
test-rabbitmqcluster-server in app-platform-01 rabbitmq-system | Feb 7, 06:49
And rabbit works fine. Can you help me understand these logs? Or should we move this to another discussion?

[michaelklishin]

@radeth what in the phrase "RabbitMQ 3.13 is out of community support" needs clarification?

Using one discussions for two questions is not appreciated neither by our team nor by those who will land here from Google in the future.

[michaelklishin]

According to

BOOT FAILED
===========
Exception during startup:
error:badarg
2025-02-07 05:49:20.703922+00:00 [info] <0.254.0> 3 transient queues from node 'rabbit@test-rabbitmqcluster-server-0.test-rabbitmqcluster-nodes.rabbitmq-system' deleted in 0.086093s�[0m
 ets:lookup/2
 args: [rabbit_registry,{exchange,direct}]
 rabbit_registry:lookup_module/2, line 57
 rabbit_exchange:type_to_module/1, line 505
 rabbit_exchange:serialise_events/1, line 84
 rabbit_exchange:serial/1, line 89
 rabbit_binding:-process_deletions/1-fun-0-/2, line 436
�[38;5;160m2025-02-07 05:49:20.704261+00:00 [error] <0.254.0> �[0m
�[38;5;160m2025-02-07 05:49:20.704261+00:00 [error] <0.254.0> BOOT FAILED�[0m
�[38;5;160m2025-02-07 05:49:20.704261+00:00 [error] <0.254.0> ===========�[0m
�[38;5;160m2025-02-07 05:49:20.704261+00:00 [error] <0.254.0> Exception during startup:�[0m
�[38;5;160m2025-02-07 05:49:20.704261+00:00 [error] <0.254.0> �[0m
�[38;5;160m2025-02-07 05:49:20.704261+00:00 [error] <0.254.0> error:badarg�[0m
�[38;5;160m2025-02-07 05:49:20.704261+00:00 [error] <0.254.0> �[0m
�[38;5;160m2025-02-07 05:49:20.704261+00:00 [error] <0.254.0> ets:lookup/2�[0m
�[38;5;160m2025-02-07 05:49:20.704261+00:00 [error] <0.254.0> args: [rabbit_registry,{exchange,direct}]�[0m
�[38;5;160m2025-02-07 05:49:20.704261+00:00 [error] <0.254.0> rabbit_registry:lookup_module/2, line 57�[0m

the internal registry cannot find the module for direct exchanges (a built-in type).

This usually means that the metadata store is not yet initialized by the time something (usually a client but can be a CLI or an HTTP API operation) tries to perform an operation that assumes that the exchanges table exists and is seeded.

4.0.6 has shipped with two health checks specifically for such scenarios.

#13153 is another improvement that will ship in 4.0.x.

That's about as much as can be said with this log snippet from one node.

[michaelklishin]

@radeth you repeatedly ignore the fact that 3.13 is out of community support and that you must first upgrade to 4.x. FYI, you won't be able to start any new discussions in this org for a month as a warning.