Skip to content

[RabbitMQ] High resource usage in AmqpConsumer::receiveBasicGet() #116

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
dkarlovi opened this issue Jun 16, 2017 · 8 comments
Closed

[RabbitMQ] High resource usage in AmqpConsumer::receiveBasicGet() #116

dkarlovi opened this issue Jun 16, 2017 · 8 comments

Comments

@dkarlovi
Copy link
Contributor

dkarlovi commented Jun 16, 2017

TLDR recieveBasicGet() (the default receive method) is using a lot of CPU and other resources as it's calling AMQPQueue::get() as fast as it can, unbound. Should be able to switch receive method to consume and/or fix this one.

I'm switching my Dockerized app from using php:7.1-alpine to alpine:3.6 and installing php7* packages required (Dockerfile), Alpine now supports PHP 7.1 and it should make my process somewhat more streamlined.

Problem is, Enqueue seems to work WAY worse in the new environment. What I mean is, resource usage goes through the roof as soon as I start a consumer (note, there are no messages produced or consumed here, it's just idle wait). When using RabbitMQ's web management console, I see my consumer's connection doing 500KB/s traffic (as I said, no messages produced, it's waiting for something to happen).

I cannot quite pinpoint why this happens as the PHP version is the same, PHP modules are the same, codebase (including Enqueue) is exactly the same, etc. Obviously, this means I shouldn't report this bug here. :)

What does stand out is the code in recieveBasicGet(), it's basically calling a non-blocking AMQPQueue::get() without any restrictions. I'm guessing this is the core cause of this problem as I'm able to avoid this completely if I add usleep(10000) to recieveBasicGet() or switch the receive method to basic_consume (both of which I'm unable to do without changing the vendor source).

@dkarlovi
Copy link
Contributor Author

Sorry, just realized I'm wrong: I'm using enqueue/amqp-ext 0.4.11 in the original branch, but 0.4.13 in my new branch, this is then a regression, I'd say. AmqpConsumer::receive() looks way different between those versions.

@dkarlovi
Copy link
Contributor Author

Confirmed, 0.4.12 and 0.4.13 are currently broken for me, 0.4.11 works as expected.

@makasim
Copy link
Member

makasim commented Jun 17, 2017

We had been using basic.consume method til we found a bug php-amqp/php-amqp#281 in AQMP extension. The bug was a serious one cuz you can get a message which is not yours (from another queue). That's why I fixed it by making basic.get a default solution.

A day later (It was really fast) the bug was fixed in the extension.

So now we have to strategies available:

  • basic_get - is the default one
  • and basic_consume which could be used only with the extension 1.9.1 and higher.

The receive method could be controlled by recive_method option:

<?php

new \Enqueue\AmqpExt\ AmqpConnectionFactory([
    'receive_method' => 'basic_get', // or 'basic_conume',
]);

@makasim
Copy link
Member

makasim commented Jun 17, 2017

@dkarlovi if you want to use basic_conume you have to set the receive_method option to basic_conume and make sure you use AMQP from the master. The fix has not been released yet.

@makasim
Copy link
Member

makasim commented Jun 17, 2017

What does stand out is the code in recieveBasicGet(), it's basically calling a non-blocking AMQPQueue::get() without any restrictions. I'm guessing this is the core cause of this problem as I'm able to avoid this completely if I add usleep(10000) to recieveBasicGet() or switch the receive method to basic_consume (both of which I'm unable to do without changing the vendor source).

That's indeed a good idea to add a usleep there

@dkarlovi
Copy link
Contributor Author

@makasim how do I enable basic_consume if using the Symfony bundle? I was looking through the code and it's read from factory, but I'm unable to specify it in config (not allowed by bundle's config validator).

@makasim
Copy link
Member

makasim commented Jun 17, 2017

Ups... I forgot to add the option to the factory.

@dkarlovi
Copy link
Contributor Author

So the problem is: you could get a message from the wrong queue? This doesn't seem to affect me as I'm using only one queue currently, this might change if I go to implement #65, but not relevant for now.

I've fixed my composer constraint to <= 0.4.11 so it's not a deal breaker for me currently, can wait for a proper fix here in both Enqueue and php-amqp.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants