Memory issue with code base of Feb24 #36
Comments
|
@defanator do you have a test machine where you can run nginx over some days and monitor the memory? Recently the memory of my standby machine filled up again and nginx restarted itself
|
|
@mimugmail, first, nginx never "restarts itself" (perhaps it was OOM killer?) Second - sure, I can set up an instance and leave it running for a few days, but it would be good to have more precise instructions on steps how to reproduce the scenario (system configuration, nginx configuration, any load patterns). Finally, what makes you think that the reason is in nginx? |
|
Hm, you're right. I started a second machine with a mostly empty config and just CRS3 rules. Will keep you updated .. |
|
Ok, first results: With code base of 24th, nginx 1.11.9 and a really basic nginx configuration there's no issue. On the other side is a the same system (OS Debian 8) but with N+, so module is compiled against 1.11.5 and the configuration is quite huge (multile balancers with plenty of stages and around 30 virtual hosts). Last day I just killed the nginx process and didn't start it again. Now the memory blowouts are gone. Next step is to just disable modsecurity on the virtual hosts, but since the config is synced regulary from the master I have to check with my colleagues .. |
|
Hi @mimugmail, Any news yet? |
|
Yesterday at 14:00 I started N+ with 33 virtual hosts, MS3 disabled, but module loaded via global config. No memissue yet. Next test is to enable MS3 again and drop external connections to be sure that there's no external process causing the issue. |
|
@zimmerle Sorry for the delay, I know started nginx with the live config bug it only listens to IP addresses which are bound to the current master, so it's just running and no connecting client can be responsible for any mem issues |
|
Till now there was no memory issue running N+ with MS3 but not listening to any running IP. Issue shoul be back within the next few hours ... |
|
@zimmerle In this time there were only 4 accesses (first three my tests and one "client"/bot/whatever) |
|
@mimugmail, could you please share system logs? Were there any messages from OOM killer? Do you have any Could you please also share the entire nginx ( |
|
@defanator Yep, here's the output [3619581.644080] nginx invoked oom-killer: gfp_mask=0x24280ca(GFP_HIGHUSER_MOVABLE|__GFP_ZERO), nodemask=0-1, order=0, oom_score_adj=0 ps aufx: Can I send you the nginx -T via N+ commercial support and you close the ticket again? Since I don't want to post here very sensitive data, but I also know MS3 in a scenario like this is not officially supported via N+ support. |
|
@mimugmail, is it happening with the "code base of Feb 24"? May I ask you to try the latest If the issue will persist, we will proceed with further investigation. |
|
@defanator Yes, it's still 24th, so I'll update it now to latest v3/master. |
|
@mimugmail, it was recompiled from sources as of Mar, 10 (after |
|
@defanator Ooooh 👍 Wasn't informed by N+ team, great news! |
|
@mimugmail, sure - please share your results here. |
|
@defanator With the package it runs very smoothly 👍 |
|
@defanator Do you know if there's a new verison of the N+ documentation (WAF guide). I switched and had massive blocks for rules 912100, 912110 and 910130. My thought was that with the current code base I don't have to disable certain rule types. |
|
@mimugmail, we're working on the documentation. You're correct, current codebase works with OWASP CRS v3.0.0 as is (no manual removing any of pre-configured rules is required). Please feel free to use N+ support channel for such kind of questions, leaving this thread focused on the memory issue - thank you! |
|
@defanator Yep, so this issue seems to be fixed now, you can close this one. Thanks for your efforts 👍 |
Hi,
I just ran Nginx+ with the code base of Feb24 on my backup machine. It just listen on port 80 for one IP and not in production. Now my monitoring system told me about full memory and indeed, all my 64GB RAM were gone and nginx restarted itself:
2017/03/02 14:29:49 [notice] 738#738: signal 17 (SIGCHLD) received
2017/03/02 14:29:49 [alert] 738#738: worker process 22623 exited on signal 9
2017/03/02 14:29:49 [notice] 738#738: start worker process 26684
2017/03/02 14:29:49 [notice] 738#738: signal 29 (SIGIO) received
2017/03/02 14:29:49 [notice] 738#738: signal 17 (SIGCHLD) received
2017/03/02 14:29:49 [alert] 738#738: worker process 22624 exited on signal 9
2017/03/02 14:29:49 [alert] 738#738: worker process 22625 exited on signal 9
2017/03/02 14:29:49 [alert] 738#738: worker process 22627 exited on signal 9
2017/03/02 14:29:49 [notice] 738#738: start worker process 26685
2017/03/02 14:29:49 [notice] 738#738: start worker process 26686
2017/03/02 14:29:49 [notice] 738#738: start worker process 26687
2017/03/02 14:29:49 [notice] 738#738: signal 29 (SIGIO) received
2017/03/02 14:29:51 [notice] 738#738: signal 17 (SIGCHLD) received
2017/03/02 14:29:51 [alert] 738#738: worker process 22643 exited on signal 9
2017/03/02 14:29:51 [notice] 738#738: start worker process 26688
2017/03/02 14:29:51 [notice] 738#738: signal 29 (SIGIO) received
I grabbed the logs of nginx but the last connect was at 11:39:02.
I had the same issue two days ago after benchmarking with "ab". I thought this was the cause so I restarted nginx, did a benchmark again and waited. Now after two days I have the same memory error again.
Sadly I cannot reproduce it.
Someone running the same code base and seeing this issue?
The text was updated successfully, but these errors were encountered: