Segfaults in kern.log

[GNU-Plus-Windows-User]

Describe the bug

ModSec has segfaults which causes CrowdSec Nginx Bouncer to stop working.

Logs and dumps

Feb  2 12:10:43 redacted kernel: [991891.066133] nginx[2386946]: segfault at 8 ip 00007f31d846001e sp 00007ffc5832da30 error 4 in libmodsecurity.so.3.0.8[7f31d8360000+114000]
Feb  2 12:10:43 redacted kernel: [991891.066146] Code: 83 c4 08 4c 89 e0 5d 41 5c c3 66 90 f3 0f 1e fa 41 57 41 56 41 55 4c 63 ee 41 54 49 89 fc 55 53 48 89 d3 48 81 ec b8 00 00 00 <4c> 8b 42 08 64 48 8b 04 25 28 00 00 00 48 89 84 24 a8 00 00 00 31
Feb  2 12:10:43 redacted kernel: [991891.109427] nginx[2422833]: segfault at 8 ip 00007f31d846001e sp 00007ffc5832da30 error 4 in libmodsecurity.so.3.0.8[7f31d8360000+114000]
Feb  2 12:10:43 redacted kernel: [991891.109441] Code: 83 c4 08 4c 89 e0 5d 41 5c c3 66 90 f3 0f 1e fa 41 57 41 56 41 55 4c 63 ee 41 54 49 89 fc 55 53 48 89 d3 48 81 ec b8 00 00 00 <4c> 8b 42 08 64 48 8b 04 25 28 00 00 00 48 89 84 24 a8 00 00 00 31

To Reproduce

Steps to reproduce the behavior:

1. Install ModSec 3.0.8 with CRS
2. Install Lua and CrowdSec Nginx Bouncer
3. wait for segfaults to show up in logs

Expected behavior

Segfaults should not be showing up in logs

Server

• ModSec v3.0.8
• Nginx 1.22.1
• Ubuntu 22.04 Proxmox LXC container
• Lua 5.1.5
• CrowdSec Nginx Bouncer

Rule Set

• CRS 3.3.4 (Sep 21, 2022)

Additional context

CrowdSec Nginx Bouncer will stop receiving decisions (IP Bans) if seg faults occur. I made a ticket on the CrowdSec Discord regarding the issue, they provided a fix for that but, if a segfault happens around the same time the bouncer is querying the CrowdSec Agent, the CrowdSec Bouncer will stop working.

Restarting Nginx temporarily fixes the issue

[martinhsv]

Hello @GNU-Plus-Windows-User ,

"Lua 5.1.5"

ModSecurity documentation states: "Note : ModSecurity v3 is compatible with Lua 5.2+."

[GNU-Plus-Windows-User]

@martinhsv I've updated Lua to 5.2 and I'm still seeing segfaults in my logs, I've also tried disabling Lua and CrowdSec Nginx Bouncer altogether but there is still segfaults.

[martinhsv]

Can you produce a core dump, and at least supply the backtrace?

[GNU-Plus-Windows-User]

@martinhsv I've generated a core dump, but I couldn't figure out how to get a backtrace.
Is there anywhere I can safely share the coredump, I don't want to post it publicly on the internet?

[GNU-Plus-Windows-User]

@martinhsv I've managed to generate a backtrace but I don't know if I did it correctly

Core was generated by `nginx: worker process                           '.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x00007f65815f3a4e in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::length (this=<optimized out>) at /usr/include/c++/11/bits/basic_string.h:927
927           { return _M_string_length; }

I still have that coredump if you want it, but I don't want to post it publicly.

[martinhsv]

Hi @GNU-Plus-Windows-User ,

That looks like the start of a backtrace. Was there more output? Usually a backtrace will show that #0 and then move up the call stack with entries for #1, #2, etc.

[GNU-Plus-Windows-User]

@martinhsv Sorry about that, I didn't read the documentation correctly.
This should be what you're after

Core was generated by `nginx: worker process                           '.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x00007f65815f3a4e in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::length (this=<optimized out>) at /usr/include/c++/11/bits/basic_string.h:927
927           { return _M_string_length; }
[Current thread is 1 (Thread 0x7f65816b3740 (LWP 1342))]
(gdb) backtrace
#0  0x00007f65815f3a4e in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::length (this=<optimized out>) at /usr/include/c++/11/bits/basic_string.h:927
#1  modsecurity::utils::string::limitTo (amount=amount@entry=200, str=<error reading variable: Cannot access memory at address 0x8>) at utils/string.cc:100
#2  0x00007f658158ea88 in modsecurity::RuleMessage::_details[abi:cxx11](modsecurity::RuleMessage const*) (rm=0x55ce45e32160) at /usr/include/c++/11/bits/shared_ptr_base.h:1295
#3  0x00007f658158f61b in modsecurity::RuleMessage::log[abi:cxx11](modsecurity::RuleMessage const*, int, int) (rm=0x55ce45e32160, props=props@entry=0, code=code@entry=-1) at rule_message.cc:88
#4  0x00007f6581577abf in modsecurity::RuleMessage::log[abi:cxx11](modsecurity::RuleMessage const*, int) (props=0, rm=<optimized out>) at ../headers/modsecurity/rule_message.h:177
#5  modsecurity::RuleMessage::log[abi:cxx11]() (this=<optimized out>) at ../headers/modsecurity/rule_message.h:162
#6  modsecurity::ModSecurity::serverLog (this=0x55ce37ac6d00, data=0x55ce463bedb0, rm=std::shared_ptr<modsecurity::RuleMessage> (use count 5, weak count 0) = {...}) at modsecurity.cc:205
#7  0x00007f658155e06c in modsecurity::Transaction::serverLog (this=this@entry=0x55ce4644c660, rm=std::shared_ptr<modsecurity::RuleMessage> (use count 5, weak count 0) = {...}) at transaction.cc:1858
#8  0x00007f6581584cbd in modsecurity::RuleWithActions::performLogging (this=this@entry=0x55ce397f1000, trans=trans@entry=0x55ce4644c660,
    ruleMessage=std::shared_ptr<modsecurity::RuleMessage> (use count 5, weak count 0) = {...}, lastLog=lastLog@entry=true, chainedParentNull=chainedParentNull@entry=true) at rule_with_actions.cc:521
#9  0x00007f658158b030 in modsecurity::RuleWithOperator::evaluate (this=0x55ce397f1000, trans=<optimized out>, ruleMessage=std::shared_ptr<modsecurity::RuleMessage> (use count 5, weak count 0) = {...})
    at rule_with_operator.cc:372
#10 0x00007f65815851d5 in modsecurity::RuleWithActions::evaluate (this=0x55ce397f1000, transaction=0x55ce4644c660) at rule_with_actions.cc:177
#11 0x00007f658157a4bd in modsecurity::RulesSet::evaluate (this=<optimized out>, phase=phase@entry=5, t=t@entry=0x55ce4644c660) at rules_set.cc:210
#12 0x00007f6581562c70 in modsecurity::Transaction::processResponseBody (this=0x55ce4644c660) at transaction.cc:1264
#13 0x00007f65816a5cea in ngx_http_modsecurity_body_filter (r=0x55ce46424f50, in=0x55ce46425ed0) at ../ModSecurity-nginx/src/ngx_http_modsecurity_body_filter.c:161
#14 0x000055ce378e16d1 in ngx_output_chain ()
#15 0x000055ce37959cf7 in ?? ()
#16 0x000055ce3791213e in ngx_http_output_filter ()
#17 0x000055ce3799785d in ?? ()
#18 0x000055ce37923f82 in ngx_http_finalize_request ()
#19 0x000055ce3791726f in ngx_http_core_rewrite_phase ()
#20 0x000055ce3790fd9d in ngx_http_core_run_phases ()
#21 0x000055ce37917a6e in ngx_http_internal_redirect ()
#22 0x000055ce37921384 in ngx_http_special_response_handler ()
#23 0x000055ce37923f82 in ngx_http_finalize_request ()
#24 0x000055ce37924735 in ngx_http_process_request_uri ()
#25 0x000055ce379259c5 in ?? ()
#26 0x000055ce379263e9 in ?? ()
#27 0x000055ce379075ab in ?? ()
#28 0x000055ce378fde99 in ngx_process_events_and_timers ()
#29 0x000055ce3790429d in ?? ()
#30 0x000055ce378f9e08 in ngx_spawn_process ()
#31 0x000055ce378fb348 in ?? ()
#32 0x000055ce3790303b in ngx_master_process_cycle ()
#33 0x000055ce378d5c37 in main ()

[martinhsv]

@GNU-Plus-Windows-User

Yes, that's it. Thanks.

One point to clarify: You mentioned that you are using ModSecurity v3.0.8. Is that the official, published version from last September? Or is that a build from more recent v3/master (which would also report its version as v3.0.8)?

[GNU-Plus-Windows-User]

@martinhsv I followed this guide to compile ModSecurity for Nginx https://www.nginx.com/blog/compiling-and-installing-modsecurity-for-open-source-nginx/
I assume this guide uses the latest published release but I'm not very knowledgeable with github

[martinhsv]

Thanks. That implies that you were using a later v3/master.

There were some recent changes to the implicated function -- although I don't immediately see how those changes could have caused a segfault.

If you are willing to do a couple of experiments, it might be interesting to know if:

During step 3 ("Download and Compile the ModSecurity 3.0 Source Code") ...

• during substep 1, use this git clone instead: git clone -b v3/master --single-branch https://github.com/SpiderLabs/ModSecurity
• during substep 2, immediately after 'cd ModSecurity' add this git checkout v3.0.8

Repeat the experiment above, but instead of the 'git checkout v3.0.8' specified above, try it with each of these:

• git checkout 62ec4edc4258971deec677b4f5e6bda188d27f26
• git checkout dabf79eec251bfcf2c24c8b19295d40ef16ce2be

Under which (if any) of those three tests the problem still occurs, could be very useful information.

[GNU-Plus-Windows-User]

@martinhsv I've recompiled ModSec using git checkout v3.0.8, I'll test all 3 and let you know if anything changes.

[martinhsv]

By the way, what compiler and version are you using?

[GNU-Plus-Windows-User]

@martinhsv I just finished all 3 tests, all of them had segfaults.
Regarding what compiler I'm using and version, I'm using whatever the Nginx guide uses along with the latest packages for Ubuntu 22.04.
The installed version of the packages listed on the Nginx guide is

• apt-utils: 2.4.8
• autoconf: 2.71
• automake: 1.16.5
• build-essentials: 12.9
• git: 2.34.1
• libcurl4-openssl-dev: 7.81.0
• libgeoip-dev: 1.6.12
• liblmdb-dev: 0.9.24
• libpcre++-dev: 0.95
• libtool: 2.4.6
• libxml2-dev: 2.9.13
• libyajl-dev: 2.1.0
• pkgconf: 1.8.0
• wget: 1.21.2
• zlib1g-dev: 1.2.11

[martinhsv]

Hi @GNU-Plus-Windows-User ,

In your original report, you mentioned your nginx version as Nginx 1.22.1.

I don't know that this is what is causing your problem, but: as of nginx 1.21.5, nginx has changed to use pcre2 by default, whereas the default in ModSecurity is pcre1.

To manage this particular difference, you need to do one of three things:

1. use an earlier version of nginx (it looks like the version of nginx available via apt in Ubuntu 22.04 is 1.18.0)
2. use a >= 1.21.5 version of nginx, use the default for ModSecurity, but make sure you build the connector with the --without-pcre2 flag (Add support for PCRE2 #2668 , Module compilation error with NGINX 1.21.5 ModSecurity-nginx#261 (comment))
3. use a >= 1.21.5 version of nginx, use the default for building the connector, but make sure you use the --with-pcre2 flag during the configure step for building ModSecurity itself (Support PCRE2 #2719).

[GNU-Plus-Windows-User]

@martinhsv I've recompiled ModSec using option 2 --without-pcre2 for the Nginx Connector, but I still encountered segfaults. I tried to do option 3 using with-prce2 but it compailed about not being able to find prce2, I using this repository for Lua and Nginx 1.22.1 https://ppa.launchpadcontent.net/ondrej/nginx/ubuntu/, not sure if it's relevant but thought I'd mention just in case.

[airween]

I tried to do option 3 using with-prce2 but it compailed about not being able to find prce2,

then you should install pcre2-dev package...

... I using this repository for Lua and Nginx 1.22.1 https://ppa.launchpadcontent.net/ondrej/nginx/ubuntu/

I don't remember now where did we discuss, but this repository has some problem (the provided Nginx package).

Btw just two suggestions:

• with the compiled libmodsecurity3, you should try ftwrunner, it's a regression test tool for CRS, but embeds libmodsecurity3, similar as Nginx (to reproduce the issue)
• you should try this repository, we supported Ubuntu 22.04 (for final solution - but this package repository provides the official Nginx version, from Ubuntu)

[GNU-Plus-Windows-User]

@martinhsv I've tried all 3 of your suggestions and I have segfaults with all 3 of them

@airween Thank you for giving me the package name, I couldn't find it earlier.
Regarding ftwrunner, when I run ./configure it can't find a header called modsecurity/modsecurity.h

[airween]

Regarding ftwrunner, when I run ./configure it can't find a header called modsecurity/modsecurity.h

You should pass the correct path for headers (and for libraries) like

export CFLAGS="-I /usr/local/include -L/usr/local/lib" && ./configure ....

(of course with the correct path)

[GNU-Plus-Windows-User]

@airween Now it's saying that the package libyaml-cpp is not installed, I'm pretty sure I installed the correct package sudo apt install libyaml-cpp-dev

[airween]

Hi @GNU-Plus-Windows-User,

@airween Now it's saying that the package libyaml-cpp is not installed, I'm pretty sure I installed the correct package sudo apt install libyaml-cpp-dev

could we move to the GH page of the tool, and open an issue there?

[GNU-Plus-Windows-User]

@airween Sure, should I just open a new issue there? I'm new to github.

[airween]

@airween Sure, should I just open a new issue there?

yes, thank you.

I'm new to github.

No worries 😃.

[GNU-Plus-Windows-User]

@airween I've given that repository you suggested a try and I'm still experiencing segfaults.
also I now get the following error from apt, but this doesn't stop me from downloading modsecurity and the nginx connector.

W: https://modsecurity.digitalwave.hu/ubuntu/dists/jammy/Release.gpg: Key is stored in legacy trusted.gpg keyring (/etc/apt/trusted.gpg), see the DEPRECATION section in apt-key(8) for details.
W: https://modsecurity.digitalwave.hu/ubuntu/dists/jammy-backports/Release.gpg: Key is stored in legacy trusted.gpg keyring (/etc/apt/trusted.gpg), see the DEPRECATION section in apt-key(8) for details.
E: The repository 'https://modsecurity.digitalwave.hu/ubuntu jammy-updates Release' does not have a Release file.
N: Updating from such a repository can't be done securely, and is therefore disabled by default.
N: See apt-secure(8) manpage for repository creation and user configuration details.

[airween]

I've given that repository you suggested a try and I'm still experiencing segfaults.

Waov. It's pretty interesting. With which Nginx have you tried? That module (both library and connector) has made for Nginx 1.18.0 (which is officially in Ubuntu 2204).

also I now get the following error from apt, but this doesn't stop me from downloading modsecurity and the nginx connector.

E: The repository 'https://modsecurity.digitalwave.hu/ubuntu jammy-updates Release' does not have a Release file.

You don't need to add the jammy-updates. That was used earlier for focal, but it's no longer needed, as the site contains:

Note: the focal-updates suite has been removed, because the Nginx package is part of regular Ubuntu focal repository. Therefore you do not need this line anymore:
...

These warnings are valid, I have to review what changed in Ubuntu 2204, and modify the page content.

W: http://modsecurity.digitalwave.hu/ubuntu/dists/jammy/Release.gpg: Key is stored in legacy trusted.gpg keyring (/etc/apt/trusted.gpg), see the DEPRECATION section in apt-key(8) for details.
W: http://modsecurity.digitalwave.hu/ubuntu/dists/jammy-backports/Release.gpg: Key is stored in legacy trusted.gpg keyring (/etc/apt/trusted.gpg), see the DEPRECATION section in apt-key(8) for details.

Thanks.

[GNU-Plus-Windows-User]

@airween I'm currently using nginx 1.18.0, the one that comes with Ubuntu 22.04 out of the box.

[martinhsv]

@GNU-Plus-Windows-User ,

If you are able to consistently produce segfaults in all of those different points in v3 history, it suggests that there is something distinctive about your build, environment, configuration, etc. (v3.0.8, in particular, can be assumed to have been in use by a large-ish number of installations).

Some options:

• try producing a core dump for those other segfaults. If it is always happening at the same point (RuleMessage::_details) then that lends weight to that being the source of the problem, rather than a somewhat random effect of the underlying problem
• if it is always the same place, I could provide a couple of experimental patches
• especially if it's not always in the same place, you may be using some less common feature or configuration that is affecting your installation -- you could experiment yourself with reducing the feature set that you're using, or supply your full configuration for me (or others) to examine.

[GNU-Plus-Windows-User]

@martinhsv I managed to find a solution to my issue.
Thank you for your help!

[martinhsv]

Reopening since the underlying bug has been identified. Thanks @airween for the work in identifying some steps to reproduce.

A fix will be committed shortly.

[martinhsv]

Resolved via #2886