modest_audit.log does not log anything, nginx

[intelbg]

Hello,
I have compiled nginx 1.12 with libmodsecurity v3, nginx connector and CRS3. Although I can't understand why modest audit log does not log anything. This is the configuration (which is 1:1 with the INSTALL steps).

cat modsec_includes.conf
include modsecurity.conf
include crs-setup.conf
include rules/REQUEST-900-EXCLUSION-RULES-BEFORE-CRS.conf
include rules/REQUEST-901-INITIALIZATION.conf
include rules/REQUEST-905-COMMON-EXCEPTIONS.conf
include rules/REQUEST-910-IP-REPUTATION.conf
include rules/REQUEST-911-METHOD-ENFORCEMENT.conf
include rules/REQUEST-912-DOS-PROTECTION.conf
include rules/REQUEST-913-SCANNER-DETECTION.conf
include rules/REQUEST-920-PROTOCOL-ENFORCEMENT.conf
include rules/REQUEST-921-PROTOCOL-ATTACK.conf
include rules/REQUEST-930-APPLICATION-ATTACK-LFI.conf
include rules/REQUEST-931-APPLICATION-ATTACK-RFI.conf
include rules/REQUEST-932-APPLICATION-ATTACK-RCE.conf
include rules/REQUEST-933-APPLICATION-ATTACK-PHP.conf
include rules/REQUEST-941-APPLICATION-ATTACK-XSS.conf
include rules/REQUEST-942-APPLICATION-ATTACK-SQLI.conf
include rules/REQUEST-943-APPLICATION-ATTACK-SESSION-FIXATION.conf
include rules/REQUEST-949-BLOCKING-EVALUATION.conf
include rules/RESPONSE-950-DATA-LEAKAGES.conf
include rules/RESPONSE-951-DATA-LEAKAGES-SQL.conf
include rules/RESPONSE-952-DATA-LEAKAGES-JAVA.conf
include rules/RESPONSE-953-DATA-LEAKAGES-PHP.conf
include rules/RESPONSE-954-DATA-LEAKAGES-IIS.conf
include rules/RESPONSE-959-BLOCKING-EVALUATION.conf
include rules/RESPONSE-980-CORRELATION.conf
include rules/RESPONSE-999-EXCLUSION-RULES-AFTER-CRS.conf

In mod security.conf I have the following configurations about logs:

SecAuditLogRelevantStatus "^(?:5|4(?!04))"
SecAuditLogParts ABIJDEFHZ
SecAuditLogType Serial
SecAuditLog /var/log/nginx/modsec_audit.log

In crs-setup.conf I have the following default action with logging:

cat crs-setup.conf| grep -i defaultac
SecDefaultAction "phase:1,log,auditlog,pass"
SecDefaultAction "phase:2,log,auditlog,pass"

Initially it worked but I haven't changed anything (I know that you heard these words a couple of times, me too :)) and now logging in audit log does not work (but in nginx error.log it logs). Where can the problem be? Also how to change it to log only 403 - is SecAuditLogRelevantStatus 403 correct one? Although how can I change the log configuration to match only header and rule id without html and additional useless information?
Thank you in advance!

[intelbg]

I found the problem but couldn't get an explanation....when the inbound score is set to 15 and outbound score to 14 logs works, but if they are on the default 5 and 3 the audit log stops:

setvar:tx.inbound_anomaly_score_threshold=15,\

Any ideas?

[intelbg]

If I change the logging type to Concurrent it doesn't work at all. I noticed that in serial logging not every request is logged, but for example 1 of 4-5 requests. Any ideas? I have userid module of nginx to set a hash in the 403.html page and the audit log too but only one hash of 5+ requests is logged and still can't understand why.

[intelbg]

One more issue, why after I remove the E part it's still logged (in serial mode)? I would like only o log IP, matched url, not the request/response body data?

[zimmerle]

Hi @intelbg,

Make sure you have your ModSecurity library compiled with JSON support. Otherwise it won't save the JSON logging in the file. That may solve your original issue about having empty logs. Please let us know. I am not sure which issue you manage to get working or not. Can you point the different problems into different issues? just to be easy for us to track it.

[intelbg]

@zimmerle is it must to be compiled with json support as I am not using json format for logging?

[intelbg]

Hi again @zimmerle , I understand that I should not describe more than one problem in one issue request, so let's focus here on the main problem. I will describe it again to eliminate the information from the other problems.

Explaining the configuration:
I have userid_module in NginX that generates and unique hash to every request. The main idea is when user is blocked and it's real user to send us the hash and through it search it in the audit logs to find which rule exactly blocked the user.

Explaining the problem.
The problem I face is that not every request is logged in audit logs. If I make 5 requests to a forbidden by rule page (for example xss url), I should see the hash which is attached to the requests 5 times logged in the audit logs, but I don't see it - in most cases it's logged one time and I can't find which rule has blocked me in the logs exactly - so the main idea is losed.

What I tried to debug the issue?

• Searching in debug log in level 9 - nothing found (or you can point me to search for a specific string?)
• Playing with the thresholds - not sure if it's completely true but I think that when threshold is set to 15 instead of 5 the logs works better, on 5 stops completely.
• Trying to play with the log parts (by default I am using all parts).
• Stopped all kinds of caching

So, you advised me to compile and use json support. Currently I am using the normal logging type. Is the issue related to the fact that I am not using json format? Do you think that enabling json will solve this issue? Why requests can be skipped and not logged? Does the userid module can be a obsticle?

[zimmerle]

Hi @intelbg, I am confused. The problem was about a JSON file being saved but empty. That you can solve by compiling it with JSON support. The advice to compile with JSON support was related to the empty file. I see now that you found other problems, and some you solved by yourself.

Please consider to create a new issue on github to detail the new problem that you are current facing. Otherwise it will be very hard for our users to search for content inside our GitHub issues repository, which is also used as knowledge base. For instance, if you check #1568, that may or may not related to the issue that you are facing.

[intelbg]

@zimmerle Initially I didn't mention json nowhere in my problem. I am using standart mod security configuration and owasp configuration which is the following one related to the logging:

SecStatusEngine On
SecAuditLogParts ABIJDEFHZ
SecAuditLogType Serial
SecAuditLog /var/log/nginx/modsec_audit.log

I am not using json logging type and libmodsecurity is not compiled with json support. The problem #1568 is not related to the mine. if you think that using json type for logging will solve my issue I will compile it and I will try. The real (original problem, let's don't think about the problems in the next posts I will separate them in separate issues if I can't solve them) is that when I make an request and have blocked it's not logged in /var/log/nginx/modsec_audit.log

[intelbg]

@zimmerle I read now that json is the default logging type in version 3 of libmodsecurity and as mine is not configured with yajl this may be exactly the problem. I had installed on my cents 6.8 yajl and yajl-devel and I see that the library is there:

ldconfig -p | grep yajl
libyajl.so.1 (libc6,x86-64) => /usr/lib64/libyajl.so.1
libyajl.so (libc6,x86-64) => /usr/lib64/libyajl.so

Although libmodsecurity didn't find it even if I explicitly set the path:

$ ./configure --with-yajl
configure:5019: GeoIP support was marked as mandatory by the utilization of --with-yajl=yes
configure:5194: error: YAJL was explicitly referenced but it was not found

What can be the reason?

[intelbg]

I even compiled yajl and still the same error.

[intelbg]

Ok, this solved the problem with compiling: ln -s /usr/local/share/pkgconfig/yajl.pc /usr/share/pkgconfig/yajl.pc. I will see if compiling with json will resolve the issue with logging.

[intelbg]

Now I have libmodsecurity compiled with json support, but the log is still empty. I give you the debug log on level 9 to review it. The problem is not only in detecting only mode but also when it's turned on.
json-debug-log-9.txt

[intelbg]

So, 1) so, logging seems to work only with concurrent / parallel logging, not with serial. But then it does not log anything with inbound and outbound thresholds 5 and 4. If I change the thresholds to 15 and 14 it starts logging. I think this is mod security problem somewhere in the code. It's not logical changing the thresholds to manage the way of logging.
Also 2) If I am blocked because of xss attack it's logged once a time. If I refresh second time I am not logged already. Is this normal? Tried cleaning the cache etc, new sessions and no more logging. Tried also to be blocked from another rule and still didn't logged no more. It logs when it wants, not when it should (when I am blocked every time). Please help.

[intelbg]

@victorhora With this branch (from Enable support for AuditLogFormat (JSON/Native) #1583) audit logging doesn't work to me too, neither with serial logging type, neither with concurrent/parallel.

[akama-aka]

idk if its a me issue or not but I've this issue now too.