SecHashEngine not working

[blackenz]

SecHashEngine on version 2.8.0 is currently not working as expected. With the following directives turned to On, mod_security is unable to change response body, in order to add the previously calculated hash:

SecDisableBackendCompression On
SecContentInjection On
SecStreamOutBodyInspection On
SecHashEngine On
SecHashKey rand KeyOnly
SecHashParam hmac
SecHashMethodRx HashHref ".php"

The Debug log, shows the following:

[17/Jun/2014:15:31:45 +0000] [OMMITED][/index.php][4] init_response_body_html_parser: assuming ISO-8859-1.
[17/Jun/2014:15:31:45 +0000] [OMMITED][/index.php][4] init_response_body_html_parser: Successfully html parser generated.
[17/Jun/2014:15:31:45 +0000] [OMMITED][/index.php][4] Signing data [index.php]
[17/Jun/2014:15:31:45 +0000] [OMMITED][/index.php][4] hash_response_body_links: Processed [0] iframe src, [0] hashed.
[17/Jun/2014:15:31:45 +0000] [OMMITED][/index.php][4] hash_response_body_links: Processed [0] frame src, [0] hashed.
[17/Jun/2014:15:31:45 +0000] [OMMITED][/index.php][4] hash_response_body_links: Processed [1] form actions, [1] hashed.
[17/Jun/2014:15:31:45 +0000] [OMMITED][/index.php][4] hash_response_body_links: Processed [0] links, [0] hashed.
[17/Jun/2014:15:31:45 +0000] [OMMITED][/index.php][4] inject_hashed_response_body: Detected encoding type [ISO-8859-1].
[17/Jun/2014:15:31:45 +0000] [OMMITED][/index.php][4] inject_hashed_response_body: Using content-type [ISO-8859-1].
[17/Jun/2014:15:31:45 +0000] [OMMITED][/index.php][4] Hash completed in 754 usec.
[17/Jun/2014:15:31:45 +0000] [OMMITED][/index.php][4] Starting phase RESPONSE_BODY.

In the resulting html, there is no reference to the hmac parameter in the href elements.
ModSecurity was compiled using the following libraries:

ModSecurity for Apache/2.8.0 (http://www.modsecurity.org/) configured.
ModSecurity: APR compiled version="1.5.1-dev"; loaded version="1.5.1-dev"
ModSecurity: PCRE compiled version="8.31 "; loaded version="8.31 2012-07-06"
ModSecurity: LUA compiled version="Lua 5.1"
ModSecurity: LIBXML compiled version="2.9.1"

Apache version is 2.4.7
I'm also using the modsecurity.conf-recommended configuration file.

[dune73]

It's possible I am held back by this bug as well:

Here is the output from my debug log:

[05/May/2016:06:29:57 +0200] [localhost/sid#d611e8][rid#7eff04002970][/form.html][4] hash_response_body_links: Processed [0] iframe src, [0] hashed.
[05/May/2016:06:29:57 +0200] [localhost/sid#d611e8][rid#7eff04002970][/form.html][4] hash_response_body_links: Processed [0] frame src, [0] hashed.
[05/May/2016:06:29:57 +0200] [localhost/sid#d611e8][rid#7eff04002970][/form.html][4] hash_response_body_links: Processed [1] form actions, [1] hashed.
[05/May/2016:06:29:57 +0200] [localhost/sid#d611e8][rid#7eff04002970][/form.html][4] hash_response_body_links: Processed [1] links, [1] hashed.
[05/May/2016:06:29:57 +0200] [localhost/sid#d611e8][rid#7eff04002970][/form.html][4] inject_hashed_response_body: Detected encoding type [ISO-8859-1].
[05/May/2016:06:29:57 +0200] [localhost/sid#d611e8][rid#7eff04002970][/form.html][4] inject_hashed_response_body: Using content-type [ISO-8859-1].
[05/May/2016:06:29:57 +0200] [localhost/sid#d611e8][rid#7eff04002970][/form.html][1] inject_hashed_response_body: Unable to inject hash into response body. Returning response without changes.
[05/May/2016:06:29:57 +0200] [localhost/sid#d611e8][rid#7eff04002970][/form.html][4] Starting phase RESPONSE_BODY.

It's the line "Unable to inject hash into response body" which is not available with the original author of this issue. So maybe this is something different.

I looked into the code and did a few additional tests, it is obvious my server fails on

xmlOutputBufferFlush(output_bug)

around line 1075 in msc_crypt.c.

My libxml2.so is version 2.9.1.

[csanders-git]

@dune73 see #1072
Generally speaking XMLOutputBufferFlush will empty out the buffer which is not the functionality that is expected by ModSec. I talked to the guys at libxml2 they seemed to think it would be fine to flush the buffer later after the work was done but I had not gotten around to really thoroughly testing the result.

[dune73]

Thanks Chaim. I'll try to get around this with the help of your description in #1072.
(Did not find that bug because of a typo in the description)

[SQLearn]

The SecHashEngine feature is currently broken in CentOS 7 and Amazon Linux since the libxml version is 2.9.1. Is there any workaround to fix this bug? Can we expect this bug to be fixed by modsecurity or it is something only libxml can fix?

Since the SecHashMethodRx Rule is currently broken is there any other way to achieve the same functionality? Can LUA be used to hash hrefs using regex to capture all the possible href elements of a CMS website and also to validate the hash using the @validateHash operator?

[dune73]

@SQLearn: I could not get it to work for me. And I tried a lot of library versions. :)

[ossie-git]

Here's a patch to at least get the links generated properly. Nothing fancy.
hmac.txt

[zimmerle]

The problem seems to be intermittent, related to a earlier cleanup. If there was an overwritten on the freed area, the data will be looking like corrupted or even nothing will be displayed. Without the overwritten, the data will look ok.

At a249574 we avoid this earlier cleanup.