[Feature Request] new operator to fetch data files via URL

[daixiang0]

After checking docs, I find that now ModSecurity only supports fetching data files by a local file path by pmFromFile.

But in the WebAssembly world, it can not read local files as usually common applications do, which means ModSecurity in WASM can not fetch data files with pmFromFile.

As @M4tteoP introduced in #2590, he uses static strings to cover CRS, refer to https://github.com/M4tteoP/minimal-modsec-istio-ext-wasm/blob/master/basemodsec/rules.cc#L5379. And check the codes, all rules including data files are removed as I said above.

I think a new operator to fetch data files via URL is time to add. If added, we can covert all CRS to strings and easily update rather than re-compile WASM binary.

On the other hand, this new operator will enable data file dynamic updating, which makes ModSecurity rules more flexible.

I am not sure rbl can fix it as I do not know how to set up an RBL database, and not sure there is any other way to fix it, if exist, please tell me.

Also, I check the RulesSet::loadRemote API, it can load rules via URL, but the data files specified in the rules it can not load which leads to related rules do not work as expected.

---

Update:

Using pm to include data files seems a workaround, but I am not sure is there a length limit for pm, or maybe other issues.

[martinhsv]

Pretend that I know nothing about WASM. (Which is actually not that far from the truth.) Please provide as much detail as possible. Expending development cycles to allow ModSecurity to work in an entirely new, previously-unsupported environment would take some justification.

What is 'the WebAssembly world' and why is it important for ModSecurity to work there?

[daixiang0]

Now with service mesh becoming more and more popular, Isito and Envoy are used widely, considered to be the de-facto standard. WAF is important for web application security, while Istio and Envoy can not provide related functions.

WebAssembly (abbreviated Wasm) is a binary instruction format for a stack-based virtual machine. Wasm is designed as a portable compilation target for programming languages and runs on many platforms. As it is a small sandbox with limited permissions(not including access to the file system at will), it is safe enough and costs little resources.

Istio and Envoy support the WASM plugin to extend those functions if ModSecurity can work with WASM, all service mesh users will get the power of ModSecurity and get benefit from it.

[martinhsv]

@daixiang0 ,

Note that pmFromFile was already changed in ModSecurity v2 to support remote https sources (https://github.com/SpiderLabs/ModSecurity/releases/tag/v2.9.0-rc1).

I have never used that functionality myself, but it looks to me like that code is all still in place.

I don't believe, however, that that functionality has ever been implemented in v3. Is that what you would be hoping for with this feature request? The same functionality that v2 has with respect to pmRemoteFile remote sources, but implemented in libModSecurity?

[daixiang0]

@martinhsv,

Sorry that I do not know the history, I only know I use v3 and fail to use pmFromFile with GitHub URL.

The function that fetches data via URL is all that I want, a new operator or using one already exists are both OK to me.

[M4tteoP]

Hi there, I can confirm that, from libModSecurity standpoint, adding the support to remote https sources would be definitely useful for our wasm-purpose. I am just not sure about possible sandbox limitations on connecting to external hosts (E.g. GitHub) from inside the WASM module. Still, it definitely may be the way to go and, if this problem arises, handle it from the WASM side.

@daixiang0, thanks for pointing out the workaround with pm, will try it!

[martinhsv]

@daixiang0 and @M4tteoP ,

An update to my comment from five days ago where I suggested the pmFromFile operator with a remote source -- but where I thought that had not been implemented in v3 ...

I was mistaken.

The functionality to use the pmFromFile operator with remote sources was indeed implemented in ModSecurity v3.

I just confirmed this in v3 with:

SecRule ARGS:theipaddress "@pmFromFile https://lists.blocklist.de/lists/bots.txt" "id:3091,phase:2,log,deny,status:403"

[daixiang0]

@martinhsv cool, please update docs/wiki in case others want to use it and will close this issue until related updated.

[martinhsv]

@daixiang0 ,

Actually, it is already mentioned near the bottom of the section for https://github.com/SpiderLabs/ModSecurity/wiki/Reference-Manual-%28v2.x%29#pmfromfile . See 'Note II : As of v2.9.0-RC1 this operator also supports ... '

Since the functionality already exists, I'll go ahead and close this.