non ascii characters causing false positives with different languages about owasp-modsecurity-crs HOT 12 CLOSED

For these who need to use it (extended solution):
As mentoned above you need:

1. Download unicode.mapping (if you don't have one) https://github.com/SpiderLabs/ModSecurity/blob/master/unicode.mapping

Add SecUnicodeMapFile /path/to/unicode.mapping
And
Add SecUnicodeCodePage 20127 (choose number from mapping file)
or if this is deprecated - use
And Add SecUnicodeMapFile /path/to/unicode.mapping 20127

3. Check if you have t:urlDecodeUni in your rules function (for example open rule you are triggering and check if it there) (if you don't have it - use "t:utf8toUnicode,t:urlDecodeUni" instead of "t:utf8toUnicode" on the next step

4. add SecRuleUpdateActionById 981243 "t:utf8toUnicode" to the last file in activated_rules (for example /activated_rules/modsecurity_crs_61_customrules.conf) (check if /activated_rules/*.conf included into your mod_security config file) (instead of 981243 use your rule number where false positive triggers, or rule ids mentioned by d-schramm )

Hope this will save some time for someone.

from owasp-modsecurity-crs.

I think you are right. We used those chars before we had the new UTF8 and Unicode mapping capabilities. With these transformation functions in place, they should all be normalized to just a single ' or ".

I will look at updating them.

from owasp-modsecurity-crs.

I'd like to elaborate on this bug report. Here's an example of a false positive as manifested in my Apache error log file:

[Fri Apr 12 11:10:10 2013] [error] [client a.b.c.d] ModSecurity: Warning. Pattern match "(?i:(?:[\\"'`\\xc2\\xb4\\xe2\\x80\\x99\\xe2\\x80\\x98]\\\\s*?\\\\*.+(?:x?or|div|like|between|and|id)\\\\W*?[\\"'`\\xc2\\xb4\\xe2\\x80\\x99\\xe2\\x80\\x98]\\\\d)|(?:\\\\^[\\"'`\\xc2\\xb4\\xe2\\x80\\x99\\xe2\\x80\\x98])|(?:^[\\\\w\\\\s\\"'`\\xc2\\xb4\\xe2\\x80\\x99\\xe2\\x80\\x98-]+(?<=and\\\\s)(?<=or|xor ..." at ARGS:term. 
[file ".../modsecurity2/activated_rules/41_modsecurity_crs_41_sql_injection_attacks.conf"] 
[line "245"] 
[id "981243"]
[data "Matched Data: \\xc2\\xa720 found within ARGS:term: 1 10\\xc2\\xa720"] [severity "CRITICAL"] 

The false positive is due to the fact that the bytes that the UTF-8 representations consist of are read as individual members of the character class ["'...], which makes 0xC2 alone match, and thus makes all Unicode characters in the range 0x80 to 0xBF match! Furthermore, non-Unicode characters 0xB4 (Latin-1: freestanding acute accent), and 0xC2 (L1: A with circumflex), 0xE2 (L1: a with circumflex) also match.

The rules mentioned are not the only ones theoretically provoking false positives (though, in practice, I've only seen them with a few). In fact the misbehavior can occur with every rule containing UTF-8 code sequences, all in modsecurity_crs_41_sql_injection_attacks.conf:

981231
981318
950901
981172
981173
981244
981255
981257
981248
981277
981250
981241
981252
981256
981245
981276
981254
981270
981240
981249
981253
981242
981246
981251
981247
981243

from owasp-modsecurity-crs.

I suggest getting rid of all the UTF-8 sequences in modsecurity_crs_41_sql_injection_attacks.conf. I prepared the file, but I don't want to include it here. Instead of character classes it uses an alternative of a character class and several byte sequences. To make the resulting expression shorter again, a feature available since PCRE v7.0 (19-Dec-06) is used: named subpatterns. This makes the payload a little higher, though, but so (probably) would switching to "t:utf8toUnicode,t:urlDecodeUni"!

Here's an example -- old rule 981318 (line 64):

"(^[\"'`´’‘;]+|[\"'`´’‘;]+$)"

New:

"(^(?:(?&QC)|;)+|(?:(?&QC)|;)+$)(?(DEFINE)(?'QC'(?:[\"'`]|\xC2?\xB4|\xE2\x80(?:\x99|\0x98))))"

from owasp-modsecurity-crs.

This can be addressed by a few changes -

1. Set SecUnicodePage (https://github.com/SpiderLabs/ModSecurity/wiki/Reference-Manual#SecUnicodeCodePage) directive with your proper codepoint

2. Use t:utf8toUnicode,t:urlDecodeUni transformation functions

3. I just modified those converted PHPIDS rules (in CRS v3.0.0) to remove the other multi-byte characters as they should now all be normalize to backtick, quote and double-quote characters.

from owasp-modsecurity-crs.

Ryan,

Is it possible to use same solution for issue that I have - please check following link owasp-modsecurity/ModSecurity#708

from owasp-modsecurity-crs.

Hi Ryan and all,

It looks like there is a big problem with Unicode. The only way we were able to support unicode across the board is with the below far from ideal removal of several rules.

SecRule REQUEST_URI "@contains /" "phase:1,t:none,pass,id:'14',nolog,ctl:ruleRemoveById=960024,ctl:ruleRemoveById=981318,ctl:ruleRemoveById=981173,ctl:ruleRemoveById=981243"

Would you please provide an update about what version to use that actually corrects the issue? Just sending bunch of Chinese characters to the backend is enough for modsecurity to complain with the rules above one at the time and in that order as we add exclusions.

Thanks!

• Nestor

from owasp-modsecurity-crs.

@nestoru could you please provide a list of which rules you've removed to prevent the issue... this is a known issue with how modsecurity uses PCRE.

from owasp-modsecurity-crs.

@csanders-git look into the SecRuleI posted above for the ids of all rules I had to remove. For what is worth I am using Ubuntu 14.04 and this plain old bash recipe https://raw.githubusercontent.com/nestoru/pob-recipes/master/common/apache/mod-security-install.sh

from owasp-modsecurity-crs.

Is it possible to remediate the false positive by reshuffling the sequence of how the transformation is apply?
Since the non asic character is transformed into unicode when t:utf8toUnicode is applied.
Then when t:urlDecodeUni is applied all the unicode resulted from non ascii character can potential be transformed to some symbol where modsecurity would deem to be an attack. i.e " 格|imag|" is first transform to unicode "%u683c|imag|" by t:utf8toUnicode
Then transform to "%u003c|imag|" by t:urlDecodeUni which in ascii is "<|imag|" . And modsecurity is treating the "<|imag|" as an attack.
What if we let the t:urlDecodeUni go before t:utf8toUnicode, then " 格|imag|" would be passed as "\xe6\xa0\xbc|imag|" to t:urlDecodeUni since no unicode exist in the original string
Then when t:utf8toUnicode is applied to "\xe6\xa0\xbc|imag|" it would only result in "%u683c|imag|" which can pass through the modsecurity.

from owasp-modsecurity-crs.

Yes and no, changing the transforms will affect how rules process the content and prevent issues with some rules but "%u683c|imag|" is not a very friendly payload at all, it will trigger 920220, dealing with other alphabets is problematic often I have to be more forgiving with those fields containing cyrilic or other non-ascii and multi byte encodings. It may be interesting to use the specific target language characters as whitelist so you allow only let's say 1000 different characters instead of 2 million characters of utf-8 not to tell all other encodings.

"Access denied with code 403 (phase 2). Invalid URL Encoding: Non-hexadecimal digits used at REQUEST_URI. [file \"/usr/share/modsecurity-crs/rules/REQUEST-920-PROTOCOL-ENFORCEMENT.conf\"] [line \"409\"] [id \"920220\"] [rev \"2\"] [msg \"URL Encoding Abuse Attack Attempt\"] [severity \"WARNING\"] [ver \"OWASP_CRS/3.0.0\"] [maturity \"6\"] [accuracy \"8\"] [tag \"application-multi\"] [tag \"language-multi\"] [tag \"platform-multi\"] [tag \"attack-protocol\"] [tag \"OWASP_CRS/PROTOCOL_VIOLATION/EVASION\"]"

from owasp-modsecurity-crs.

This issue has timed out as it has not received any update in over 2 years.
If this is still a problem please open a new issue.

from owasp-modsecurity-crs.