Comments (9)
The question buried in here is:
Is there any significant set of rules which runs by default on JSON, but not XML-parsed fields? If not (and I couldn't find any), I am at a total loss to explain the roughly 650% increase in processing time in JSON compared to XML requests
from owasp-modsecurity-crs.
UPDATE!
I've been able to confirm that the latency with JSON has something to do with the inclusion of a large list within the payload.
from owasp-modsecurity-crs.
Question for the CRS team now:
Why do JSON Lists cause so much latency with CRS rules?
from owasp-modsecurity-crs.
Payload for the "Fast" JSON test outlined above:
{
"lorem": "At vero eos et accusamus et iusto odio dignissimos ducimus qui blanditiis praesentium voluptatum deleniti atque corrupti quos dolores et quas molestias excepturi sint occaecati cupiditate non provident, similique sunt in culpa qui officia deserunt mollitia animi, id est laborum et dolorum fuga. Et harum quidem rerum facilis est et expedita distinctio. Nam libero tempore, cum soluta nobis est eligendi optio cumque nihil impedit quo minus id quod maxime placeat facere possimus, omnis voluptas assumenda est, omnis dolor repellendus. Temporibus autem quibusdam et aut officiis debitis aut rerum necessitatibus saepe eveniet ut et voluptates repudiandae sint et molestiae non recusandae. Itaque earum rerum hic tenetur a sapiente delectus, ut aut reiciendis voluptatibus maiores alias consequatur aut perferendis doloribus asperiores repellat.",
"ipsum":"On the other hand, we denounce with righteous indignation and dislike men who are so beguiled and demoralized by the charms of pleasure of the moment, so blinded by desire, that they cannot foresee the pain and trouble that are bound to ensue; and equal blame belongs to those who fail in their duty through weakness of will, which is the same as saying through shrinking from toil and pain. These cases are perfectly simple and easy to distinguish. In a free hour, when our power of choice is untrammelled and when nothing prevents our being able to do what we like best, every pleasure is to be welcomed and every pain avoided. But in certain circumstances and owing to the claims of duty or the obligations of business it will frequently occur that pleasures have to be repudiated and annoyances accepted. The wise man therefore always holds in these matters to this principle of selection: he rejects pleasures to secure other greater pleasures, or else he endures pains to avoid worse pains.",
"lorem1": "At vero eos et accusamus et iusto odio dignissimos ducimus qui blanditiis praesentium voluptatum deleniti atque corrupti quos dolores et quas molestias excepturi sint occaecati cupiditate non provident, similique sunt in culpa qui officia deserunt mollitia animi, id est laborum et dolorum fuga. Et harum quidem rerum facilis est et expedita distinctio. Nam libero tempore, cum soluta nobis est eligendi optio cumque nihil impedit quo minus id quod maxime placeat facere possimus, omnis voluptas assumenda est, omnis dolor repellendus. Temporibus autem quibusdam et aut officiis debitis aut rerum necessitatibus saepe eveniet ut et voluptates repudiandae sint et molestiae non recusandae. Itaque earum rerum hic tenetur a sapiente delectus, ut aut reiciendis voluptatibus maiores alias consequatur aut perferendis doloribus asperiores repellat.",
"ipsum1":"On the other hand, we denounce with righteous indignation and dislike men who are so beguiled and demoralized by the charms of pleasure of the moment, so blinded by desire, that they cannot foresee the pain and trouble that are bound to ensue; and equal blame belongs to those who fail in their duty through weakness of will, which is the same as saying through shrinking from toil and pain. These cases are perfectly simple and easy to distinguish. In a free hour, when our power of choice is untrammelled and when nothing prevents our being able to do what we like best, every pleasure is to be welcomed and every pain avoided. But in certain circumstances and owing to the claims of duty or the obligations of business it will frequently occur that pleasures have to be repudiated and annoyances accepted. The wise man therefore always holds in these matters to this principle of selection: he rejects pleasures to secure other greater pleasures, or e",
"lorem2": "At vero eos et accusamus et iusto odio dignissimos ducimus qui blanditiis praesentium voluptatum deleniti atque corrupti quos dolores et quas molestias excepturi sint occaecati cupiditate non provident, similique sunt in culpa qui officia deserunt mollitia animi, id est laborum et dolorum fuga. Et harum quidem rerum facilis est et expedita distinctio. Nam libero tempore, cum soluta nobis est eligendi optio cumque nihil impedit quo minus id quod maxime placeat facere possimus, omnis voluptas assumenda est, omnis dolor repellendus. Temporibus autem quibusdam et aut officiis debitis aut rerum necessitatibus saepe eveniet ut et voluptates repudiandae sint et molestiae non recusandae. Itaque earum rerum hic tenetur a sapiente delectus, ut aut reiciendis voluptatibus maiores alias consequatur aut perferendis doloribus asperiores repellat.",
"ipsum2":"On the other hand, we denounce with righteous indignation and dislike men who are so beguiled and demoralized by the charms of pleasure of the moment, so blinded by desire, that they cannot foresee the pain and trouble that are bound to ensue; and equal blame belongs to those who fail in their duty through weakness of will, which is the same as saying through shrinking from toil and pain. These cases are perfectly simple and easy to distinguish. In a free hour, when our power of choice is untrammelled and when nothing prevents our being able to do what we like best, every pleasure is to be welcomed and every pain avoided. But in certain circumstances and owing to the claims of duty or the obligations of business it will frequently occur that pleasures have to be repudiated and annoyances accepted. The wise man therefore always holds in these matters to this principle of selection: he rejects pleasures to secure other greater pleasures, or else he endures pains to avoid worse pains.",
"lorem3": "At vero eos et accusamus et iusto odio dignissimos ducimus qui blanditiis praesentium voluptatum deleniti atque corrupti quos dolores et quas molestias excepturi sint occaecati cupiditate non provident, similique sunt in culpa qui officia deserunt mollitia animi, id est laborum et dolorum fuga. Et harum quidem rerum facilis est et expedita distinctio. Nam libero tempore, cum soluta nobis est eligendi optio cumque nihil impedit quo minus id quod maxime placeat facere possimus, omnis voluptas assumenda est, omnis dolor repellendus. Temporibus autem quibusdam et aut officiis debitis aut rerum necessitatibus saepe eveniet ut et voluptates repudiandae sint et molestiae non recusandae. Itaque earum rerum hic tenetur a sapiente delectus, ut aut reiciendis voluptatibus maiores alias consequatur aut perferendis doloribus asperiores repellat.",
"ipsum3":"On the other hand, we denounce with righteous indignation and dislike men who are so beguiled and demoralized by the charms of pleasure of the moment, so blinded by desire, that they cannot foresee the pain and trouble that are bound to ensue; and equal blame belongs to those who fail in their duty through weakness of will, which is the same as saying through shrinking from toil and pain. These cases are perfectly simple and easy to distinguish. In a free hour, when our power of choice is untrammelled and when nothing prevents our being able to do what we like best, every pleasure is to be welcomed and every pain avoided. But in certain circumstances and owing to the claims of duty or the obligations of business it will frequently occur that pleasures have to be repudiated and annoyances accepted. The wise man therefore always holds in these matters to this principle of selection: he rejects pleasures to secure other greater pleasures, or else he endures pains to avoid worse pains.",
"lorem4": "At vero eos et accusamus et iusto odio dignissimos ducimus qui blanditiis praesentium voluptatum deleniti atque corrupti quos dolores et quas molestias excepturi sint occaecati cupiditate non provident, similique sunt in culpa qui officia deserunt mollitia animi, id est laborum et dolorum fuga. Et harum quidem rerum facilis est et expedita distinctio. Nam libero tempore, cum soluta nobis est eligendi optio cumque nihil impedit quo minus id quod maxime placeat facere possimus, omnis voluptas assumenda est, omnis dolor repellendus. Temporibus autem quibusdam et aut officiis debitis aut rerum necessitatibus saepe eveniet ut et voluptates repudiandae sint et molestiae non recusandae. Itaque earum rerum hic tenetur a sapiente delectus, ut aut reiciendis voluptatibus maiores alias consequatur aut perferendis doloribus asperiores repellat.",
"ipsum4":"On the other hand, we denounce with righteous indignation and dislike men who are so beguiled and demoralized by the charms of pleasure of the moment, so blinded by desire, that they cannot foresee the pain and trouble that are bound to ensue; and equal blame belongs to those who fail in their duty through weakness of will, which is the same as saying through shrinking from toil and pain. These cases are perfectly simple and easy to distinguish. In a free hour, when our power of choice is untrammelled and when nothing prevents our being able to do what we like best, every pleasure is to be welcomed and every pain avoided. But in certain circumstances and owing to the claims of duty or the obligations of business it will frequently occur that pleasures have to be repudiated and annoyances accepted. The wise man therefore always holds in these matters to this principle of selection: he rejects pleasures to secure other greater pleasures, or else he endures pains to avoid worse pains."
}
from owasp-modsecurity-crs.
I'm afraid we can't help with the available information.
The JSON contents above are very different: the first one has so many keys, and all child have more child items. The second one has 10 keys (as I see) without any sub-child.
If a rule has an argument XML:/*
, it means all keys will checked. That could be so many time...
Yes, you're right - the XML content doesn't trigger this latency - I have no idea why. But I think this means this issue is not CRS related.
You should start to turn off each rule set, start with 901. If this modification has no effect, turn back and take next one. Iterate this while you get a better result. Then you found the source of your problem, and can check each rule in that file.
from owasp-modsecurity-crs.
Thanks @airween !
Agree, there's not enough info yet to draw conclusions.
For now, my current theory is that something about lists with multiple items, multiple nested keys, might cause high latency in some particular rule(s).
I'm pretty sure XML vs JSON doesn't really matter. I just happened to run my original test with a particularly "bad" JSON. I ran a "List" version of this payload earlier today too, and that did not produce similar high latency.
I also attempted running Modsecurity with CRS disabled on the original "bad" JSON, and that too did not produce high latency - that's the only reason I was eyeing something in CRS.
Will attempt to identify which rule(s) seem to take excess time on the "bad" JSON.
from owasp-modsecurity-crs.
I made some researches, let me share with you the results. I sent all of three payloads above to my test Nginx with curl
, and reviewed the modsec_debug.log
.
The first column in the line is the line number in file (all requests logged into same file). The first line of three pairs is the beginning of random rule, the last is same with next one. Just see the difference between the first and last line numbers. That means, how many steps required to execute a rule.
- 'High-latency JSON' payload
407 [158573518154.925648] [/] [4] (Rule: 942100) Executing operator "DetectSQLi against REQUEST_COOKIES|...|ARGS|XML:/*.
4268 [158573518154.925648] [/] [4] (Rule: 942140) Executing operator "Rx" with param "(?i:\b(...)" against REQUEST_COOKIES|REQUEST_COOKIES_NAMES|ARGS_NAMES|ARGS|XML:/*.
3861 lines.
- XML payload
129999 [158573526493.497825] [/] [4] (Rule: 942100) (Rule: 942100) Executing operator "DetectSQLi against REQUEST_COOKIES|...|ARGS|XML:/*.
130014 [158573526493.497825] [/] [4] (Rule: 942140) Executing operator "Rx" with param "(?i:\b(...)" against REQUEST_COOKIES|REQUEST_COOKIES_NAMES|ARGS_NAMES|ARGS|XML:/*.
15 lines.
- JSON with 10 keys
132705 [158573628522.775075] [/] [4] (Rule: 942100) Executing operator "DetectSQLi against REQUEST_COOKIES|...|ARGS|XML:/*.
132834 [158573628522.775075] [/] [4] (Rule: 942140) Executing operator "Rx" with param "(?i:\b(...)" against REQUEST_COOKIES|REQUEST_COOKIES_NAMES|ARGS_NAMES|ARGS|XML:/*.
129 lines.
The other lines between two quotes lines above contains all steps, including transformations, and all operator evaluations.
Please check these values at your side - I think you can see the reason, why is so slowly the JSON payload what you included.
Hope this helps.
from owasp-modsecurity-crs.
@airween thank you for the analysis!
Just one last Q - is this behavior consistent between all rules for the "High Latency" JSON? Or just for the 942100
rule?
from owasp-modsecurity-crs.
No, not just for the rule 942100
- that's just a random chosed :).
It's consistent between all rules (which affected on your chosed PL), just check your log.
The "problem" is that JSON parser converts the tree hierarchy into a flat structure, and the engine handles it as ARGS
and ARGS_NAMES
. Any rule which contains these variables will check all 'flatted' list.
Just check your modsec_debug.log
.
from owasp-modsecurity-crs.
Related Issues (20)
- SOAPUI SOAP Tx multipart/related call False Positive id: 920470 HOT 4
- DOS protection is invalid
- Crazy Long Processing time of XML of a certain kinda payload body. HOT 2
- Easy to trigger these rule id blocks just with keywords [932115, 942360]
- DoS rule triggering with static (png) file
- SQLi bypass at PL1(CRS 3.2.0) HOT 1
- XSS Attack Detected via libinjection for AWS AWSALBCORS Cookie HOT 4
- Block QQGameHall in UA HOT 4
- Monthly Chat Agenda April (2020-04-06) HOT 1
- NextCloud False Positive HOT 9
- WordPress JetPack False Positive
- Rule 920450 and modsec 3x HOT 4
- Password Scrubbing within the libinjection rule HOT 1
- Monthly Chat Agenda May (2020-05-04) HOT 1
- rule 920300 title / details mismatch HOT 1
- Note config change of tx.allowed_request_content_type in the v3.3 release notes
- false positive on rule 932110
- Incompatible with ModSecurity 3.x? HOT 1
- False positive with WordPress when hosted from http://example.com/update-prefix HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from owasp-modsecurity-crs.