Code Monkey home page Code Monkey logo

Comments (9)

rsbrisci avatar rsbrisci commented on August 11, 2024

The question buried in here is:

Is there any significant set of rules which runs by default on JSON, but not XML-parsed fields? If not (and I couldn't find any), I am at a total loss to explain the roughly 650% increase in processing time in JSON compared to XML requests

from owasp-modsecurity-crs.

rsbrisci avatar rsbrisci commented on August 11, 2024

UPDATE!

I've been able to confirm that the latency with JSON has something to do with the inclusion of a large list within the payload.

image

from owasp-modsecurity-crs.

rsbrisci avatar rsbrisci commented on August 11, 2024

Question for the CRS team now:

Why do JSON Lists cause so much latency with CRS rules?

from owasp-modsecurity-crs.

rsbrisci avatar rsbrisci commented on August 11, 2024

Payload for the "Fast" JSON test outlined above:

{
 "lorem": "At vero eos et accusamus et iusto odio dignissimos ducimus qui blanditiis praesentium voluptatum deleniti atque corrupti quos dolores et quas molestias excepturi sint occaecati cupiditate non provident, similique sunt in culpa qui officia deserunt mollitia animi, id est laborum et dolorum fuga. Et harum quidem rerum facilis est et expedita distinctio. Nam libero tempore, cum soluta nobis est eligendi optio cumque nihil impedit quo minus id quod maxime placeat facere possimus, omnis voluptas assumenda est, omnis dolor repellendus. Temporibus autem quibusdam et aut officiis debitis aut rerum necessitatibus saepe eveniet ut et voluptates repudiandae sint et molestiae non recusandae. Itaque earum rerum hic tenetur a sapiente delectus, ut aut reiciendis voluptatibus maiores alias consequatur aut perferendis doloribus asperiores repellat.",
"ipsum":"On the other hand, we denounce with righteous indignation and dislike men who are so beguiled and demoralized by the charms of pleasure of the moment, so blinded by desire, that they cannot foresee the pain and trouble that are bound to ensue; and equal blame belongs to those who fail in their duty through weakness of will, which is the same as saying through shrinking from toil and pain. These cases are perfectly simple and easy to distinguish. In a free hour, when our power of choice is untrammelled and when nothing prevents our being able to do what we like best, every pleasure is to be welcomed and every pain avoided. But in certain circumstances and owing to the claims of duty or the obligations of business it will frequently occur that pleasures have to be repudiated and annoyances accepted. The wise man therefore always holds in these matters to this principle of selection: he rejects pleasures to secure other greater pleasures, or else he endures pains to avoid worse pains.",
"lorem1": "At vero eos et accusamus et iusto odio dignissimos ducimus qui blanditiis praesentium voluptatum deleniti atque corrupti quos dolores et quas molestias excepturi sint occaecati cupiditate non provident, similique sunt in culpa qui officia deserunt mollitia animi, id est laborum et dolorum fuga. Et harum quidem rerum facilis est et expedita distinctio. Nam libero tempore, cum soluta nobis est eligendi optio cumque nihil impedit quo minus id quod maxime placeat facere possimus, omnis voluptas assumenda est, omnis dolor repellendus. Temporibus autem quibusdam et aut officiis debitis aut rerum necessitatibus saepe eveniet ut et voluptates repudiandae sint et molestiae non recusandae. Itaque earum rerum hic tenetur a sapiente delectus, ut aut reiciendis voluptatibus maiores alias consequatur aut perferendis doloribus asperiores repellat.",
"ipsum1":"On the other hand, we denounce with righteous indignation and dislike men who are so beguiled and demoralized by the charms of pleasure of the moment, so blinded by desire, that they cannot foresee the pain and trouble that are bound to ensue; and equal blame belongs to those who fail in their duty through weakness of will, which is the same as saying through shrinking from toil and pain. These cases are perfectly simple and easy to distinguish. In a free hour, when our power of choice is untrammelled and when nothing prevents our being able to do what we like best, every pleasure is to be welcomed and every pain avoided. But in certain circumstances and owing to the claims of duty or the obligations of business it will frequently occur that pleasures have to be repudiated and annoyances accepted. The wise man therefore always holds in these matters to this principle of selection: he rejects pleasures to secure other greater pleasures, or e",
"lorem2": "At vero eos et accusamus et iusto odio dignissimos ducimus qui blanditiis praesentium voluptatum deleniti atque corrupti quos dolores et quas molestias excepturi sint occaecati cupiditate non provident, similique sunt in culpa qui officia deserunt mollitia animi, id est laborum et dolorum fuga. Et harum quidem rerum facilis est et expedita distinctio. Nam libero tempore, cum soluta nobis est eligendi optio cumque nihil impedit quo minus id quod maxime placeat facere possimus, omnis voluptas assumenda est, omnis dolor repellendus. Temporibus autem quibusdam et aut officiis debitis aut rerum necessitatibus saepe eveniet ut et voluptates repudiandae sint et molestiae non recusandae. Itaque earum rerum hic tenetur a sapiente delectus, ut aut reiciendis voluptatibus maiores alias consequatur aut perferendis doloribus asperiores repellat.",
"ipsum2":"On the other hand, we denounce with righteous indignation and dislike men who are so beguiled and demoralized by the charms of pleasure of the moment, so blinded by desire, that they cannot foresee the pain and trouble that are bound to ensue; and equal blame belongs to those who fail in their duty through weakness of will, which is the same as saying through shrinking from toil and pain. These cases are perfectly simple and easy to distinguish. In a free hour, when our power of choice is untrammelled and when nothing prevents our being able to do what we like best, every pleasure is to be welcomed and every pain avoided. But in certain circumstances and owing to the claims of duty or the obligations of business it will frequently occur that pleasures have to be repudiated and annoyances accepted. The wise man therefore always holds in these matters to this principle of selection: he rejects pleasures to secure other greater pleasures, or else he endures pains to avoid worse pains.",
"lorem3": "At vero eos et accusamus et iusto odio dignissimos ducimus qui blanditiis praesentium voluptatum deleniti atque corrupti quos dolores et quas molestias excepturi sint occaecati cupiditate non provident, similique sunt in culpa qui officia deserunt mollitia animi, id est laborum et dolorum fuga. Et harum quidem rerum facilis est et expedita distinctio. Nam libero tempore, cum soluta nobis est eligendi optio cumque nihil impedit quo minus id quod maxime placeat facere possimus, omnis voluptas assumenda est, omnis dolor repellendus. Temporibus autem quibusdam et aut officiis debitis aut rerum necessitatibus saepe eveniet ut et voluptates repudiandae sint et molestiae non recusandae. Itaque earum rerum hic tenetur a sapiente delectus, ut aut reiciendis voluptatibus maiores alias consequatur aut perferendis doloribus asperiores repellat.",
"ipsum3":"On the other hand, we denounce with righteous indignation and dislike men who are so beguiled and demoralized by the charms of pleasure of the moment, so blinded by desire, that they cannot foresee the pain and trouble that are bound to ensue; and equal blame belongs to those who fail in their duty through weakness of will, which is the same as saying through shrinking from toil and pain. These cases are perfectly simple and easy to distinguish. In a free hour, when our power of choice is untrammelled and when nothing prevents our being able to do what we like best, every pleasure is to be welcomed and every pain avoided. But in certain circumstances and owing to the claims of duty or the obligations of business it will frequently occur that pleasures have to be repudiated and annoyances accepted. The wise man therefore always holds in these matters to this principle of selection: he rejects pleasures to secure other greater pleasures, or else he endures pains to avoid worse pains.",
"lorem4": "At vero eos et accusamus et iusto odio dignissimos ducimus qui blanditiis praesentium voluptatum deleniti atque corrupti quos dolores et quas molestias excepturi sint occaecati cupiditate non provident, similique sunt in culpa qui officia deserunt mollitia animi, id est laborum et dolorum fuga. Et harum quidem rerum facilis est et expedita distinctio. Nam libero tempore, cum soluta nobis est eligendi optio cumque nihil impedit quo minus id quod maxime placeat facere possimus, omnis voluptas assumenda est, omnis dolor repellendus. Temporibus autem quibusdam et aut officiis debitis aut rerum necessitatibus saepe eveniet ut et voluptates repudiandae sint et molestiae non recusandae. Itaque earum rerum hic tenetur a sapiente delectus, ut aut reiciendis voluptatibus maiores alias consequatur aut perferendis doloribus asperiores repellat.",
"ipsum4":"On the other hand, we denounce with righteous indignation and dislike men who are so beguiled and demoralized by the charms of pleasure of the moment, so blinded by desire, that they cannot foresee the pain and trouble that are bound to ensue; and equal blame belongs to those who fail in their duty through weakness of will, which is the same as saying through shrinking from toil and pain. These cases are perfectly simple and easy to distinguish. In a free hour, when our power of choice is untrammelled and when nothing prevents our being able to do what we like best, every pleasure is to be welcomed and every pain avoided. But in certain circumstances and owing to the claims of duty or the obligations of business it will frequently occur that pleasures have to be repudiated and annoyances accepted. The wise man therefore always holds in these matters to this principle of selection: he rejects pleasures to secure other greater pleasures, or else he endures pains to avoid worse pains."
}

from owasp-modsecurity-crs.

airween avatar airween commented on August 11, 2024

I'm afraid we can't help with the available information.

The JSON contents above are very different: the first one has so many keys, and all child have more child items. The second one has 10 keys (as I see) without any sub-child.

If a rule has an argument XML:/*, it means all keys will checked. That could be so many time...

Yes, you're right - the XML content doesn't trigger this latency - I have no idea why. But I think this means this issue is not CRS related.

You should start to turn off each rule set, start with 901. If this modification has no effect, turn back and take next one. Iterate this while you get a better result. Then you found the source of your problem, and can check each rule in that file.

from owasp-modsecurity-crs.

rsbrisci avatar rsbrisci commented on August 11, 2024

Thanks @airween !

Agree, there's not enough info yet to draw conclusions.

For now, my current theory is that something about lists with multiple items, multiple nested keys, might cause high latency in some particular rule(s).

I'm pretty sure XML vs JSON doesn't really matter. I just happened to run my original test with a particularly "bad" JSON. I ran a "List" version of this payload earlier today too, and that did not produce similar high latency.

I also attempted running Modsecurity with CRS disabled on the original "bad" JSON, and that too did not produce high latency - that's the only reason I was eyeing something in CRS.

Will attempt to identify which rule(s) seem to take excess time on the "bad" JSON.

from owasp-modsecurity-crs.

airween avatar airween commented on August 11, 2024

I made some researches, let me share with you the results. I sent all of three payloads above to my test Nginx with curl, and reviewed the modsec_debug.log.

The first column in the line is the line number in file (all requests logged into same file). The first line of three pairs is the beginning of random rule, the last is same with next one. Just see the difference between the first and last line numbers. That means, how many steps required to execute a rule.

  1. 'High-latency JSON' payload
   407  [158573518154.925648] [/] [4] (Rule: 942100) Executing operator "DetectSQLi against REQUEST_COOKIES|...|ARGS|XML:/*.
  4268  [158573518154.925648] [/] [4] (Rule: 942140) Executing operator "Rx" with param "(?i:\b(...)" against REQUEST_COOKIES|REQUEST_COOKIES_NAMES|ARGS_NAMES|ARGS|XML:/*.

3861 lines.

  1. XML payload
129999  [158573526493.497825] [/] [4] (Rule: 942100) (Rule: 942100) Executing operator "DetectSQLi against REQUEST_COOKIES|...|ARGS|XML:/*.
130014  [158573526493.497825] [/] [4] (Rule: 942140) Executing operator "Rx" with param "(?i:\b(...)" against REQUEST_COOKIES|REQUEST_COOKIES_NAMES|ARGS_NAMES|ARGS|XML:/*.

15 lines.

  1. JSON with 10 keys
132705  [158573628522.775075] [/] [4] (Rule: 942100) Executing operator "DetectSQLi against REQUEST_COOKIES|...|ARGS|XML:/*.
132834  [158573628522.775075] [/] [4] (Rule: 942140) Executing operator "Rx" with param "(?i:\b(...)" against REQUEST_COOKIES|REQUEST_COOKIES_NAMES|ARGS_NAMES|ARGS|XML:/*.

129 lines.

The other lines between two quotes lines above contains all steps, including transformations, and all operator evaluations.

Please check these values at your side - I think you can see the reason, why is so slowly the JSON payload what you included.

Hope this helps.

from owasp-modsecurity-crs.

rsbrisci avatar rsbrisci commented on August 11, 2024

@airween thank you for the analysis!

Just one last Q - is this behavior consistent between all rules for the "High Latency" JSON? Or just for the 942100 rule?

from owasp-modsecurity-crs.

airween avatar airween commented on August 11, 2024

No, not just for the rule 942100 - that's just a random chosed :).

It's consistent between all rules (which affected on your chosed PL), just check your log.
The "problem" is that JSON parser converts the tree hierarchy into a flat structure, and the engine handles it as ARGS and ARGS_NAMES. Any rule which contains these variables will check all 'flatted' list.

Just check your modsec_debug.log.

from owasp-modsecurity-crs.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.