[Meta] Address Non-Performant Rules with Wildcard Usage about detection-rules HOT 5 OPEN

Update 03-03-2024

• All rules shared from SDH have been reviewed and adjusted, with draft pull requests for each
• As intended, each query was reviewed for a list of performance checks that are listed below
• Note - each pull request is in draft until we can identify performance changes accurately
  • Each query was EQL and we do not have an EQL profiler to assess performance changes introduced, our initial changes are based on the assumption of EQL functions vs wildcard usage where wildcard usage is more expensive
• Global alert telemetry was reviewed for each rule in an attempt to identify efficacy tuning by removing false-positives or making query adjustments
• Comments were added to each query in an attempt to better explain the logic, especially if complex
• ### Performance headers were added where an investigation guide existed. Content beneath this was added to explain potential performance impact of the rule and why according to the logic
• note was added to each rule where it did not exist prior to. The value added explained potential performance impact fo the rule and why according to the logic
• Rules were reviewed for deprecation if efficacy remained low and could not be adjusted

Considerations for performance:

Lookback Time Window Adequacy:

• Ensure the lookback time window is set to an adequate duration to capture the intended events without overlapping or redundancy.

Index Scoping:

• Scope indexes tightly to specific data streams expected to contain the relevant events, based on event category, to reduce querying across unnecessary data sources.

Use of EQL Sequence Queries:

• If using EQL sequence queries, consider converting to non-sequence queries if possible to simplify logic and reduce processing time.
• For sequence queries, ensure a maxspan is defined to limit the volume of events captured. Adjust as necessary to balance coverage and performance.
• Optimize each sequence to filter out noise or false positives, reducing the volume of events held in memory.

Event Type Specification:

• Start EQL queries by specifying the event category to focus the search and avoid querying all event categories.

Wildcard and Function Use:

• Evaluate the use of wildcards and consider replacing them with EQL functions like startsWith and stringContains to maintain logical targeting for the threat while potentially improving performance.
• If using wildcards, they can only be used with specific operators : and like

Filtering Redundancy:

• Review and reduce the amount of filtering to eliminate redundant logic that checks for similar values or conditions, streamlining the query.

Join Operations and Process Termination:

• If joins are used (e.g., for process.entity_id), consider adding an until operation explicitly for process ending to optimize search duration.
• If different event categories are used in each sequence, do we need to do additional joint operations on specific fields

from detection-rules.

Update 03-04-2024

While we await feedback from the query team, we were introduced to https://github.com/elastic/rally which may help us benchmark the searches for each rule adjustment we have done so far. This will help us determine if adjustments are more performant or not. We will move on to testing this while we await feedback.

from detection-rules.

Update 03-27-2024

• The rule performance concerns that relate to these rules has been solved via architecture adjustments to the customers ingestion, shard allocation and more
• We have unsuccessfully been able to conclude that changing from asterisks to EQL functions where plausible increases performance when the rule is executed due to a lack of benchmarking and performance testing options for EQL
• This issue is no longer tied directly to the urgent SDH and will be treated as a meta to tune several rules for not only performance, but false-positives as well.
• Rules will need to be revisited, re-ran and the appropriate details shared within each pull request before requesting review.

from detection-rules.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

from detection-rules.

This has been closed due to inactivity. If you feel this is an error, please re-open and include a justifying comment.

from detection-rules.