sigmahq / sigma-cli Goto Github PK
View Code? Open in Web Editor NEWThe Sigma command line interface based on pySigma
The Sigma command line interface based on pySigma
Hello,
I tried to use sigma plugin install <backend>
as described in the README (for several different backends), but I got Error: No such command 'plugin'.
Hello! We've been using sigma for a while to do some conversion to splunk and in the workflow automation I have setup, we noticed the sigma check
when running it through a python subprocess.run()
is giving a returncode of 0 when it should be a 1 in regards to an invalid tags
value.
When running the check manually it has the error in the output like so
(venv) ➜ sec-sigma-rules (main) ✗ sigma check -e rules/production/wazuh/wazuh_Threat_Kubectl_Rule.yml
Parsing Sigma rules [------------------------------------] 0%
Check error: Sigma tag must start with namespace separated with dot from remaining tag.
It doesn't fail and continues on to try and push the rule to splunk which fails down the line.
In python it looks like this:
CompletedProcess(args='sigma check -e rules/production/wazuh/wazuh_Threat_Kubectl_Rule.yml', returncode=0, stdout='', stderr='Parsing Sigma rules\nCheck error: Sigma tag must start with namespace separated with dot from remaining tag.\n')
I notice with other errors it does return a returncode=1
so our tests fail before it gets to the point of trying to push to splunk with the API. Here's and example of the status
field of a sigma rule being invalid and the returncode is appropriate:
CompletedProcess(args='sigma check -e rules/production/wazuh/wazuh_Threat_Kubectl_Rule.yml', returncode=1, stdout="=== Sigma Rule Errors ===\n'blahblahblah' is no valid Sigma rule status in /Users/cchaffee/kayak_repos/sec-sigma-rules/rules/production/wazuh/wazuh_Threat_Kubectl_Rule.yml\n\n=== Summary ===\nFound 1 errors, 0 condition errors and 0 issues.\n\nRule error summary:\n+-------+------------------+\n| Count | Rule Error |\n+-------+------------------+\n| 1 | SigmaStatusError |\n+-------+------------------+\nNo condition errors found.\nNo validation issues found.\nCheck failure\n", stderr='Parsing Sigma rules\nChecking Sigma rules\n')
Could an error on an invalid tag value also return a 1? Maybe there is a better way to do this with python(?), but up until now the checks have been failing/succeeding appropriately until the tags.
Thanks!
issue=NumberAsStringIssue severity=low description="A number was expressed as string" rules=[D:\rootme\sigma\rules\windows\builtin\printservice\win_exploit_cve_2021_1675_printspooler_operational.yml] string= 123
issue=NumberAsStringIssue severity=low description="A number was expressed as string" rules=[D:\rootme\sigma\rules\windows\builtin\printservice\win_exploit_cve_2021_1675_printspooler_operational.yml] string= 1234
In the rules :
keywords:
- 'UNIDRV.DLL, kernelbase.dll, '
- ' 123 '
- ' 1234 '
- 'mimispool'
there is a space before and after so this is a string .
Sigma successfully installed, but it throws an error (python3.10):
:~$ sigma
Traceback (most recent call last):
File "/usr/local/bin/sigma", line 5, in <module>
from sigma.cli.main import main
ModuleNotFoundError: No module named 'sigma.cli'
:~$ python3 -m pip install sigma-cli
Defaulting to user installation because normal site-packages is not writeable
Requirement already satisfied: sigma-cli in /usr/local/lib/python3.10/dist-packages (1.0.2)
Requirement already satisfied: prettytable<4.0.0,>=3.1.1 in ./.local/lib/python3.10/site-packages (from sigma-cli) (3.10.0)
Requirement already satisfied: colorama<0.5.0,>=0.4.6 in ./.local/lib/python3.10/site-packages (from sigma-cli) (0.4.6)
Requirement already satisfied: click<9.0.0,>=8.0.3 in ./.local/lib/python3.10/site-packages (from sigma-cli) (8.1.7)
Requirement already satisfied: pysigma<0.12.0,>=0.11.3 in ./.local/lib/python3.10/site-packages (from sigma-cli) (0.11.7)
Requirement already satisfied: wcwidth in ./.local/lib/python3.10/site-packages (from prettytable<4.0.0,>=3.1.1->sigma-cli) (0.2.13)
Requirement already satisfied: jinja2<4.0,>=3.1 in ./.local/lib/python3.10/site-packages (from pysigma<0.12.0,>=0.11.3->sigma-cli) (3.1.2)
Requirement already satisfied: packaging<24.0,>=22.0 in ./.local/lib/python3.10/site-packages (from pysigma<0.12.0,>=0.11.3->sigma-cli) (23.2)
Requirement already satisfied: requests<3.0,>=2.31 in ./.local/lib/python3.10/site-packages (from pysigma<0.12.0,>=0.11.3->sigma-cli) (2.31.0)
Requirement already satisfied: pyparsing<4.0,>=3.1 in ./.local/lib/python3.10/site-packages (from pysigma<0.12.0,>=0.11.3->sigma-cli) (3.1.2)
Requirement already satisfied: pyyaml<7.0,>=6.0 in ./.local/lib/python3.10/site-packages (from pysigma<0.12.0,>=0.11.3->sigma-cli) (6.0.1)
Requirement already satisfied: MarkupSafe>=2.0 in ./.local/lib/python3.10/site-packages (from jinja2<4.0,>=3.1->pysigma<0.12.0,>=0.11.3->sigma-cli) (2.1.3)
Requirement already satisfied: urllib3<3,>=1.21.1 in ./.local/lib/python3.10/site-packages (from requests<3.0,>=2.31->pysigma<0.12.0,>=0.11.3->sigma-cli) (2.2.1)
Requirement already satisfied: charset-normalizer<4,>=2 in ./.local/lib/python3.10/site-packages (from requests<3.0,>=2.31->pysigma<0.12.0,>=0.11.3->sigma-cli) (3.3.2)
Requirement already satisfied: certifi>=2017.4.17 in /usr/lib/python3/dist-packages (from requests<3.0,>=2.31->pysigma<0.12.0,>=0.11.3->sigma-cli) (2020.6.20)
Requirement already satisfied: idna<4,>=2.5 in /usr/lib/python3/dist-packages (from requests<3.0,>=2.31->pysigma<0.12.0,>=0.11.3->sigma-cli) (3.3)
Hello, I tred to use sigma convert, but I got a error in both windows 11 and linux when I run the following sigma command. And the elasticsearch backend is installed successfully.
└─# ./sigma plugin install elasticsearch 1 ⨯
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
Successfully installed plugin 'elasticsearch'
┌──(rootkali)-[/usr/local/software/Python-3.9.10/bin]
└─# ./sigma convert -t elasticsearch -p sysmon /home/kali/Downloads/proc_creation_win_webshell_recon_detection.yml
Usage: sigma convert [OPTIONS] INPUT...
Try 'sigma convert --help' for help.
Error: Invalid value for '--target' / '-t': 'elasticsearch' is not one of . - run sigma plugin list --plugin-type backend for a list of available plugins.```
Hello !
When trying to convert a sigma rule into savedseaches format i get an error from splunk backend (backends/splunk/splunk.py
).
Conversion goes smoothly without the -f savedsearches
flag or with -f default
.
It looks like a bug, but maybe I am missusing the tool.
I am trying to dig deeper into the code to understand the error, but I am not too familiar with it yet.
module | version |
---|---|
pysigma | 0.9.2 |
pysigma-backend-splunk | 1.0.2 |
pysigma-pipeline-sysmon | 1.0.2 |
pysigma-pipeline-windows | 1.1.0 |
sigma-cli | 0.6.0 |
Command
sigma convert -t splunk -f savedsearches -p windows-audit sigma/rules/windows/process_creation/proc_creation_win_susp_whoami.yml
Stack trace
Traceback (most recent call last):
File "/home/<user>/.local/bin/sigma", line 8, in <module>
sys.exit(main())
File "/home/<user>/.local/lib/python3.9/site-packages/sigma/cli/main.py", line 17, in main
cli()
File "/home/<user>/.local/lib/python3.9/site-packages/click/core.py", line 1130, in __call__
return self.main(*args, **kwargs)
File "/home/<user>.local/lib/python3.9/site-packages/click/core.py", line 1055, in main
rv = self.invoke(ctx)
File "/home/<user>/.local/lib/python3.9/site-packages/click/core.py", line 1657, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/home/<user>/.local/lib/python3.9/site-packages/click/core.py", line 1404, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/home/<user>t/.local/lib/python3.9/site-packages/click/core.py", line 760, in invoke
return __callback(*args, **kwargs)
File "/home/<user>/.local/lib/python3.9/site-packages/sigma/cli/convert.py", line 147, in convert
result = backend.convert(rule_collection, format)
File "/home/<user>/.local/lib/python3.9/site-packages/sigma/conversion/base.py", line 97, in convert
return self.finalize(queries, output_format or self.default_format)
File "/home/<user>/.local/lib/python3.9/site-packages/sigma/conversion/base.py", line 356, in finalize
return self.__getattribute__("finalize_output_" + output_format)(queries)
File "/home/<user>.local/lib/python3.9/site-packages/sigma/backends/splunk/splunk.py", line 126, in finalize_output_savedsearches
return f"\n[default]" + self._generate_settings(self.output_settings) + "\n" + "\n".join(queries)
File "/home/<user>/.local/lib/python3.9/site-packages/sigma/backends/splunk/splunk.py", line 98, in _generate_settings
output += f"\n{k} = " + " \\\n".join(v.split("\n")) # cannot use \ in f-strings
AttributeError: 'NoneType' object has no attribute 'split'
(sorry for the multiple issues recently)
Hello, I am wondering if you would be open to some modifications to this project. I know this is designed as a cli however my team would really like to use this project inside of notebooks to test and convert our sigma rules. The trouble is most of the functions we wish to use (such as the convert functions) have click annotations and do not return the data we need. So what I'm really asking is would you be open to changes that allow this to be used in a notebook? or do you think that there is a better way to meet this usecase?
Right now, there is a tight coupling between backends and sigma-cli. Although you're defining output formats in the backend, to use them with sigma-cli, you need to modify its code to add new output format(s). It would be easier if backend provided that information to sigma-cli, so no code modification is required.
Successfully installed sigma-cli-0.7.2
% sigma
Traceback (most recent call last):
File "/opt/homebrew/bin/sigma", line 5, in <module>
from sigma.cli.main import main
File "/opt/homebrew/lib/python3.9/site-packages/sigma/cli/main.py", line 3, in <module>
from .list import list_group
File "/opt/homebrew/lib/python3.9/site-packages/sigma/cli/list.py", line 6, in <module>
plugins = InstalledSigmaPlugins.autodiscover()
File "/opt/homebrew/lib/python3.9/site-packages/sigma/plugins.py", line 65, in autodiscover
backends = cls._discover_module_directories(sigma.backends, "backends", include_backends)
File "/opt/homebrew/lib/python3.9/site-packages/sigma/plugins.py", line 53, in _discover_module_directories
imported_module = importlib.import_module(module.name)
File "/opt/homebrew/Cellar/[email protected]/3.9.16/Frameworks/Python.framework/Versions/3.9/lib/python3.9/importlib/__init__.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "/opt/homebrew/lib/python3.9/site-packages/sigma/backends/insight_idr/__init__.py", line 1, in <module>
from .insight_idr import InsightIDRBackend
File "/opt/homebrew/lib/python3.9/site-packages/sigma/backends/insight_idr/insight_idr.py", line 12, in <module>
class InsightIDRBackend(TextQueryBackend):
File "/opt/homebrew/lib/python3.9/site-packages/sigma/backends/insight_idr/insight_idr.py", line 14, in InsightIDRBackend
backend_processing_pipeline : ClassVar[ProcessingPipeline] = insight_idr_pipeline()
File "/opt/homebrew/lib/python3.9/site-packages/sigma/pipelines/insight_idr/insight_idr.py", line 65, in insight_idr_pipeline
ProcessingItem(
File "<string>", line 14, in __init__
File "/opt/homebrew/lib/python3.9/site-packages/sigma/processing/pipeline.py", line 125, in __post_init__
raise SigmaTypeError(f"Detection item processing condition '{str(detection_item_condition)}' is not a DetectionItemProcessingCondition")
sigma.exceptions.SigmaTypeError: Detection item processing condition 'IncludeFieldCondition(fields=['CurrentDirectory', 'IntegrityLevel', 'imphash', 'Imphash', 'LogonId'], type='plain')' is not a DetectionItemProcessingCondition
Hello,
The SentinelOne backend doesnt seem to handle lengthy lists within a field's key such as CommandLine|contains|any. For example, proc_creation_macos_disable_security_tools.yml ends up exceeding S1's max operators limit of 20, thus needing manual modification.
I havent run into this for other rules I've been testing with, but I may just not have encountered another case yet. Thanks for supporting this backend!
Hi, I'm trying to configure a YAML config for sigma check
to remove some validators.
However, I don't know how to go from sigma check
's output:
to knowing which entries to add to my YAML config.
This seems like a UX issue. Perhaps a command sigma check list
that outputs all validators, or change the output of sigma check
to output the name of the validators that cause issues? I'm not sure, but something is currently missing and it makes it hard to know what to input in the config.
sigma check
should have an argument like --exclude
that ignores a given list of validators. It would be used like this:
sigma check ./rules --exclude InvalidRelatedIdValidator InvalidRelatedSubfieldValidator InvalidRelatedTypeValidator
It would allow backend devs and detection-as-code pipelines to disable default validators that we do not want to include in validations.
Thoughts?
The -P
short option is used for two different options:
-P, --without-pipeline Proceed with conversion without processing
pipeline, even if it is mandatory for the
target.
-P, --file-pattern TEXT Pattern for file names to be included in
recursion into directories. [default:
*.yml]
Hi,
Are there any plans to add support for reading rules from standard input?
I have a wrapper built in nodejs that is currently using sigmac
for rule conversion
I have the same issue as the #29
How did you manage to resolve it ?
I don't understand why there is no probleme with the splunk module
We have tried both plugin qradar & ibm-qradar-aql and the available pipelines but there is an error like the one in title. Here is the command
cmd>sigma convert -t ibm-qradar-aql -p qradar-aql-fields D:\user\sigma-master\rules\windows\process_creation
Parsing Sigma rules [####################################] 100%
Error while conversion: field 'Description' is not supported
Any workaround for this
Steps to repeat:
Windows 10 Professional operating system. Standard Poetry install.
git clone https://github.com/SigmaHQ/sigma-cli.git
cd sigma-cli
poetry install
poetry shell
sigma plugin install sentinelone-pq
plugin
commands, meaning you can't even attempt undo your mistake.poetry remove pySigma
then poetry add pySigma==0.11.6
(the version it was before the plugin install)This could possibly be called an issue with the backend rather than sigma-cli but I would argue that the sigma plugin install
command should not be writing over existing dependencies. If I were to install one of these backends myself, I would expect poetry or pip to throw up a dependency incompatibility.
As found when checking stdin in #21, the check
functionality currently treats an empty Path specification as normal, giving the output as if rules were processed.
(sigma-cli-py3.11) ❯❯❯❯ cat rules/windows/process_creation/certutil_susp_download.yml | sigma check && echo $?
DEBUG: input: ()
Parsing Sigma rules [####################################] 100%
Checking Sigma rules [####################################] 100%
=== Summary ===
Found 0 errors, 0 condition errors and 0 issues.
No rule errors found.
No condition errors found.
No validation issues found.
0
(sigma-cli-py3.11) ❯❯❯❯ sigma check && echo $?
DEBUG: input: ()
Parsing Sigma rules [####################################] 100%
Checking Sigma rules [####################################] 100%
=== Summary ===
Found 0 errors, 0 condition errors and 0 issues.
No rule errors found.
No condition errors found.
No validation issues found.
0
This issue could extend #21 in allowing standard input for check
while providing a message supplied by the program to indicate that no rules were found and as such no rules were processed (indicating as such in the summary).
Preferably this would produce an exit(1)
, but this is open to debate, or a potential configuration option.
Hello, thank you for maintaining tools :)
I found the following issue, so I would appreciate it if you could check it.
Describe the bug
sigma list pipelines
command failed with latest simga-cli(0.5.3).
Step to Reproduce
python3 -m pip install sigma-cli
sigma list pipelines
Expected behavior
sigma list pipelines
command succeeds.
Actual behavior
The command failed with the following stack trace.
...% sigma list pipelines
Traceback (most recent call last):
File "/Users/fukusuke/.pyenv/versions/3.11.0/bin/sigma", line 8, in <module>
sys.exit(main())
^^^^^^
File "/Users/fukusuke/.pyenv/versions/3.11.0/lib/python3.11/site-packages/sigma/cli/main.py", line 15, in main
cli()
File "/Users/fukusuke/.pyenv/versions/3.11.0/lib/python3.11/site-packages/click/core.py", line 1130, in __call__
return self.main(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/fukusuke/.pyenv/versions/3.11.0/lib/python3.11/site-packages/click/core.py", line 1055, in main
rv = self.invoke(ctx)
^^^^^^^^^^^^^^^^
File "/Users/fukusuke/.pyenv/versions/3.11.0/lib/python3.11/site-packages/click/core.py", line 1657, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/fukusuke/.pyenv/versions/3.11.0/lib/python3.11/site-packages/click/core.py", line 1657, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/fukusuke/.pyenv/versions/3.11.0/lib/python3.11/site-packages/click/core.py", line 1404, in invoke
return ctx.invoke(self.callback, **ctx.params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/fukusuke/.pyenv/versions/3.11.0/lib/python3.11/site-packages/click/core.py", line 760, in invoke
return __callback(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/fukusuke/.pyenv/versions/3.11.0/lib/python3.11/site-packages/sigma/cli/list.py", line 49, in list_pipelines
pipeline = pipeline_resolver.resolve_pipeline(name)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/fukusuke/.pyenv/versions/3.11.0/lib/python3.11/site-packages/sigma/processing/resolver.py", line 32, in resolve_pipeline
return pipeline()
^^^^^^^^^^
File "/Users/fukusuke/.pyenv/versions/3.11.0/lib/python3.11/site-packages/sigma/pipelines/splunk/splunk.py", line 58, in splunk_windows_pipeline
items=[
^
File "/Users/fukusuke/.pyenv/versions/3.11.0/lib/python3.11/site-packages/sigma/pipelines/splunk/splunk.py", line 61, in <listcomp>
transformation=AddConditionTransformation({ "source": "WinEventLog:" + source}),
~~~~~~~~~~~~~~~^~~~~~~~
TypeError: can only concatenate str (not "list") to str
Environment
I confirmed in the following environments.
Additional context
Above error occurs in the following code. It seems that originally source
was expected to be of type str
(not list
)
However, the windows_logsource_mapping
value now contains the type list
from the following commits.(pySigma 0.8.10
)
Because the pySigma version is 0.8.1
in poetry.lock
, this issue did not occur when I did with poetry install
.
It seems to be an issue related to pySigma-backend-splunk
, so it may be better to create an issue in the pySigma-backend-splunk repository
?
Command:
sigma convert -t qradar -p qradar_pipeline.yaml rules\windows\dns_query\dns_query_win_anonymfiles_com.yml
Error:
Error while conversion: The QRadar savedsearches Sigma backend supports only the following fields for process_creation log source
I think there should be something after "log source" in the error message but there is not.
Not working:
name: Qradar Pipeline
priority: 100
transformations:
Not working:
name: Qradar Pipeline
priority: 100
transformations:
Working:
name: Qradar Pipeline
priority: 100
transformations:
Working:
name: Qradar Pipeline
priority: 100
transformations:
Currently, it is not supported to use sigma convert --pipeline ./my_pipelines_directory [...]
We're starting to use processing pipelines internally and I wanted to segment processing pipelines by files in a same folder.
Now, when converting, I need to specify manually all the pipelines files, but I think it would be cleaner to be able to specify a directory and that all the YAML pipelines definitions are used in the conversion.
Thoughts?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.