New TAXII Miner for MineMeld
License: Apache License 2.0
This is an unofficial repository for AWS and Azure templates to deploy VM-Series Next-Generation firewalls from Palo Alto Networks into the respective public clouds.
minemeld-web service throws this error when starting with minemeld-taxii-ng version 0.2a4.
[2021-06-25 14:54:06 EDT] [1877] [INFO] Loading blueprint from taxiing
[2021-06-25 14:54:06 +0000] [1877] [ERROR] Exception in worker process
Traceback (most recent call last):
......
File "/opt/minemeld/engine/core/minemeld/flask/main.py", line 3, in <module>
app = create_app()
File "/opt/minemeld/engine/core/minemeld/flask/__init__.py", line 113, in create_app
app.register_blueprint(bprint)
File "/opt/minemeld/engine/current/local/lib/python2.7/site-packages/flask/app.py", line 64, in wrapper_func
return f(self, *args, **kwargs)
File "/opt/minemeld/engine/current/local/lib/python2.7/site-packages/flask/app.py", line 941, in register_blueprint
if blueprint.name in self.blueprints:
AttributeError: 'function' object has no attribute 'name'
minemeld-web starts normally.
minemeld-web.log reports this error and exits.
[2021-06-25 14:54:06 EDT] [1877] [INFO] Loading blueprint from taxiing
[2021-06-25 14:54:06 +0000] [1877] [ERROR] Exception in worker process
Traceback (most recent call last):
File "/opt/minemeld/engine/current/local/lib/python2.7/site-packages/gunicorn/arbiter.py", line 517, in spawn_worker
worker.init_process()
File "/opt/minemeld/engine/current/local/lib/python2.7/site-packages/gunicorn/workers/ggevent.py", line 190, in init_process
super(GeventWorker, self).init_process()
File "/opt/minemeld/engine/current/local/lib/python2.7/site-packages/gunicorn/workers/base.py", line 122, in init_process
self.load_wsgi()
File "/opt/minemeld/engine/current/local/lib/python2.7/site-packages/gunicorn/workers/base.py", line 132, in load_wsgi
self.wsgi = self.app.wsgi()
File "/opt/minemeld/engine/current/local/lib/python2.7/site-packages/gunicorn/app/base.py", line 67, in wsgi
self.callable = self.load()
File "/opt/minemeld/engine/current/local/lib/python2.7/site-packages/gunicorn/app/wsgiapp.py", line 65, in load
return self.load_wsgiapp()
File "/opt/minemeld/engine/current/local/lib/python2.7/site-packages/gunicorn/app/wsgiapp.py", line 52, in load_wsgiapp
return util.import_app(self.app_uri)
File "/opt/minemeld/engine/current/local/lib/python2.7/site-packages/gunicorn/util.py", line 357, in import_app
__import__(module)
File "/opt/minemeld/engine/core/minemeld/flask/main.py", line 3, in <module>
app = create_app()
File "/opt/minemeld/engine/core/minemeld/flask/__init__.py", line 113, in create_app
app.register_blueprint(bprint)
File "/opt/minemeld/engine/current/local/lib/python2.7/site-packages/flask/app.py", line 64, in wrapper_func
return f(self, *args, **kwargs)
File "/opt/minemeld/engine/current/local/lib/python2.7/site-packages/flask/app.py", line 941, in register_blueprint
if blueprint.name in self.blueprints:
AttributeError: 'function' object has no attribute 'name'
Workaround:
Delete the extension from the /opt/minemeld/local/library/ folder.
Restart minemeld-web.
Re-import extension and activate it.
I would like to use minemeld-taxii-ng plugin with MineMeld to parse STIX 2.0 indicators transported via taxii_11 protocol.
My indicator looks like:
{
"type": "indicator",
"id": "indicator--ded913b4-ef6e-43e6-9b3c-ddfdf286eda6",
"created": "2022-03-22T18:10:28.000Z",
"modified": "2022-03-22T18:10:28.000Z",
"name": "Malware variant",
"description": "Each of these file hashes indicates that a variant of Win32/Emotet.CD trojan is present.",
"pattern": "[file:hashes.'SHA-256'='6100fb1ff5b9fa6c73c60c3e57c61879b7f5c9a73f110d34f7e4e7c3aeef6e18'] OR [file:hashes.'SHA-1'='c9c358f69a90a1433e99d44f9940e0e9400d78e6'] OR [file:hashes.'MD5'='48132ad061b7d2190bbe797a764b880b']",
"valid_from": "2022-03-22T18:10:28Z",
"valid_until": "2022-03-24T18:10:28Z",
"labels": [
"malicious-activity"
]
},
But INDICATOR displayed by MineMeld looks like:
[file:hashes.'SHA-256'='6100fb1ff5b9fa6c73c60c3e57c61879b7f5c9a73f110d34f7e4e7c3aeef6e18'] OR [file:hashes.'SHA-1'='c9c358f69a90a1433e99d44f9940e0e9400d78e6'] OR [file:hashes.'MD5'='48132ad061b7d2190bbe797a764b880b']
I'm unable to filter hash types like 'sha256', 'sha1', 'md5' and use only one hash type in minemeld processor (aggregator).
I tried to modify the regex STIX2_SIMPLE_PATTERN in pattern.py file but with no success.
If I use aggregator based on minemeld.ft.op.AggregateFT CLASS with infilters only accepting all (accept) i can get output with the whole "pattern" field.
This looks like this:
[file:hashes.'SHA-256'='6100fb1ff5b9fa6c73c60c3e57c61879b7f5c9a73f110d34f7e4e7c3aeef6e18'] OR [file:hashes.'SHA-1'='c9c358f69a90a1433e99d44f9940e0e9400d78e6'] OR [file:hashes.'MD5'='48132ad061b7d2190bbe797a764b880b']
So, please what should I modify to correctly parse only the usable types from the "pattern" field
There is also file: encode.py which contains some try to TYPE_CONVERSION:
TYPE_CONVERSION = {
'IPv4': 'ipv4-addr:value',
'IPv6': 'ipv6-addr:value',
'domain': 'domain-name:value',
'URL': 'url:value',
'sha256': "file:hashes.'SHA-256'",
'sha1': "file:hashes.'SHA-1'",
'sha512': "file:hashes.'SHA-512'",
'md5': "file:hashes.MD5",
'ssdeep': "file:hashes.ssdeep"
}
According this, I guess, types parsing is planned and should be functional
In MineMeld LOG I can also see correctly parsed/processed fields like stix2_created, stix2_id, stix2_name, sources, first_seen, etc.
Example:
{
"stix2_created": 1647972628000,
"_age_out": 1650563662181,
"stix2_id": "indicator--ded913b4-ef6e-43e6-9b3c-ddfdf286eda6",
"stix2_name": "Malware variant",
"_last_run": 1647971662181,
"sources": [
"eti_eset_botnet"
],
"first_seen": 1647972628000,
"last_seen": 1647971662181,
"type": "stix2-pattern",
"stix2_labels": [
"malicious-activity"
],
"stix2_modified": 1647972628000
}
Thank you very much for your help :-)
This issue was opened by a bot called Community Health (PANW) because this repo has failed too many community health checks.
Repo maintainers: Please take the time to fix the issues in the table to reach the target score. These improvements will help others find your work and contribute to it. This issue will update as your score improves until it hits the target score.
Click More info for instructions to fix each item.
| Health Check | Pass | Score | More Info |
|---|---|---|---|
| Contains a meaningful README.md file | 20 / 20 | More info | |
| SUPPORT.md file exists | 0 / 20 | More info | |
| Repo has a description | 15 / 15 | More info | |
| Has a recognized open source license | ✅ | 15 / 15 | More info |
| Has a descriptive repo name | 15 / 15 | More info | |
| Required topics attached to repo | 0 / 15 | More info | |
| CONTRIBUTING.md file with contribution guidelines | 0 / 5 | More info | |
| Has custom issue and pull request templates | 0 / 5 | More info |
Current score: 65
Target threshold: 100
Total possible: 110
The miner fails to parse indicators from the XML when polling the TAXII server.
After polling the server, the XML is parsed, reports success and updates the indicators.
When polling the server for TAXII data the miner does not parse the XML, reports success, and never adds new indicators.
When requesting TAXII data from a server that is sending back its response with Content-Encoding: gzip the miner is unable to parse the XML. This is due to the use of requests.Response.raw which is passed to the XML iterparse (here). The raw data is accessed before the response is GZIP decompressed (as opposed to, for example, requests.Response.text).
A simple workaround is to add on Accept-Encoding: none or Accept-Encoding: identity to request headers. Sample code below for this would start after this line:
reqhdrs['Accept-Encoding'] = 'none'
gzip or compress content encodingThis prevents us from the ability to retrieve indicators from TAXII servers.
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.