punch-cyber / stoq-plugins-public Goto Github PK

stoQ Public Plugins

License: Apache License 2.0

Python 84.65% Smarty 0.36% Java 0.21% YARA 14.78%

python3 malware-analysis malware-research malware-analyzer security-automation security malware-detection yara stoq-plugins

stoq-plugins-public's Introduction

Overview

stoQ is a automation framework that helps to simplify the mundane and repetitive tasks an analyst is required to do. It allows analysts and DevSecOps teams the ability to quickly transition between different data sources, databases, decoders/encoders, and numerous other tasks using enriched and consistent data structures. stoQ was designed to be enterprise ready and scalable, while also being lean enough for individual security researchers.

Documentation

If you're interested in learning more about stoQ, to include how to develop your own plugins, checkout the full documentation.

This git repository contains publicly available plugins that have been created for use with stoQ. The core stoQ repository can be found here.

Installation

Details on how to install these plugins can be found here.

Plugin List

Below is a listing of all public stoQ plugins, a description, and their respective plugin class.

Plugin Name	Description	Plugin Type
acce	Scan payloads using ACCE	Worker
azure_blob	Save results and archive payloads with Azure Blob Storage	Archiver, Connector
b64decode	Decode base64 encoded payloads	Worker
decompress	Extract content from a multitude of archive formats	Worker
dirmon	Monitor a directory for newly created files for processing	Provider
entropy	Calculate shannon entropy of a payload	Worker
es-search	Saves results to ElasticSearch	Connector
exif	Processes a payload using ExifTool	Worker
falcon-sandbox	Scan payloads using Falcon Sandbox	Worker
filedir	Ingest a file or directory for processing	Provider, Connector, Archiver
gcs	Read and write data to Google Cloud Storage	Archiver, Connector
hash	Hash content	Worker
hash_ssdeep	Generate a ssdeep hash of payloads	Worker
iocextract	Regex routines to extract and normalize IOC's from a payload	Worker
javaclass	Decodes and extracts information from Java Class files	Worker
jinja	Decorate results using a template	Connector, Decorator
kafka-queue	Publish and consume messages from a Kafka server	Archiver, Connector, Provider
lief	Parse and abstract PE, ELF and MachO files using LIEF	Worker
mimetype	Determine mimetype of a payload	Worker
mongodb	Save results and archive payloads to/from mongodb	Archiver, Connector
mraptor	Port of mraptor3 from oletools	Worker
ole	Carve OLE streams within Microsoft Office Documents	Worker
opswat	Scan payloads using OPSWAT MetaDefender	Worker
pecarve	Carve portable executable files from a data stream	Worker
peinfo	Gather relevant information about an executable using pefile	Worker
pubsub	Interact with Google Cloud Pub/Sub	Archiver, Connector, Provider
redis-queue	Interact with Redis server	Archiver, Connector, Provider
rtf	Extract objects from RTF payloads	Worker
s3	Read and write data to Amazon S3 buckets	Archiver, Connector
sentinel	Save results to Azure Sentinel	Connector
smtp	SMTP Parser Worker	Worker
stdout	Sends results to STDOUT	Connector
swfcarve	Carve and decompress SWF files from payloads	Worker
symhash	Calculate symbol table hashes of a Mach-O executable file	Worker
tika	Upload content to a Tika server for automated text extraction	Worker
tnef	TNEF File Extractor	Worker
trid	Identify file types from their TrID signature	Worker
vtmis-filefeed	Process VTMIS File Feed	Provider, Worker
vtmis-search	Search VTMIS for sha1 hash of a payload or from results of `iocextract` plugin	Worker, Dispatcher
xdpcarve	Carve and decode streams from XDP documents	Worker
xordecode	Decode XOR encoded payloads	Worker
xorsearch	Scan a payload using xorsearch	Worker
xyz	Extract Zip file metadata	Worker
yara	Process a payload using yara	Worker, Dispatcher

stoq-plugins-public's People

Contributors

Stargazers

Watchers

stoq-plugins-public's Issues

Missing MANIFEST.in

Most plug-ins have a MANIFEST.in referencing their stoq file(s)...
include decompress/*.stoq
...but these three omit it?:

jinja
peinfom
yara

TrID Results Count

Is your feature request related to a problem? Please describe.
Default size of TrID results.

Describe the solution you'd like
The TrID tool provides an option to set the limit on the number of results it will return...
-n:nn Number of matches to show (default: 5)
It would be nice to provide a trid.stoq option to allow setting that to a value beside the default.

VTMIS-Search Attribute Name /w Space

The VTMIS-Search output can include at least one key with a space in it's name...
results.workers.vtmis-search.additional_info.sigcheck."link date"
...which can be problematic for some tools.

Replacing the space with an underscore may be preferable...
results.workers.vtmis-search.additional_info.sigcheck.link_date

OPSWAT URL Problem?

I'm having trouble with the OPSWAT plugin. Using...

[opswat]
opswat_url = https://api.metadefender.com/v4/file
apikey = XXX

...set in stoq.cfg (Note: apikey is the public one), I get the following error on analyses...

[2019-12-12 16:20:28,856 ERROR] stoq: deep dispatch:failed to scan (pass 1/1)
Traceback (most recent call last):
  File "/home/pass/.stoq/.venv/lib/python3.7/site-packages/stoq/core.py", line 702, in _single_scan
    payload, request_meta
  File "/home/pass/.stoq/plugins/opswat/opswat.py", line 84, in scan
    response.raise_for_status()
  File "/home/pass/.stoq/.venv/lib/python3.7/site-packages/requests/models.py", line 940, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 400 Client Error: Bad Request for url: https://api.metadefender.com/v4/file

Performing a curl based request on the same URL...

curl -X POST https://api.metadefender.com/v4/file -H "apikey: XXX" -H 'content-type: application/octet-stream' -d @file_to_scan

Returns expected results...
{"data_id":"bzE5MTIxMkJ5ZnZYQVF4Q1NIa1hEWDBYZ0Fy","status":"inqueue","in_queue":2,"queue_priority":"normal","sha1":"2EF98367FBDCE44619C481DDEEBDD1D3E7D2113E","sha256":"1D32D496E4C58E673EB7693BF4BB39BC2A4FCFD29DEF74EED306DD8827C54903"}

Adding Yara Rules

Problem: Using the YARA worker does not seem to work. Results are always blank.

In looking at the stoq.yar file it seems to be an index yara that is including the alienvault and fireeye yara files. So to add more yara rules, is it just as simple as editing stoq.yar and adding the new entries?

For example, if I clone the main yara-rules repo, can I just replace the index.yar from there and have stoq use them instead?

IOCregex whitelist behavior

IOCregex reader plugin has a whitelist that differentiates between domain and url types. In most use cases, these lists are likely identical and easier to maintain as a single list.

Suggest merging the two into a single whitelist indicator type (recommend domain).

For additional flexibility, allow for type exception cases for either domain or url regex extraction. I
i.e., if .cnn.com is added to the whitelist, allow for an additional indicator type that excludes the whitelist entry from either url or domain matches.

i.e.
domain:.cnn.com
excepturl:.cnn.com

The above example would not include cnn.com to the url indicator type whitelist, but it would remain for the domain type.

VTMIS-Search References Worker Data in Dispatcher?

I'm confused by the VTMIS dispatcher...

    def get_dispatches(
        self, payload: Payload, request_meta: RequestMeta
    ) -> DispatcherResponse:
        """
        Check if `iocextract` plugin has results, if so, dispatch to `vtmis-search` worker

        """
        dr = DispatcherResponse()
        if 'iocextract' in payload.worker_results:
            dr.plugin_names.append('vtmis-search')
        return dr

It seems to references payload.worker_results, but I thought such were only available to Deep Dispatchers?

Decompress Tools Documentation

The TrID plug-in's documentation mentions that TrID must be installed...

Note: TRiD binary and definitions need to be installed for this plugin to operate properly.

The Decompression plug-in fails to mention that in it's default configuration it needs...

7z
upx
gzip
tar
ace

...also since 7z supports: tar, and gzip it might be preferable to minimize the number of dependencies by utilizing 7z for those types as well.

Recommend using YARA 3.11.0 for stoQ v3

Changing this plugin requirement to yara-python>=3.11 for stoq >= v3.0.0 would permit consistent use of rules with new keywords.

stoq-plugins-public/yara/requirements.txt

Line 1 in 1640340

yara-python>=3.8

Like xor, which could replace 254 rules with one that generates all 254.

stoq-plugins-public/yara/yarascan/rules/dispatcher.yar

Line 139 in 1640340

$xord_1 = { 55 69 68 72 21 71 73 6e 66 73 60 6c }

To copy directly from the YARA docs:

        $xor_string = "This program cannot" xor(0x01-0xff)

Add option to not save the body of an e-mail

SMTP Worker

Decompress Meta-data

The decompress plugin currently outputs nothing (besides additional payloads) when run...

"decompress": {}

It would be good to provide meta meta-data for archives. This could occur even if archives aren't or can't be extracted (i.e. too large to extract, or have other issues).

7z provides meta-data with...

7z l -slt file

The meta-data alone may be sufficient for analysis in some cases, so a mode to just extract meta-data without decompressing may also be useful.

Add SSL support to rabbitmq source plugin

maxdepth / iterations / files

A maximum_size variable is documented, does this plug-in not account for potentially malicious archives with:

very deep layers of nesting
very large numbers of tiny duplicate files
etc

suricata-dirmon

Noticed there was a suricata-dirmon: https://github.com/PUNCH-Cyber/stoq-plugins-public/tree/3d792144e3cc65f3a3984bae10538b3baf55ce69/source/suricata-dirmon that doesn't seem to be publicized. It included this comment...

    # A bit of a race condition here. Suricata writes a tmp file, until the stream is complete.
    # At that point, suricata moves file to a filename minus the .tmp extension which is not
    # detected by the filesystem as an on_created() event, thus we will never see it. Let's
    # assume that if the file name ends with a ".tmp" extension, by the time we get to loading
    # the file itself, suricata has finished writing to disk. Ugly? yes.

You can detect the finalized files being ready using the on_moved() event instead, which is what the renames should appear as.

Smtp plugin does not decode body or html sections

Hi, While testing a noticed that the smtp plugin does not decode base64 encoded body or html sections of the email. To resolve this I added the following code at line 108 in the scan() function (also replacing lines 109 & 110):

Extract the e-mail body, to include HTML if available

        if message.is_multipart():
            retrieve_body = False
            retrieve_html = False
            for part in message.walk():
                ctype = part.get_content_type()
                cdispo = str(part.get('Content-Disposition'))
                #print(type(ctype), ctype, cdispo)
                if ctype == 'text/plain' and 'attachment' not in cdispo and not retrieve_body:
                    body = str(part.get_payload(decode=True))
                    message_json['body'] = body
                    retrieve_body = True
                elif ctype == 'text/html' and 'attachment' not in cdispo and not retrieve_html:
                    body_html = str(part.get_payload(decode=True))
                    message_json['body_html'] = body_html
                    retrieve_html = True
        else:
            message_json['body'] = str(message.get_body(preferencelist=('plain')))
            message_json['body_html'] = str(message.get_body(preferencelist=('html')))

File / binaries repository in support of unit tests

I'm looking into adding unit tests to the latest LIEF plugin, pull request #44, in order to be inline with issue #22 and I want to ask what would be the best approach to obtain the necessary test data:

Create a submodule on the data directory pointing to existing binary samples repositories, for example, LIEF test samples or JonathanSalwan Binary Samples
Copy the binary samples from such repositories, but include the respective licensing information
Create stoQ own collection on a separate repository (e.g. stoq-samples) that would be linked to the data directory as a submodule and that could be used by other plugins like peinfo

I'm of the opinion that even though it would take longer, I'm inclined to go for the 3rd option.

Initial inputs to the discussion from @mlaferrera:

That is really good question and one I've been grappling with. I think there is a lot of utility in creating a separate repo for test data, especially since there is overlap in filetypes amongst plugins. This would require a bit of testing to ensure there aren't complications with using a submodule. I've had some bad experiences using them, so I may seem be a bit apprehensive in their use. With that said, I'm not sure there is a better option.

The only problem I have with submodules is forgetting to update the reference to the latest version. Usually this means that the build breaks because there are submodule changes that are needed by the main project but such changes aren't available because it still references old submodule code :D

Plugin Errors and Dependencies

I'm having issues with dependencies on plug-ins that error. They seem to still generate worker subsections, just without any populated data. The one example deep-dispatcher uses logic like this to check for plug-in dependencies...
if 'iocextract' in worker_result:
...but that is likely to fail when worker_results.iocextract exists, but doesn't include content due to error.

Should the dependency check logic actually look something like...
if 'iocextract' in worker_result and ('errors' not in worker_result or 'iocextract' not in worker_result.errors):
...I'm not really clear how to cleanly access errors from within workers / dispatchers. I'm guessing it may be better to just test for empty workers instead.

Also, payload.errors is an object, but payload.workers is an array; how do things work out with errors when a worker is run multiple times? Each plugin's errors is an array so I'd guess each error would appear in that array, but the different structure likely makes it a pain to correlate errors with their related worker.

Add priority handling to rabbitmq source plugin

Develop plugins for use with oletools

https://github.com/decalage2/oletools

Vtmis-search - whois results are not in json format, just show as a long string

Was doing some more testing with the vtmis-search plugin and noticed that the whois results aren't being converted into json format (this is a vtmis problem and not a problem with the plugin). To resolve this I added the following code to line 95 in the _query_api() function:

if 'whois' in result:
whois = result['whois']
temp = '{"' + whois + '"}'
temp = temp.replace('\n', '","')
temp = temp.replace(': ', '": "')
temp = json.loads(temp)
result['whois'] = temp

I should also mention I needed to import the json module for this fix.

smtp example uses incompatible dirmon

when using the latest dockerimage and the example in the smtp plugin the following error is produced
[2019-12-24 18:00:32,918 WARNING] stoq: Plugin dirmon not compatible with this version of stoQ. Unpredictable results may occur!

Avoiding Recursive Decompress Errors

Hoping for some guidance on avoiding exceptions from the Decompress plug-in when it encounters uncompressed files (particularly if used recursively).

I'm presuming I need a dispatcher to handle such. The only available one is Yara, so would I need a Yara rule to pick up on the supported MIME types and hand off to Decompress, Or would I need a custom deep dispatcher to handle it, since I'd guess I'd want to check against MIME data from the mimetype plug-in.

I'm also presuming the the recursion against decompress payloads restarts the plug-in chain (i.e. each file output is processed by a dispatcher, rather than by a deep dispatcher), but wasn't clear.

Uncompress from YaRa dispatch randomly failing

Hi, I am new to Stoq. Only started playing with it yesterday. Stoq installed from Git. Python 3.6.9.

I am using command below to process Maildir files with further dispatch to YaRa. I am testing with single RFC822-compliant forwarded email sample that contains ZIP attachment. ZIP contains PE. What I am trying to achieve is trivial:

YaRa identifies attachment as ZIP and dispatches to decompress, which extracts and archives PE

stoq run --log-level debug -P dirmon -R yara -A filedir -C filedir -C stdout  -s smtp \
  --plugin-opts yara:dispatch_rules=`pwd`/dispatcher.yar \
  yara:worker_rules=`pwd`/stoq.yar  \
  dirmon:source_dir=`pwd`/samples \
  filedir:archive_dir=`pwd`/archive \
  filedir:results_dir=`pwd`/results \
  smtp:always_dispatch=hash,mimetype smtp:archive_attachments=True smtp:extract_iocs=True

Command produces inconsistent results. In most cases it fails with error:

    "errors": [
        {
            "error": "worker:failed to scan: File \"/home/rtops/.stoq/plugins/decompress/decompress.py\", line 151, in scan ; KeyError: 'mimetype'",
            "plugin_name": "decompress",
            "payload_id": "6c1c06c3-19c6-423d-b6ef-73e4af995b2c"
        }
    ],

On every run mimetype is identified properly by plugin and in payload meta:
payload meta:

            "payload_meta": {
                "should_archive": true,
                "should_scan": true,
                "extra_data": {
                    "charset": null,
                    "content-description": null,
                    "disposition": "attachment",
                    "filename": "INVOICE COPY CONFIRMATION.pdf.zip",
                    "type": "application/zip"

mimetype plugin:

            "workers": {
                "mimetype": {
                    "mimetype": "application/zip"
                },

Occasionally it works and I get PE extracted and archived. Please help to identify the issue.

TrID Failure

This file seems to cause trouble for the TrID plugin...
file.6.gz
...no output or error occurs, though it's recognized as a gz by the command line version.

stoq scan file.6.gz -a trid
{
    "results": [
        {
            "payload_id": "2b51d6d8-f9d7-42b6-95f8-8b8ffcb710f8",
            "size": 1580,
            "payload_meta": {
                "should_archive": true,
                "extra_data": {
                    "filename": "file.6.gz"
                },
                "dispatch_to": []
            },
            "workers": [
                {
                    "trid": {}
                }
            ],
            "archivers": {},
            "plugins_run": {
                "workers": [
                    [
                        "trid"
                    ]
                ],
                "archivers": []
            },
            "extracted_from": null,
            "extracted_by": null
        }
    ],
    "request_meta": {
        "archive_payloads": true,
        "source": null,
        "extra_data": {}
    },
    "errors": {},
    "time": "2019-04-26T00:02:20.536012",
    "decorators": {},
    "scan_id": "af45d546-0d07-4e63-89e4-5f0a26e8530e"
}

LANG=/usr/lib/locale/C.UTF-8
trid file.6.gz

TrID/32 - File Identifier v2.24 - (C) 2003-16 By M.Pontello
Definitions found:  11448
Analyzing...

Collecting data from file: file.6.gz
100.0% (.GZ/GZIP) GZipped data (3000/1)

Mimetype: 'mime' is not defined?

On Ubuntu 19.04 I'm seeing this error from the mimetype plugin...

"worker:failed to scan: File "/home/pass/.stoq/plugins/mimetype/mimetype.py", line 45, in scan ; NameError: name 'mime' is not defined"

Something seems to be in conflict with stoQ's mime handling, I'm not able to replicate the error with only stoQ and the mimetype plug-in installed; still trying to pin down what else being installed specifically results in the error.

IOC Extract doesn't extract Ambiguous IP Addresses

The ipaddress module does not accept IP addresses with ambiguous decimal/octal values.
In this example, the IP address is rejected because of the ambiguous final octet.

>>> from ipaddress import ip_address
>>> ip_address("192.000.002.012")
ValueError: '192.000.002.012' does not appear to be an IPv4 or IPv6 address

Because iocextract uses ip_address to validate IP candidates, this causes iocextract to fail to extract similar ambiguous IP addresses.

Although the IP address is technically ambiguous, I recommend logging the ambiguity and then trying again with an address with the leading zeroes removed (to prefer decimal).

>>> re.sub(r"0+([0-9])", r"\1", "192.000.002.012")
'192.0.2.12'

Moved from inappropriate location, PUNCH-Cyber/stoq#140

S3 plugin has python2 dependency

S3 plugin imports boto library. This particular library uses python 2.X syntax "import urllib2" and will not work under python 3.X

TrID Extensionless Results

TrID sometimes recognizes file types without extensions. These are output as follows...

"trid": {
	"TRD": [
		{
			"likely": "50.0%",
			"type": "TrID defs package (14009/2/5)"
		}
	],
	"": [
		{
			"likely": "14.2%",
			"type": "Generic RIFF container (4000/1)"
		}
	]
}

The empty string used as a key can be problematic for some tools. An alternate format that captured the extensions as an attribute value rather than as a key might be preferable. Though such a format change would likely impact current users of the trid plugin.

Alternately, it might be nice to provide an trid.stoq option to allow providing a name for extensionless types...
extensionless = "___"
...

"trid": {
	"TRD": [
		{
			"likely": "50.0%",
			"type": "TrID defs package (14009/2/5)"
		}
	],
	"___": [
		{
			"likely": "14.2%",
			"type": "Generic RIFF container (4000/1)"
		}
	]
}

Encoding Errors with VTMIS-Search / TrID

I'm hitting a UnicodeEncodeError when trying to use the VTMIS-Search plugin against the attached zip or the exe inside it...
dnetc.zip

The exe can be retrieved directly from the Distributed.net site if my upload is untrusted: http://http.distributed.net/pub/dcti/current-client/dnetc-win64-amd64.zip

LANG=/usr/lib/locale/C.UTF-8
stoq scan dnetc.zip -a decompress vtmis-search trid
Traceback (most recent call last):
  File "/home/user/.stoq/.venv/bin/stoq", line 11, in <module>
    sys.exit(main())
  File "/home/user/.stoq/.venv/lib/python3.6/site-packages/stoq/cli.py", line 271, in main
    print(response)
UnicodeEncodeError: 'ascii' codec can't encode character '\xa9' in position 7736: ordinal not in range(128)

LANG=/usr/lib/locale/C.UTF-8
stoq scan dnetc.exe -a vtmis-search trid
Traceback (most recent call last):
  File "/home/user/.stoq/.venv/bin/stoq", line 11, in <module>
    sys.exit(main())
  File "/home/user/.stoq/.venv/lib/python3.6/site-packages/stoq/cli.py", line 271, in main
    print(response)
UnicodeEncodeError: 'ascii' codec can't encode character '\xa9' in position 6549: ordinal not in range(128)

LANG=/usr/lib/locale/C.UTF-8
...seems to be required for the TrID plugin to run without this error...

    "errors": {
        "trid": [
            "trid: loadlocale.c:129: _nl_intern_locale_data: Assertion `cnt < (sizeof (_nl_value_type_LC_TIME) / sizeof (_nl_value_type_LC_TIME[0]))' failed.\n"
        ]
    },

TrID Text Files

When analyzing text files the trid plugin outputs nothing...

"trid": {}

The TrID tool outputs...

trid example.txt

TrID/32 - File Identifier v2.24 - (C) 2003-16 By M.Pontello
Definitions found:  11309
Analyzing...

Collecting data from file: scene.org.txt

Warning: file seems to be plain text/ASCII
         TrID is best suited to analyze binary files!

100.0% (.BIN) PrintFox/Pagefox bitmap (var. P) (1000/1)

trid test

TrID/32 - File Identifier v2.24 - (C) 2003-16 By M.Pontello
Definitions found:  11309
Analyzing...

Collecting data from file: test

Warning: file seems to be plain text/ASCII
         TrID is best suited to analyze binary files!

       Unknown!

It may be be preferable to indicate identification as a text file...

"trid": {
	"TXT": [
		{
			"type": "ASCII text"
		}
	]
}

...though unclear on matching the TrID format for such. and/or preserve the warning.

TrID Error Details

If TrID errors we get the following...

"Exception: worker:failed to scan: File \"/home/user/.stoq/plugins/trid/trid.py\", line 63, in scan ; StoqPluginException(\"Failed gathering TRiD data: Command '['trid', '-d:/home/user/triddefs.trd', '/tmp/tmp78nnddq9']' died with <Signals.SIGABRT: 6>.\",)"

It would be nice if the actual TrID error output was exposed allowing the specific issue to be more easily identified. The two I've seen...

When the local isn't set appropriately (i.e. normally "LANG=en_US.UTF-8", but TrID seems to want a path like "LANG=/usr/lib/locale/C.UTF-8")

trid: loadlocale.c:129: _nl_intern_locale_data: Assertion `cnt < (sizeof (_nl_value_type_LC_TIME) / sizeof (_nl_value_type_LC_TIME[0]))' failed.
Aborted (core dumped)

When TrID DB is missing:

TrID/32 - File Identifier v2.24 - (C) 2003-16 By M.Pontello

File /nodb not found!
No definitions available! You can download an up to date
defs library from TrID's page at http://mark0.net

Add option to extract e-mail body and save separately

SMTP Worker
For privacy laws/requirements

TrID Text Files

The TrID plug-in generates errors when analyzing many text file types, rather than more standard "type" output. This can be even more irksome when analyzing archives, as the errors build up across child files resulting in an errors.trid field that can look like...

"file seems to be plain text/ASCII, file seems to be plain text/ASCII, file seems to be plain text/ASCII, file seems to be plain text/ASCII, file seems to be plain text/ASCII, file seems to be plain text/ASCII, file seems to be plain text/ASCII, file seems to be plain text/ASCII, file seems to be plain text/ASCII, file seems to be plain text/ASCII, file seems to be plain text/ASCII, file seems to be plain text/ASCII, file seems to be plain text/ASCII, file seems to be plain text/ASCII, file seems to be plain text/ASCII, file seems to be plain text/ASCII, file seems to be plain text/ASCII, file seems to be plain text/ASCII, file seems to be plain text/ASCII, file seems to be plain text/ASCII, file seems to be plain text/ASCII, file seems to be plain text/ASCII, file seems to be plain text/ASCII, file seems to be plain text/ASCII, file seems to be plain text/ASCII, file seems to be plain text/ASCII, file seems to be plain text/ASCII, file seems to be plain text/ASCII, file seems to be plain text/ASCII, file seems to be plain text/ASCII, file seems to be plain text/ASCII, file seems to be plain text/ASCII, file seems to be plain text/ASCII, file seems to be plain text/ASCII, file seems to be plain text/ASCII, file seems to be plain text/ASCII, file seems to be plain text/ASCII, file seems to be plain text/ASCII, file seems to be plain text/ASCII, file seems to be plain text/ASCII, file seems to be plain text/ASCII, file seems to be plain text/ASCII, file seems to be plain text/ASCII, file seems to be plain text/ASCII, file seems to be plain text/ASCII, file seems to be plain text/ASCII, file seems to be plain text/ASCII, file seems to be plain text/ASCII"

It would be nice if it could optionally generate a results.workers.trid.TXT field instead (Though I'm unclear appropriate (#/#) / % data can be populated).

This would probably be more acceptable, if worker errors better aligned with their related payloads and didn't build up.

AllInfo for VT Queries

It would be nice if the VTMIS-Search plug-in supported the VirusTotal API's "allinfo=true" parameter to pull the larger dataset that offers. Some of the outputs likely overlap those of stoQ (i.e. PE info), but others, like "harmless_votes" and "malicious_votes", can be helpful for analysis.

I tried making a quick edit to a copy...

params = {
	'allinfo': 'true', #Should return additional data, but this isn't working out?
	'apikey': self.apikey,
	key: query
}

...but such didn't work out. I think I'm missing something about how plug-ins work / are updated.

Python Shebang?

A minor nitpick here, only a limited set of plug-in files include a Python3 shebang (#!/usr/bin/env python3)...

test_exif.py
falcon-sandbox.py
hash,py
test_hash.py
test_ssdeep.py
test_javaclass.py
mraptor.py
ole.py
opsway.py
pecarve.py
rtf.py
smtp.py
swfcarve.py
symhash.py
tika.py
tnet.py
vtmis-filefeed.py
vtmis-search.py
xpdcarve.py
test_xorsearch.py
test_yara.py
yarascan

Further consistency to include it or not would probably be good.

RSDS and NB10 debug info never gets included in peinfo

debug_data data is a byte string, therefore the if conditions debug_data[:4] == 'RSDS' and `debug_data[:4] == 'NB10' will never be True, and we will never get the additional GUID, PDB, etc. meta-data.

I changed the logic to:

if debug_data[:4] == b'RSDS':                
...
elif debug_data[:4] == b'NB10':
...

which fixed the problem, but the GUID string does not look great, it has extra b'' characters in it...

Fix relative paths for plugins

Some plugins are using relative paths for required files (i.e., trid, yara, floss). This should be fixed to ensure full path is defined.

Better Handling of Excessively Large Compressed Files

The decompress maximum_size option is carefully worded (i.e. "to process")...

# Maximum size of payloads to process (original compressed file and extracted files)
# Default: 50MB
maximum_size = 50000000

No checks seem to be made with regards to file size before attempting extraction, so StoQ will attempt to extract archives with huge content (presuming the extraction doesn't exceed the timeout). A lot of data can be output on an SSD in 45s (20-90Gb) which could cause trouble.

It may be preferable to consider archive meta-data before extraction.

Related to #43, regarding capturing meta-data for archives.

Decompress Missing Exception / Error Details When Contents to Large For Disk

Analyzing the .vhd file in this zip with an "application/octet-stream" mime type result in...

    "errors": {
        "decompress": [
            "Unable to access extracted content: "
        ]
    },

sample12.zip

Decompressing it outputs a large 34gb sample12.img file, so I believe it's failing due to insufficient disk space, which should raise an exception or at least provide some more specific details in the errors.decompress field.

Vtmis-search Appropriate Handling of Rate Limits

Many users of the Vtmis-search plugin are likely to be using public API keys, which limit look-ups to only 4/h, which can easily be exceeded with common stoQ analyses. Even paid packages have similar, though higher limits. Repeated queries when over limits can result in the suspension of VirusTotal accounts.

Revisions are needed to detect rate limit errors, and impose delays preventing repeated look-up until the next timeframe a query will be responded to.

TRID Description Typo

https://github.com/PUNCH-Cyber/stoq-plugins-public/blob/master/trid/setup.py
Description is missing final e in "signature"...
description="Identify file types from their TrID signatur",

Add tests to v2 plugins

TrID Index Out of Range

Running the trid plugin against this COM file...
wh.zip
...results in...

"errors": {
        "trid": [
            "worker:failed to scan: File \"/home/pass/.stoq/plugins/trid/trid.py\", line 87, in scan ; IndexError: list index out of range"
        ]
    },

The output from TrID looks like...

trid wh.com

TrID/32 - File Identifier v2.24 - (C) 2003-16 By M.Pontello
Definitions found:  11309
Analyzing...

Collecting data from file: wh.com
       Unknown!

IOCRegex Plugin - TLD Download Error

Received an error message when attempting to run IOCRegex with an older TLD file. I was able to manually download the latest TLD file and replace the existing one - then the plugin worked normally. Looks like there might be a path issue with the download. See error message below:

(.stoq-pyenv) stoq@SYSTEM:~$ python ./stoq-cli.py iocextract -t -F APT_notes/2008/ -C file
[INFO] stoq.reader.iocregex: IANA TLD file is 43 days old
[INFO] stoq.reader.iocregex: Downloading latest IANA TLD file from https://data.iana.org/TLD/tlds-alpha-by-domain.txt
Traceback (most recent call last):
File "./stoq-cli.py", line 87, in
worker = stoq.load_plugin(options.command, 'worker')
File "/usr/local/stoq/.stoq-pyenv/lib/python3.4/site-packages/stoq-0.9.37-py3.4.egg/stoq/plugins.py", line 287, in load_plugin
File "/usr/local/stoq/plugins/worker/iocextract/iocextract.py", line 54, in activate
self.load_reader('iocregex')
File "/usr/local/stoq/.stoq-pyenv/lib/python3.4/site-packages/stoq-0.9.37-py3.4.egg/stoq/plugins.py", line 693, in load_reader
File "/usr/local/stoq/.stoq-pyenv/lib/python3.4/site-packages/stoq-0.9.37-py3.4.egg/stoq/plugins.py", line 287, in load_plugin
File "/usr/local/stoq/plugins/reader/iocregex/iocregex.py", line 52, in activate
iana_tlds = self.__parse_iana()
File "/usr/local/stoq/plugins/reader/iocregex/iocregex.py", line 298, in __parse_iana
self.__download_iana()
File "/usr/local/stoq/plugins/reader/iocregex/iocregex.py", line 313, in __download_iana
self.stoq.write(content, filename=self.iana_tld_file, binary=True, overwrite=True)
File "/usr/local/stoq/.stoq-pyenv/lib/python3.4/site-packages/stoq-0.9.37-py3.4.egg/stoq/core.py", line 412, in write
File "/usr/local/stoq/.stoq-pyenv/lib/python3.4/genericpath.py", line 19, in exists
os.stat(path)
TypeError: stat: can't specify None for path argument

Filedir "'ascii' codec can't encode characters"

I'm seeing character encoding issues again in v2.0.3 /w the filedir plugin; unclear it's good to force ascii rather than unicode encoding...

[2019-04-23 14:45:37,645 ERROR] stoq: Failed to save results using filedir: {
    "results": [
        {
            "payload_id": "3bf03de9-cf75-431a-80e1-52c45a8d71fe",
            "size": 72704,
            "payload_meta": {
                "should_archive": true,
                "extra_data": {
                    "filename": "file.224"
                },
                "dispatch_to": []
            },
            "workers": [
                {
                    "hash": {
                        "sha256": "09a1c17ac55cde962b4f3bcd61140d752d86362296ee74736000a6a647c73d8c",
                        "md5": "2a9d0d06d292a4cbbe4a95da4650ed54",
                        "sha1": "44c32dfae9ac971c3651adbd82c821971a5400dc"
                    },
                    "trid": {
                        "EXE": [
                            {
                                "likely": "48.1%",
                                "type": "Win32 Executable MS Visual C++ (generic) (31206/45/13)"
                            },
                            {
                                "likely": "25.4%",
                                "type": "Microsoft Visual C++ compiled executable (generic) (16529/12/5)"
                            },
                            {
                                "likely": "6.9%",
                                "type": "Win32 Executable (generic) (4508/7/1)"
                            },
                            {
                                "likely": "3.1%",
                                "type": "OS/2 Executable (generic) (2029/13)"
                            }
                        ],
                        "DLL": [
                            {
                                "likely": "10.1%",
                                "type": "Win32 Dynamic Link Library (generic) (6578/25/2)"
                            }
                        ]
                    },
                    "clamav": {
                        "found": true,
                        "result": "Win.Trojan.Farfli-444"
                    },
                    "mimetype": {
                        "mimetype": "application/x-dosexec"
                    }
                },
                {
                    "vtmis-search": [
                        {
                            "vhash": "07402d556095z100131mz3fz",
                            "submission_names": [
                                "SogouPY Config",
                                "Config.exe",
                                "854137.exe",
                                "0.exe",
                                "/854137.exe",
                                "/root/Desktop/0.exe",
                                "Malware.ex___",
                                "08.exe",
                                "uncompressed",
                                "C:\\Documents and Settings\\Administrator\\My Documents\\Downloads\\malware samples\\854137.exe\\854137.exe",
                                "41414221412rddwqdqw.exe",
                                "DSA9B0.dscapture.net_2017-01-16T09.44.38+0700_192.168.81.70-64866_192.168.81.140-80_2a9d0d06d292a4cbbe4a95da4650ed54_1.exe",
                                "D:\\0xFFFFFFFFFF\\\ucef4\ud4e8\ud130\\\uc545\uc131\ucf54\ub4dc\ubd84\uc11d\\0.exe\\0.exe",
                                "Lab1.exe",
                                "\ub4dc\ub86d\ud37c.exe",
                                "sample.exe",
                                "Ghost.exe",
                                "evil-shit.exe",
                                "C:\\Users\\hrd\\0.exe",
                                "854137.exe.vir",
                                "0.exe.vir",
                                "Trojan.W32.Downloader-Uad.Farfli-444.exe",
                                "2a9d0d06d292a4cbbe4a95da4650ed54",
                                "23f82686258760c273af981b69cf4251041b8f0b",
                                "2A9D0D06D292A4CBBE4A95DA4650ED54.VIR",
                                "9100173",
                                "/var/www/clean-mx/virusesevidence/output.9100173.txt",
                                "C:\\Downloads\\Files1\\0.exe",
                                "c:\\downloads\\files1\\0.exe",
                                "E:\\TEKDEFENSE\\854137.exe"
                            ],
                            "scan_date": "2019-03-24 23:35:14",
                            "first_seen": "2013-01-14 22:05:50",
                            "times_submitted": 136,
                            "additional_info": {
                                "magic": "PE32 executable for MS Windows (GUI) Intel 80386 32-bit",
                                "exiftool": {
                                    "UninitializedDataSize": "0",
                                    "LinkerVersion": "6.0",
                                    "ImageVersion": "0.0",
                                    "FileVersionNumber": "1.0.0.1",
                                    "LanguageCode": "Chinese (Simplified)",
                                    "FileFlagsMask": "0x003f",
                                    "ImageFileCharacteristics": "No relocs, Executable, No line numbers, No symbols, 32-bit",
                                    "CharacterSet": "Unicode",
                                    "InitializedDataSize": "71680",
                                    "EntryPoint": "0x15a2",
                                    "OriginalFileName": "Config.exe",
                                    "MIMEType": "application/octet-stream",
                                    "LegalCopyright": "? 2010 Sogou.com Inc. All rights reserved.",
                                    "FileVersion": "5.0.0.3787",
                                    "TimeStamp": "2011:03:22 16:36:10+01:00",
                                    "FileType": "Win32 EXE",
                                    "PEType": "PE32",
                                    "InternalName": "SogouPY Config",
                                    "ProductVersion": "5.0.0.3787",
                                    "SubsystemVersion": "4.0",
                                    "OSVersion": "4.0",
                                    "FileOS": "Windows NT 32-bit",
                                    "Subsystem": "Windows GUI",
                                    "MachineType": "Intel 386 or later, and compatibles",
                                    "CompanyName": "Sogou.com Inc.",
                                    "CodeSize": "0",
                                    "FileSubtype": "0",
                                    "ProductVersionNumber": "1.0.0.1",
                                    "FileTypeExtension": "exe",
                                    "ObjectFileType": "Executable application"
                                },
                                "trid": "Win32 Executable MS Visual C++ (generic) (48.1%)\nMicrosoft Visual C++ compiled executable (generic) (25.4%)\nWin32 Dynamic Link Library (generic) (10.1%)\nWin32 Executable (generic) (6.9%)\nOS/2 Executable (generic) (3.1%)",
                                "pe-imphash": "03f2c2376dbaab48c69a23e5f572970b",
                                "pe-resource-list": {
                                    "934bff4e3799007028d2fb8ecf30013dec9fcfdd91cf4ec2e15ec1120683ee7e": "ASCII text",
                                    "96e3d5cf15f4ad9ae0abe2c55e485b7b9a072ae4748f0f58f9ee9cf8498de1d2": "data",
                                    "dd69a739e398ce71ee9e05b92db9e9b12447c23eba896ac3f73adf50ca9071de": "data",
                                    "a92f60b25322592e7ddd13d88e4006c097666f4d87c8cb0c21ffdccd53b31d78": "Lotus 1-2-3",
                                    "9ee45783d72da6e3ca955b6333b50d4512695c99209c2b11fd675184cc9b1ca6": "data",
                                    "0717dfca923df0beca176f2cb47bdf066cd80d7365dac55184d1a6282bb81b26": "data",
                                    "391109432ba2df9f3ebc74e0144f42a490405f7c8ecb51da01b4ce793be72f25": "application/x-ms-dos-executable",
                                    "35b7d03732d6f5834ca165995ac2985880c2ac0c13b0d9c60a23edc9e0ae11e3": "ASCII text",
                                    "519122f5886bcca7e78f1537961c526d3128675006ed0c04b459ac49409176be": "data"
                                },
                                "peid": "Armadillo v1.71",
                                "pe-resource-langs": {
                                    "CHINESE SIMPLIFIED": 9
                                },
                                "contacted_domains": [
                                    "www.wikiplum.com"
                                ],
                                "contacted_ips": [
                                    "208.91.197.46"
                                ],
                                "deepguard": "Suspicious:W32/Malware!Online",
                                "sigcheck": {
                                    "product": "\u641c\u72d7\u62fc\u97f3\u8f93\u5165\u6cd5",
                                    "description": "\u641c\u72d7\u62fc\u97f3\u8f93\u5165\u6cd5 \u8bbe\u7f6e\u7a0b\u5e8f",
                                    "copyright": "? 2010 Sogou.com Inc. All rights reserved.",
                                    "original name": "Config.exe",
                                    "authentihash": "3bad0e636b23c59cbf300ebbf3df53380288b7035f8c2ba130f3735ab3b3a2d1",
                                    "file version": "5.0.0.3787",
                                    "internal name": "SogouPY Config",
                                    "link date": "4:36 PM 3/22/2011"
                                },
                                "compressed_parents": [
                                    "ccac5ae298c791f3fc3c7e98817e318ee86694c0ab02936c61a8933828761f48",
                                    "35f8662cfae89266708e5faaeb539db4ac9158a2a379cd3b283c97278d669034",
                                    "86bdb2ca9cabab6335ce2c2ff8204d7e6f2a342471aaf7856c0c0494f099dde0",
                                    "19d5b3d83bb2c366f7daf443e07492d406708f2cef4b73396f087b569b059693",
                                    "c79ac8a613c7a25793b2a0167d48a6a5e8e7c811ccdaf01d0a47efc7dff99dbd",
                                    "4967fa8105bb39ff58c2ebd2dcb9e3767f7ccc8713f36f73627eaaeaad28a1f6",
                                    "c60373d02dc3309de283fc9081e23d78caa152cc420727351b6693e3cd5331f3",
                                    "e3443db4619946094b683d1290b02b38266b7844053562bd612b0a497e7eb6ad",
                                    "0425e34cae3f701cf17dd64155f29cca0a77799a4029df42320ab741c2e96ed1",
                                    "66797f88850ce377c6ddf41856799ab47644a277b982e11994ec7e2a40415c3e",
                                    "780d3b7a7427bf86190722c24b483a6b0866a0fd0e1c3000e196c5109ccd6ec6",
                                    "ac3084a0404db903e66796ff7adfbb078c8b8285d0bc73721f1e85d1101a0339",
                                    "1dd806fc41e7ce89609e056301a150945e88b47331e523e46fbcd8de9cc9f193",
                                    "a81d15158decfd7bc39870714a7f5053bcff14150529f80e3e80416242675eba",
                                    "ef13fa473820ec1b67851ace3338ef486bfa4f7acfdddd1e2249010a32006799",
                                    "56ab6024ac67cabbafb80a5839a83f45a611d58604944a53c3d5a44578c63c37",
                                    "8e7b4017a0e0702627835f0ef853bfa86d97b3a4e4d9cbe7ebc4162ff67fd37f"
                                ],
                                "positives_delta": 1,
                                "pe-resource-detail": [
                                    {
                                        "lang": "CHINESE SIMPLIFIED",
                                        "chi2": 987762.3125,
                                        "filetype": "application/x-ms-dos-executable",
                                        "entropy": 6.1942267417907715,
                                        "sha256": "391109432ba2df9f3ebc74e0144f42a490405f7c8ecb51da01b4ce793be72f25",
                                        "type": "CPP"
                                    },
                                    {
                                        "lang": "CHINESE SIMPLIFIED",
                                        "chi2": 56994.4375,
                                        "filetype": "data",
                                        "entropy": 0.7523787021636963,
                                        "sha256": "519122f5886bcca7e78f1537961c526d3128675006ed0c04b459ac49409176be",
                                        "type": "RT_CURSOR"
                                    },
                                    {
                                        "lang": "CHINESE SIMPLIFIED",
                                        "chi2": 12890.708984375,
                                        "filetype": "data",
                                        "entropy": 1.92000412940979,
                                        "sha256": "9ee45783d72da6e3ca955b6333b50d4512695c99209c2b11fd675184cc9b1ca6",
                                        "type": "RT_BITMAP"
                                    },
                                    {
                                        "lang": "CHINESE SIMPLIFIED",
                                        "chi2": 1830.888671875,
                                        "filetype": "data",
                                        "entropy": 1.9447168111801147,
                                        "sha256": "0717dfca923df0beca176f2cb47bdf066cd80d7365dac55184d1a6282bb81b26",
                                        "type": "RT_MENU"
                                    },
                                    {
                                        "lang": "CHINESE SIMPLIFIED",
                                        "chi2": 11565.8759765625,
                                        "filetype": "data",
                                        "entropy": 2.8630785942077637,
                                        "sha256": "96e3d5cf15f4ad9ae0abe2c55e485b7b9a072ae4748f0f58f9ee9cf8498de1d2",
                                        "type": "RT_DIALOG"
                                    },
                                    {
                                        "lang": "CHINESE SIMPLIFIED",
                                        "chi2": 7893.99951171875,
                                        "filetype": "ASCII text",
                                        "entropy": 0.9609531760215759,
                                        "sha256": "934bff4e3799007028d2fb8ecf30013dec9fcfdd91cf4ec2e15ec1120683ee7e",
                                        "type": "RT_STRING"
                                    },
                                    {
                                        "lang": "CHINESE SIMPLIFIED",
                                        "chi2": 1797.600341796875,
                                        "filetype": "Lotus 1-2-3",
                                        "entropy": 2.0192408561706543,
                                        "sha256": "a92f60b25322592e7ddd13d88e4006c097666f4d87c8cb0c21ffdccd53b31d78",
                                        "type": "RT_GROUP_CURSOR"
                                    },
                                    {
                                        "lang": "CHINESE SIMPLIFIED",
                                        "chi2": 68214.3046875,
                                        "filetype": "data",
                                        "entropy": 3.580381155014038,
                                        "sha256": "dd69a739e398ce71ee9e05b92db9e9b12447c23eba896ac3f73adf50ca9071de",
                                        "type": "RT_VERSION"
                                    },
                                    {
                                        "lang": "CHINESE SIMPLIFIED",
                                        "chi2": 4716.19970703125,
                                        "filetype": "ASCII text",
                                        "entropy": 5.106089115142822,
                                        "sha256": "35b7d03732d6f5834ca165995ac2985880c2ac0c13b0d9c60a23edc9e0ae11e3",
                                        "type": "RT_MANIFEST"
                                    }
                                ],
                                "first_seen_itw": "2011-03-22 08:36:10",
                                "pe-resource-types": {
                                    "RT_DIALOG": 1,
                                    "RT_GROUP_CURSOR": 1,
                                    "RT_STRING": 1,
                                    "RT_MANIFEST": 1,
                                    "RT_MENU": 1,
                                    "CPP": 1,
                                    "RT_BITMAP": 1,
                                    "RT_CURSOR": 1,
                                    "RT_VERSION": 1
                                },
                                "pe-timestamp": 1300808170,
                                "imports": {
                                    "ADVAPI32.dll": [
                                        "RegOpenKeyA",
                                        "RegCloseKey",
                                        "OpenServiceA",
                                        "ChangeServiceConfigA",
                                        "RegSetValueExA",
                                        "ControlService",
                                        "StartServiceA",
                                        "RegCreateKeyExA",
                                        "OpenSCManagerA"
                                    ],
                                    "KERNEL32.dll": [
                                        "GetStartupInfoA",
                                        "SizeofResource",
                                        "GetWindowsDirectoryA",
                                        "Sleep",
                                        "GetModuleHandleA",
                                        "LoadResource",
                                        "LockResource",
                                        "WaitForSingleObject",
                                        "DeleteFileA",
                                        "CreateEventA",
                                        "WriteFile",
                                        "GetTickCount",
                                        "CloseHandle",
                                        "CreateFileA",
                                        "GetModuleFileNameA",
                                        "GetProcAddress",
                                        "FindResourceA",
                                        "LoadLibraryA",
                                        "FreeResource"
                                    ],
                                    "MSVCRT.dll": [
                                        "_except_handler3",
                                        "rand",
                                        "_acmdln",
                                        "_adjust_fdiv",
                                        "srand",
                                        "__p__commode",
                                        "__p__fmode",
                                        "_controlfp",
                                        "__setusermatherr",
                                        "exit",
                                        "sprintf",
                                        "__getmainargs",
                                        "_exit",
                                        "__set_app_type",
                                        "_initterm",
                                        "_XcptFilter"
                                    ],
                                    "USER32.dll": [
                                        "LoadCursorA",
                                        "RegisterClassA",
                                        "LoadIconA"
                                    ],
                                    "GDI32.dll": [
                                        "GetStockObject"
                                    ]
                                },
                                "pe-entry-point": 5538,
                                "sections": [
                                    [
                                        ".data",
                                        4096,
                                        3020,
                                        3072,
                                        "5.82",
                                        "2a6a06117a251a3d3aef8f00b73876a2"
                                    ],
                                    [
                                        ".rsrc",
                                        8192,
                                        69632,
                                        68608,
                                        "6.13",
                                        "74a468373ff0f87c6a068b0bfbcb969b"
                                    ]
                                ],
                                "pe-machine-type": 332
                            },
                            "size": 72704,
                            "scan_id": "09a1c17ac55cde962b4f3bcd61140d752d86362296ee74736000a6a647c73d8c-1553470514",
                            "total": 71,
                            "harmless_votes": 0,
                            "verbose_msg": "Scan finished, information embedded",
                            "sha256": "09a1c17ac55cde962b4f3bcd61140d752d86362296ee74736000a6a647c73d8c",
                            "type": "Win32 EXE",
                            "scans": {
                                "Bkav": {
                                    "detected": true,
                                    "version": "1.3.0.9899",
                                    "result": "W32.SogouQhupgfLnr.Trojan",
                                    "update": "20190320"
                                },
                                "MicroWorld-eScan": {
                                    "detected": true,
                                    "version": "14.0.297.0",
                                    "result": "Gen:Variant.Symmi.72359",
                                    "update": "20190324"
                                },
                                "CMC": {
                                    "detected": true,
                                    "version": "1.1.0.977",
                                    "result": "Trojan-GameThief.Win32.Magania!O",
                                    "update": "20190321"
                                },
                                "CAT-QuickHeal": {
                                    "detected": true,
                                    "version": "14.00",
                                    "result": "Backdoor.Farfli.O",
                                    "update": "20190324"
                                },
                                "McAfee": {
                                    "detected": true,
                                    "version": "6.0.6.653",
                                    "result": "Generic Dropper.abs",
                                    "update": "20190324"
                                },
                                "Cylance": {
                                    "detected": true,
                                    "version": "2.3.1.101",
                                    "result": "Unsafe",
                                    "update": "20190325"
                                },
                                "Zillya": {
                                    "detected": true,
                                    "version": "2.0.0.3781",
                                    "result": "Trojan.Magania.Win32.59362",
                                    "update": "20190324"
                                },
                                "TheHacker": {
                                    "detected": true,
                                    "version": "6.8.0.5.4098",
                                    "result": "Trojan/Magania.enxs",
                                    "update": "20190324"
                                },
                                "BitDefender": {
                                    "detected": true,
                                    "version": "7.2",
                                    "result": "Gen:Variant.Symmi.72359",
                                    "update": "20190324"
                                },
                                "K7GW": {
                                    "detected": true,
                                    "version": "11.34.30381",
                                    "result": "Password-Stealer ( 0022e0431 )",
                                    "update": "20190324"
                                },
                                "K7AntiVirus": {
                                    "detected": true,
                                    "version": "11.34.30381",
                                    "result": "Password-Stealer ( 0022e0431 )",
                                    "update": "20190324"
                                },
                                "Arcabit": {
                                    "detected": true,
                                    "version": "1.0.0.844",
                                    "result": "Trojan.Symmi.D11AA7",
                                    "update": "20190324"
                                },
                                "TrendMicro": {
                                    "detected": true,
                                    "version": "10.0.0.1040",
                                    "result": "TROJ_SPNR.15JQ11",
                                    "update": "20190324"
                                },
                                "Baidu": {
                                    "detected": true,
                                    "version": "1.0.0.2",
                                    "result": "Win32.Backdoor.DarkAngle.a",
                                    "update": "20190318"
                                },
                                "Babable": {
                                    "detected": false,
                                    "version": "9107201",
                                    "result": null,
                                    "update": "20180918"
                                },
                                "F-Prot": {
                                    "detected": true,
                                    "version": "4.7.1.166",
                                    "result": "W32/Backdoor.Q.gen!Eldorado",
                                    "update": "20190324"
                                },
                                "Symantec": {
                                    "detected": true,
                                    "version": "1.8.0.0",
                                    "result": "Trojan.Dropper",
                                    "update": "20190324"
                                },
                                "TotalDefense": {
                                    "detected": false,
                                    "version": "37.1.62.1",
                                    "result": null,
                                    "update": "20190324"
                                },
                                "TrendMicro-HouseCall": {
                                    "detected": true,
                                    "version": "10.0.0.1040",
                                    "result": "TROJ_SPNR.15JQ11",
                                    "update": "20190324"
                                },
                                "Paloalto": {
                                    "detected": true,
                                    "version": "1.0",
                                    "result": "generic.ml",
                                    "update": "20190325"
                                },
                                "ClamAV": {
                                    "detected": true,
                                    "version": "0.101.1.0",
                                    "result": "Win.Trojan.Farfli-444",
                                    "update": "20190324"
                                },
                                "Kaspersky": {
                                    "detected": true,
                                    "version": "15.0.1.13",
                                    "result": "Trojan-GameThief.Win32.Magania.ensu",
                                    "update": "20190324"
                                },
                                "Alibaba": {
                                    "detected": false,
                                    "version": "0.2.0.3",
                                    "result": null,
                                    "update": "20190306"
                                },
                                "NANO-Antivirus": {
                                    "detected": true,
                                    "version": "1.0.134.24576",
                                    "result": "Trojan.Win32.Dwn.tshuf",
                                    "update": "20190324"
                                },
                                "ViRobot": {
                                    "detected": true,
                                    "version": "2014.3.20.0",
                                    "result": "Trojan.Win32.PSW-Magania.72704",
                                    "update": "20190324"
                                },
                                "SUPERAntiSpyware": {
                                    "detected": true,
                                    "version": "5.6.0.1032",
                                    "result": "Trojan.Agent/Gen-Farfli",
                                    "update": "20190321"
                                },
                                "Avast": {
                                    "detected": true,
                                    "version": "18.4.3895.0",
                                    "result": "Win32:Downloader-UAD [Trj]",
                                    "update": "20190324"
                                },
                                "Rising": {
                                    "detected": true,
                                    "version": "25.0.0.24",
                                    "result": "Backdoor.Farfli!1.64A3 (CLOUD)",
                                    "update": "20190324"
                                },
                                "Endgame": {
                                    "detected": true,
                                    "version": "3.0.8",
                                    "result": "malicious (high confidence)",
                                    "update": "20190322"
                                },
                                "Trustlook": {
                                    "detected": false,
                                    "version": "1.0",
                                    "result": null,
                                    "update": "20190325"
                                },
                                "Sophos": {
                                    "detected": true,
                                    "version": "4.98.0",
                                    "result": "Troj/Farfli-Gen",
                                    "update": "20190322"
                                },
                                "Comodo": {
                                    "detected": true,
                                    "version": "30620",
                                    "result": "TrojWare.Win32.Farfli.~hon@4k8xs5",
                                    "update": "20190325"
                                },
                                "F-Secure": {
                                    "detected": true,
                                    "version": "12.0.86.52",
                                    "result": "Trojan.TR/Spy.Gen",
                                    "update": "20190324"
                                },
                                "DrWeb": {
                                    "detected": true,
                                    "version": "7.0.34.11020",
                                    "result": "Trojan.DownLoader4.44699",
                                    "update": "20190324"
                                },
                                "VIPRE": {
                                    "detected": true,
                                    "version": "73920",
                                    "result": "Trojan-Dropper.Win32.Farfli.e (v)",
                                    "update": "20190324"
                                },
                                "Invincea": {
                                    "detected": true,
                                    "version": "6.3.6.26157",
                                    "result": "heuristic",
                                    "update": "20190313"
                                },
                                "McAfee-GW-Edition": {
                                    "detected": true,
                                    "version": "v2017.3010",
                                    "result": "Generic Dropper.abs",
                                    "update": "20190324"
                                },
                                "Trapmine": {
                                    "detected": true,
                                    "version": "3.1.48.748",
                                    "result": "malicious.high.ml.score",
                                    "update": "20190301"
                                },
                                "Emsisoft": {
                                    "detected": true,
                                    "version": "2018.4.0.1029",
                                    "result": "Gen:Variant.Symmi.72359 (B)",
                                    "update": "20190324"
                                },
                                "SentinelOne": {
                                    "detected": true,
                                    "version": "1.0.24.302",
                                    "result": "DFI - Malicious PE",
                                    "update": "20190317"
                                },
                                "Cyren": {
                                    "detected": true,
                                    "version": "6.2.0.1",
                                    "result": "W32/Backdoor.Q.gen!Eldorado",
                                    "update": "20190324"
                                },
                                "Jiangmin": {
                                    "detected": true,
                                    "version": "16.0.100",
                                    "result": "Trojan/PSW.Magania.auqv",
                                    "update": "20190324"
                                },
                                "Webroot": {
                                    "detected": true,
                                    "version": "1.0.0.403",
                                    "result": "W32.Backdoor.Gen",
                                    "update": "20190325"
                                },
                                "Avira": {
                                    "detected": true,
                                    "version": "8.3.3.8",
                                    "result": "TR/Spy.Gen",
                                    "update": "20190324"
                                },
                                "MAX": {
                                    "detected": true,
                                    "version": "2018.9.12.1",
                                    "result": "malware (ai score=100)",
                                    "update": "20190325"
                                },
                                "Antiy-AVL": {
                                    "detected": true,
                                    "version": "3.0.0.1",
                                    "result": "Trojan[GameThief]/Win32.Magania",
                                    "update": "20190324"
                                },
                                "Kingsoft": {
                                    "detected": true,
                                    "version": "2013.8.14.323",
                                    "result": "Win32.Troj.Generic.(kcloud)",
                                    "update": "20190325"
                                },
                                "Microsoft": {
                                    "detected": true,
                                    "version": "1.1.15800.1",
                                    "result": "TrojanDropper:Win32/Farfli.E",
                                    "update": "20190324"
                                },
                                "AegisLab": {
                                    "detected": true,
                                    "version": "4.2",
                                    "result": "Trojan.Win32.Magania.4!c",
                                    "update": "20190324"
                                },
                                "ZoneAlarm": {
                                    "detected": true,
                                    "version": "1.0",
                                    "result": "Trojan-GameThief.Win32.Magania.ensu",
                                    "update": "20190324"
                                },
                                "Avast-Mobile": {
                                    "detected": false,
                                    "version": "190324-00",
                                    "result": null,
                                    "update": "20190324"
                                },
                                "GData": {
                                    "detected": true,
                                    "version": "A:25.21250B:25.14682",
                                    "result": "Gen:Variant.Symmi.72359",
                                    "update": "20190324"
                                },
                                "AhnLab-V3": {
                                    "detected": true,
                                    "version": "3.15.0.23609",
                                    "result": "Dropper/Win32.OnlineGameHack.R3269",
                                    "update": "20190324"
                                },
                                "Acronis": {
                                    "detected": false,
                                    "version": "1.0.1.40",
                                    "result": null,
                                    "update": "20190322"
                                },
                                "VBA32": {
                                    "detected": true,
                                    "version": "4.0.0",
                                    "result": "BScope.Trojan.Downloader",
                                    "update": "20190322"
                                },
                                "ALYac": {
                                    "detected": true,
                                    "version": "1.1.1.5",
                                    "result": "Gen:Variant.Symmi.72359",
                                    "update": "20190324"
                                },
                                "TACHYON": {
                                    "detected": true,
                                    "version": "2019-03-24.02",
                                    "result": "Trojan-PWS/W32.WebGame.72704.AX",
                                    "update": "20190324"
                                },
                                "Ad-Aware": {
                                    "detected": true,
                                    "version": "3.0.5.370",
                                    "result": "Gen:Variant.Symmi.72359",
                                    "update": "20190324"
                                },
                                "Malwarebytes": {
                                    "detected": true,
                                    "version": "2.1.1.1115",
                                    "result": "Backdoor.Farfli.Gen",
                                    "update": "20190324"
                                },
                                "Zoner": {
                                    "detected": true,
                                    "version": "1.0",
                                    "result": "Trojan.Win32.9143",
                                    "update": "20190325"
                                },
                                "ESET-NOD32": {
                                    "detected": true,
                                    "version": "19081",
                                    "result": "Win32/Farfli.DV",
                                    "update": "20190324"
                                },
                                "Tencent": {
                                    "detected": true,
                                    "version": "1.0.0.1",
                                    "result": "Trojan.Win32.Magania.nlz",
                                    "update": "20190325"
                                },
                                "Yandex": {
                                    "detected": true,
                                    "version": "5.5.1.3",
                                    "result": "Trojan.PWS.Magania!d9Mad2m07yY",
                                    "update": "20190324"
                                },
                                "Ikarus": {
                                    "detected": true,
                                    "version": "0.1.5.2",
                                    "result": "Trojan-Spy.Win32.Insain",
                                    "update": "20190324"
                                },
                                "eGambit": {
                                    "detected": true,
                                    "version": "v4.3.6",
                                    "result": "Unsafe.AI_Score_95%",
                                    "update": "20190325"
                                },
                                "Fortinet": {
                                    "detected": true,
                                    "version": "5.4.247.0",
                                    "result": "W32/Onlinegames.BNLQ!tr",
                                    "update": "20190324"
                                },
                                "AVG": {
                                    "detected": true,
                                    "version": "18.4.3895.0",
                                    "result": "Win32:Downloader-UAD [Trj]",
                                    "update": "20190324"
                                },
                                "Cybereason": {
                                    "detected": true,
                                    "version": "1.2.449",
                                    "result": "malicious.6d292a",
                                    "update": "20190324"
                                },
                                "Panda": {
                                    "detected": true,
                                    "version": "4.6.4.2",
                                    "result": "Generic Malware",
                                    "update": "20190324"
                                },
                                "CrowdStrike": {
                                    "detected": true,
                                    "version": "1.0",
                                    "result": "win/malicious_confidence_100% (W)",
                                    "update": "20190212"
                                },
                                "Qihoo-360": {
                                    "detected": true,
                                    "version": "1.0.0.1120",
                                    "result": "Win32/Trojan.GameThief.cda",
                                    "update": "20190325"
                                }
                            },
                            "tags": [
                                "peexe",
                                "armadillo"
                            ],
                            "authentihash": "3bad0e636b23c59cbf300ebbf3df53380288b7035f8c2ba130f3735ab3b3a2d1",
                            "unique_sources": 116,
                            "positives": 65,
                            "ssdeep": "1536:jWZpTtLcWyeYd4//yEZc1GJf7/QP4uirySj5e:+pZTvnyEZiGJ7/QguiryS5e",
                            "md5": "2a9d0d06d292a4cbbe4a95da4650ed54",
                            "permalink": "https://www.virustotal.com/file/09a1c17ac55cde962b4f3bcd61140d752d86362296ee74736000a6a647c73d8c/analysis/1553470514/",
                            "sha1": "44c32dfae9ac971c3651adbd82c821971a5400dc",
                            "resource": "44c32dfae9ac971c3651adbd82c821971a5400dc",
                            "response_code": 1,
                            "community_reputation": -59,
                            "malicious_votes": 5,
                            "ITW_urls": [
                                "http://34.240.31.94/854137.exe",
                                "http://200.129.137.121/Malware.ex___",
                                "http://199.193.71.91:89/0.exe"
                            ],
                            "last_seen": "2019-03-09 18:46:24"
                        }
                    ]
                }
            ],
            "archivers": {},
            "plugins_run": {
                "workers": [
                    [
                        "hash",
                        "trid",
                        "clamav",
                        "mimetype"
                    ],
                    [
                        "vtmis-search"
                    ]
                ],
                "archivers": []
            },
            "extracted_from": null,
            "extracted_by": null
        }
    ],
    "request_meta": {
        "archive_payloads": true,
        "source": null,
        "extra_data": {}
    },
    "errors": {
    },
    "time": "2019-04-23T14:45:37.636438",
    "decorators": {},
    "scan_id": "c25068fd-f0f2-4f7d-b988-d7712ef4d132"
}
Traceback (most recent call last):
  File "/home/pass/.stoq/.venv/lib/python3.6/site-packages/stoq/core.py", line 546, in scan_payload
    connector.save(response)
  File "/home/pass/.stoq/plugins/filedir/filedir.py", line 139, in save
    outfile.write(f'{helpers.dumps(response, compactly=self.compactly)}\n')
UnicodeEncodeError: 'ascii' codec can't encode characters in position 1418-1420: ordinal not in range(128)
--- Logging error ---
Traceback (most recent call last):
  File "/home/pass/.stoq/.venv/lib/python3.6/site-packages/stoq/core.py", line 546, in scan_payload
    connector.save(response)
  File "/home/pass/.stoq/plugins/filedir/filedir.py", line 139, in save
    outfile.write(f'{helpers.dumps(response, compactly=self.compactly)}\n')
UnicodeEncodeError: 'ascii' codec can't encode characters in position 1418-1420: ordinal not in 
range(128)

Vtmis-search not displaying IOC that was sent for analysis in results

When running the vtmis-search plugin, the IOC (or SHA1) are not included in the plugin results. This makes it difficult to associate the results with the IOC in the decorator stage (or post analysis). I added the following code on lines 76 and 81 to fix this error in my dev environment:

if response:
response['ioc'] = ioc

Hope this helps.

iocextract Documentation Error

https://github.com/PUNCH-Cyber/stoq-plugins-public/tree/master/iocextract

The descriptions for the url and domain indicator types seem to be reversed, though they'll also seem to need some rewording, since "Pattern is identical to domain" isn't appropriate if transitioned to Domain.

Decompress Error: Unable to determine archive type

Using the decompress plugin on the attached file...
ses-manse.zip

Results in..

stoq scan manse.zip -a mimetype decompress
[2019-03-16 18:48:31,835 ERROR] stoq: worker:failed to scan
Traceback (most recent call last):
  File "/home/pass/.stoq/.venv/lib/python3.6/site-packages/stoq/core.py", line 653, in _single_scan
    worker_response = plugin.scan(payload, request_meta)  # pyre-ignore[16]
  File "/home/pass/.stoq/plugins/decompress/decompress.py", line 156, in scan
    f'Unable to determine archive type, mimetype: {mimetype}'
stoq.exceptions.StoqPluginException: Unable to determine archive type, mimetype: text/plain
{
    "results": [
        {
            "payload_id": "eb41f67a-d223-4d47-83db-d1f08d0a6756",
            "size": 853606,
            "payload_meta": {
                "should_archive": true,
                "extra_data": {
                    "filename": "manse.zip"
                },
                "dispatch_to": []
            },
            "workers": [
                {
                    "mimetype": {
                        "mimetype": "application/zip"
                    },
                    "decompress": {}
                }
            ],
            "archivers": {},
            "plugins_run": {
                "workers": [
                    [
                        "mimetype",
                        "decompress"
                    ]
                ],
                "archivers": []
            },
            "extracted_from": null,
            "extracted_by": null
        },
        {
            "payload_id": "41f6c260-de53-4aba-bbd2-f086b79609c8",
            "size": 850868,
            "payload_meta": {
                "should_archive": true,
                "extra_data": {
                    "filename": "MANSEFIT.PRG"
                },
                "dispatch_to": []
            },
            "workers": [
                {
                    "mimetype": {
                        "mimetype": "application/octet-stream"
                    },
                    "decompress": {}
                }
            ],
            "archivers": {},
            "plugins_run": {
                "workers": [
                    [
                        "mimetype",
                        "decompress"
                    ]
                ],
                "archivers": []
            },
            "extracted_from": "eb41f67a-d223-4d47-83db-d1f08d0a6756",
            "extracted_by": "decompress"
        },
        {
            "payload_id": "655f5ccc-1f22-468b-8cd7-2539a534e7ea",
            "size": 897,
            "payload_meta": {
                "should_archive": true,
                "extra_data": {
                    "filename": "README.NOW"
                },
                "dispatch_to": []
            },
            "workers": [
                {
                    "mimetype": {
                        "mimetype": "application/octet-stream"
                    },
                    "decompress": {}
                }
            ],
            "archivers": {},
            "plugins_run": {
                "workers": [
                    [
                        "mimetype",
                        "decompress"
                    ]
                ],
                "archivers": []
            },
            "extracted_from": "eb41f67a-d223-4d47-83db-d1f08d0a6756",
            "extracted_by": "decompress"
        },
        {
            "payload_id": "d4eb6781-ec35-4a04-bf00-823e8177c803",
            "size": 3377,
            "payload_meta": {
                "should_archive": true,
                "extra_data": {
                    "filename": "scene.org.txt"
                },
                "dispatch_to": []
            },
            "workers": [
                {
                    "mimetype": {
                        "mimetype": "text/plain"
                    }
                }
            ],
            "archivers": {},
            "plugins_run": {
                "workers": [
                    [
                        "mimetype",
                        "decompress"
                    ]
                ],
                "archivers": []
            },
            "extracted_from": "eb41f67a-d223-4d47-83db-d1f08d0a6756",
            "extracted_by": "decompress"
        }
    ],
    "request_meta": {
        "archive_payloads": true,
        "source": null,
        "extra_data": {}
    },
    "errors": {
        "decompress": [
            "worker:failed to scan: File \"/home/pass/.stoq/plugins/decompress/decompress.py\", line 156, in scan ; stoq.exceptions.StoqPluginException: Unable to determine archive type, mimetype: text/plain"
        ]
    },
    "time": "2019-03-16T18:48:31.837008",
    "decorators": {},
    "scan_id": "3cb2d161-f275-4cf8-aaf5-b77474de59d7"
}

Not quite clear what's happening. I first though it might be the file detected as an octet-stream (MANSEFIT.PRG ) is not a compressed file, but the mime type indicated in the error is text/plain.

peinfo has some unused member methods

The member method: _is_suspicious is not called. Why did you not call and include this in results? Same question with _is_valid, however I noticed that peutils.is_valid does not do anything.

Connector Failure Breaks All Connector

It seems that if exceptions occur with one Connector plug-in's output, output of others also fail (perhaps dependent on their process order). For instance, given this command...
stoq scan file -a hash -C es-search stdout filedir
...an exception with the es-search plugin causes the stdout and filedir outputs to not be processed (or perhaps also fail).

Outputs should be handled independently, so that a failure on one will not cause others to fail.