lmco / laikaboss Goto Github PK

First - thanks for contributing this tool! At $dayjob we're looking at piloting laikaboss as the engine for automated file analysis of files extracted from a large grid of network sensors - so here's the first of several newb questions:

Right now I have a networked instance of laika running and accepting requests from cloudscan.py just fine. I plan to leverage cloudscan to send files to laikad from each sensor. However I'd like the results of those scans to be sent from the centralized laikad instance to a log aggregation point. It looks like I can accomplish this with the log_fluent module - however I'm having trouble figuring out how I might configure it to send these logs to my fluentd endpoint. Is this done in laikad.conf or someplace else? Would really appreciate it if someone could point me in the right direction. Once I get this figured out I'd be happy to write documentation on it and contribute it back.

Many thanks!

Hi,

Do you have already an IMAP client so i can get email from my spamtraps and send it to laikad or should I just write one?

Also can you share slides or other documentations?

Thanks

I've grabbed the Docker image for Lockheed Martin's Laikaboss, and I've ran:

sudo docker run --rm -it -v ~/laikaboss-workdir:/home/nonroot/workdir wzod/laikaboss

I understand I must use a networked instance, so I ran:

sudo docker run --rm -it -p 5558:5558 -v ~/laikaboss-workdir:/home/nonroot/workdir wzod/laikaboss

Before running Laika BOSS, create the ~/laikaboss-workdir and then run chmod a+xwr to make accessible globally but I still get errors from Docker:

Docker error: Cannot start service ...: network 7808732465bd529e6f20e4071115218b2826f198f8cb10c3899de527c3b637e6 not found

I am using ubuntu18.04 LTS

Among the many functions of laikaboss, we only want the file name, file type as the result.

Is there any way?

example)
#laika.py test.txt

result
{
"source": "CLI",
"scan_result": [
{
"objectHash": "96ac2cfac312d8dabf974187ed381e37",
"uuid": "6a9344d3-c1ee-457c-b512-04f04bce3994",
"fileType": [],
"filename": "test.txt",
}

So I was having an issue with getting Postfix to talk to Laika but I have fixed that and updated this comment. I was trying to spawn laikamilter.py from master.cf and that was a failure. Setting the py script to execute on system boot and then throwing this into the main.cf solved my initial issue:

smtpd_milters = inet:<my hosts IP>:7226

If there is any detailed documents on setup, examples, etc, I would be interested in seeing those.
Thanks! :)

I set PF to talk to Laika but I have fixed that and updated this comment, look at psaux | gerp laikamilter ,this was trying to spawn laikamilter.py from master.cf and that was a failure.

smtpd_milters = inet:<my hosts IP>:7226

I seen similar thread, any ideas?

Do you use it as a proxy and send the data via http proxy over certain port? I don't really get how to use it as networked instance. (And would love to try it out.)
Thanks.

Hello, I am a student who is using laikaboss

I have a question about network instances and distributed processing.

As far as I'm concerned,
According to the white paper, laikaboss has brokers and walkers, which makes it faster.

If what I understand is correct, the more walkers, the faster the file scanning should be.
However, the more walkers there are, the slower it seems.

Do you happen to know the solution?

The composition of my experiment is as follows.

broker: ./laikad.py --broker-frontend=tcp://:5558 --broker-backend=tcp://:5559 --worker-connect=tcp://localhost:5559 -p 8

Client: ./cloudscan.py ~/FILE PATH/ -R -r -a tcp://"brokerIP:5559"

worker1: ./laikad.py --no-broker --worker-connect=tcp://brokerIP:5559 -a -p 8
worker2: ./laikad.py --no-broker --worker-connect=tcp://brokerIP:5559 -a -p 8
worker3: ./laikad.py --no-broker --worker-connect=tcp://brokerIP:5559 -a -p 8

Thank you.

Is there any appetite to switch to python3? Has this already been looked into?

When processing a portable executable with out of the box Ubuntu configuration, I get the following error:

$ python laika.py ../malware.zip | jq '.scan_result[] | { "file name" : .filename, "file type" : .fileType, "flags" : .flags, "md5" : .objectHash }'
ERROR:root:error on 19ac5658-3fd7-4c55-94ad-09054798e08e running module META_PE. exception details below: 0:00) ETA:  --:--:--
Traceback (most recent call last):
  File "/home/user/laikaboss-master/laikaboss/si_module.py", line 43, in run
    moduleResult = self._run(scanObject, result, depth, args)
  File "/home/user/laikaboss-master/laikaboss/modules/meta_pe.py", line 53, in _run
    moduleResult.append(ModuleObject(buffer=section.get_data(),externalVars=ExternalVars(filename=secName)))
TypeError: get_data() takes at least 2 arguments (1 given)
[snip results]

In this case, the PE is contained in a .zip. I get the same error processing the PE by itself. In both cases, I still get the scan result.

I also tried running the FCIV executable from Microsoft through it and got the same result:

$ python laika.py ../Windows-KB841290-x86-ENU.exe | jq '.scan_result[] | { "file name" : .filename, "file type" : .fileType, "flags" : .flags, "md5" : .objectHash }'
ERROR:root:error on 84121a80-9d90-4fb5-8980-c90dd867b194 running module META_PE. exception details below: 0:00) ETA:  --:--:--
Traceback (most recent call last):
  File "/home/user/laikaboss-master/laikaboss/si_module.py", line 43, in run
    moduleResult = self._run(scanObject, result, depth, args)
  File "/home/user/laikaboss-master/laikaboss/modules/meta_pe.py", line 53, in _run
    moduleResult.append(ModuleObject(buffer=section.get_data(),externalVars=ExternalVars(filename=secName)))
TypeError: get_data() takes at least 2 arguments (1 given)
100%[#######################################################] Processed: 1/1 total files (Elapsed Time: 0:00:00) Time: 0:00:00
{
  "md5": "58dc4df814685a165f58037499c89e76",
  "flags": [],
  "file type": [
    "pe",
    "cab"
  ],
  "file name": "../Windows-KB841290-x86-ENU.exe"
}

Did I miss something important in the setup?

Thanks!

When trying to compile, this is a constant error. Any ideas?

How do you recommend invoking laikaboss? I don't think the testfiles directory exists that is mentioned in README.md.

Not sure why this keeps saying permission denied. Do you have example pieces of malware we can run to try using laika? Using the download from Practical Malware Analysis. I figured it would be good to scan since it unpacks a lot of malware.
I have tried

./laika.py PracticalMalwareAnalysis-Lab.exe | jq -c . | (there was something else here)
./laika.py PracticalMalwareAnalysis-Lab.exe | jq -c
./laika.py PracticalMalwareAnalysis-Lab.exe
./laika.py <PracticalMalwareAnalysis-Lab.exe> out.txt
./laika.py
python laika.py
python laikad.py

I have been attempting to integrate the rules from the Yara-Rules repo into laikaboss and find that it is giving the following error after adding the index.yar to laikas signatures file:

ERROR:root:util: yara on demand scan failed with rule /etc/laikaboss/modules/scan-yara/signatures.yara
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/laikaboss-2.0-py2.7.egg/laikaboss/util.py", line 67, in yara_on_demand
    yara_on_demand_rules[rule] = yara.compile(rule, externals=externalVars)
SyntaxError: /home/test/laikaboss-master/rules/./malware/MALW_Mirai_Okiru_ELF.yar(35): duplicated identifier "is__elf"
ERROR:root:error on 1107370e-783f-4242-ae53-8ee2043fb518 running module SCAN_YARA. exception details below: 
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/laikaboss-2.0-py2.7.egg/laikaboss/si_module.py", line 43, in run
    moduleResult = self._run(scanObject, result, depth, args)
  File "/usr/local/lib/python2.7/dist-packages/laikaboss-2.0-py2.7.egg/laikaboss/modules/scan_yara.py", line 87, in _run
    matches = yara_on_demand(config.yarascanrules, scanObject.buffer, externalVars=externalVars)
  File "/usr/local/lib/python2.7/dist-packages/laikaboss-2.0-py2.7.egg/laikaboss/util.py", line 67, in yara_on_demand
    yara_on_demand_rules[rule] = yara.compile(rule, externals=externalVars)
SyntaxError: /home/test/laikaboss-master/rules/./malware/MALW_Mirai_Okiru_ELF.yar(35): duplicated identifier "is__elf"

I can't see how to resolve the issue, if anyone has some insight it would be appreciated

What's the reasoning behind each character in the bad_chars list in cleanKey?

It translates things like SHELL32.dll into SHELL32_dll

Is the timer actually starting? Or is it just processing the files too fast to populate a time? It isn't scanning them all instantly, they all take at about a second to work. Is the minimum time 1 second?

Hey, is this project still in dev?

TIA!

Platform: Operating on Ubuntu 14.04 EC2 instance
Overview: Attempting to run a networked instance of laikaboss and run files through a custom module
Problem: After starting a sever "./laikad.py" and attempting to use cloudscan as a client (as described here: #18), the output from the cloudscan request contains no module data.

Details:
$ ./laika.py {file}
"scanModules": ["SCAN_YARA", "META_HASH", "SCAN_TEST"]
works correctly w/ added scanModules and meta data
$ ./laikad.py + $ ./cloudscan.py {file}
"scanModules": []

• Using default cloudscan and laikad conf files.

Any help would be appreciated.

Hi guys,

Its been really fascinating experimenting with the laikaboss for web scanning. It was working well earlier, but after an apt update (cannot really put a finger on what specifically changed), I'm steadily receiving the error on below.

ERROR:root:error on 47ca1da3-faf5-4002-a7d4-f411592d0adb running module META_PE. exception details below:
Traceback (most recent call last):
File "/home/hostname/Programs/laikaboss/laikaboss/si_module.py", line 43, in run
moduleResult = self._run(scanObject, result, depth, args)
File "/home/hostname/Programs/laikaboss/laikaboss/modules/meta_pe.py", line 201, in _run
self.module_name, 'Rich Header', self.parseRich(pe))
File "/home/hostname/Programs/laikaboss/laikaboss/modules/meta_pe.py", line 247, in parseRich
result['Hashes'] = self.richHeaderHashes(pe)
File "/home/hostname/Programs/laikaboss/laikaboss/modules/meta_pe.py", line 259, in richHeaderHashes
rich_end = data.index(0x68636952)
ValueError: 1751345490 is not in list

Additionally, I'm now having issues on laikaboss/laikaboss/dispatch.py where it says the following.
Traceback (most recent call last):
File "laika.py", line 25, in
from laikaboss.dispatch import Dispatch, close_modules
File "/home/hostname/Programs/laikaboss/laikaboss/dispatch.py", line 18, in
from util import get_scanObjectUID, listToSSV, yara_on_demand,
ModuleNotFoundError: No module named 'util'

Just wondering since it's been sometime since the previous issue was closed out. Is the effort to develop a python3 compatible version of Laika still in the works?

laika.py missing json import for EXT_METADATA load.

Traceback (most recent call last):
File "laika.py", line 442, in
sys.exit(main())
File "laika.py", line 144, in main
EXT_METADATA = json.loads(metafile.read())
NameError: global name 'json' is not defined

Has there been any thought of including a flag in the cloudscan client that allows the user to specify a database (e.g. mongodb) to add the scan output to as opposed to printing it to standard out?

I have ran in to this use case and would be willing to help implement this feature if there is interest.

Thanks!

I created a gist that has instructions on how to get this working on CentOS 7.x. The only piece that is missing from the default repos is Yara and unrar. I created RPMs for those and put it into a Package Cloud, free repo (included in the instructions).

https://gist.github.com/dcode/a828034a883cc06634f9

meta_pe.py is throwing the following error on some PE samples:

Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/laikaboss-2.0-py2.7.egg/laikaboss/si_module.py", line 43, in run
    moduleResult = self._run(scanObject, result, depth, args)
  File "/usr/local/lib/python2.7/dist-packages/laikaboss-2.0-py2.7.egg/laikaboss/modules/meta_pe.py", line 201, in _run
    self.module_name, 'Rich Header', self.parseRich(pe))
  File "/usr/local/lib/python2.7/dist-packages/laikaboss-2.0-py2.7.egg/laikaboss/modules/meta_pe.py", line 247, in parseRich
    result['Hashes'] = self.richHeaderHashes(pe)
  File "/usr/local/lib/python2.7/dist-packages/laikaboss-2.0-py2.7.egg/laikaboss/modules/meta_pe.py", line 259, in richHeaderHashes
    rich_end = data.index(0x68636952)
ValueError: 1751345490 is not in list

I'd rather not share details about the sample here, but will say that the data list variable is not empty. I can share the sample and details through an alternate channel.

ERROR:root:error on f8188a62-a22a-4553-9ada-dbb183cb38f4 running module META_JAVA_CLASS. exception details below:
Traceback (most recent call last):
File "/home/user/Desktop/laikaboss/laikaboss/si_module.py", line 43, in run
moduleResult = self._run(scanObject, result, depth, args)
File "/home/user/Desktop/laikaboss/laikaboss/modules/meta_java_class.py", line 27, in _run
class_obj = unpack_class(scanObject.buffer)
File "/usr/local/lib/python2.7/dist-packages/javatools/init.py", line 2201, in unpack_class
o.unpack(up, magic=magic)
File "/usr/local/lib/python2.7/dist-packages/javatools/init.py", line 430, in unpack
self.cpool, is_method=False))
File "/usr/local/lib/python2.7/dist-packages/javatools/pack.py", line 131, in unpack_objects
obj.unpack(self)
File "/usr/local/lib/python2.7/dist-packages/javatools/init.py", line 976, in unpack
self.attribs.unpack(unpacker)
File "/usr/local/lib/python2.7/dist-packages/javatools/init.py", line 345, in unpack
self[cval(name)] = unpacker.read(size)
File "/usr/local/lib/python2.7/dist-packages/javatools/pack.py", line 198, in read
raise UnpackException(None, count, avail)
File "/usr/local/lib/python2.7/dist-packages/javatools/pack.py", line 312, in init
Exception.init(self.template % (fmt, wanted, present))

All seems to be up and running correctly, laikad, worker, broker, milter all on one box. A simple test email from command line to postfix the message is passed to milter, then I get this error:

Sep 19 08:59:56 hostname laikamilter: 0002f3446bf5 ERROR EOM: RETURNING DEFAULT (0) ['Traceback (most recent call last):\n', ' File "/var/opt/laika/laikaboss-master/milter/laikamilter.py", line 290, in eom\n self.rtnToMTA = self._dispositionMessage()\n', ' File "/var/opt/laika/laikaboss-master/milter/laikamilter.py", line 395, in _dispositionMessage\n success = dispositioner.zmqGetFlagswithRetry(self.milterConfig.zmqMaxRetry, self)\n', ' File "/var/opt/laika/laikaboss-master/milter/laikamilter.py", line 588, in zmqGetFlagswithRetry\n sendResponse = self._zmqGetFlags(numRetries, milterContext)\n', ' File "/var/opt/laika/laikaboss-master/milter/laikamilter.py", line 616, in _zmqGetFlags\n gotResponseFromScanner = self._zmqSendBuffer(milterContext, numRetries, REQUEST_TIMEOUT, SERVER_ENDPOINT)\n', ' File "/var/opt/laika/laikaboss-master/milter/laikamilter.py", line 636, in _zmqSendBuffer\n source=milterContext.milterConfig.milterName+"-"+str(myhostname[:myhostname.index(".")]),\n', 'ValueError: substring not found\n']

hi
I have a question about exiftool at laika.py.

When scanning a file, the MAC in exiftool displays the current time.

Do you know how to solve this?

thank you

The link at the desciption doesn't work
wget http://pefile.googlecode.com/files/pefile-1.2.10-139.tar.gz
When i clone the pefile version that exists now on git, i get warnings when running laika.py
Solve it when downloading the exact version from here
wget http://pkgs.fedoraproject.org/repo/pkgs/python-pefile/pefile-1.2.10-139.tar.gz/f10a94320bfaca356fff4d28c41e9278/pefile-1.2.10-139.tar.gz

Receiving this KeyError in the META_PE module when trying to scan directories with the latest release:

ERROR:root:error on 63c576ee-3139-4f31-a560-be23b0061843 running module META_PE. exception details below:
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/laikaboss-2.0-py2.7.egg/laikaboss/si_module.py", line 43, in run
moduleResult = self._run(scanObject, result, depth, args)
File "/usr/local/lib/python2.7/dist-packages/laikaboss-2.0-py2.7.egg/laikaboss/modules/meta_pe.py", line 83, in _run
for imp_symbol in dump_dict['Imported symbols']:
KeyError: 'Imported symbols'

This isn't in the Laikaboss code, but the installation instructions list a specific, old version of pefile to use.

Under this version, rich headers are not extracted by pefile, and never make it into metadata. Instead, you just see an empty dictionary under the "Rich Header" label.

This can be fixed by upgrading to the latest version (v2018.8.8 as of this posting) - we haven't had any compatibility issues upgrading.
Note: We saw this bug under pefile v2016.3.4, which is even newer than the one listed in the instructions. I'm not sure at what point it was fixed, but sometime in the last two years...

I might be overlooking something simple, but I get the following error when trying to run Laikaboss as a standalone mode.

[c@localhost laikaboss]$ ./laika.py badness.exe
ERROR:root:error on f3279c32-c38a-4f55-8dbc-49b502b6434e running module META_EXIFTOOL. exception details below:
Traceback (most recent call last):
File "/home/c/laikaboss/laikaboss/si_module.py", line 43, in run
moduleResult = self._run(scanObject, result, depth, args)
File "/home/c/laikaboss/laikaboss/modules/meta_exiftool.py", line 45, in _run
with exiftool.ExifTool() as et:
File "/usr/lib/python2.7/site-packages/exiftool.py", line 191, in enter
self.start()
File "/usr/lib/python2.7/site-packages/exiftool.py", line 174, in start
stderr=devnull)
File "/usr/lib64/python2.7/subprocess.py", line 711, in init
errread, errwrite)
File "/usr/lib64/python2.7/subprocess.py", line 1327, in _execute_child
raise child_exception
OSError: [Errno 2] No such file or directory

Would it be helpful to the community to setup an automated Docker build for the project?

https://docs.docker.com/docker-hub/builds/

I suspect the owner of the repo would be the best person to create a special GitHub account with access to this repo that links both the Docker automated build and the repo. That way it's not reliant on a community member that may or may not keep up with it.

Hello,

There is an undefined variable in the exception handler when open() fails in laika.py:

Traceback (most recent call last):
  File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/usr/local/lib/python2.7/dist-packages/laikaboss-2.0-py2.7.egg/EGG-INFO/scripts/laika.py", line 402, in run
    self.result_queue.put(answer)
NameError: global name 'answer' is not defined

Extract of the code:

396             try:
397                 with open(next_task) as nextfile:
398                     file_buffer = nextfile.read()
399             except IOError:
400                 logging.debug("Error opening: %s" % (next_task))
401                 self.task_queue.task_done()
402                 self.result_queue.put(answer)
403                 continue

I am not really sure how to fix this, sorry.

Suggested planning for support of CybOX and MAEC
Ref:
http://cybox.mitre.org/
http://maec.mitre.org/

Is it possible to write a more lightweight client that implements a tripwire type detection. Im thinking in terms of a periodic scan of the file system, comparing hashes and then passing suspect files via the network to a laikaboss host. webscan.py still requires a lot of libraries to be installed.

I'm getting an error, seems to be related to yara (3.4 installed)

dispositioner.py
from yara import SyntaxError
ImportError: cannot import name SyntaxError

full message:
ERROR:root:Import Exception for dispositioner.py module: ['Traceback (most recent call last):\n', ' File "/etc/laikaboss/laikaboss/modules/init.py", line 34, in \n _temp = import(module[:-3], locals(), globals(), [module[:-3].upper()], -1)\n', ' File "/etc/laikaboss/laikaboss/modules/dispositioner.py", line 20, in \n from yara import SyntaxError\n', 'ImportError: cannot import name SyntaxError\n']
Traceback (most recent call last):
File "/etc/laikaboss/laikaboss/modules/init.py", line 34, in
_temp = import(module[:-3], locals(), globals(), [module[:-3].upper()], -1)
File "/etc/laikaboss/laikaboss/modules/dispositioner.py", line 20, in
from yara import SyntaxError
ImportError: cannot import name SyntaxError
ERROR:root:Scan worker died, shutting down