withsecurelabs / snake Goto Github PK

snake - a malware storage zoo

License: BSD 3-Clause "New" or "Revised" License

Shell 88.20% Dockerfile 11.80%

countercept python snake

snake's Introduction

Introduction

Snake is a malware storage zoo that was built out of the need for a centralised and unified storage solution for malicious samples that could seamlessly integrate into the investigation pipeline.

Snake is designed to provide just enough information to allow analysts to quickly and efficiently pivot to the most suitable tools for the task at hand. That being said there will be times where the information provided by Snake is more than sufficient. It is a Python based application built on top of Tornado and MongoDB. Scales provide Snake with a variety of functionality from static analysis through to interaction with external services.

For more information, please see: Wiki

The Snake Family

There is more to Snake than just the above, below is a summary:

snake: The malware storage zoo.
- core: The main guts of Snake and the RESTful API.
- pit: The celery based workers that are used to execute static based commands.
snake-charmer: The regression based test suite.
snake-scales: The official repository of snake scales (plugins).
snake-skin: The Web UI.
snake-tail: The UNIX based command line UI.

Install

There are a few ways to install Snake, but the install scripts below will install Snake and the Web UI (Snake Skin).

Note: To install these components individually refer to their respective repositories.

Docker

Snake can be run simply with the following commands:

# Get the lastest version of Snake
git clone https://github.com/countercept/snake.git
git submodule init
git submodule update

# Run Snake
sudo docker-compose up

Snake scales can be installed by exec'ing into the Snake container and running snake install:

# Exec into the Snake container
sudo docker exec -it snake_snake_1 /entrypoint.sh /bin/bash

# Install a scale
snake install SCALE_NAME

Production

This is the preferred method and will install Snake and the Web UI (Snake Skin) into the UNIX system.

Dependencies

There are a few dependencies to install Snake and Web UI (Snake Skin).

Required

(Snake) LibYAML
(Snake) MongoDB 3.4 or greater
(Snake) Python 3.5 or greater
(Snake) Redis
(Snake Skin) NodeJS 8 or greater
(Snake Skin) NPM

Optional

(Snake) libfuzzy & ssdeep

The above can be installed like so:

Ubuntu 17.10

# Install dependencies
sudo apt-get install libyaml-dev mongodb nodejs npm python3-dev python3-pip redis-server libfuzzy-dev ssdeep

Ubuntu 16.04

# Install cURL
sudo apt-get install curl

# Add repository for MongoDB 3.6
sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv 2930ADAE8CAF5059EE73BB4B58712A2291FA4AD5
echo "deb [ arch=amd64,arm64 ] https://repo.mongodb.org/apt/ubuntu xenial/mongodb-org/3.6 multiverse" | sudo tee /etc/apt/sources.list.d/mongodb-org-3.6.list

# Add repository for nodejs 8
curl -sL https://deb.nodesource.com/setup_8.x | sudo -E bash -

# Install dependencies
sudo apt-get update
sudo apt-get install libyaml-dev mongodb-org nodejs python3-dev python3-pip redis-server libfuzzy-dev ssdeep

# Update pip and setuptools
sudo -H pip3 install --upgrade pip setuptools

git clone https://github.com/countercept/snake.git
cd snake
sys/install.sh

To start Snake:

# Start Snake Pit and Snake services
systemctl start snake-pit
systemctl start snake

To serve Snake Skin (port: 3000):

# Start Snake Skin
systemctl start snake-skin

Scales (Plugins)

By default Snake only provides three core scales:

hashes: a command based scale used to perform a variety of hashing techniques on a sample.
strings: a command based module to run strings on a sample.
url: an upload based component used to upload samples to Snake from URLs.

Installing Additional Scales

Additional Scales are available at snake-scales

Snake provides a wrapper around pip to ease the installation of scales. A scale can be installed with this utility like so:

snake install virustotal

A scale can be checked at any time to see if it will successfully load in Snake.

snake check virustotal

Note: Whenever a new scale is installed, Snake and Celery must be restarted.

To create a scale, please see Scale Documentation

Usage

Both installations will serve Snake on port 5000 (API) and Snake Skin on port 8000.

To communicate with the WebUI:

Visit http://127.0.0.1:8000

To communicate with the API:

curl http://127.0.0.1:5000

Screenshots

Details View

An overview of a sample that has been uploaded to Snake, with additional data enrichment from Cuckoo and VirusTotal.

Notes View

Stores an user written notes about the sample.

Analysis View

This view is used to execute and view commands on a sample.

Interfaces View

This view is used to communicate with external services in relation to a sample.

Configuration

For an overview of Snake's settings, please see Snake

For an overview of Snake Skin's settings, please see Snake Skin

snake's People

Contributors

Stargazers

Watchers

snake's Issues

VirusTotal: "snake has encountered an Error!"

Hi,

I have installed the VirusTotal Scale (snake-scales) and get multiple "snake has encountered an Error!" messages on the application.

Is this a known issue?

Questions

Where are the samples stored on the system?
I want to know where the samples are stored so that I can properly document the location for future reference.
Can samples be deleted from the malware zoo after being uploaded for management of samples?
If so, how do I remove/delete/archive a sample? If not, I recommend that a method of deleting or archiving samples be added to the program so that the Snake Malware Zoo will not get too full with samples?
Is there any way to make the program accessible outside of localhost?
I want the program to be accessed by multiple users in different locations. Is there a way to do this?
How do you format the text in the notes page to have titles and tables?

Upload Failed...

I'm just using the docker-compose script with only one change, the skin port changed to 80 from 8080 and I'm not able to upload any files.

logs from snake docker container as follows:

snake_1 | [W 200604 06:41:45 snake_handler:208] upload/file - no 'file' in part
snake_1 | [W 200604 06:41:45 web:2106] 422 POST /upload/file (172.19.0.5) 2.38ms

Utilise Cortex Analyzers?

Thoughts on implementing Cortex Analyzers (https://github.com/TheHive-Project/Cortex-Analyzers) alongside existing Snake Scales?

Potentially a quick and easy way to integrate more analysers into the project. They are pretty actively worked on.

Install Issues

This could be do to lack of coffee, but I am struggling to install Snake (core and scale). I have a clean install of Ubuntu 18.04 server and after installing all the dependencies listed I attempt to run the sys/user.sh (same if i try the production sys/install.sh) script, it appears to die after doing all the submodule path with no error message.

errored in pydeep due to a lack of gcc

i was trying to get this working on Ubuntu 16.04 vps
web gui installed successfully but upload functions did not work

install script i quickly put together
https://pastebin.com/7hDeTB9g

errors from the install
https://pastebin.com/iEPaN7GK

issue per talking with dev Alex

"Looks like it errored in pydeep due to a lack of gcc, which then seems to have stopped snake backend from installing. Feel free to open an issue on github where it is easier to track and debug. I will make sure to update the install script to stop when it hits an error."

Add Python 3.7 Support

This is currently blocked until the latest version of celery supports Python 3.7.

Need help with installing snake!

Hey guys!

I am having a little difficulty installing snake. Can someone help me out? ^^

Install script only enable localhost access

After installing and running, the API endpoints work, but the Web-UI does not. It loads, but if you try to upload a file it hangs and on refresh shows no file was uploaded.

I tried running the tests through Snake Charmer and it fails when starting celery.
Starting mongodb... Done! Starting redis... Done! Starting Celery...Could not start celery

I've verified that celery is installed, so I'm not sure why it can't start

I've tried reinstalling which did not work

I'm running this on Ubuntu:
Distributor ID: Ubuntu Description: Ubuntu 18.04.1 LTS Release: 18.04 Codename: bionic

The log files I checked didn't seem to helpful, is there anything else I can try to get this working?

Mongo sorting error after adding more samples

My number of samples is increasing steadily (28832) and today snake-skin stopped working due to an error from snake-core, see below. I'll have a look myself but it seems like if passed a threshold.

[E 200401 11:50:25 web:1621] Uncaught exception GET /store?limit=10&order=-1&sort=timestamp (127.0.0.1)
    HTTPServerRequest(protocol='http', host='localhost:5000', method='GET', uri='/store?limit=10&order=-1&sort=timestamp', version='HTTP/1.1', remote_ip='127.0.0.1')
    Traceback (most recent call last):
      File "lib/python3.7/site-packages/tornado/web.py", line 1543, in _execute
        result = yield result
      File "lib/python3.7/site-packages/tornado/gen.py", line 1099, in run
        value = future.result()
      File "lib/python3.7/site-packages/snake-1.0.1-py3.7.egg/snake/routes/store.py", line 56, in get
        while await cursor.fetch_next:
      File "lib/python3.7/concurrent/futures/thread.py", line 57, in run
        result = self.fn(*self.args, **self.kwargs)
      File "lib/python3.7/site-packages/pymongo-3.10.1-py3.7-amd64.egg/pymongo/cursor.py", line 1073, in _refresh
        self.__send_message(q)
      File "lib/python3.7/site-packages/pymongo-3.10.1-py3.7-amd64.egg/pymongo/cursor.py", line 955, in __send_message
        address=self.__address)
      File "lib/python3.7/site-packages/pymongo-3.10.1-py3.7-amd64.egg/pymongo/mongo_client.py", line 1346, in _run_operation_with_response
        exhaust=exhaust)
      File "lib/python3.7/site-packages/pymongo-3.10.1-py3.7-amd64.egg/pymongo/mongo_client.py", line 1464, in _retryable_read
        return func(session, server, sock_info, slave_ok)
      File "lib/python3.7/site-packages/pymongo-3.10.1-py3.7-amd64.egg/pymongo/mongo_client.py", line 1340, in _cmd
        unpack_res)
      File "lib/python3.7/site-packages/pymongo-3.10.1-py3.7-amd64.egg/pymongo/server.py", line 136, in run_operation_with_response
        _check_command_response(first)
      File "lib/python3.7/site-packages/pymongo-3.10.1-py3.7-amd64.egg/pymongo/helpers.py", line 159, in _check_command_response
        raise OperationFailure(msg % errmsg, code, response)
    pymongo.errors.OperationFailure: Executor error during find command :: caused by :: errmsg: "Sort operation used more than the maximum 33554432 bytes of RAM. Add an index, or specify a smaller limit."

Uploading errors

I'have some problems during the upload step of samples.

I have installed Snake on Ubuntu Server 18.04.

snake and snake-pit services are working correctly.

When i try to upload a sample:

On NGINX error log:
2019/12/18 13:48:42 [error] 1394#1394: *4 open() "/var/www/snake-skin/api/store" failed (2: No such file or directory), client: XXX.XXX.XXX.XXX, server: _, request: "GET /api/store?file_type=file&limit=10&sort=timestamp HTTP/1.1", host: "XXX.XXX.XXX.XXX:8000", referrer: "http://XXX.XXX.XXX.XXX:8000/" 2019/12/18 13:48:42 [error] 1393#1393: *1 open() "/var/www/snake-skin/api/store" failed (2: No such file or directory), client: XXX.XXX.XXX.XXX, server: _, request: "GET /api/store?file_type=memory&limit=10&sort=timestamp HTTP/1.1", host: "XXX.XXX.XXX.XXX:8000", referrer: "http://XXX.XXX.XXX.XXX:8000/" 2019/12/18 13:48:44 [error] 1393#1393: *6 open() "/var/www/snake-skin/api/scales" failed (2: No such file or directory), client: XXX.XXX.XXX.XXX, server: _, request: "GET /api/scales?file_type=file HTTP/1.1", host: "XXX.XXX.XXX.XXX:8000", referrer: "http://XXX.XXX.XXX.XXXm:8000/" 2019/12/18 13:48:58 [error] 1393#1393: *6 client intended to send too large body: 2409045 bytes, client: XXX.XXX.XXX.XXX, server: _, request: "POST /api/upload/file HTTP/1.1", host: "XXX.XXX.XXX.XXX:8000", referrer: "http://XXX.XXX.XXX.XXX:8000/"

Amazing project, will it be further updated?

Hi, love the project and I find it very useful. Can I check if the project will still be updated/supported further?

Tim

Issues with Web UI

Hello

I think that I have made some errors when it came to the installation of Snake on my Ubuntu 20.04 Desktop Machine. I have followed all of the instructions provided for the installation and startup of snake, snake-pit, and snake-skin. However, when I try to access the webui page, I get the following message:

I have also checked the :6379 page and get this message:

If I could get some help regarding on how to start to fix these problems this would be greatly appreciated.

Yara rules creation

Does snake have ability to automatically produce yara rules for malware samples?

What are your Contribution Guidelines?

All,

Are you able to define contribution guidelines. I am keen to contribute to the project but would like to know if there are any contribution guidelines?

Cheers!

vue-cli-service build fails

I am getting the following error message when installing the production version of snake on a CentOS 7 server:

> [email protected] build /home/user/snake/snake-skin
> vue-cli-service build

/home/user/snake/snake-skin/node_modules/open/index.js:16
const wslToWindowsPath = async path => {
                               ^^^^

SyntaxError: Unexpected identifier
    at createScript (vm.js:56:10)
    at Object.runInThisContext (vm.js:97:10)
    at Module._compile (module.js:549:28)
    at Object.Module._extensions..js (module.js:586:10)
    at Module.load (module.js:494:32)
    at tryModuleLoad (module.js:453:12)
    at Function.Module._load (module.js:445:3)
    at Module.require (module.js:504:17)
    at require (internal/module.js:20:19)
    at Object.<anonymous> (/home/user/snake/snake-skin/node_modules/@vue/cli-shared-utils/lib/openBrowser.js:9:14)

npm ERR! Linux 3.10.0-957.27.2.el7.x86_64
npm ERR! argv "/usr/bin/node" "/bin/npm" "run" "build"
npm ERR! node v6.17.1
npm ERR! npm  v3.10.10
npm ERR! code ELIFECYCLE
npm ERR! [email protected] build: `vue-cli-service build`
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the [email protected] build script 'vue-cli-service build'.
npm ERR! Make sure you have the latest version of node.js and npm installed.
npm ERR! If you do, this is most likely a problem with the snake-skin package,
npm ERR! not with npm itself.
npm ERR! Tell the author that this fails on your system:
npm ERR!     vue-cli-service build
npm ERR! You can get information on how to open an issue for this project with:
npm ERR!     npm bugs snake-skin
npm ERR! Or if that isn't available, you can get their info via:
npm ERR!     npm owner ls snake-skin
npm ERR! There is likely additional logging output above.

npm ERR! Please include the following file with any support request:
npm ERR!     /home/user/snake/snake-skin/npm-debug.log
mv: cannot stat ‘dist’: No such file or directory

Install instructions need updating

Have tried all mentioned methods on new/clean installs of Ubuntu 16.04 , 18.04 , 20.04 and errors occur at various steps. These are all on new AWS EC2 instances and can be tested quickly.

Part of the issue seems to be the services not being created correctly in the setup sh

Add docker support

I found your project while looking for a place to house some malware and wanted to give it a quick look, however I didn't want to run it on my host machine and have been having some issues getting it to work in docker (no systemd, 404 on web ui when trying to run everything individually).

I see that the last update to snake was about 11 months at this point - are there plans to continue support for this application? If so it would be great to have a docker-compose file so that spinning up an instance is trivial. I'd be happy to help with this if there is interest - seems like there should be 3 or 4 docker containers - mongo/redis/snake backend/snake frontend.

Cheers,
Pieter

Configurable SSL

Allow for configurable SSL for the web interface. The current default HTTP is not very secure when combined with the lack of authentication. If this is already possible, expand on this in documentation.

Failed at the [email protected] build script.

Hi,

Been trying to install this on a VPS Hosting Ubuntu 16.04 but keep getting this error. Have you encountered this before?

[email protected] build /root/snake/snake-skin
node build/build.js
sudo
⠧ building for production...Killed
npm ERR! code ELIFECYCLE
npm ERR! errno 137
npm ERR! [email protected] build: node build/build.js
npm ERR! Exit status 137
npm ERR!
npm ERR! Failed at the [email protected] build script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.
npm ERR! A complete log of this run can be found in:
npm ERR! /root/.npm/_logs/2018-03-16T19_04_43_447Z-debug.log
mv: cannot stat 'dist': No such file or directory

Thanks in advance!

Expose samples on filesystem

Ever thought of exposing the samples to the filesystem?
This way other tools can be utilised where a scale has not been made or can't be. I'm thinking use of IDA and then storing the db with the sample?
Or even storing a config file with the sample?