intelowlproject / intelowl Goto Github PK

http://iplists.firehol.org/

We could integrate an analyzer for the FireHOL blocklists. In this way, we could check if a specific IP was reported as malicious by one of those lists.

We should manage the download of these lists internally in the project. Then we should update these lists regularly to allow the IntelOwl users to get fresh results.

Matteo Lodi:

• We could add indexes for the fields that are being used for search purposes.
  Obviously, for testing and for a "classic" usage of this project this is not necessary, but to let it scale we should index the most important ones. https://docs.djangoproject.com/en/2.2/ref/models/indexes/

• Also, at the moment there is a cronjob in crons.py that performs a cleanup of old data. The retention days are hardcoded. We could move this value to the configuration and let the intel owl users decide the retention days.

We could add an analyzer that checks if a URL is in the Phishtank phish list.

https://www.phishtank.com/api_info.php

We should integrate Droidbox as a new analyzer. As stated on the honeynet website,

Droidbox is a dynamic analysis platform for android applications. Droidbox was developed by Patrik Lantz as part of GSoc 2011. You can try it out by downloading Android Reverse Engineering virtual machine, which bundels droidbox as well as additional android malware analysis tools.

We should add an analyzer to extract info from Auth0.

https://auth0.com/signals/docs/#get-full-ip-address-reputation-info

Suggestions Please @mlodic.

We could integrate APKiD which is an Android Application Identifier for Packers, Protectors, Obfuscators and Oddities - PEiD for Android.

GitHub: https://github.com/rednaga/APKiD

Task #1: (from eshaan's GSoC proposal)

• login, logout API endpoints in Django application
• add the django-cors-headers package to deal with preflight CORS requests.

https://cloud.google.com/web-risk

We could create an analyzer for the WebRisk API. In this way, we can check a specific URL against the provided blacklists. This is the commercial version of the "GoogleSafeBrowsing" APIs.

We could add a Manet analyzer to get a screenshot from a specified URL that we would like to analyze.

https://github.com/vbauer/manet

We should add the docker image as optional in the docker-compose configuration and use the Manet APIs to run the analysis.

ATM, there's no way to dynamically pass in a configuration for an analyzer at the time of requesting a scan. For any configuration changes, the user needs to edit the analyzer_config.json manually.

Breaking past this limit could be one of the most exciting and powerful features of Intel Owl.

Some use cases:

• an option that allows requesting an analysis with X instances of a single analyzer with different/dynamic configuration.
• ...

It's not really clear on first use what analyzers are available, or how the platform is to be used. For example, I try to search an MD5 once logged in (eg. Mimikatz) and get no results. Do I need to enable some more analyzers, or is there something else I must do to populate a local DB?

We should add support for some common authentication methods that could be leveraged to avoid the pain to create and manage new users directly in IntelOwl.

• LDAP support
• GSuite via SAML
• others, if requested

We should add an analyzer to analyze files with PEframe

https://github.com/guelfoweb/peframe

Get WHOIS for a specific IP with the RIPE API:

https://github.com/RIPE-NCC/whois/wiki/WHOIS-REST-API
https://www.ripe.net/manage-ips-and-asns/db/support/documentation/ripe-database-query-reference-manual#tables-of-query-types-supported-by-the-ripe-database
http://jaluther.blogspot.com/2016/04/python-for-network-engineers-part-2.html

We could add another analyzer for VirusTotal that extracts the sandbox reports available for a specific file.

https://developers.virustotal.com/v3.0/reference#file-behaviour-summary

At the moment this analyzer (https://github.com/intelowlproject/IntelOwl/blob/master/api_app/script_analyzers/observable_analyzers/maxmind.py) leverages only the "Country" DB.

We could add the download of the GeoCity DB weekly and the related check.

Example usage of the library: https://geoip2.readthedocs.io/en/latest/

We should add an analyzer to scan a given file against all the yara rules available in Malpedia.

https://malpedia.caad.fkie.fraunhofer.de/usage/api#apiscanbinary

Under script_analyzers/, we have too much duplicate code in each analyzer's run() function. For example, getting additional_config_params, verifying them, setting up report, catching and handling errors. Can't we take more of an Object oriented approach here ?

This is just a very basic overview of what I have in mind:

from abc import ABC, abstractmethod
import requests

class AnalyzerRunNotImplemented(Exception):
    detail = "run() is not implemented for analyzer {}"

class Analyzer(ABC):
    analyzer_name = None
    md5 = None
    observable_classification = None
    # and more variables
    ...

    @abstractmethod
    def run(self, *args, **kwargs):
        # this should be overwritten in 
        # child class
        raise AnalyzerRunNotImplemented(self.analyzer_name)

    def __init__(self, *args, **kwargs):
        # initialize shared variables
    
    def get_config(self, *args, **kwargs):
        # fetch additional_config_params
        # verify params, API keys, etc

    def set_report_and_cleanup(self, *args, **kwargs):
        # sets report logic
    def handle_error(self, *args, **kwargs):
        # handles error logic


class Shodan(Analyzer):
    analyzer_name = "shodan"
    observable_classification = "ip"
    url = "https://api.shodan.io/shodan/host/"
    
    def run(self):
        try:
            data = {"ip": self.observable_value}
            resp = requests.post(self.url, json=self.data)
            return super(self).set_report(resp)
        except Exception:
            return super(self).handle_error(resp)

Advantages:

• reduce a lot of duplicate code
• code looks more clean
• makes it easier for new contributors to integrate an analyzer.
• don't have to pass a bunch of variables to the run function, since we can just store them as member variables of the class. def run(analyzer_name, job_id, observable_name, observable_classification, additional_config_params) - This does not look good.
• in some analyzers, the error/report handling is different - for those we can just overwrite the set_report_and_cleanup and handle_error functions too. But for all others, we don't need to.
• We already have code for handling error, setting up report and all in general.py, we would mostly need to refactor it work with objects/classes.

Using f-strings increases code readability and are known to be faster than str.format()

In the tests/test_observables.py, For HybridAnalysis I noticed that we are passing HA_Get as the analyzer name in all 3 instances, shouldn't it be HybridAnalysis_Get_Observable, as defined in the analyzers_configuration.json ? or am I wrong?

All the 3 instances referenced:
https://github.com/certego/IntelOwl/blob/af05c109e253100ae9a2b0087c96676bdb19dae4/tests/test_observables.py#L93
https://github.com/certego/IntelOwl/blob/af05c109e253100ae9a2b0087c96676bdb19dae4/tests/test_observables.py#L157
https://github.com/certego/IntelOwl/blob/af05c109e253100ae9a2b0087c96676bdb19dae4/tests/test_observables.py#L244

We could separate the requirements.txt file into small files such as development/local.txt and requirements/production.txt.

Originally posted by @Andrew-Chen-Wang in #95 (comment)

Browser extensions in Chrome are identified by an alphanumeric identifier.

For example, Google Keep has the following URL:

https://chrome.google.com/webstore/detail/google-keep-chrome-extens/lpcaedmchfhocbbapmcbpinfpgnhiddi

which leads to the extension ID lpcaedmchfhocbbapmcbpinfpgnhiddi.

Extension IDs are used to determine the installation folder, e.g.

C:\Users\[login_name]\AppData\Local\Google\Chrome\User Data\Default\Extensions\lpcaedmchfhocbbapmcbpinfpgnhiddi

Also, in normal mode (not in incognito windows) local resources injected by an extension inside web pages is loaded by using urls of this form:

chrome-extension://lpcaedmchfhocbbapmcbpinfpgnhiddi/resource-name.js

Knowing whether an extension is malicious or not is a very useful task in many cases, and several public services/APIs already exist for this purpose, for example:

• https://crxcavator.io/
• https://robwu.nl/crxviewer/

Being able to analyze CRX IDs would be very useful IMHO.

Hey @eshaan7. I'm a contributor on cookiecutter-django, and I think the way you've set up this project could take some inspiration from that repository.

For one, the compose files could be labeled a little better. Other things like the .env files could be in their own directory, the Dockerfiles can be in their own, and the requirements could be separated to make the Docker builds smaller.

The one main part is Traefik: auto-renewal of certs and easy to configure with compose. You can take a look at production.yml and compose/production/traefik for how we configure it. It may come useful for those looking for self deployment.

If people want to scale, I made a complete-walkthrough repository for those looking to do AWS ECS deployment via git push to GitHub. It can be made into fargate if you don't integrate celery. (Which is the other thing: I don't think using Fargate with this project would be wise since you're using celery. The first two answers in this SO makes it clear why.).

We should add an analyzer to extract info about an IP/URL/hash from ThreatMiner.

https://www.threatminer.org/api.php

Suggestions Please @mlodic.

Issue I am facing:
When I create a new analyze request using the pyintelowl client, the job gets created but it keeps saying `status: "running" and the command hangs, see here:

And if I visit the server's GUI to query the database, this is what I get:

The job just hangs there finally throws me time-out error.

What I tried:

I have tried with different analyzers, different files and observables as well. I am facing the same issue.
There are no errors in my docker-compose terminal.

Any help would be appreciated.

Some IDEs like having .env files because their extensions can easily digest those and show syntax errors. Additionally, it's just habit/conventional to have these "." files that are either unnecessary or "secretive" stay as . so that gitignore files can have something like:

.*
!.travis.yml
etc.

We should add a new analyzer to allow users to analyze files using a Drakvuf instance:

https://github.com/tklengyel/drakvuf

At the moment, IntelOwl can be used mainly via APIs because its major intent is to be integrated with other security tools.
So now we can find only a very basic interface that allow the users:

• to authenticate
• to query the database for analysis results
• to view the results via a JSON file

We should start improving the web interface to allow the project to be more user friendly:

• Add the chance to analyze an observable from the web interface
• Add the chance to view the analyzer configuration and to change it
• etc...more ideas are welcome

We could create an analyzer that checks a specific hash against the malware database of Team Cymru

https://www.team-cymru.com/mhr.html

• A tags column in the Job table in the database. That way, a user can distinguish between or group different jobs.

https://urlhaus-api.abuse.ch

Check a URL, domain or IP against the URLhaus database

Stumbled upon GreyNoise.
They have an "IP Context" API endpoint - https://docs.greynoise.io/#ip-context.

Should we integrate this ? @mlodic

Integrate psf/black and flake8 for better linting and consistent styling.

We should add an analyzer to extract info about an IP using Shodan APIs.

https://developer.shodan.io/api
https://shodan.readthedocs.io/en/latest/

We should add an analyzer to analyze malicious Javascript files using Box-js

https://github.com/CapacitorSet/box-js

We can integrate wigle.net's API for wireless network’s analysis.

Interesting API endpoints:

• https://api.wigle.net/swagger#/Bluetooth_search_and_information_tools/detail
• https://api.wigle.net/swagger#/Cell_search_and_information_tools/mccMnc

We should add an analyzer to extract info from Securitytrials

https://securitytrails.com/corp/api

Suggestions Please @mlodic.

IntelOwl was designed to be integrated with other security tools very easily.
In particular, it is a perfect fit for Threat Intelligence Platforms because it can enrich the data available in those platform and can be customized on its own needs.

So, in order to help the community to achieve this result, we should add new connectors for the most used open source TIPs, for example: MISP

We should study their project and add a relative connector to integrate IntelOwl with MISP.
An idea could be to create a new expansion module (https://github.com/MISP/misp-modules) to allow MISP users to enrich an indicator via IntelOwl and integrate the results on the relative event.
We should leverage the official IntelOwl client for this purpose.

We should leverage the new service created by Abuse.ch to create an analyzer that checks a file hash against the Malware Bazaar DB.

https://bazaar.abuse.ch/api/#query_hash

We should integrate Thug as a new analyzer: https://github.com/buffer/thug

Even if Thug can be easily installed with pip for python3, it really requires a lot of dependencies to be installed in the machine (in particular https://github.com/area1/stpyv8/). We need to try to study how to add it to the project without "poisoning" the base docker image too much. For this purpose we could just add the official Thug docker image (https://hub.docker.com/r/buffer/thug/dockerfile) to the docker-compose file but, then, we should expose that container to the main container to let it run Thug via API.

Some other requirements:

• give the chance to setup custom proxy
• give the chance to setup custom browser
• give the chance to analyze only URLs or HTML files
• use all DOM events as default

Custom column fields and searchable fields in admin view models for Job and Tag DB models.

We should add a new analyzer to allow users to analyze files with ClamAV.

https://github.com/Cisco-Talos/clamav-devel.

It's important to:

• if the license is permissive, add public-available rules by default

• give the chance to customize the rules used to scan a file with ClamAV, in particular to run its own rules

Adding to the awesome list already, we can also integrate the following sources to fetch intel:

intelx.io [Data Breaches] : https://blog.intelx.io/2020/04/13/new-python-api-wrapper-and-command-line-interface/
binaryedge.io [~Shodan]: https://docs.binaryedge.io/api-v2/
spiderfoot [Mega-Platform for many such sources]: https://www.spiderfoot.net/documentation/
phonebook.cz [A part of Intelx.io]: https://phonebook.cz/

We should add an analyzer to extract info from opswat metadefender cloud

https://onlinehelp.opswat.com/mdcloud/Description_of_file_categories.html

Suggestions Please @mlodic

ONYPHE is a search engine for open-source and cyber threat intelligence data collected by crawling various sources available on the Internet or by listening to Internet background noise.

They have many scanning services available for an IP/Domain/URL.
Please take a look at API docs and suggest the ones we need.

There's a python client, pyonyphe available.
Plus, an expansion-plugin in MISP - MISP/....../expansion/onyphe.py

Should we try to integrate this with the MISP analzyer we have or use the python client available or just the API ?

IntelOwl was designed to be integrated with other security tools very easily.
In particular, it is a perfect fit for Threat Intelligence Platforms because it can enrich the data available in those platform and can be customized on its own needs.

So, in order to help the community to achieve this result, we should add new connectors for the most used open source TIPs, for example: OpenCTI

We should study their project and add a relative connector to integrate IntelOwl with OpenCTI. A related issue was already opened: OpenCTI-Platform/connectors#43.
We should leverage the official IntelOwl client for this purpose

https://tranco-list.eu/api_documentation

We should add an analyzer to extract info from URLscan

https://urlscan.io/about-api/

Suggestions Please @mlodic.