intelowlproject / intelowl Goto Github PK
View Code? Open in Web Editor NEWIntelOwl: manage your Threat Intelligence at scale
Home Page: https://intelowlproject.github.io
License: GNU Affero General Public License v3.0
IntelOwl: manage your Threat Intelligence at scale
Home Page: https://intelowlproject.github.io
License: GNU Affero General Public License v3.0
We could integrate an analyzer for the FireHOL blocklists. In this way, we could check if a specific IP was reported as malicious by one of those lists.
We should manage the download of these lists internally in the project. Then we should update these lists regularly to allow the IntelOwl users to get fresh results.
Matteo Lodi:
We could add indexes for the fields that are being used for search purposes.
Obviously, for testing and for a "classic" usage of this project this is not necessary, but to let it scale we should index the most important ones. https://docs.djangoproject.com/en/2.2/ref/models/indexes/
Also, at the moment there is a cronjob in crons.py
that performs a cleanup of old data. The retention days are hardcoded. We could move this value to the configuration and let the intel owl users decide the retention days.
We could add an analyzer that checks if a URL is in the Phishtank phish list.
We should integrate Droidbox as a new analyzer. As stated on the honeynet website,
Droidbox is a dynamic analysis platform for android applications. Droidbox was developed by Patrik Lantz as part of GSoc 2011. You can try it out by downloading Android Reverse Engineering virtual machine, which bundels droidbox as well as additional android malware analysis tools.
We should add an analyzer to extract info from Auth0.
https://auth0.com/signals/docs/#get-full-ip-address-reputation-info
Suggestions Please @mlodic.
We could integrate APKiD which is an Android Application Identifier for Packers, Protectors, Obfuscators and Oddities - PEiD for Android.
GitHub: https://github.com/rednaga/APKiD
Task #1: (from eshaan's GSoC proposal)
django-cors-headers
package to deal with preflight CORS requests.https://cloud.google.com/web-risk
We could create an analyzer for the WebRisk API. In this way, we can check a specific URL against the provided blacklists. This is the commercial version of the "GoogleSafeBrowsing" APIs.
We could add a Manet analyzer to get a screenshot from a specified URL that we would like to analyze.
https://github.com/vbauer/manet
We should add the docker image as optional in the docker-compose configuration and use the Manet APIs to run the analysis.
ATM, there's no way to dynamically pass in a configuration for an analyzer at the time of requesting a scan. For any configuration changes, the user needs to edit the analyzer_config.json
manually.
Breaking past this limit could be one of the most exciting and powerful features of Intel Owl.
Some use cases:
It's not really clear on first use what analyzers are available, or how the platform is to be used. For example, I try to search an MD5 once logged in (eg. Mimikatz) and get no results. Do I need to enable some more analyzers, or is there something else I must do to populate a local DB?
We should add support for some common authentication methods that could be leveraged to avoid the pain to create and manage new users directly in IntelOwl.
We should add an analyzer to analyze files with PEframe
Get WHOIS for a specific IP with the RIPE API:
https://github.com/RIPE-NCC/whois/wiki/WHOIS-REST-API
https://www.ripe.net/manage-ips-and-asns/db/support/documentation/ripe-database-query-reference-manual#tables-of-query-types-supported-by-the-ripe-database
http://jaluther.blogspot.com/2016/04/python-for-network-engineers-part-2.html
We could add another analyzer for VirusTotal that extracts the sandbox reports available for a specific file.
https://developers.virustotal.com/v3.0/reference#file-behaviour-summary
At the moment this analyzer (https://github.com/intelowlproject/IntelOwl/blob/master/api_app/script_analyzers/observable_analyzers/maxmind.py) leverages only the "Country" DB.
We could add the download of the GeoCity DB weekly and the related check.
Example usage of the library: https://geoip2.readthedocs.io/en/latest/
We should add an analyzer to scan a given file against all the yara rules available in Malpedia.
https://malpedia.caad.fkie.fraunhofer.de/usage/api#apiscanbinary
Under script_analyzers/, we have too much duplicate code in each analyzer's run()
function. For example, getting additional_config_params
, verifying them, setting up report, catching and handling errors. Can't we take more of an Object oriented approach here ?
This is just a very basic overview of what I have in mind:
from abc import ABC, abstractmethod
import requests
class AnalyzerRunNotImplemented(Exception):
detail = "run() is not implemented for analyzer {}"
class Analyzer(ABC):
analyzer_name = None
md5 = None
observable_classification = None
# and more variables
...
@abstractmethod
def run(self, *args, **kwargs):
# this should be overwritten in
# child class
raise AnalyzerRunNotImplemented(self.analyzer_name)
def __init__(self, *args, **kwargs):
# initialize shared variables
def get_config(self, *args, **kwargs):
# fetch additional_config_params
# verify params, API keys, etc
def set_report_and_cleanup(self, *args, **kwargs):
# sets report logic
def handle_error(self, *args, **kwargs):
# handles error logic
class Shodan(Analyzer):
analyzer_name = "shodan"
observable_classification = "ip"
url = "https://api.shodan.io/shodan/host/"
def run(self):
try:
data = {"ip": self.observable_value}
resp = requests.post(self.url, json=self.data)
return super(self).set_report(resp)
except Exception:
return super(self).handle_error(resp)
Advantages:
def run(analyzer_name, job_id, observable_name, observable_classification, additional_config_params)
- This does not look good.set_report_and_cleanup
and handle_error
functions too. But for all others, we don't need to.Using f-strings
increases code readability and are known to be faster than str.format()
In the tests/test_observables.py
, For HybridAnalysis I noticed that we are passing HA_Get
as the analyzer name in all 3 instances, shouldn't it be HybridAnalysis_Get_Observable
, as defined in the analyzers_configuration.json ? or am I wrong?
All the 3 instances referenced:
https://github.com/certego/IntelOwl/blob/af05c109e253100ae9a2b0087c96676bdb19dae4/tests/test_observables.py#L93
https://github.com/certego/IntelOwl/blob/af05c109e253100ae9a2b0087c96676bdb19dae4/tests/test_observables.py#L157
https://github.com/certego/IntelOwl/blob/af05c109e253100ae9a2b0087c96676bdb19dae4/tests/test_observables.py#L244
We could separate the requirements.txt
file into small files such as development/local.txt and requirements/production.txt.
Originally posted by @Andrew-Chen-Wang in #95 (comment)
Browser extensions in Chrome are identified by an alphanumeric identifier.
For example, Google Keep has the following URL:
https://chrome.google.com/webstore/detail/google-keep-chrome-extens/lpcaedmchfhocbbapmcbpinfpgnhiddi
which leads to the extension ID lpcaedmchfhocbbapmcbpinfpgnhiddi
.
Extension IDs are used to determine the installation folder, e.g.
C:\Users\[login_name]\AppData\Local\Google\Chrome\User Data\Default\Extensions\lpcaedmchfhocbbapmcbpinfpgnhiddi
Also, in normal mode (not in incognito windows) local resources injected by an extension inside web pages is loaded by using urls of this form:
chrome-extension://lpcaedmchfhocbbapmcbpinfpgnhiddi/resource-name.js
Knowing whether an extension is malicious or not is a very useful task in many cases, and several public services/APIs already exist for this purpose, for example:
Being able to analyze CRX IDs would be very useful IMHO.
Hey @eshaan7. I'm a contributor on cookiecutter-django, and I think the way you've set up this project could take some inspiration from that repository.
For one, the compose files could be labeled a little better. Other things like the .env files could be in their own directory, the Dockerfiles can be in their own, and the requirements could be separated to make the Docker builds smaller.
The one main part is Traefik: auto-renewal of certs and easy to configure with compose. You can take a look at production.yml and compose/production/traefik for how we configure it. It may come useful for those looking for self deployment.
If people want to scale, I made a complete-walkthrough repository for those looking to do AWS ECS deployment via git push to GitHub. It can be made into fargate if you don't integrate celery. (Which is the other thing: I don't think using Fargate with this project would be wise since you're using celery. The first two answers in this SO makes it clear why.).
We should add an analyzer to extract info about an IP/URL/hash from ThreatMiner.
https://www.threatminer.org/api.php
Suggestions Please @mlodic.
Issue I am facing:
When I create a new analyze request using the pyintelowl client, the job gets created but it keeps saying `status: "running" and the command hangs, see here:
And if I visit the server's GUI to query the database, this is what I get:
The job just hangs there finally throws me time-out error.
What I tried:
I have tried with different analyzers, different files and observables as well. I am facing the same issue.
There are no errors in my docker-compose terminal.
Any help would be appreciated.
Some IDEs like having .env files because their extensions can easily digest those and show syntax errors. Additionally, it's just habit/conventional to have these "." files that are either unnecessary or "secretive" stay as . so that gitignore files can have something like:
.*
!.travis.yml
etc.
We should add a new analyzer to allow users to analyze files using a Drakvuf instance:
At the moment, IntelOwl can be used mainly via APIs because its major intent is to be integrated with other security tools.
So now we can find only a very basic interface that allow the users:
We should start improving the web interface to allow the project to be more user friendly:
We could create an analyzer that checks a specific hash against the malware database of Team Cymru
Check a URL, domain or IP against the URLhaus database
Stumbled upon GreyNoise.
They have an "IP Context" API endpoint - https://docs.greynoise.io/#ip-context.
Should we integrate this ? @mlodic
We should add an analyzer to extract info about an IP using Shodan APIs.
https://developer.shodan.io/api
https://shodan.readthedocs.io/en/latest/
We should add an analyzer to analyze malicious Javascript files using Box-js
We can integrate wigle.net's API for wireless network’s analysis.
Interesting API endpoints:
We should add an analyzer to extract info from Securitytrials
https://securitytrails.com/corp/api
Suggestions Please @mlodic.
IntelOwl was designed to be integrated with other security tools very easily.
In particular, it is a perfect fit for Threat Intelligence Platforms because it can enrich the data available in those platform and can be customized on its own needs.
So, in order to help the community to achieve this result, we should add new connectors for the most used open source TIPs, for example: MISP
We should study their project and add a relative connector to integrate IntelOwl with MISP.
An idea could be to create a new expansion module (https://github.com/MISP/misp-modules) to allow MISP users to enrich an indicator via IntelOwl and integrate the results on the relative event.
We should leverage the official IntelOwl client for this purpose.
We should leverage the new service created by Abuse.ch to create an analyzer that checks a file hash against the Malware Bazaar DB.
We should integrate Thug as a new analyzer: https://github.com/buffer/thug
Even if Thug can be easily installed with pip for python3, it really requires a lot of dependencies to be installed in the machine (in particular https://github.com/area1/stpyv8/). We need to try to study how to add it to the project without "poisoning" the base docker image too much. For this purpose we could just add the official Thug docker image (https://hub.docker.com/r/buffer/thug/dockerfile) to the docker-compose file but, then, we should expose that container to the main container to let it run Thug via API.
Some other requirements:
Custom column fields and searchable fields in admin view models for Job and Tag DB models.
We should add a new analyzer to allow users to analyze files with ClamAV.
https://github.com/Cisco-Talos/clamav-devel.
It's important to:
if the license is permissive, add public-available rules by default
give the chance to customize the rules used to scan a file with ClamAV, in particular to run its own rules
Adding to the awesome list already, we can also integrate the following sources to fetch intel:
intelx.io [Data Breaches] : https://blog.intelx.io/2020/04/13/new-python-api-wrapper-and-command-line-interface/
binaryedge.io [~Shodan]: https://docs.binaryedge.io/api-v2/
spiderfoot [Mega-Platform for many such sources]: https://www.spiderfoot.net/documentation/
phonebook.cz [A part of Intelx.io]: https://phonebook.cz/
We should add an analyzer to extract info from opswat metadefender cloud
https://onlinehelp.opswat.com/mdcloud/Description_of_file_categories.html
Suggestions Please @mlodic
ONYPHE is a search engine for open-source and cyber threat intelligence data collected by crawling various sources available on the Internet or by listening to Internet background noise.
They have many scanning services available for an IP/Domain/URL.
Please take a look at API docs and suggest the ones we need.
There's a python client, pyonyphe available.
Plus, an expansion-plugin in MISP - MISP/....../expansion/onyphe.py
Should we try to integrate this with the MISP analzyer we have or use the python client available or just the API ?
IntelOwl was designed to be integrated with other security tools very easily.
In particular, it is a perfect fit for Threat Intelligence Platforms because it can enrich the data available in those platform and can be customized on its own needs.
So, in order to help the community to achieve this result, we should add new connectors for the most used open source TIPs, for example: OpenCTI
We should study their project and add a relative connector to integrate IntelOwl with OpenCTI. A related issue was already opened: OpenCTI-Platform/connectors#43.
We should leverage the official IntelOwl client for this purpose
We should add an analyzer to extract info from URLscan
Suggestions Please @mlodic.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.