Code Monkey home page Code Monkey logo

intel_collector's Introduction

intel_collector

intel collector is a Python library to query various sources of threat intelligence for data on domains, file hashes, and IP addresses. Responses are returned in JSON format and written to CSV.

CrowdStrike Falcon and Microsoft Defender for Endpoint customers can also query their tenant for the presence of indicators within their own environment.

Supported APIs

Free Resources

BinaryEdge (binaryedge.io)
Circl.lu (hashlookup.circl.lu)
Echotrail (echotrail.io)
Filescan.io (filescan.io)
GreyNoise Community API (api.greynoise.io)
Hybrid Analysis (hybrid-analysis.com)
LeakIX (leakix.net)
Netlas (app.netlas.io)
Onyphe Free Tier (onyphe.io)
PulseDive (pulsedive.com)
Shodan (shodan.io)
Stalkphish (stalkphish.io)
Stratosphere IPS (stratosphereips.org)
Triage (tria.ge)
Urlhaus (urlhaus-api.abuse.ch)
Urlscan.io (urlscan.io)
VirusTotal (virustotal.com)

Paid Resources

CrowdStrike Falcon Intel (api.crowdstrike.com)
Emerging Threats Intelligence (api.emergingthreats.net)
Microsoft Defender for Endpoint (api.securitycenter.windows.com)

Setting API keys

API keys are set from within the library for each intel source.

# binaryedge.py (BinaryEdge)
binaryedge_api_key = 'your binary edge api key'

# crwd.py (CrowdStrike)
crwd_client_id = 'your crowdstrike api client id'
crwd_client_secret = 'your crowdstrike api client secret'

# echotrail.py (Echotrail.io)
echotrail_api_key = 'your echotrail api key'

# etintel.py (Emerging Threats Intelligence)
etintel_api_key = 'your emerging threats intelligence api key'

# filescan.py (Filescan.io)
filescan_api_key = 'your filescan.io api key'

# greynoise.py (GreyNoise.io)
greynoise_api_key = 'your greynoise community api key'

# hybrid.py (Hybrid Analysis)
hybrid_api_key = 'your hybrid analysis api key'

# leakix.py (LeakIX)
leakix_api_key = 'your leakix api key'

# msde.py (Microsoft Defender for Endpoint)
msft_tenant_id = 'your M365 tenant id'
msft_client_id = 'your M365 client id'
msft_client_secret = 'your M365 client secret'

# netlas.py (Netlas.io)
netlas_api_key 'your netlas api key'

# onyphe.py (Onyphe)
onyphe_api_key = 'your onyphe api key'

# pulsedive.py (Pulsedive)
pulsedive_api_key = 'your pulsedive api key'

# shodanpy.py (Shodan)
shodan_api_key = 'your shodan api key'

# stalkphish.py (Stalkphish)
stalkphish_api_key = 'Token your stalkphish api key'

# triage.py (Tria.ge)
triage_api_key = 'your tria.ge api key'

# urlscan.py (Urlscan.io)
urlscan_api_key = 'your urlscan.io api key'

# virustotal.py (VirusTotal)
virustotal_api_key = 'your virustotal api key'

Disabling Modules

All modules are enabled by default. Modules within each function can be disabled if you don't have an API key or don't wish to utilize them. Add # to the beginning of these lines as needed:

# Free Resources 
import binaryedge
import circl
import echotrail
import filescan
import greynoise
import hybrid
import leakix
import netlas
import onyphe
import pulsedive
import shodanpy
import stalkphish
import strato
import triage
import urlhaus
import urlscan
import virustotal

# Paid Resources
import crwd
import msde
import etintel

find_domain

    results["Emerging Threats"] = etintel.domain(domain)        # Emerging Threats
    results["Microsoft"] = msde.domain(domain)                  # Microsoft Defender for Endpoint
    results["Netlas"] = netlas.iocs(domain)                     # Netlas.io
    results["Onyphe"] = onyphe.domain(domain)                   # Onyphe
    results["Pulsedive"] = pulsedive.iocs(domain)               # Pulsedive
    results["Shodan"] = shodanpy.domain(domain)                 # Shodan
    results["Tria.ge"] = triage.iocs(domain,'domain')           # Tria.ge
    results["URLhaus"] = urlhaus.iocs(domain,'host')            # Urlhaus
    results["URLScan"] = urlscan.domain(domain)                 # Urlscan.io
    results["VirusTotal"] = virustotal.domain(domain)           # VirusTotal

find_hash

    results["Circl.lu"] = circl.hash(hash,'md5')         # Circl.lu
    results["CrowdStrike"] = crwd.iocs(hash,'md5')       # CrowdStrike Falcon
    results["Echotrail"] = echotrail.hash(hash)          # Echotrail.io
    results["Emerging Threats"] = etintel.hash(hash)     # Emerging Threats
    results["FileScan.io"] = filescan.hash(hash,'md5')   # Filescan.io
    results["Hybrid Analysis"] = hybrid.hash(hash)       # Hybrid Analysis
    results["Microsoft"] = msde.hash(hash)               # Microsoft Defender for Endpoint            
    results["Tria.ge"] = triage.iocs(hash,'sha1')        # Tria.ge
    results["URLhaus"] = urlhaus.iocs(hash,'sha256')     # Urlhaus
    results["URLScan"] = urlscan.hash(hash)              # Urlscan.io
    results["VirusTotal"] = virustotal.hash(hash)        # VirusTotal

find_ip

    results["BinaryEdge"] = binaryedge.ip(ip)            # BinaryEdge.io
    results["CrowdStrike"] = crwd.iocs(ip,'ipv4')        # CrowdStrike Falcon
    results["Emerging Threats"] = etintel.ip(ip)         # Emerging Threats
    results["GreyNoise"] = greynoise.ip(ip)              # GreyNoise
    results["LeakIX"] = leakix.ip(ip)                    # LeakIX
    results["Microsoft"] = msde.ip(ip)                   # Microsoft Defender for Endpoint
    results["Netlas"] = netlas.iocs(ip)                  # Netlas.io
    results["Onyphe"] = onyphe.ip(ip)                    # Onyphe
    results["Pulsedive"] = pulsedive.iocs(ip)            # Pulsedive
    results["Shodan"] = shodanpy.ip(ip)                  # Shodan
    results["Stalkphish"] = stalkphish.ip(ip)            # Stalkphish
    results["Stratosphere IPS"] = strato.ip(ip)          # Stratosphere IPS
    results["Tria.ge"] = triage.iocs(ip,'ip')            # Tria.ge
    results["URLhaus"] = urlhaus.iocs(ip,'host')         # Urlhaus
    results["URLScan"] = urlscan.ip(ip)                  # Urlscan.io
    results["VirusTotal"] = virustotal.ip(ip)            # VirusTotal

Usage

# Import the library
from intel_collector import intel_collector

# Initialize client API keys and base URLs  
go = intel_collector()

# Get information on a domain
go.find_domain('bkdata.vn')

# Get information on an IP address
go.find_ip('103.161.17.242')

# Get information on a file hash
go.find_hash('870c31aa344b2950d0ea4849a472dafed312ecee8aa212c47bf543668bbee8e9')
go.find_hash('1e5bc9d7e413ddd7902c2932e418702b84d0cc07')
go.find_hash('177f3c8a2623d4efb41b0020d680be83')

Helpful hints for searching file hashes:

  • The Circl.lu API supports the following indicator types (md5, sha1, sha256)
  • The Crowdstrike Falcon API for custom IOCs supports the following indicator types (md5, sha256)
  • The Echotrail API supports the following indicator types (md5, sha256)
  • The ET Intel API supports the (md5) indicator type
  • The Filescan.io API supports the following indicator types (md5, sha1, sha256)
  • The Hybrid Analysis API supports the following indicator types (md5, sha1, sha256)
  • The Microsoft Defender for Endpoint API supports the following indicator types (sha1, sha256)
  • The Tria.ge API supports the following indicator types (md5, sha1, sha256, sha512)
  • The Urlhaus API supports the following indicator types (md5, sha256)
  • The Urlscan.io API supports the (sha256) indicator type
  • The VirusTotal API supports the following indicator types (md5, sha1, shad256)

Sample Output

go.find_domain('bkdata.vn')
    Found in Microsoft Defender for Endpoint - Domains
    Found in Netlas.io
    Found in Onyphe
    Found in Shodan
go.find_hash('870c31aa344b2950d0ea4849a472dafed312ecee8aa212c47bf543668bbee8e9')
    Response from Echotrail.io
    Found in Filescan.io
    Found in Hybrid Analysis
    Found in Microsoft Defender for Endpoint - Global File Stats
    Found in Tria.ge
    Found in VirusTotal - Files
go.find_ip('103.161.17.242')
    Found in Emerging Threats - Events
    Response from GreyNoise
    Found in Microsoft Defender for Endpoint - IP Stats
    Found in Onyphe
    Found in Shodan
    Found in Stalkphish
    Response from URLscan.io

Contributing

Pull requests are welcome. For major changes, please open an issue to discuss what you would like to change.

Authors

Jason Ford

License

GPLv3

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.