tenzir / threatbus Goto Github PK

Hi @0ortmann and team,

ich bin zurueck =/

Trying to finish up an app plugin for threatbus to send out to another threat intel system (CIFv3). Setup is working fine and I can get threatbus started up with the cif3 plugin. The primary objective is to take submissions from MISP and eventually send them on to CIF.

However, the issue I've run into is that while the zmq sub to MISP seems to be picking up new attributes and sending them out to threatbus/intel topic, the call to queue.Queue().get() at https://github.com/mdavis332/threatbus/blob/master/plugins/apps/threatbus_cif3/plugin.py#L42 is hanging. If I take the if watched_queue.empty() bit out and just let it rely on the get() timeout, it cycles back around fine every 15 seconds. However, the second an Intel item is picked up by the queue, the loop never cycles again and never moves forward.

Any thoughts? Thanks again for your help.

It would be desirable to (optionally) completely exclude indicators received from MISP without the to_ids flag in MISP from retromatching.

Hello sir,
While solving the issue #82 I have encountered new issue. I have tried installing into fresh system. And reinstalled all the components again freshly. So here are the following conclusion I have reached till this point:
1- when I am compiling the threat bus and other tools like inmem, misp, zeek from the cloning the github, its not installing properly for some reasons. threatbus was stucking at :
2020-12-16 17:57:31 INFO [threatbus] Starting plugins...

2- What is working for me in that in "venv" envoirnment running the following command for installing threatbus and related utilities using:

pip install threatbus
pip install threatbus-inmem
pip install threatbus-misp
pip install threatbus-zeek

3- Seems like since the day zeek has been updated there is some issue started occurring between zeek and threatbus communication, because zeek version i am running is: 3.2.3

4- Threatbus misp communication is working fine for me.
So now the persistent issue is as shown in screenshot: (I will replace the API key later so didnt hide it here, once all fixed, atleast I hope so)
//Also tried with http://localhost
error in /opt/zeek/share/zeek/base/frameworks/broker/./log.zeek, line 80: Broker error (Broker::PEER_UNAVAILABLE): (invalid-node, *localhost:47760, "unable to connect to remote peer")

Sir, if my issue can be resolved, I will be very grateful to you.Thank you very much.

Kind Regards
Nillay

Hi!
First of all thank you for your great job!

I have some troubles with Threatbus using with Kafka. I want to ship IOCs from MISP to kafka via ThreatBus
That am i dooing wrong?

I have installed misp and dockerized kafka, manually created topic with name indicator, and run threatbus with my config-file.

ThreatBus messages:

ThreatBus host tcpdump:

Kafka host tcpdump:

I saw nothing in my indicator topic, only this in __consumer_offsets topic:

My ThreatBus config:

logging:
  console: true
  console_verbosity: INFO        # One of "DEBUG", "INFO", "WARNING", "ERROR", "CRITICAL".
  file: false

plugins:
  backbones:
    inmem: {}
  apps:
    misp:
      api:
        host: https://localhost
        ssl: false
        key: zBkLoUb7Q*******************Vqz
      filter:
        - orgs: 
            - "2"
      kafka:
        topics:
        - indicator
        poll_interval: 1.0
      #  # All config entries are passed as-is to librdkafka
      #  # https://github.com/edenhill/librdkafka/blob/master/CONFIGURATION.md
        config:
          bootstrap.servers: "kafka:9092"
          group.id: "threatbus"
          auto.offset.reset: "earliest"

My dockerized kafka:

version: '3.5'

networks:
  default:
    name: threatbus
    external: false
volumes:
  kafka_data:
  zookeeper_data:
services:
  zookeeper:
    image: docker.io/bitnami/zookeeper:3.7
    restart: unless-stopped
    ports:
      - "2181:2181"
    volumes:
      - "zookeeper_data:/bitnami"
      - "./zookeeper_data/:/bitnami/zookeeper/"
    environment:
      - ALLOW_ANONYMOUS_LOGIN=yes
      - ZOO_SERVER_ID=1 
  kafka:
    image: docker.io/bitnami/kafka:2
    restart: unless-stopped
    ports:
      - "9092:9092"
    volumes:
      - "kafka_data:/bitnami"
      - "./kafka_data/:/bitnami/kafka/data"
    environment:
      - KAFKA_BROKER_ID=1 
      - KAFKA_ADVERTISED_PORT=9092
      - KAFKA_CFG_LISTENERS=PLAINTEXT://:9092
#      - KAFKA_CFG_ADVERTISED_LISTENERS=PLAINTEXT://kafka:9092
      - KAFKA_CFG_ADVERTISED_LISTENERS=PLAINTEXT://kafka:9092
      - KAFKA_CFG_PLAINTEXT_HOST://localhost:9092
      - KAFKA_CFG_ZOOKEEPER_CONNECT=zookeeper:2181
      - KAFKA_CFG_AUTO_CREATE_TOPICS_ENABLE=false
      - KAFKA_CFG_LOG_RETENTION_BYTES=21474836480
      - KAFKA_CFG_LOG_RETENTION_HOURS=24
      - ALLOW_PLAINTEXT_LISTENER=yes
      - KAFKA_CFG_DELETE_TOPIC_ENABLE=true
    depends_on:
      - zookeeper

Thank you!

Observed Behavior

i have install threatbus threatbus-zeek threatbus-misp threatbus-inmem.
i can't subscribed to p2p_topic

1. i don't have zmq on my centos. if i setup
   zmq:
   host: localhost
   port: 50000
   can it work?

2. if i want to use kafka.
   kafka:
   topics:
   - misp_attribute
   poll_interval: 1.0
   config:
   bootstrap.servers: "x.x.x.x:9092"
   group.id: "threatbus"
   auto.offset.reset: "earliest"
   i get error : Kafka error: KafkaError{code=UNKNOWN_TOPIC_OR_PART,val=3,str="Subscribed topic not available: misp_attribute: Broker: Unknown topic or partition"}

Environment Context

zeek version 3.0.8
full config.yml:

logging:
console: true
console_verbosity: DEBUG
file: false
file_verbosity: DEBUG
filename: threatbus.log

plugins:
backbones:
inmem:

apps:
zeek:
host: "localhost"
port: 47761
module_namespace: Tenzir
misp:
api:
host: http://x.x.x.x
ssl: false
key: xxxxxxxxxxxxxxxxxxxx
zmq:
host: localhost
port: 50000
# kafka:
# topics:
# - misp_attribute
# poll_interval: 1.0
# config:
# bootstrap.servers: "x.x.x.x:9092"
# group.id: "threatbus"
# auto.offset.reset: "earliest"

Hello,

i would like to develop an own Plugin for threatbus. In the description are links to a guide on how to write a plugin (https://docs.tenzir.com/threatbus/plugins/plugin-development) and different plugin types (https://docs.tenzir.com/threatbus/plugins/overview). Unfortunately both links are redirected to the main page of vast.io, where i am not able to find the guides. Are the guides now located anywhere else and I am unable to find them? If not would it be possible to still get access to the guides?

Thanks in advance

Kind regards

Observed Behavior

Guten Tag!
Running tenzir/threatbus:latest docker container and mounting config.yaml as volume. I believe it's successfully talking to my MISP instance, but there's no indication the backbone is getting traffic (debug for console turned on but nothing showing up in STDOUT on docker-compose logs -f). As an aside, I'm not using the zeek portion of the config at all (no Zeek instance).

Ultimately, I'm looking at writing a threatbus plugin to take the MISP messages that are published to in_mem and submit those as indicators to another threat intel platform (CIF) using that python client.

Expected Behavior

I've got an alpha CIF plugin for threatbus written, but it doesn't appear to be getting any data, which makes me think the publisher for MISP perhaps isn't publishing like I would expect? I've been adding logger.debug statements everywhere to hopefully find where something isn't going right, but so far no luck.

Steps to reproduce

1. docker-compose up -d
2. docker-compose logs -f
3. See just startup info

Attaching to threatbus
threatbus    | 2020-05-22 20:23:04 INFO     [threatbus] Starting plugins...
threatbus    | 2020-05-22 20:23:04 INFO     [threatbus_zeek.plugin] Broker: endpoint listening - localhost:47761
threatbus    | 2020-05-22 20:23:04 INFO     [threatbus_zeek.plugin] Zeek plugin started
threatbus    | WARNING [api.py:82 - __init__() ] The version of PyMISP recommended by the MISP instance (2.4.126) is newer than the one you're using now (2.4.125). Please upgrade PyMISP.
threatbus    | 2020-05-22 20:23:04 INFO     [threatbus_misp.plugin] MISP plugin started
threatbus    | 2020-05-22 20:23:04 INFO     [threatbus_inmem.plugin] In-memory backbone started.

Environment Context

• Version: latest docker image
• Compiler:
• Operating System: host is Ubuntu 1804

Config.yaml

logging:
  console: true
  console_verbosity: DEBUG
  file: false
  file_verbosity: DEBUG
  filename: threatbus.log

plugins:
  apps:
    zeek:
      host: localhost
      port: 47761
      module_namespace: Tenzir
    misp:
      api:
        host: https://192.168.0.2:8006
        ssl: false
        key: <MISP_API_KEY>
      zmq:
        host: 192.168.0.2
        port: 50000

docker-compose.yml

version: '3.0'
services:
  threatbus:
    image: tenzir/threatbus:latest
    container_name: threatbus
    hostname: threatbus
    volumes:
      - ./config.yaml:/opt/tenzir/threatbus/config.yaml
    command: -c config.yaml

vielen Dank!

When using an external command to transform a sighting context, the stderr output from the proc.communicate() call is checked and an error message is logged if that output is non-empty (https://github.com/tenzir/threatbus/blob/master/apps/vast/vast_threatbus/vast_threatbus.py#L579). However, this triggers an error message in vast-threatbus each time the external command outputs any log information on stderr. This is not uncommon and is not always a clear indication of an actual error. For example, I am calling a command that simply prints the location of its configuration file to stderr on startup.

I propose to use the returncode attribute that the communicate() call sets implicitly to check the returned error code of the external process and to use that information to trigger the error detection. This avoids spamming vast-threatbus's log with non-error messages.

Hello sir,
I am very new to the threatbus and misp. So sincere apology for asking the naive question here.
I am trying to figure out how threatbus works with zeek and misp. Also I referred the youtube video put my tenzir team regarding that. I am using misp and zeek all on same host inside a ubuntu 20.04 vm. I am able to access the misp through https://localhost from inside And using https://localhost:8443 from outside vm. I am struggling through the following problem, I will be very grateful if someone can take a look about why I am getting this error:

Observed Behavior

(venv) root@ubuntu-pc:/home/ubuntu# threatbus -c config-new.yaml
2020-12-09 20:52:48 INFO [threatbus] Starting plugins...
2020-12-09 20:52:48 INFO [threatbus_inmem.plugin] In-memory backbone started.
2020-12-09 20:52:48 INFO [threatbus_inmem.plugin] Adding subscription to: threatbus/snapshotrequest
2020-12-09 20:52:48 INFO [threatbus_inmem.plugin] Adding subscription to: threatbus/snapshotenvelope
2020-12-09 20:52:48 INFO [threatbus_zeek.plugin] Zeek plugin started
2020-12-09 20:52:48 ERROR [threatbus_misp.plugin] Cannot subscribe to MISP at https://localhost, using SSL: True
2020-12-09 20:52:48 ERROR [threatbus_misp.plugin] Failed to start MISP plugin

• Version:
• Compiler:
• Operating System: Ubuntu 20.04
• ..