Cannot subscribe to MISP and Failed to start MISP plugin about threatbus HOT 71 CLOSED

Hi @nillay
Please open a new issue when you can confirm that the original issue - MISP <-> Threat Bus communication works fine on your end.
Thanks and regards!

from threatbus.

Hi @nillay,

Sorry, I'm at a loss understanding why these sightings won't show up in your MISP web view.
This does not seem to relate to Threat Bus, because even though the cURL response indicates success, your sightings view is not updated. Please report the issue in the MISP repo. Please use the curl command as example in the new issue.

from threatbus.

You're welcome! Please come over to our chat for future questions about running Threat Bus.

from threatbus.

Can you try setting ssl: false in the misp section of the config.yaml file, please?

from threatbus.

Thanks a lot @0ortmann sir. I can't thank you enough for helping me out, in which I was stuck for so long.

from threatbus.

In the default installation, MISP creates a self-signed certificate. The certificate check errors, because it is not a valid certificate. Using ssl: false disables that the Python MISP library (PyMISP) checks the certificate. That does not influence the MISP<->Threat Bus communication.

from threatbus.

You're very welcome :)

from threatbus.

Hi @nillay
I see you closed this issue again, did you resolve the problem?
It could be that the MISP web view simply did not update. Does it still show the wrong count when you reload the page?

from threatbus.

No sir no sighting shown in MISP. Reloaded multiple times. Zeek and threatbus talking perfectly. Generating Intel logs and so on. But MISP seems to into 1-Way communication mode.

from threatbus.

I will try to reproduce the behavior with the MISP version I see in your screenshot 2.4.135. I'll get back to you by end of the day (CET).

from threatbus.

Sure @0ortmann sir I will try to fresh install everything on new ubuntu 20.04.... thanks a lot for giving your precious time in addressing my issue. Really means a lot.

regards

The logs are quite clear about the error: Zeek cannot reach Threat Bus at the specified address. Could it be that a firewall rule is blocking the connection?
According to your previous messages, i.e., this one, you already had a working connection between Threat Bus and Zeek. If no firewall is blocking the connection, I would kindly ask you to restore your setup back to that point.

from threatbus.

Hi @nillay

Can you please try our misp ioc-sender for debugging? Configure it with your MISP API key and URL, similar to your Threat Bus configuration. Then run the script. If all goes well, it should create a new test event and attribute in MISP.

Does the tool report any errors?

from threatbus.

@nillay The misp-ioc-sender test tool does not generate a sighting, we only used that to verify your API connection is intact.

Please run Threat Bus as you did in your original question and generate a sighting with Zeek.

from threatbus.

Does the web-interface show the sighting that you added manually?

from threatbus.

Hi @nillay
Your Threat Bus config.yaml shows a wrong value for the zmq MISP endpoint - it should use the IP address from the MISP instance, not localhost.

misp:
 api:
  ...
 zmq:
  host: 10.0.3.9       # <- this
  port: 50000

You also need to enable the zmq plugin:

• Go to Administration -> Server Settings & Maintenance -> Diagnostics Tab
• Find the ZeroMQ plugin section and enable it
• Go to Administration -> Server Settings & Maintenance -> Plugin settings Tab
• Set the entry Plugin.ZeroMQ_attribute_notifications_enable to true

from threatbus.

Hi @nillay
I need some more information from you to debug this.

• Can you please paste the contents of your config.yaml file? You can redact the MISP API key.
• How did you setup and install MISP?
• What Python version are you using?

from threatbus.

Sir,
thanks a lot for your reply. Sir this is version I am using in python:

============
(venv) root@ubuntu-pc:/home/ubuntu# python
Python 3.8.5 (default, Jul 28 2020, 12:59:40)
[GCC 9.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.

============

config-new.yaml content:
logging:
console: true
console_verbosity: DEBUG
file: false
file_verbosity: DEBUG
filename: threatbus.log

plugins:
backbones:
inmem:
apps:
zeek:
host: "127.0.0.1"
port: 47760
module_namespace: Tenzir
misp:
api:
host: https://localhost
ssl: true
key: GBV9jL***********************IJyWXxoUri7X2tfgkcd3N
zmq:
host: localhost
port: 50000

Sir as suggested in here. I have installed misp using:

===================
Sir I installed misp using following command ): 
# Please check the installer options first to make the best choice for your install
wget -O /tmp/INSTALL.sh https://raw.githubusercontent.com/MISP/MISP/2.4/INSTALL/INSTALL.sh
bash /tmp/INSTALL.sh

# This will install MISP Core
wget -O /tmp/INSTALL.sh https://raw.githubusercontent.com/MISP/MISP/2.4/INSTALL/INSTALL.sh
bash /tmp/INSTALL.sh -c
=============

from threatbus.

Wow, sir magically it started. But my question is that even if my misp is running on https:// will it affect the working of the threatbus+misp integration ?

from threatbus.

Thank you @0ortmann sir for such an crucial information from bottom of my heart. A great learning for me today. Sir, You really made my day.

from threatbus.

@0ortmann sir, sorry to bother you again but I am little stuck in one more issue.

Sir I will be very thankful if you could guide me for the same, I know I am doing very silly mistake like before, my apologies for same:

So with threatbus plugin, I am successfully able to detect the intel into the zeek and threatbus... Although I am not sure that will these intel will be deleted when we remove it from MISP, or some other config will be required to do that !

But the primary issue I am facing is that "After sighting got recorded in zeek" as well as "Threatbus" console. Misp is "Not" reflecting the same. Its stuck to 0 count. I published the event with IDS checkbox "on". but sighting not been updating to MISP.
Sir, sorry again for such a silly question. Please pardon me for the same.

Attaching screenshot here:

Kind regards

from threatbus.

Hello @0ortmann sir,
Thanks for your reply. No sir issue didn't solved. I thought my question is silly. So closed it because of that. Although problem is not solved yet.

from threatbus.

From the logs in the screenshot you provided, we can see that the MISP plugin reports back the sighting to your MISP instance. When you reload the MISP event page in your browser (CRTL+Shift+R), does the it show the sighting?

from threatbus.

@0ortmann Thank you sir.

from threatbus.

Hi @nillay
I reconstructed the setup using with the following application versions:

• MISP 2.4.135
• Zeek v3.2.0
• Threat Bus 2020.11.26 and Threat Bus GitHub master

Unfortunately, I cannot reproduce your error. Sightings get reported back to MISP as they should. We just merged a PR #87 to add error logging in case the sighting reporting to MISP errors.

Can I kindly ask you to install the latest Threat Bus version from GitHub master and retry? In case you see error logs, please paste them in this GitHub issue.
Cheers!

from threatbus.

sure sir doing the same, will let you know asap. thanks a lot for checking the issue from your side. @0ortmann.

from threatbus.

@0ortmann Sir I pulled the threatbus, threatbus misp, threatbus inmem from pip3..... can you please refer me to the document where i can manully build all three tools from source code..... or can you suggest me the right way to install the most updated threatbus..... ? Also is there a way to check the current running version, if yes please let me know. I will be very grateful for you. thanks.

from threatbus.

@0ortmann Sir seems like I am already into:
Threat Bus: 2020.11.26
Zeek Version: 3.2.2
MISP Version: 2.4.135

from threatbus.

I don't want to derail the conversation, and have limited ability to aid in debugging, but it looks like there's an error on the Zeek side: can show the contents of threatbus2.zeek near line 255, e.g., +/- 10 lines?

from threatbus.

@mavam Sir, my apology, actually that second tab is not related to the that, I am doing some log related experiment in zeek script, so I duplicated the zeek script for that. Otherwise with original script its working fine. But I thank you for adding your observation to it. Please guide me through in handling the issue. These screenshots are only meant to show the versions i am running.....
Attaching the working version below:

from threatbus.

@0ortmann Sir, After re-updating everything I am facing following error, can you help me out in what I am missing: Seems like Zeek in not seeing threatbus for some reason: But MISP talking to threatbus.... And threatbus opened a listening port too....
Thanks a lot.

I have also enabled the verbose on broker.conf
and getting the debug messages as follows w.r.t. above mentioned error:

from threatbus.

Hi @nillay

You seem to run into multiple issues at once. When you first opened this issue, the Threat Bus <-> Zeek connection worked fine as can be seen in the first screenshot you provided here. Please restore your setup back to that point.

As per the question how you can install Threat Bus from source, without PyPI:

• clone the tenzir/threatbus repository via git clone [email protected]:tenzir/threatbus.git
• navigate to the repository root folder cd threatbus
• install the plugin directly from source: pip3 install . && pip3 install plugins/apps/threatbus_misp && pip3 install plugins/apps/threatbus_zeek && pip3 install plugins/backbones/threatbus_inmem

from threatbus.

@0ortmann Sir thank you for your revert. I am trying hard to restore the setup of last known good config but I seems like no luck in that. Just tried as per recommendation #and still getting the same error, sir, should I create a separate thread for this issue ? Also If you think I should try it in fresh installed system then I will try to do that too please let me know, Thanks :

Hi @nillay

You seem to run into multiple issues at once. When you first opened this issue, the Threat Bus <-> Zeek connection worked fine as can be seen in the first screenshot you provided here. Please restore your setup back to that point.

As per the question how you can install Threat Bus from source, without pip3:

* clone the [tenzir/threatbus](https://github.com/tenzir/threatbus/) repository via `git clone [email protected]:tenzir/threatbus.git`

* navigate to the repository root folder `cd threatbus`

* install the plugin directly from source: `pip3 install . && pip3 install plugins/apps/threatbus_misp && pip3 install plugins/apps/threatbus_zeek && pip3 install plugins/backbones/threatbus_inmem`

from threatbus.

Hi @nillay please paste the contents of your Threat Bus config.yaml in here.

from threatbus.

@0ortmann Sir, here it is:

from threatbus.

From the logs you posted:

line 80: Broker error (Broker::PEER_UNAVAILABLE): (invalid-node, *localhost:47761, "unable to connect to remote peer")

This means that Zeek cannot connect to Threat Bus via Broker on localhost:47761.

With your config.yaml you configure Threat Bus to bind a Broker endpoint to 127.0.0.1.

It could be that your /etc/hosts is configured in a way that localhost does not resolve to 127.0.01. You can explicitly override the Tenzir::broker_host option of the Zeek script. Please start Zeek by specifying this option and set it to 127.0.0.1:

zeek -C threatbus.zeek -- "Tenzir::broker_host=127.0.0.1"

from threatbus.

@0ortmann sir, Tried that too now instead of localhost zeek throwing error for 127.0.0.1, Actually it broke when I was trying to update the treatbus, zeek etc unable to find what must be happened while doing so, totally out of clue since then:

from threatbus.

The logs are quite clear about the error: Zeek cannot reach Threat Bus at the specified address. Could it be that a firewall rule is blocking the connection?
According to your previous messages, i.e., this one, you already had a working connection between Threat Bus and Zeek. If no firewall is blocking the connection, I would kindly ask you to restore your setup back to that point.

from threatbus.

@0ortmann sir, hope you are doing well. I have tried it on fresh system. And reinstalled all the components again freshly. So here are the following conclusion I have reached till this point:
1- when I am compiling the threat bus and other tools like inmem, misp, zeek from the cloning the github, its not installing properly for some reasons. threatbus was stucking at :
2020-12-16 17:57:31 INFO [threatbus] Starting plugins...

2- What is working for me in that in "venv" envoirnment running the following command for installing threatbus and related utilities using:

pip install threatbus
pip install threatbus-inmem
pip install threatbus-misp
pip install threatbus-zeek

3- Seems like since the day zeek has been updated there is some issue started occurring between zeek and threatbus communication, because zeek version i am running is: 3.2.3

4- Threatbus misp communication is working fine for me.
So now the persistent issue is as shown in screenshot: (I will replace the API key later so didnt hide it here, once all fixed, atleast I hope so)
//Also tried with http://localhost
error in /opt/zeek/share/zeek/base/frameworks/broker/./log.zeek, line 80: Broker error (Broker::PEER_UNAVAILABLE): (invalid-node, *localhost:47760, "unable to connect to remote peer")

sir, if you want i can open a separate issue here in github for this problem i am facing. Thank you very much.

Kind Regards
Nillay

from threatbus.

What is the status of this, did you make any progress with the communication from Threat Bus to MISP?
If not, I'd kindly ask you to update to all latest versions of the Threat Bus package family:

pip install --upgrade threatbus threatbus-inmem threatbus-misp threatbus-zeek

from threatbus.

@0ortmann sir, I tried with updated "threatbus threatbus-inmem threatbus-misp threatbus-zeek" as mentioned in #89 but MISP is not reflecting the intel seen by threatbus/zeek. that issue is persisting as it was at the time of opening this thread. MISP is updated to the latest version. Broker already fixed under your guidance to 1.4.0, Zeek is 3.2.2.

from threatbus.

Hi @nillay

Can you please try our misp ioc-sender for debugging? Configure it with your MISP API key and URL, similar to your Threat Bus configuration. Then run the script. If all goes well, it should create a new test event and attribute in MISP.

Does the tool report any errors?

@0ortmann sir, thanks for your kind reply. I tested the threatbus<--->misp communication with the ioc script you have provided. Luckily looks like its working fine. Attaching screenshot for the same.

Thanks a lot.

Kind Regards,
Nillay

from threatbus.

Hi @nillay
From the logs and screenshots you provide, it looks like Threat Bus correctly reports the Sighting back to MISP. The debug logs indicate that the Sighting is reported without errors. The debug tool also works as intended. Please see the attached screenshot about where sightings are tracked in MISP, just to make sure we're on the same page:

Sightings in MISP are tracked at the Attribute, so we'd expect the sightings-count to go up there. The screenshot shows an attribute with 3 sightings. Can you confirm that, after Threat Bus has logged Reported sighting ..., it does not change that counter in the MISP web-view, even when you reload the page?

from threatbus.

@0ortmann sir, thanks a lot for revert. Sir after reloading the page too MISP count is Not increasing for sure.

Actually I thought that this debug tool just create ioc inside MISP. That part is working fine. But Count is "0". Thanks.

from threatbus.

The debug tool does only produce an attribute and toggles the to_ids flag, it does not affect the Sighting counter. That was only a test to check the connection.

The attribute count is increased when MISP is notified about a sighting of that attribute, i.e., when Threat Bus reports the sighting. And according to your logs here Threat Bus reports a Sighting to MISP without errors.

from threatbus.

Hi @nillay

another idea: can you please go to your MISP settings (click Administration -> Server Settings & Maintenance) and then find the Plugin Settings tab. When you search for sighting in the searchbar, it will show the following options:

Make sure that your MISP user account has the required permissions to see sightings. I.e., it could be that your MISP is configured to only show sightings to the Event Owner. If your user doesn't own the event, it could be that you cannot see them.

from threatbus.

@0ortmann sir, although my setting was set to "Event Owner", I switched it to "Everyone" for testing.
Again ran the script which created a new event in MISP. But seems like sighting is still stuck to "0".
Attaching screenshot for the same. Thank you sir.

from threatbus.

@0ortmann sir, Just checked, seems like no changes in sighting, only modification graph which was flat has been change for once... Thanks a lot.

from threatbus.

@nillay The Modification map shows an upwards trend. That correlates with the Threat Bus logs, which indicate a successful reporting. This all seems to be a visibility issue in your MISP instance.

Dumb question: when you manually add a Sighting in your MISP web-interface, does it show up? Also, when you login as Administrator in MISP, can you then see the Sightings?

from threatbus.

@nillay The Modification map shows an upwards trend. That correlates with the Threat Bus logs, which indicate a successful reporting. This all seems to be a visibility issue in your MISP instance.

Dumb question: when you manually add a Sighting in your MISP web-interface, does it show up? Also, when you login as Administrator in MISP, can you then see the Sightings?

@0ortmann sir, sounds interesting. Yeap sir after being frustrated a lot I started added the sighting manually when sighting was occurred 😁 at least I was feeling little better while doing so 😆
Also sir since I am new to MISP too I am not taking any risk of permissions, so I am logging
in using admin account in MISP. Thanks a lot sir. 🙏

from threatbus.

Does the web-interface show the sighting that you added manually?

Sorry @0ortmann sir, just saw the message. somehow I missed this message. Yes sir, manual sighting is shown in web-interface. Please accept my apology for the late reply.

from threatbus.

Hi @nillay

I now extended the misp testutil so it also reports an artificial sighting. Could you please download the updated script and run it again? Please note that you need to update the config.yaml to include report_sighting: true (see the example config here).

Please paste the logs in this issue. It should create an attribute (if it does not exist already) and report a sighting.

from threatbus.

@0ortmann sir, thanks a lot for providing the updated script. I ran this one and found that with this script MISP is reflecting the sighting properly. Ran this script 2 times and got 2 sightings back.
Please take a look at the screenshot attached below: Thank you very much.

from threatbus.

Hi @nillay
Thanks for running the script. I'm at a loss understanding why the sightings from Threat Bus don't show up in the UI, while this test-sighting did.

Could you please run another test?

• Start Zeek to monitor your standard network interface
• Start Threat Bus as you did in your original question, with MISP and Zeek connected.
• Run the test script again -> that should result in Threat Bus forwarding the IoC to Zeek
• Now do a simple curl test-2020-12-23.vast (that is the test IoC from the script) -> that should trigger a sighting in Zeek, that forwards it to Threat Bus and ideally it would show up in MISP.

from threatbus.

@0ortmann sir, I tried as you instructed, so here is the outcome as follows:

• I started the threatbus
• Then the zeek for monitoring the test interface
• Then I ran the script you have provided
• Two things happened simultaneously
  • a. threatbus and zeek both registered the ioc
  • b. Misp created new ioc and _updated the count to 1
• Then I went to my test interface and generated a event using the curl command as "curl test-2020-12-23.vast"
• This time threatbus and zeek both responded when sighting occcured.
• But Misp is "Not" updated the sighting count. It is frozen to 1, which was increased from 0 to 1 while triggering the test misp-ioc script.

Attaching the screenshots for the same.
Thank you.

from threatbus.

Hi @nillay
thanks for your patience and all the tests you have done so far! I really appreciate that :)
I created a new debug branch for the MISP plugin. Please install that branch and run the above test scenario again. The updated version will log the MISP response from the API call to report the sighting.
You can install the branch like this:

pip uninstall threatbus-misp
pip install git+https://github.com/tenzir/threatbus.git@topic/misp-debug#subdirectory=plugins/apps/threatbus_misp

Once installed, please run the test again and post the new Threat Bus logs.

from threatbus.

@0ortmann sir, thanks a lot for your kind words. Sir, I am the newbie in these domain. So I am grateful that you are giving your precious time and effort in finding the root cause of the issue. I have no words to thank you. In this noble work, if I can become useful in anyway, I will consider myself lucky enough.

Sir, I have ran the procedures as you instructed, Please find the screenshots of the same.Although its was just for debugging as you said but just for the record Count is still not increased.

Kind Regards
Nillay

from threatbus.

Hi @nillay,

Happy new year and thanks for your patience! Were you able to make any progress yet?
I have pushed another debug version of the MISP plugin to the debug branch. It produces more logs. Can you please run that again, using the same instructions as above?

pip uninstall threatbus-misp
pip install git+https://github.com/tenzir/threatbus.git@topic/misp-debug#subdirectory=plugins/apps/threatbus_misp

Then generate a sighting with Zeek as we did before.

from threatbus.

Hello @0snap sir,
Wishing you a warm Happy new year 2021. Thank you for you kind reply. Sir, actually I am kind of clueless about the misp sighting issue. So I'm still stuck at the same spot. Also I had one question in mind, pardon me if its not related to this issue.

• What if I want to use MISP which is not on the same machine as threatbus ? Do I need to do some specific changes in order to do so or in the config.yaml just need to mention the ip of the MISP host in place of 'localhost' ?

• Now coming to the updates, sir I re-ran all the requirements as your instructed. Please find the screenshots below for the same. just for the records, MISP is still not updating the count.

from threatbus.

Thanks for running the test, @nillay
I'm trying to figure out what the problem is with MISP not updating the sighting. From the new DEBUG logs we can see that

• The MISP API is queried, the API TOKEN is correct
• MISP responds with a generic message that prints a description for the API endpoint. That message usually indicates that the API call is executed without parameters. So now we log the JSON object that is sent to MISP.

Going forward with the debug output from the new logs, can you please run the following cURL command against your MISP instance?

curl --insecure --header "Authorization: <API_TOKEN> " --header "Accept: application/json" --header "Content-Type: application/json" https://<MISP_IP>/sightings/add -d '{"id": 504, "type": 0, "timestamp": "1609755019"}' 

Please replace <API_TOKEN> and <MISP_IP> with the actual values of your installation.
The command instructs MISP to do the same as Threat Bus does: it should add a sighting to the Attribute with id 504.
When you run that command, what is the API response from MISP? Does it add a sighting?

To answer your other question:

• MISP and Threat Bus are designed to run on different host systems. Simply place the correct IP address in the config.yaml.

from threatbus.

@0snap sir, thank you for your kind reply. After running the API call seems like its not responding to MISP for some reasons.
no ioc has been added.

from threatbus.

Hi @nillay,
The cURL command was successful. We can see that, because the API returns the Sighting JSON structure. When you check the event with ID 89, it should have new sightings attached, one for each of the cURL commands you have run.
Do these sightings show up in the Web UI?

from threatbus.

Hi @nillay,

The cURL command was successful. We can see that, because the API returns the Sighting JSON structure. When you check the event with ID 89, it should have new sightings attached, one for each of the cURL commands you have run.

Do these sightings show up in the Web UI?

Thanks sir for reply. No sir, No sightings were attached in webui.

from threatbus.

When the web-ui does not show the sightings, even when added directly with cURL, this is an issue for the MISP Project.

How did you install MISP? I still cannot reproduce this issue (I use the pre-built virtual machines provided on the MISP website, i.e., this one).

from threatbus.

When the web-ui does not show the sightings, even when added directly with cURL, this is an issue for the MISP Project.

How did you install MISP? I still cannot reproduce this issue (I use the pre-built virtual machines provided on the MISP website, i.e., this one).

Sir I installed from misp script provided in their documentation. I think they created 2 shell scripts for the same. Seems like some internal config issue is occurring when we are not using their prebuilt MISP.

Sir let me use their Vm and hopefully then I don't need to manually look under the hood.

Will update you about at first either I am able to connect the misp MV with threatbus and secondly if it's responding well or not. Thanks a lot sir taking your precious time and efforts to look into my issue. It really means a lot to me. 🙏

from threatbus.

@0snap sir,
I just setup the cicle misp vm. And for some reasons on misp vm when I am adding an ioc. Its not reflecting on threatbus, seems like I am doing something wrong. I will be thankful if you could guide me to solve the issue. Thank you.
Attaching screenshots below:

from threatbus.

@0snap Sir,
Thank you very much. I was doing silly mistake. Your solution worked like charm. But the issue in Circle Misp is looking as same as my local MISP copy. Now sightings are reflected in threatbus and zeek when I add them in Misp. But agan Misp sight count is not increasing. I have checked the sighting settings and set it for everyone for now.

Also I gave curl command as in previous instruction you gave, throwing some kind of error:
Attaching the screenshot for the same:
Thank a lots sir again.

Kind Regards

from threatbus.

Hi @nillay

Thanks for your tests. In the last curl command, please replace the "id": 504 with "id": 4. Since you now use a new MISP installation, you need to update the attribute ID. We can see the ID in the Threat Bus logs.

from threatbus.

@0snap sir,
Thank you for your kind reply.
Please find the output below when I ran curl command, Thanks:

from threatbus.

The curl commands indicate success, they return a Sighting JSON structure. Did the sightings web-view update, i.e., are these sightings visible now?

from threatbus.

no sir, sightings are not visible, even new event or any attribute is not added.....

from threatbus.

Hi @nillay,

Sorry, I'm at a loss understanding why these sightings won't show up in your MISP web view.
This does not seem to relate to Threat Bus, because even though the cURL response indicates success, your sightings view is not updated. Please report the issue in the MISP repo. Please use the curl command as example in the new issue.

Hi @0snap sir,
Sure. But I would like to thank you from bottom of my heart for the humongous time and efforts you spend in helping with my issue. It means a lot and I learnt a lot in your company. It was a truly special communication for me. And will always be. So Thank you again for everything.... And my apology for any inconvenience caused to you because of me.

Kind Regards,
Nillay

from threatbus.