Malware Bazar Connector Error about connectors HOT 2 OPEN

Hi there - apologies for the delay in getting back on this. I wanted to update to 6.1.2 from 6.0.10 before looking into it. I have not observed this behavior on my instance before or since updating, so I am not able to make an educated guess based just upon the errors above. It appears there are two errors in the "Actual Output" - first is a failure to connect to abuse.ch, and second is the MISSING_REFERENCE_ERROR. The timestamp wasn't included in the connection failure error, so I don't have any way to determine how close these two occurred in time to each other.

It is entirely possible that there could have been a platform update/outage/bug occurring upstream at Malware Bazaar, and the error is a result of corrupted/incomplete data attempted to be consumed from Malware Bazaar during the course of this event.

Would have to see more examples of this happening to get an idea for what is going on. That said, from my experience the Malware Bazaar Recent Additions connector seems to fetch all of the recent additions of the past 60 minutes, regardless of the last connector state (https://github.com/OpenCTI-Platform/connectors/blob/master/external-import/malwarebazaar-recent-additions/src/malwarebazaar-recent-additions.py#L187), and then will try to determine if the entity was already consumed into OpenCTI by performing a query (https://github.com/OpenCTI-Platform/connectors/blob/master/external-import/malwarebazaar-recent-additions/src/malwarebazaar-recent-additions.py#L117) before ingest. In this particular case, the MISSING_REFERENCE_ERROR in the upload_artifact call suggests that the upload failed and the entity was not constructed (correct me if this assumption is a wrong interpretation of the error message).

So, hypothetically if this occurred once and the connector is configured to attempt a pull multiple times per hour, and then successive attempts to pull recent additions don't fail, then the entities in question can reasonably be presumed to have been consumed successfully.

If you set the log level to info instead of error for the connector, then if this error keeps occurring over and over, the connector will also report above it which item from Malware Bazaar was attempting to be ingested at that time (https://github.com/OpenCTI-Platform/connectors/blob/master/external-import/malwarebazaar-recent-additions/src/malwarebazaar-recent-additions.py#L92), and in that case the raw data from MBRA can be pulled down and we can look at what is missing from the data.

from connectors.

@ckane I see that you participate to the code "recently". Do you have any idea about how to fix this? For me it might be that some entities are not created before a relationship that link them but I might be wrong.

from connectors.