quarkslab / irma-brain Goto Github PK

Description:
The error is visible in the celery brain log file. If this occurs, for instance, the probes are not listed in the web front end. It occurs on 1 out of 2 uploads.

Occurrence:
This occurs sporadic.

When:
It happens when the main page of the interface is called and when the upload is finished.

Log Trace:

[2014-04-28 11:25:07,589: ERROR/MainProcess] Received unregistered task of type u'brain.tasks.probe_list'.
The message has been ignored and discarded.

Did you remember to import the module containing this task?
Or maybe you are using relative imports?
Please see http://bit.ly/gLye1c for more information.

The full contents of the message body was:
{u'utc': True, u'chord': None, u'args': [], u'retries': 0, u'expires': None, u'task': u'brain.tasks.probe_list', u'callbacks': None, u'errbacks': None, u'timelimit': [None, None], u'taskset': None, u'kwargs': {}, u'eta': None, u'id': u'bd2cbbd2-5a04-4343-bb87-c71012690670'} (259b)
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/celery/worker/consumer.py", line 455, in on_task_received
strategies[name](message, body,
KeyError: u'brain.tasks.probe_list'

I am a developer for the ArchAssault project and we are trying to add this to our repo, but it does not appear to be packaged correctly through setup.py.

I have followed the instructions here https://github.com/quarkslab/irma-brain/blob/master/docs/brain/install.rst

It creates a filestructure like so:
brain config docs extras irma_brain_app-1.1.0-py2.7.egg-info MANIFEST.in requirements.txt scripts setup.cfg setup.py

Shouldnt these be under /usr/lib/python_versio/site-packages or dist-packages if your one debian in a irma-brian folder. It seems that irma-{brian,probe,frontend} all install this way. Correct me if I am wrong, but how is the user supposed to use this application.

I've seen many instances of code (if not all) that uses raw string formatting such as:

# https://github.com/quarkslab/irma-brain/blob/1e5e52ca2b1cab3ea279776085f102f90c91e244/brain/tasks.py#L68
log.info("{0}: Found user".format(frontend_scanid) + 
         "quota remaining {0}/{1}".format(remaining, quota)) 

The right way™ is to pass parameters to the log.{debug,info,warning,error} methods and use printf placeholders, so that string formatting, which is quite an expensive computation, is only done when it's needed. Eg. if I disable logging or set it to warning level, there is no need to compute debug and info strings.

log.info("%s: Found user quota remaining %d/%d", frontend_scanid, remaining, quota)

Note this is not only a performance improvement, it also prevents the program from crashing because of an exception during the string formatting in a message that was not supposed to be shown:

logging.basicConfig(level=logging.WARNING)
n = "not a number"
logging.debug("this is not shown, does not crash %d", n)
logging.debug("this is not shown, but crashes %d" % n)

more convenient for firewall rules

Hi,

The IRMA architecture (particularly the brain) falls into an incoherent state when applying the following scenario:

• stop rabbitmq-server
• start rabbitmq-server 10 seconds later (simple restart does not seem to provoke the bug)

The result is that after that, the frontend ends up accepting scans, the celery.scan service seems to see the probes, but the celery.result service never starts any task (although its logs are saying that it is correctly connected to rabbitmq).

Final result is that every service is up and happy, the frontend accepts scans, but they never finish. Would you have a fix for that?

Do not hesitate if you need more info. Thanks!

Florent

If you have at least 2 workers for scan_app, simultaneous calls to probe_list fails.
This issue explains a lot of flip-flop effect in the probe list.

In [1]: from brain.tasks import *
In [2]: a,b = probe_list.delay(),probe_list.delay()
In [3]: a.get(), b.get()
Out[3]:
([0,
[u'AVGAntiVirusFree',
u'BitdefenderForUnices',
u'ClamAV',
u'EScan',
u'VirusBlokAda',
u'EsetNod32',
u'McAfeeVSCL',
u'McAfee-Daemon',
u'Zoner']],
[0, []])

Hi,

When using vsftpd rather than pureftpd in IRMA, there is an issue with the file deletion. For example, the brain tries to delete the file fontend/dir1/frontend/dir1/file1 instead of frontend/dir1/file1.

The root of the problem seems to be that the return format of NLST frontend for vsftpd is:

frontend/dir1
frontend/dir2
frontend/dir3

and not:

dir1
dir2
dir3

which seems to be the format returned by pureftpd (according to the irma code, could you confirm?). I would say that both implementations can be OK, since the FTP RFCs are pretty ambiguous.

This means that to be implementation independent, irma.ftp.handler.FtpTls.list should either cd to the path, list the cwd (NLST without any argument) and cd back to the old cwd, or only keep the basename of the paths retrieved from ftps.nlst(path).

I think that being able to use any FTP implementation is quite an interesting point in such a modular architecture, and vsftpd is known to be a really good one. What do you think?

Florent

Probe now registers on brain and send a regexp to filter mimetype.
This registration is stored in db and refresh on brain restart but handling of a probe shutting down is missing.

• Allow irma-brain to use SSL with RabbitMQ
• Make RabbitMQ running SSL mode

when large number of probes are connected, concurrent access leads to an error in result_app.
Seen in /var/log/supervisor/results_app.log:IrmaDatabaseResultNotFound:

[2015-01-13 13:53:41,312: ERROR/MainProcess] Task brain.tasks.job_success[85b16c15-60a8-4586-8860-185899c91
fbd] raised unexpected: IrmaDatabaseResultNotFound("The given id (63) doesn't exist in irma_job",)
IrmaDatabaseResultNotFound: The given id (63) doesn't exist in irma_job