blue-yonder / bonfire Goto Github PK

Is there currently a way to use a SOCKS or other web proxy with bonfire? It would be incredibly helpful for those on VPNs, or other tunnels into private networks. Typically its the supporting HTTP library where the support may or may not live.

Dear All,

Is there a workaround for this issue ?
I am located in TZ Europe/Vienna.
When I query with bonfire the last 10 minutes than I get the 10 minutes one hour ago.
When I change the timezone of my graylog server to UTC, bonfire is working well but the web-interface of graylog doesn't show the last hour.

Maybe there is anywhere in program code a location where I can add or subtract 1 hour ?
I know this is only an individual solution. But better than nothing.

Kind regards
Hans

If I execute:
bonfire -k --search-from 2022-10-05\ 23:35:22.000 --search-to 2022-10-07\ 23:39:22.000 :docker
it gives me the correct result, but if I add -f it starts rolling with current logs, so it is not considering the time for the search, it should just show the logs for that time range and stay in standby

This RuntimeError occurs not for the first printed output, but the second after a 1s wait.

Obviously its generated at this line.

Given that a -f can't specify a limit, what should I be doing to use it correctly?

Do you plan to release a version supporting the api endpoint?

Hello,

When we active the follow mode and we specify some fields, it work for the first query, but then, the message field is removed from the selection for the next queries.

Found a fix, pull request incoming

Romain

Hello

Can you tell me how to use tls with bonfire?

When I try to connect to graylog with tls options, it tells me that message:

[SSL: CERTIFICATE_VERIFY_FAILED]

Any clue? Thank you!

Ragards.

There doesn't seem to be a way to get datetime or timestamp when in dump format (-d).

Hi, I installed bonfire only yesterday but I can't seem to get any output when using the -f nor --follow option. Wondering if I'm just doing it wrongly or perhaps I'm using a wrong syntax. Query works fine w/out the "follow" option but this flag is what I'm looking forward to use. For example:

• bonfire -@ "2021-07-11T23:30:00.831Z" -f "facility:* AND source:server1"
• bonfire -@ "2021-07-11T23:30:00.831Z" -f

No error whatsoever but no output either and I have to control+c to get back the prompt. Let me know if you have any tips, thanks..

hi,
I am using the RC2 and it doesn't read keyring and follow options asI have them here:

[node:default]
host=graylog.domain.com
tls=True
port=
endpoint=/api/
username=user
keyring = True
follow = True

Also, since my graylog server is running on 443, I had to leave it empty, otherwise it is malforming a URL like graylog.domain.com

We have launched the beta of the Graylog Marketplace: https://marketplace.graylog.org/

The Marketplace is the central directory of all Graylog add-ons and integrations, including GELF libraries.

You would help us a lot if you could submit your tool there. :)

I seem to get too many logs OR the client doesn't use chunks:

$ bonfire -h our-log.server.net -u login -k -t -f -@'1 second ago' application:airflow
Please select a stream to query:
0: Stream 'All' (id: 586b17d304e6a30001fb4559)
Enter stream number: [0]: 
Traceback (most recent call last):
  File "/Users/daniellamblin/VEnvs/pythonToolsEnv/bin/bonfire", line 11, in <module>
    sys.exit(run())
  File "/Users/daniellamblin/VEnvs/pythonToolsEnv/lib/python2.7/site-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/Users/daniellamblin/VEnvs/pythonToolsEnv/lib/python2.7/site-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/Users/daniellamblin/VEnvs/pythonToolsEnv/lib/python2.7/site-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/daniellamblin/VEnvs/pythonToolsEnv/lib/python2.7/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/Users/daniellamblin/VEnvs/pythonToolsEnv/lib/python2.7/site-packages/bonfire/cli.py", line 196, in run
    run_logprint(gl_api, q, formatter, follow, interval, latency, output)
  File "/Users/daniellamblin/VEnvs/pythonToolsEnv/lib/python2.7/site-packages/bonfire/output.py", line 23, in run_logprint
    result = run_logprint(api, query, formatter, follow=False, output=output)
  File "/Users/daniellamblin/VEnvs/pythonToolsEnv/lib/python2.7/site-packages/bonfire/output.py", line 35, in run_logprint
    result = api.search(query, fetch_all=True)
  File "/Users/daniellamblin/VEnvs/pythonToolsEnv/lib/python2.7/site-packages/bonfire/output.py", line 35, in run_logprint
    result = api.search(query, fetch_all=True)
  File "/Users/daniellamblin/VEnvs/pythonToolsEnv/lib/python2.7/site-packages/bonfire/graylog_api.py", line 125, in search
    raise RuntimeError("Query returns more than 10000 log entries. Use offsets to query in chunks.")
RuntimeError: Query returns more than 10000 log entries. Use offsets to query in chunks.

This would be related to the used of the -f flag; Without it the command would return 10 lines, and without -t and a more generous -@ it seems to work correctly.

Hi,

the TLS support is a great feature, but the 0.0.7 release of bonfire cannot be installed via pip
from the package index site.

Please upload the package

Currently, you need to define any query in the config file and you cannot leave it empty and specify it as an argument with something like bonfire -q container:slurm :cluster

Hi, why is six version pinned to 1.9.0 ?

This makes bonfire raise this error after another package has upgraded the six package:
pkg_resources.DistributionNotFound: The 'six==1.9.0' distribution was not found and is required by bonfire

Could it possibly be changed to require six>=1.9.0 ?

Hello

Does hit package have support for Graylog 2.1 and can it also have option for PATH as Graylog 2.1 allows you to have API on same port eg //graylog.com/api/

thanks

Command:
bonfire --host myhost --port 9000 --endpoint /api --stream "Linux Syslog" --username test --search-from "2021-07-07 00:00:00" --search-to "now" --output output_log.txt

Getting the following error:
Traceback (most recent call last):
File "/home/pm9448/.local/bin/bonfire", line 8, in
sys.exit(run())
File "/home/pm9448/.local/lib/python3.6/site-packages/click/core.py", line 722, in call
return self.main(*args, **kwargs)
File "/home/pm9448/.local/lib/python3.6/site-packages/click/core.py", line 697, in main
rv = self.invoke(ctx)
File "/home/pm9448/.local/lib/python3.6/site-packages/click/core.py", line 895, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/home/pm9448/.local/lib/python3.6/site-packages/click/core.py", line 535, in invoke
return callback(*args, **kwargs)
File "/home/pm9448/.local/lib/python3.6/site-packages/bonfire/cli.py", line 199, in run
run_logprint(gl_api, q, formatter, follow, interval, latency, output)
File "/home/pm9448/.local/lib/python3.6/site-packages/bonfire/output.py", line 45, in run_logprint
if isinstance(output, basestring):
NameError: name 'basestring' is not defined

Any suggestions?

Hello,

I have installed bonfire on windows with:

setup.py bdist_wininst

Then I installed with the generated installer.

When i launch bonfire, it told me that it couldn't import syslog.

I commented syslog in graylog_api and formats

Then I have the following error:

output.py", line 36, in run_logprint
formatted_msgs = map(reverse, result.messages)
NameError: name 'reverse' is not defined

Could you tell me how to install bonfire on windows?

Regards

Some UTF8 messages are badly detected and crash bonfire with:

Traceback (most recent call last):
  File "/usr/local/bin/bonfire", line 9, in <module>
    load_entry_point('bonfire==0.0.6.post0.dev6+g8261729.dirty', 'console_scripts', 'bonfire')()
  File "/home/pdepaepe/.local/lib/python2.7/site-packages/click/core.py", line 610, in __call__
    return self.main(*args, **kwargs)
  File "/home/pdepaepe/.local/lib/python2.7/site-packages/click/core.py", line 590, in main
    rv = self.invoke(ctx)
  File "/home/pdepaepe/.local/lib/python2.7/site-packages/click/core.py", line 782, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/pdepaepe/.local/lib/python2.7/site-packages/click/core.py", line 416, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/bonfire/cli.py", line 185, in run
    run_logprint(gl_api, q, formatter, follow, interval, latency, output)
  File "/usr/local/lib/python2.7/dist-packages/bonfire/output.py", line 23, in run_logprint
    result = run_logprint(api, query, formatter, follow=False, output=output)
  File "/usr/local/lib/python2.7/dist-packages/bonfire/output.py", line 36, in run_logprint
    formatted_msgs = map(formatter, result.messages)
  File "/usr/local/lib/python2.7/dist-packages/bonfire/formats.py", line 52, in format
    field_text="; ".join(field_text))
UnicodeEncodeError: 'ascii' codec can't encode character u'\xf3' in position 214: ordinal not in range(128)

Hi, thanks for building and maintaining this tool.

I run Bonfire on a Graylog instance where I don’t have full permissions. This causes it to prompt me for a stream to query. But regardless of which stream I select (including “All messages”), I get no results. However, if I remove this check from the code, it doesn’t pass the filter parameter to Graylog, and then I do get the expected results.

I could make a fix for this, but I don’t know enough about Graylog to decide on the proper course of action:

• Is it wrong for Bonfire to assume “absent [*] permissions, a stream must be selected”? Should it always default to “no stream filter” and only prompt if e.g. --stream prompt is specified?
• Should Bonfire provide an explicit “force disable stream filter” option, e.g. --stream all and a corresponding option in the prompt?
• Should I just use the “All messages” stream? Is it wrong/unusual that Bonfire gives no results from it? I do get the results when I query this stream in the Graylog UI. (But I don’t control this instance, and if it’s somehow misconfigured, it’s unlikely to change.)

Hello,

is it possible to search in all stream instead of selecting one?

Regards.

Add a man page to get help on how to use this tool.

note: I would be interested by contributing to this repository on this very specific issue

with arrow==0.15.4, the tests throw the following error:

_____________________________ test_datetime_parser _____________________________

    def test_datetime_parser():
        now = arrow.now()

        ts_tuples = [
            ("10 minutes ago", lambda x: x.replace(minutes=-10, microsecond=0, tzinfo='local')),
            ("1 day ago", lambda x: x.replace(days=-1, microsecond=0, tzinfo='local')),
            ("yesterday midnight", lambda x: x.replace(days=-1, hour=0, minute=0, second=0, microsecond=0, tzinfo='local')),
            ("1986-04-24 00:51:24+02:00", lambda x: arrow.get("1986-04-24 00:51:24+02:00")),
            ("2001-01-01 01:01:01", lambda x: arrow.get("2001-01-01 01:01:01").replace(tzinfo="local")),
            (now, lambda x: now)]

        for (s, ts) in ts_tuples:
>           assert datetime_parser(s) == ts(arrow.now())

tests/test_dateutils.py:25:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
tests/test_dateutils.py:17: in <lambda>
    ("10 minutes ago", lambda x: x.replace(minutes=-10, microsecond=0, tzinfo='local')),
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <Arrow [2019-12-22T18:24:15.958968+00:00]>
kwargs = {'microsecond': 0, 'minutes': -10, 'tzinfo': 'local'}
absolute_kwargs = {}, key = 'minutes', value = -10

    def replace(self, **kwargs):
        """ Returns a new :class:`Arrow <arrow.arrow.Arrow>` object with attributes updated
        according to inputs.

        Use property names to set their value absolutely::

            >>> import arrow
            >>> arw = arrow.utcnow()
            >>> arw
            <Arrow [2013-05-11T22:27:34.787885+00:00]>
            >>> arw.replace(year=2014, month=6)
            <Arrow [2014-06-11T22:27:34.787885+00:00]>

        You can also replace the timezone without conversion, using a
        :ref:`timezone expression <tz-expr>`::

            >>> arw.replace(tzinfo=tz.tzlocal())
            <Arrow [2013-05-11T22:27:34.787885-07:00]>

        """

        absolute_kwargs = {}

        for key, value in kwargs.items():

            if key in self._ATTRS:
                absolute_kwargs[key] = value
            elif key in ["week", "quarter"]:
                raise AttributeError("setting absolute {} is not supported".format(key))
            elif key != "tzinfo":
>               raise AttributeError('unknown attribute: "{}"'.format(key))
E               AttributeError: unknown attribute: "minutes"

Using the current master branch (d0af9ca), on Mac, when tailing and writing the output to a file, the output gets written in one line.

Among other things this makes it very hard to read.

Exemplary command:

bonire --host mygraylog.mydomain --endpoint /api --port 9000 -o test.log \
    -r STREAMID -u USER -n 1000 -@ "2 days ago" "tag:help"

Is this intended behavior?

Starting Python 3, map returns an iterator instead of a list.

Instead of using the map built-in function use:

formatted_msgs = [formatter(m) for m in result.messages]

in output.py, line 36

Here is the original error:

Traceback (most recent call last):
  File "/home/me/bin/bonfire", line 9, in <module>
    load_entry_point('bonfire==0.0.6.post0.dev12+g0c56782', 'console_scripts', 'bonfire')()
  File "/home/me/lib/python3.5/site-packages/click/core.py", line 610, in __call__
    return self.main(*args, **kwargs)
  File "/home/me/lib/python3.5/site-packages/click/core.py", line 590, in main
    rv = self.invoke(ctx)
  File "/home/me/lib/python3.5/site-packages/click/core.py", line 782, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/me/lib/python3.5/site-packages/click/core.py", line 416, in invoke
    return callback(*args, **kwargs)
  File "/home/me/lib/python3.5/site-packages/bonfire/cli.py", line 185, in run
    run_logprint(gl_api, q, formatter, follow, interval, latency, output)
  File "/home/me/lib/python3.5/site-packages/bonfire/output.py", line 23, in run_logprint
    result = run_logprint(api, query, formatter, follow=False, output=output)
  File "/home/me/lib/python3.5/site-packages/bonfire/output.py", line 38, in run_logprint
    formatted_msgs.reverse()
AttributeError: 'map' object has no attribute 'reverse'

The README file suggests that there's still a bug but the Closed issue #25 suggests otherwise.

My local machine time and Graylog Server both are in America/Toronto timezone and I'm getting some issues with incorrect time range.

It will be great if someone could clarify.

Also, this tool is amazing!

Getting this error:

requests.exceptions.HTTPError: 404 Client Error: Not Found for url: <hostname>:<port>/users/<username>

This is after the execution of this line:

        r = requests.get(self.base_url + url, params=params, headers=self.get_header,
                         auth=(self.username, self.password), proxies=self.proxies)

in graylog_api.py

@mahrz - Hey Malte, it seems like this project might not be maintained anymore. Any chances of becoming a contributor / taking over the maintenance? Or should we just fork, and do our own thing? Appreciate any feedback.

Hi,

It'd be great to be able to query 2 or more nodes with a single query without having to resort to a custom outside script to call bonfire.

Thanks 👍
fred