Problem Understanding Filters about fluent-plugin-systemd HOT 12 CLOSED

Hi,

Great question, we certainly need to improve the documentation around filtering somewhat...

The interface we have here is just a very thin wrapper over what you would do to filter entries with journalctl...

I made an attempt to explain this better below, by borrowing the examples from the journalctl docs, let me know if they help? And I will work on a properly written page of documentation...

AFACT systemd does not support wildcards in these filters ... so to get everything that contains any value in CONTAINER_NAME you would probably want to read all the messages from the journal, then filter things down futher e.g. with a grep filter in fluentd https://docs.fluentd.org/v1.0/articles/filter_grep

https://www.freedesktop.org/software/systemd/man/journalctl.html

Without arguments, all collected logs are shown unfiltered:

journalctl

This is the default if you don't specify any filters in the config

filters []

With one match specified, all entries with a field matching the expression are shown:

journalctl _SYSTEMD_UNIT=avahi-daemon.service

filters [{"_SYSTEMD_UNIT": "avahi-daemon.service"}]

If two different fields are matched, only entries matching both expressions at the same time are shown:

journalctl _SYSTEMD_UNIT=avahi-daemon.service _PID=28097

filters [{"_SYSTEMD_UNIT": "avahi-daemon.service", "_PID": 28097}]

If two matches refer to the same field, all entries matching either expression are shown:

journalctl _SYSTEMD_UNIT=avahi-daemon.service _SYSTEMD_UNIT=dbus.service

Fields with Arrays as values are treated as an OR statement, since a ruby hash can only have one value per key.

filters [{"_SYSTEMD_UNIT": ["avahi-daemon.service", "dbus.service"]}]

This could also be expressed as two separate filter hashes...

filters [{"_SYSTEMD_UNIT": "avahi-daemon.service"}, {"_SYSTEMD_UNIT": "dbus.service"}]

The form you choose only matters if you need to filter on multiple fields

If the separator "+" is used, two expressions may be combined in a logical OR. The following will show all messages from the Avahi service process with the PID 28097 plus all messages from the D-Bus service (from any of its processes):

journalctl _SYSTEMD_UNIT=avahi-daemon.service _PID=28097 + _SYSTEMD_UNIT=dbus.service

filters [{"_SYSTEMD_UNIT": "avahi-daemon.service", "_PID": 28097}, {"_SYSTEMD_UNIT": "dbus.service"}]

Show all logs generated by the D-Bus executable:

journalctl /usr/bin/dbus-daemon

filters [{"_exe": "/usr/bin/dbus-daemon"}]

from fluent-plugin-systemd.

Thanks so much for the excellent description!

from fluent-plugin-systemd.

Honestly I haven't thought about it much ... but it is a goal for 1.0 to have better documentation.

/docs on master seems reasonable to start with, I think the README is already a bit too long... so think we should start to split off some topic pages and index them all in the README...

Honestly though if you want to spend some time on this ... do whatever you feel works best...

from fluent-plugin-systemd.

If I was to put in a PR to expand the documentation around this particular topic, which branch should I base it off of? Also, would you like to have documentation branch out into separate files in a /docs directory, or just continue to expand off of README.md?

from fluent-plugin-systemd.

I'll submit separate PRs to both.

from fluent-plugin-systemd.

I'm writing up the documentation now, but I found an edge case that you didn't address. Is there any way to specify an OR condition between two expressions instead of the default AND?

IE:

# journalctl _PID=2345 _SYSTEMD_UNIT=docker.service
... <logical AND result here> ...
# journalctl _PID=2345 + _SYSTEMD_UNIT=docker.service
... <logical OR result here> ...

from fluent-plugin-systemd.

I think one of your examples might actually hit this question.

You gave an example of:

filters [{"_SYSTEMD_UNIT": "avahi-daemon.service", "_PID": 28097}, {"_SYSTEMD_UNIT": "dbus.service"}]

Would the two separate hashes define a logical OR condition @errm? IE: [{"THING1": "value"}, {"THING2": "value"}] would match any logs with THING1=value OR THING2=value?

from fluent-plugin-systemd.

Correct:

• Within a single Hash the field matches are logical AND
• Each hash in the array is a logical OR
• Array values are logical OR (for that value)

from fluent-plugin-systemd.

There, I've created 2 PRs, one against the master branch and the other cherry-picking commits into the 1.0.0 branch.

from fluent-plugin-systemd.

Sure I wouldn't worry about the v1.0.0 branch I am going to merge it into master just before we release v1 anyway ... people will see the docs on master when they look for them ...

from fluent-plugin-systemd.

Since my issue has been resolved and master documentation has been merged, I'm happy that this issue is fully dealt with. Thanks again for your help!

from fluent-plugin-systemd.

Thanks for your help :)

from fluent-plugin-systemd.