Comments (10)
edsiper
commented on July 19, 2024
hi @pnovotnak,
thanks for reaching out. If I am not wrong, the in_tail plugin should work on BSD, if you want to test now (I can do it later this week), you can try to remove the plugin from the "Linux" conditional here:
https://github.com/fluent/fluent-bit/blob/master/plugins/CMakeLists.txt#L66
I mean, just move this part outside of the conditional:
REGISTER_IN_PLUGIN("in_tail")
from fluent-bit.
pnovotnak
commented on July 19, 2024
Nice! I'll test tonight / tomorrow and post results.
…On Jan 31, 2017 21:05, "Eduardo Silva" ***@***.***> wrote:
hi @pnovotnak <https://github.com/pnovotnak>,
thanks for reaching out. If I am not wrong, the in_tail plugin should work
on BSD, if you want to test now (I can do it later this week), you can try
to remove the plugin from the "Linux" conditional here:
https://github.com/fluent/fluent-bit/blob/master/
plugins/CMakeLists.txt#L66
I mean, just move this part outside of the conditional:
REGISTER_IN_PLUGIN("in_tail")
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#167 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AFctXWMWElLo0GrfXolaJCuSk-SflFWaks5rYBKGgaJpZM4LzS4g>
.
from fluent-bit.
pnovotnak
commented on July 19, 2024
Seems like wildcards don't work
[2.3.3-DEVELOPMENT][[email protected]]/root: ~/fluent-bit-0.10.1/build/bin/fluent-bit -i tail -p path=/var/log/*.log -p db=~/nginx.log.db -o stdout
/root/fluent-bit-0.10.1/build/bin/fluent-bit: No match.
When specifying a single file by full path, it reads the entire contents of the file in 1-minute intervals;
[2.3.3-DEVELOPMENT][[email protected]]/var/log: ~/fluent-bit-0.10.1/build/bin/fluent-bit -i tail -p path=/var/log/nginx.log -o stdout
Fluent-Bit v0.10.1
Copyright (C) Treasure Data
[2017/02/01 20:51:52] [ info] [engine] started
[0] tail.0: [1485982312, {"log"=>"Feb 1 20:46:30 pfSense pfsense.localdomain nginx: ::1 - - [01/Feb/2017:20:46:30 +0000] \"GET / HTTP/1.1\" 301 178 \"-\" \"curl/7.52.1\""}]
[1] tail.0: [1485982312, {"log"=>"Feb 1 20:46:39 pfSense pfsense.localdomain nginx: ::1 - - [01/Feb/2017:20:46:39 +0000] \"GET / HTTP/1.1\" 200 3998 \"-\" \"curl/7.52.1\""}]
[2] tail.0: [1485982312, {"log"=>"Feb 1 20:48:05 pfSense pfsense.localdomain nginx: ::1 - - [01/Feb/2017:20:48:05 +0000] \"GET / HTTP/1.1\" 200 3998 \"-\" \"curl/7.52.1\""}]
[3] tail.0: [1485982312, {"log"=>"Feb 1 20:48:11 pfSense pfsense.localdomain nginx: ::1 - - [01/Feb/2017:20:48:11 +0000] \"GET / HTTP/1.1\" 200 3998 \"-\" \"curl/7.52.1\""}]
[4] tail.0: [1485982312, {"log"=>"Feb 1 20:48:24 pfSense pfsense.localdomain nginx: ::1 - - [01/Feb/2017:20:48:24 +0000] \"GET / HTTP/1.1\" 200 3998 \"-\" \"curl/7.52.1\""}]
[5] tail.0: [1485982312, {"log"=>"Feb 1 20:48:25 pfSense pfsense.localdomain nginx: ::1 - - [01/Feb/2017:20:48:25 +0000] \"GET / HTTP/1.1\" 200 3998 \"-\" \"curl/7.52.1\""}]
[6] tail.0: [1485982312, {"log"=>"Feb 1 20:48:26 pfSense pfsense.localdomain nginx: ::1 - - [01/Feb/2017:20:48:26 +0000] \"GET / HTTP/1.1\" 200 3998 \"-\" \"curl/7.52.1\""}]
[7] tail.0: [1485982312, {"log"=>"Feb 1 20:48:26 pfSense pfsense.localdomain nginx: ::1 - - [01/Feb/2017:20:48:26 +0000] \"GET / HTTP/1.1\" 200 3998 \"-\" \"curl/7.52.1\""}]
[8] tail.0: [1485982312, {"log"=>"Feb 1 20:48:26 pfSense pfsense.localdomain nginx: ::1 - - [01/Feb/2017:20:48:26 +0000] \"GET / HTTP/1.1\" 200 3998 \"-\" \"curl/7.52.1\""}]
[9] tail.0: [1485982312, {"log"=>"Feb 1 20:48:31 pfSense pfsense.localdomain nginx: ::1 - - [01/Feb/2017:20:48:31 +0000] \"GET / HTTP/1.1\" 301 178 \"-\" \"curl/7.52.1\""}]
[10] tail.0: [1485982312, {"log"=>"Feb 1 20:48:32 pfSense pfsense.localdomain nginx: ::1 - - [01/Feb/2017:20:48:32 +0000] \"GET / HTTP/1.1\" 301 178 \"-\" \"curl/7.52.1\""}]
[0] tail.0: [1485982372, {"log"=>"Feb 1 20:46:30 pfSense pfsense.localdomain nginx: ::1 - - [01/Feb/2017:20:46:30 +0000] \"GET / HTTP/1.1\" 301 178 \"-\" \"curl/7.52.1\""}]
[1] tail.0: [1485982372, {"log"=>"Feb 1 20:46:39 pfSense pfsense.localdomain nginx: ::1 - - [01/Feb/2017:20:46:39 +0000] \"GET / HTTP/1.1\" 200 3998 \"-\" \"curl/7.52.1\""}]
[2] tail.0: [1485982372, {"log"=>"Feb 1 20:48:05 pfSense pfsense.localdomain nginx: ::1 - - [01/Feb/2017:20:48:05 +0000] \"GET / HTTP/1.1\" 200 3998 \"-\" \"curl/7.52.1\""}]
[3] tail.0: [1485982372, {"log"=>"Feb 1 20:48:11 pfSense pfsense.localdomain nginx: ::1 - - [01/Feb/2017:20:48:11 +0000] \"GET / HTTP/1.1\" 200 3998 \"-\" \"curl/7.52.1\""}]
[4] tail.0: [1485982372, {"log"=>"Feb 1 20:48:24 pfSense pfsense.localdomain nginx: ::1 - - [01/Feb/2017:20:48:24 +0000] \"GET / HTTP/1.1\" 200 3998 \"-\" \"curl/7.52.1\""}]
[5] tail.0: [1485982372, {"log"=>"Feb 1 20:48:25 pfSense pfsense.localdomain nginx: ::1 - - [01/Feb/2017:20:48:25 +0000] \"GET / HTTP/1.1\" 200 3998 \"-\" \"curl/7.52.1\""}]
[6] tail.0: [1485982372, {"log"=>"Feb 1 20:48:26 pfSense pfsense.localdomain nginx: ::1 - - [01/Feb/2017:20:48:26 +0000] \"GET / HTTP/1.1\" 200 3998 \"-\" \"curl/7.52.1\""}]
[7] tail.0: [1485982372, {"log"=>"Feb 1 20:48:26 pfSense pfsense.localdomain nginx: ::1 - - [01/Feb/2017:20:48:26 +0000] \"GET / HTTP/1.1\" 200 3998 \"-\" \"curl/7.52.1\""}]
[8] tail.0: [1485982372, {"log"=>"Feb 1 20:48:26 pfSense pfsense.localdomain nginx: ::1 - - [01/Feb/2017:20:48:26 +0000] \"GET / HTTP/1.1\" 200 3998 \"-\" \"curl/7.52.1\""}]
[9] tail.0: [1485982372, {"log"=>"Feb 1 20:48:31 pfSense pfsense.localdomain nginx: ::1 - - [01/Feb/2017:20:48:31 +0000] \"GET / HTTP/1.1\" 301 178 \"-\" \"curl/7.52.1\""}]
[10] tail.0: [1485982372, {"log"=>"Feb 1 20:48:32 pfSense pfsense.localdomain nginx: ::1 - - [01/Feb/2017:20:48:32 +0000] \"GET / HTTP/1.1\" 301 178 \"-\" \"curl/7.52.1\""}]
[0] tail.0: [1485982432, {"log"=>"Feb 1 20:46:30 pfSense pfsense.localdomain nginx: ::1 - - [01/Feb/2017:20:46:30 +0000] \"GET / HTTP/1.1\" 301 178 \"-\" \"curl/7.52.1\""}]
[1] tail.0: [1485982432, {"log"=>"Feb 1 20:46:39 pfSense pfsense.localdomain nginx: ::1 - - [01/Feb/2017:20:46:39 +0000] \"GET / HTTP/1.1\" 200 3998 \"-\" \"curl/7.52.1\""}]
[2] tail.0: [1485982432, {"log"=>"Feb 1 20:48:05 pfSense pfsense.localdomain nginx: ::1 - - [01/Feb/2017:20:48:05 +0000] \"GET / HTTP/1.1\" 200 3998 \"-\" \"curl/7.52.1\""}]
[3] tail.0: [1485982432, {"log"=>"Feb 1 20:48:11 pfSense pfsense.localdomain nginx: ::1 - - [01/Feb/2017:20:48:11 +0000] \"GET / HTTP/1.1\" 200 3998 \"-\" \"curl/7.52.1\""}]
[4] tail.0: [1485982432, {"log"=>"Feb 1 20:48:24 pfSense pfsense.localdomain nginx: ::1 - - [01/Feb/2017:20:48:24 +0000] \"GET / HTTP/1.1\" 200 3998 \"-\" \"curl/7.52.1\""}]
[5] tail.0: [1485982432, {"log"=>"Feb 1 20:48:25 pfSense pfsense.localdomain nginx: ::1 - - [01/Feb/2017:20:48:25 +0000] \"GET / HTTP/1.1\" 200 3998 \"-\" \"curl/7.52.1\""}]
[6] tail.0: [1485982432, {"log"=>"Feb 1 20:48:26 pfSense pfsense.localdomain nginx: ::1 - - [01/Feb/2017:20:48:26 +0000] \"GET / HTTP/1.1\" 200 3998 \"-\" \"curl/7.52.1\""}]
[7] tail.0: [1485982432, {"log"=>"Feb 1 20:48:26 pfSense pfsense.localdomain nginx: ::1 - - [01/Feb/2017:20:48:26 +0000] \"GET / HTTP/1.1\" 200 3998 \"-\" \"curl/7.52.1\""}]
[8] tail.0: [1485982432, {"log"=>"Feb 1 20:48:26 pfSense pfsense.localdomain nginx: ::1 - - [01/Feb/2017:20:48:26 +0000] \"GET / HTTP/1.1\" 200 3998 \"-\" \"curl/7.52.1\""}]
[9] tail.0: [1485982432, {"log"=>"Feb 1 20:48:31 pfSense pfsense.localdomain nginx: ::1 - - [01/Feb/2017:20:48:31 +0000] \"GET / HTTP/1.1\" 301 178 \"-\" \"curl/7.52.1\""}]
[10] tail.0: [1485982432, {"log"=>"Feb 1 20:48:32 pfSense pfsense.localdomain nginx: ::1 - - [01/Feb/2017:20:48:32 +0000] \"GET / HTTP/1.1\" 301 178 \"-\" \"curl/7.52.1\""}]
However, when using an sqlite3 database it offsets properly after each read 👍 though it still seems not to actively watch the file, polling it at 1m intervals instead.
[2.3.3-DEVELOPMENT][[email protected]]/root: ~/fluent-bit-0.10.1/build/bin/fluent-bit -i tail -p path=/var/log/nginx.log -p db=/root/nginx.log.db -o stdout
Fluent-Bit v0.10.1
Copyright (C) Treasure Data
[2017/02/01 21:05:12] [ info] [engine] started
[0] tail.0: [1485983112, {"log"=>"Feb 1 20:46:30 pfSense pfsense.localdomain nginx: ::1 - - [01/Feb/2017:20:46:30 +0000] \"GET / HTTP/1.1\" 301 178 \"-\" \"curl/7.52.1\""}]
[1] tail.0: [1485983112, {"log"=>"Feb 1 20:46:39 pfSense pfsense.localdomain nginx: ::1 - - [01/Feb/2017:20:46:39 +0000] \"GET / HTTP/1.1\" 200 3998 \"-\" \"curl/7.52.1\""}]
[2] tail.0: [1485983112, {"log"=>"Feb 1 20:48:05 pfSense pfsense.localdomain nginx: ::1 - - [01/Feb/2017:20:48:05 +0000] \"GET / HTTP/1.1\" 200 3998 \"-\" \"curl/7.52.1\""}]
[3] tail.0: [1485983112, {"log"=>"Feb 1 20:48:11 pfSense pfsense.localdomain nginx: ::1 - - [01/Feb/2017:20:48:11 +0000] \"GET / HTTP/1.1\" 200 3998 \"-\" \"curl/7.52.1\""}]
[4] tail.0: [1485983112, {"log"=>"Feb 1 20:48:24 pfSense pfsense.localdomain nginx: ::1 - - [01/Feb/2017:20:48:24 +0000] \"GET / HTTP/1.1\" 200 3998 \"-\" \"curl/7.52.1\""}]
[5] tail.0: [1485983112, {"log"=>"Feb 1 20:48:25 pfSense pfsense.localdomain nginx: ::1 - - [01/Feb/2017:20:48:25 +0000] \"GET / HTTP/1.1\" 200 3998 \"-\" \"curl/7.52.1\""}]
[6] tail.0: [1485983112, {"log"=>"Feb 1 20:48:26 pfSense pfsense.localdomain nginx: ::1 - - [01/Feb/2017:20:48:26 +0000] \"GET / HTTP/1.1\" 200 3998 \"-\" \"curl/7.52.1\""}]
[7] tail.0: [1485983112, {"log"=>"Feb 1 20:48:26 pfSense pfsense.localdomain nginx: ::1 - - [01/Feb/2017:20:48:26 +0000] \"GET / HTTP/1.1\" 200 3998 \"-\" \"curl/7.52.1\""}]
[8] tail.0: [1485983112, {"log"=>"Feb 1 20:48:26 pfSense pfsense.localdomain nginx: ::1 - - [01/Feb/2017:20:48:26 +0000] \"GET / HTTP/1.1\" 200 3998 \"-\" \"curl/7.52.1\""}]
[9] tail.0: [1485983112, {"log"=>"Feb 1 20:48:31 pfSense pfsense.localdomain nginx: ::1 - - [01/Feb/2017:20:48:31 +0000] \"GET / HTTP/1.1\" 301 178 \"-\" \"curl/7.52.1\""}]
[10] tail.0: [1485983112, {"log"=>"Feb 1 20:48:32 pfSense pfsense.localdomain nginx: ::1 - - [01/Feb/2017:20:48:32 +0000] \"GET / HTTP/1.1\" 301 178 \"-\" \"curl/7.52.1\""}]
[0] tail.0: [1485983292, {"log"=>"Feb 1 21:07:58 pfSense pfsense.localdomain nginx: ::1 - - [01/Feb/2017:21:07:58 +0000] \"GET / HTTP/1.1\" 301 178 \"-\" \"curl/7.52.1\""}]
[1] tail.0: [1485983292, {"log"=>"Feb 1 21:08:06 pfSense pfsense.localdomain nginx: ::1 - - [01/Feb/2017:21:08:06 +0000] \"GET / HTTP/1.1\" 200 3998 \"-\" \"curl/7.52.1\""}]
from fluent-bit.
edsiper
commented on July 19, 2024
For wildcard, modify this:
-p path=/var/log/*.log
to
-p 'path=/var/log/*.log'
on that way we can avoid wildcard expansion in the shell.
In order to check the build flags and internals, would you please run fluent-bit -h and paste the output of the last section called Internal ?
from fluent-bit.
pnovotnak
commented on July 19, 2024
Oops! Here they are;
[2.3.3-DEVELOPMENT][[email protected]]/root: ~/fluent-bit-0.10.1/build/bin/fluent-bit -i tail -p 'path=/var/log/*.log' -o stdout
Fluent-Bit v0.10.1
Copyright (C) Treasure Data
[2017/02/01 21:14:27] [ info] [engine] started
[2017/02/01 21:14:29] [error] [:328 errno=2] No such file or directory
[2017/02/01 21:14:29] [error] [:328 errno=2] No such file or directory
[0] tail.0: [1485983667, {"log"=>"Feb 1 20:30:10 pfSense dhclient: PREINIT"}]
[1] tail.0: [1485983667, {"log"=>"Feb 1 20:30:10 pfSense dhclient[5178]: DHCPDISCOVER on em0 to 255.255.255.255 port 67 interval 1"}]
...
[2.3.3-DEVELOPMENT][[email protected]]/root: fluent-bit-0.10.1/build/bin/fluent-bit -h
Usage: fluent-bit [OPTION]
Available Options
-c --config=FILE specify an optional configuration file
-d, --daemon run Fluent Bit in background mode
-f, --flush=SECONDS flush timeout in seconds (default: 5)
-i, --input=INPUT set an input
-m, --match=MATCH set plugin match, same as '-p match=abc'
-o, --output=OUTPUT set an output
-p, --prop="A=B" set plugin configuration property
-e, --plugin=FILE load an external plugin (shared lib)
-l, --log_file=FILE write log info to a file
-t, --tag=TAG set plugin tag, same as '-p tag=abc'
-v, --verbose enable verbose mode
-q, --quiet quiet mode
-V, --version show version number
-h, --help print this help
Inputs
tail Tail files
head Head Input
health Check TCP server health
serial Serial input
stdin Standard Input
tcp TCP
mqtt MQTT, listen for Publish messages
forward Fluentd in-forward
random Random
Outputs
counter Records counter
es Elasticsearch
forward Forward (Fluentd protocol)
http HTTP Output
influxdb InfluxDB Time Series
null Throws away events
plot Generate data file for GNU Plot
stdout Prints events to STDOUT
td Treasure Data
flowcounter FlowCounter
Internal
Event Loop = select
Build Flags = JSMN_PARENT_LINKS JSMN_STRICT FLB_HAVE_TLS FLB_HAVE_SQLDB FLB_HAVE_FLUSH_LIBCO FLB_HAVE_FORK FLB_HAVE_PROXY_GO FLB_HAVE_C_TLS FLB_HAVE_SETJMP FLB_HAVE_ACCEPT4
from fluent-bit.
edsiper
commented on July 19, 2024
thanks.
So if I understand correctly, in_tail after a long period is not longer watching the files ?
notes: BSD system using select(2) backend to watch logs.
from fluent-bit.
pnovotnak
commented on July 19, 2024
Ah, yes, someday I will learn how to read. I suppose supporting every kernel starts to become a rabbithole. Looks like the in_tail plugin works! Thanks @edsiper!
from fluent-bit.
edsiper
commented on July 19, 2024
awesome!. Before to close the issue, would you please describe your use case, is it for production ?
from fluent-bit.
pnovotnak
commented on July 19, 2024
Sure.
We want to collect logs from our pfSense box and aggregate them in
Stackdriver so that we can have offsite log backups, do alerting, etc.
Target is production, but "production" here means 5 users on a wlan + 2
headless workstations, so it's a pretty small deployment.
from fluent-bit.
edsiper
commented on July 19, 2024
Closing this ticket as "help wanted", we need more contributors to implement specific OS stuff.
from fluent-bit.
Related Issues (20)
- Core: Fluent Bit's http server does not respond when enabling keepalive and failed to upgrade to HTTP/2 HOT 1
- Collecting Traces from Container stdout and Sending to Jaeger HOT 1
- Windows: generate_chunk_name function generate name probably the same
- Error with INPUT tail on Windows Server 2019 to monitor logs on network share drive HOT 1
- the latest release 3.0.7 for red head enterprise 8.* crashes when Daemon is On HOT 1
- IPv6 does not seem to work with Windows
- Gzip Decompression Failure Due to 100MB Limit in Fluent Bit 3.0.7 HOT 3
- Bigquery Output Plugin HTTP buffer size is not configurable.
- Fluent Bit 3.1.0 build breaks (luajit) HOT 8
- Fluent Bit 3.1.0 record_modifier filter problem HOT 9
- Build fails if git shows signatures HOT 3
- Server/Hostname should print on payload using filter eventhough in payload value is not present HOT 2
- It seems that record_modifier is not included in image 3.1.0 HOT 4
- record_modifier filter plugin is missing in 3.1.0 but present in 3.0.x HOT 4
- opentelemetry output failed to add TraceId, SpanID field of the opentelemtry logs data model in 3.1.0 HOT 4
- record_modifier plugin doesn't exist HOT 1
- INPUT elasticsearch : Authenticate with elasticsearch cluster
- how to set file roll over base on file size or day when use file out put plugin HOT 1
- the Upstream configuration file doesn't support environmental variables
- How to get Fluint-bit logs are get in the Grafana without install Prometheus?
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from fluent-bit.