Code Monkey home page Code Monkey logo

flowgger's Introduction

Flowgger

CI Build License: BSD2

New major version: 0.3.0 (2022-03-14)

Breaking Changes


Flowgger is a fast, simple and lightweight data collector written in Rust.

It reads log entries over a given protocol, extracts them, decodes them using a given format, re-encodes them into a different format, and asynchronously pushes the result into a remote data store.

Flowgger is designed to be:

  • Paranoid: it carefully validates input data to prevent injection of malformed/incomplete records down the chain.
  • Safe: written in Rust, without any unsafe code.
  • Fast: even though messages are systematically parsed and validated, Flowgger is orders of magnitude faster than Logstash and Fluentd.
  • Standalone: it comes as a single executable file, and doesn't require a JVM.

Flowgger supports common input types: stdin, UDP, TCP, TLS and Redis, as well as multiple input formats: JSON (GELF), LTSV, Cap'n Proto and RFC5424. Normalized messages can be sent to Kafka, Graylog, to downstream Flowgger servers, or to other log collectors for further processing.

flowgger's People

Contributors

cahakgeorge avatar chills42 avatar crisidev avatar jedisct1 avatar jehuty0shift avatar kdnakt avatar kurojishi avatar markatwood avatar neuodev avatar pdepaepe avatar rejao avatar salpalvv avatar trojan295 avatar vche avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

flowgger's Issues

TCP/Tls output

Awesome code, thanks a lot.

May you please consider to offer also a TCP/TLS output?

Enhance kafka commit

Hello,

I think, it could be very useful to enhance "Kafka output" with a kind of timer option and associate it with the actual kafka_coalesce parameter.

kafka_coalesce=100
kafka_flush_interval=5s

If one of theses conditions is met, commit the batch to Kafka.

Thanks

milliseconds resolution in Gelf Decoder.

We would like to be able to send up to milliseconds resolution message in gelf_decoder.rs since it is supported by Graylog. Is it possible ? (I know the answer since the patch does not seem too big :-)

Wildcard support on "ltsv_schema"

Hello,

I think it can make sense to have such support:

Given

[input.ltsv_suffixes]
 f64  = "_double"
 i64  = "_int"
 bool = "_bool"

Such schema will helps

[input.ltsv_schema]
 *_int = "i64"
 *_double = "double"
 *_bool = "bool"

Metrics

Awesome code, thanks a lot.

May you please consider to expose some metrics?
Such as: uptime, number of active sessions, message received, sent, droped (charset or bad format).

is this project dead?

There has been no commits or releases in over 2 years.

is it that stable/mature or is it dead ?

Log file naming hardcode the time and extensions separators

In log file name the separator with the datetime string and the extension strings are hardcoded.

So, using another char than "-" or no extension doesn't work.

file_path = "var/output/logs/fwsyslog.log"
  file_rotation_timeformat = ".%Y-%m-%d-%H-%M"
  file_rotation_time = 60

  # want fwsyslog.2021-12-17-15-06.log
  # get fwsyslog-.2021-12-17-15-39.log
 file_path = "var/output/logs/fwsyslog"
  file_rotation_timeformat = ".%Y-%m-%d-%H-%M"
  file_rotation_time = 60

  # want fwsyslog.2021-12-17-15-06
  # get fwsyslog-.2021-12-17-15-39.log

Update new_file.set_file_name(&format!("{}-{}.{}"

  • The datetime separator must be removed and put in the config string.
  • The ".{}" suffix should only be appended if the filename contains a suffix

input from nginx error_log show unsupported version

I use nginx-1.12.0 version and flowgger-master on macOS 10.12.4

in the nginx configuration file nginx.conf, I instruct nginx to send error log to local syslog server

error_log   syslog:server=127.0.0.1 debug;

in flowgger.toml

[input]

### Syslog over UDP
type = "udp"
listen = "0.0.0.0:514"

[output]

### Debug output (stdout)
type = "stdout"
grep -rnw 'src' -e 'Unsupported version'
src/flowgger/decoder/rfc5424_decoder.rs:79:        return Err("Unsupported version");
nano +79 src/flowgger/decoder/rfc5424_decoder.rs

// I add a println! to this function to show line value
fn parse_pri_version(line: &str) -> Result<Pri, &'static str> {
    println!("line:{}", line); // <----show line value
    if !line.starts_with('<') {
        return Err("The priority should be inside brackets");
    }
    let mut parts = line[1..].splitn(2, '>');
    let pri_encoded: u8 =
        try!(try!(parts.next().ok_or("Empty priority")).parse().or(Err("Invalid priority")));
    let version = try!(parts.next().ok_or("Missing version"));
    if version != "1" {
        return Err("Unsupported version");
    }
    Ok(Pri {
           facility: pri_encoded >> 3,
           severity: pri_encoded & 7,
       })
}

after I recompile and restart flowgger, start nginx and make some error to generate a error log

target/release/flowgger
Flowgger 0.2.6
<184> Apr
Unsupported version

Do you have a plan to support input nginx log (both error_log and access_log)?

LTSV decoder doesn't respect output.ltsv_extra add or replace fields

Found this while writing tests

As for https://github.com/jedisct1/flowgger/wiki/LTSV-Encoder the encoders should support the ability to add or replace, without duplicating tags and fields.

LTSV is not only not overriding it, but get duplicated as the encoding is done by appending data to a String:

https://github.com/awslabs/flowgger/blob/master/src/flowgger/encoder/ltsv_encoder.rs#L41

left: "some_info:foo\tsome_info:bar\thost:example.org\ttime:1385053862.3072\tmessage:A short message\tfull_message:Backtrace here with more\tlevel:1\tappname:appname\tprocid:44\tmsgid:msg_id",
right: "some_info:bar\thost:example.org\ttime:1385053862.3072\tmessage:A short message\tfull_message:Backtrace here with more\tlevel:1\tappname:appname\tprocid:44\tmsgid:msg_id"', src/flowgger/encoder/ltsv_encoder.rs:175:9

Question: Buffers

If the buffer ever gets full, new messages will not be discarded: clients simply won't be able to push new messages until the queue drains.

How is this working with the UDP input?

Other question,
Would it be possible to add a file_buffer instead of memory buffer?

Even better would be a secondary buffer, ex:
main_buffer_type=memory
main_buffer_max_message=5000
secondary_buffer_type=file
secondary_buffer_path=/tmp/flowgger.buffer
secondary_buffer_max_size=5G

Then when the output comes back online Flowgger can dump the 5gig of messages :)

Lua Parsing

Would you be open to accepting a PR to support a Lua parsing?
This way some more custom log formats can be added without having to recompile, or add into source.

Multiple inputs

Especially with file input (#20), it may be desirable to merge the logs from multiple sources (each with their own decoder).

Enhancement request: NATS Output

It would be great if Flowlogger supported output to NATS (https://nats.io/).

Kafka is quite complex and has JVM dependency, NATS is fast becoming a popular alternative for people who don't need all the bells and whistles of Kafka.

Flowgger 0.2.0 does not support multiline in gelf anymore.

Example of working message on flowgger 0.1.X to gelf input with multiline:
echo -e '{"version":"1.1", "host": "example.org", "short_message": "A short GELF message that helps you identify what is going on", "full_message": "Backtrace here\n\nmore stuff", "timestamp": 1470749969, "level": 1, "_user_id": 9001, "_some_info": "foo", "some_metric_num": 42.0}\0' | openssl s_client -quiet -no_ign_eof -connect localhost:12202

[input]
type = "tls"
format = "gelf"
listen = "0.0.0.0:12202"
framing = "nul"
...

Now with flowgger 0.2.0, I have the following error:

Invalid GELF input, unable to parse as a JSON object: [{"version":"1.1", "host": "example.org", "short_message": "A short GELF message that helps you identify what is going on", "full_message": "Backtrace here

more stuff", "timestamp": 1470749909, "level": 1, "_user_id": 9001, "_some_info": "foo", "some_metric_num": 42.0}]

File input

I may have missed it, but I didn't find in the docs how to get input from a log file.

It would need to:

  • Not read the whole file from the start, but seek to the end and read from there (like tail -f)
  • Keep working if a logrotate service renames the log file and creates a new file with the previous name (flowgger should start reading the new file).

Timezone ?

Hello,
We are trying flowgger for forwarding logs from remote site to our central Graylog server.
Everything on our infrastructure is with timezone CEST.
Flowgger seems to force the timestamp of the GELF to UTC
Is there a way to configure and change this behavior ?
Thanks in advance
Flowgger seems to be a nice tool !!

Cross build workflow fails again

Build fails due to workflow and cross compilation being broken. I've enabled manual runs for the workflow and re ran it on the current release (which previously passed) https://github.com/awslabs/flowgger/actions/runs/9127101215 and it fails.

Since we're using a cross build and openssl is not included anymore, we were using an older version. we'll have to fix this.
Find how to build a cross image with ssh, or maybe have cross build validation without enabled ?

Build fail with redis feature

Hi. I'm getting errors while trying to build flowgger with --all-features (rust 1.62.0) as pointed out in https://gitlab.alpinelinux.org/alpine/aports/-/issues/13952#note_243393.

Changing the declaration to let mut ... (like the log suggests) makes the build pass but I honestly don't know whether things will break. This seems to be happen after redis-rs 0.11.0-beta1 release (ref: https://github.com/redis-rs/redis-rs/blob/main/CHANGELOG.md#0110-beta1---2019-05-30)

warning: trait objects without an explicit `dyn` are deprecated
  --> src/flowgger/input/tcp/tcpco_input.rs:52:18
   |
52 |     decoder: Box<Decoder>,
   |                  ^^^^^^^
   |
   = note: `#[warn(bare_trait_objects)]` on by default
   = warning: this is accepted in the current edition (Rust 2018) but is a hard error in Rust 2021!
   = note: for more information, see <https://doc.rust-lang.org/nightly/edition-guide/rust-2021/warnings-promoted-to-error.html>
help: use `dyn`
   |
52 -     decoder: Box<Decoder>,
52 +     decoder: Box<dyn Decoder>,
   | 

warning: trait objects without an explicit `dyn` are deprecated
  --> src/flowgger/input/tcp/tcpco_input.rs:53:18
   |
53 |     encoder: Box<Encoder>,
   |                  ^^^^^^^
   |
   = warning: this is accepted in the current edition (Rust 2018) but is a hard error in Rust 2021!
   = note: for more information, see <https://doc.rust-lang.org/nightly/edition-guide/rust-2021/warnings-promoted-to-error.html>
help: use `dyn`
   |
53 -     encoder: Box<Encoder>,
53 +     encoder: Box<dyn Encoder>,
   | 

warning: trait objects without an explicit `dyn` are deprecated
  --> src/flowgger/input/tcp/tcpco_input.rs:29:22
   |
29 |         decoder: Box<Decoder + Send>,
   |                      ^^^^^^^^^^^^^^
   |
   = warning: this is accepted in the current edition (Rust 2018) but is a hard error in Rust 2021!
   = note: for more information, see <https://doc.rust-lang.org/nightly/edition-guide/rust-2021/warnings-promoted-to-error.html>
help: use `dyn`
   |
29 -         decoder: Box<Decoder + Send>,
29 +         decoder: Box<dyn Decoder + Send>,
   | 

warning: trait objects without an explicit `dyn` are deprecated
  --> src/flowgger/input/tcp/tcpco_input.rs:30:22
   |
30 |         encoder: Box<Encoder + Send>,
   |                      ^^^^^^^^^^^^^^
   |
   = warning: this is accepted in the current edition (Rust 2018) but is a hard error in Rust 2021!
   = note: for more information, see <https://doc.rust-lang.org/nightly/edition-guide/rust-2021/warnings-promoted-to-error.html>
help: use `dyn`
   |
30 -         encoder: Box<Encoder + Send>,
30 +         encoder: Box<dyn Encoder + Send>,
   | 

warning: trait objects without an explicit `dyn` are deprecated
  --> src/flowgger/input/tls/tlsco_input.rs:52:18
   |
52 |     decoder: Box<Decoder>,
   |                  ^^^^^^^
   |
   = warning: this is accepted in the current edition (Rust 2018) but is a hard error in Rust 2021!
   = note: for more information, see <https://doc.rust-lang.org/nightly/edition-guide/rust-2021/warnings-promoted-to-error.html>
help: use `dyn`
   |
52 -     decoder: Box<Decoder>,
52 +     decoder: Box<dyn Decoder>,
   | 

warning: trait objects without an explicit `dyn` are deprecated
  --> src/flowgger/input/tls/tlsco_input.rs:53:18
   |
53 |     encoder: Box<Encoder>,
   |                  ^^^^^^^
   |
   = warning: this is accepted in the current edition (Rust 2018) but is a hard error in Rust 2021!
   = note: for more information, see <https://doc.rust-lang.org/nightly/edition-guide/rust-2021/warnings-promoted-to-error.html>
help: use `dyn`
   |
53 -     encoder: Box<Encoder>,
53 +     encoder: Box<dyn Encoder>,
   | 

warning: trait objects without an explicit `dyn` are deprecated
  --> src/flowgger/input/tls/tlsco_input.rs:29:22
   |
29 |         decoder: Box<Decoder + Send>,
   |                      ^^^^^^^^^^^^^^
   |
   = warning: this is accepted in the current edition (Rust 2018) but is a hard error in Rust 2021!
   = note: for more information, see <https://doc.rust-lang.org/nightly/edition-guide/rust-2021/warnings-promoted-to-error.html>
help: use `dyn`
   |
29 -         decoder: Box<Decoder + Send>,
29 +         decoder: Box<dyn Decoder + Send>,
   | 

warning: trait objects without an explicit `dyn` are deprecated
  --> src/flowgger/input/tls/tlsco_input.rs:30:22
   |
30 |         encoder: Box<Encoder + Send>,
   |                      ^^^^^^^^^^^^^^
   |
   = warning: this is accepted in the current edition (Rust 2018) but is a hard error in Rust 2021!
   = note: for more information, see <https://doc.rust-lang.org/nightly/edition-guide/rust-2021/warnings-promoted-to-error.html>
help: use `dyn`
   |
30 -         encoder: Box<Encoder + Send>,
30 +         encoder: Box<dyn Encoder + Send>,
   | 

warning: trait objects without an explicit `dyn` are deprecated
  --> src/flowgger/input/tcp/tcpco_input.rs:61:51
   |
61 |         "capnp" => Box::new(CapnpSplitter) as Box<Splitter<_>>,
   |                                                   ^^^^^^^^^^^
   |
   = warning: this is accepted in the current edition (Rust 2018) but is a hard error in Rust 2021!
   = note: for more information, see <https://doc.rust-lang.org/nightly/edition-guide/rust-2021/warnings-promoted-to-error.html>
help: use `dyn`
   |
61 -         "capnp" => Box::new(CapnpSplitter) as Box<Splitter<_>>,
61 +         "capnp" => Box::new(CapnpSplitter) as Box<dyn Splitter<_>>,
   | 

warning: trait objects without an explicit `dyn` are deprecated
  --> src/flowgger/input/tcp/tcpco_input.rs:62:49
   |
62 |         "line" => Box::new(LineSplitter) as Box<Splitter<_>>,
   |                                                 ^^^^^^^^^^^
   |
   = warning: this is accepted in the current edition (Rust 2018) but is a hard error in Rust 2021!
   = note: for more information, see <https://doc.rust-lang.org/nightly/edition-guide/rust-2021/warnings-promoted-to-error.html>
help: use `dyn`
   |
62 -         "line" => Box::new(LineSplitter) as Box<Splitter<_>>,
62 +         "line" => Box::new(LineSplitter) as Box<dyn Splitter<_>>,
   | 

warning: trait objects without an explicit `dyn` are deprecated
  --> src/flowgger/input/tcp/tcpco_input.rs:63:53
   |
63 |         "syslen" => Box::new(SyslenSplitter) as Box<Splitter<_>>,
   |                                                     ^^^^^^^^^^^
   |
   = warning: this is accepted in the current edition (Rust 2018) but is a hard error in Rust 2021!
   = note: for more information, see <https://doc.rust-lang.org/nightly/edition-guide/rust-2021/warnings-promoted-to-error.html>
help: use `dyn`
   |
63 -         "syslen" => Box::new(SyslenSplitter) as Box<Splitter<_>>,
63 +         "syslen" => Box::new(SyslenSplitter) as Box<dyn Splitter<_>>,
   | 

warning: trait objects without an explicit `dyn` are deprecated
  --> src/flowgger/input/tcp/tcpco_input.rs:64:47
   |
64 |         "nul" => Box::new(NulSplitter) as Box<Splitter<_>>,
   |                                               ^^^^^^^^^^^
   |
   = warning: this is accepted in the current edition (Rust 2018) but is a hard error in Rust 2021!
   = note: for more information, see <https://doc.rust-lang.org/nightly/edition-guide/rust-2021/warnings-promoted-to-error.html>
help: use `dyn`
   |
64 -         "nul" => Box::new(NulSplitter) as Box<Splitter<_>>,
64 +         "nul" => Box::new(NulSplitter) as Box<dyn Splitter<_>>,
   | 

warning: use of deprecated associated function `may::Config::set_io_workers`: use `set_workers` only
  --> src/flowgger/input/tls/tlsco_input.rs:33:23
   |
33 |         may::config().set_io_workers(tls_config.threads);
   |                       ^^^^^^^^^^^^^^
   |
   = note: `#[warn(deprecated)]` on by default

warning: trait objects without an explicit `dyn` are deprecated
  --> src/flowgger/input/tls/tlsco_input.rs:68:51
   |
68 |         "capnp" => Box::new(CapnpSplitter) as Box<Splitter<_>>,
   |                                                   ^^^^^^^^^^^
   |
   = warning: this is accepted in the current edition (Rust 2018) but is a hard error in Rust 2021!
   = note: for more information, see <https://doc.rust-lang.org/nightly/edition-guide/rust-2021/warnings-promoted-to-error.html>
help: use `dyn`
   |
68 -         "capnp" => Box::new(CapnpSplitter) as Box<Splitter<_>>,
68 +         "capnp" => Box::new(CapnpSplitter) as Box<dyn Splitter<_>>,
   | 

warning: trait objects without an explicit `dyn` are deprecated
  --> src/flowgger/input/tls/tlsco_input.rs:69:49
   |
69 |         "line" => Box::new(LineSplitter) as Box<Splitter<_>>,
   |                                                 ^^^^^^^^^^^
   |
   = warning: this is accepted in the current edition (Rust 2018) but is a hard error in Rust 2021!
   = note: for more information, see <https://doc.rust-lang.org/nightly/edition-guide/rust-2021/warnings-promoted-to-error.html>
help: use `dyn`
   |
69 -         "line" => Box::new(LineSplitter) as Box<Splitter<_>>,
69 +         "line" => Box::new(LineSplitter) as Box<dyn Splitter<_>>,
   | 

warning: trait objects without an explicit `dyn` are deprecated
  --> src/flowgger/input/tls/tlsco_input.rs:70:53
   |
70 |         "syslen" => Box::new(SyslenSplitter) as Box<Splitter<_>>,
   |                                                     ^^^^^^^^^^^
   |
   = warning: this is accepted in the current edition (Rust 2018) but is a hard error in Rust 2021!
   = note: for more information, see <https://doc.rust-lang.org/nightly/edition-guide/rust-2021/warnings-promoted-to-error.html>
help: use `dyn`
   |
70 -         "syslen" => Box::new(SyslenSplitter) as Box<Splitter<_>>,
70 +         "syslen" => Box::new(SyslenSplitter) as Box<dyn Splitter<_>>,
   | 

warning: trait objects without an explicit `dyn` are deprecated
  --> src/flowgger/input/tls/tlsco_input.rs:71:47
   |
71 |         "nul" => Box::new(NulSplitter) as Box<Splitter<_>>,
   |                                               ^^^^^^^^^^^
   |
   = warning: this is accepted in the current edition (Rust 2018) but is a hard error in Rust 2021!
   = note: for more information, see <https://doc.rust-lang.org/nightly/edition-guide/rust-2021/warnings-promoted-to-error.html>
help: use `dyn`
   |
71 -         "nul" => Box::new(NulSplitter) as Box<Splitter<_>>,
71 +         "nul" => Box::new(NulSplitter) as Box<dyn Splitter<_>>,
   | 

error[E0596]: cannot borrow `redis_cnx` as mutable, as it is not declared as mutable
   --> src/flowgger/input/redis_input.rs:105:46
    |
99  |         let redis_cnx = self.redis_cnx;
    |             --------- help: consider changing this to be mutable: `mut redis_cnx`
...
105 |             let dummy: RedisResult<String> = redis_cnx.rpoplpush(queue_key_tmp, queue_key);
    |                                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ cannot borrow as mutable

error[E0596]: cannot borrow `redis_cnx` as mutable, as it is not declared as mutable
   --> src/flowgger/input/redis_input.rs:110:38
    |
99  |         let redis_cnx = self.redis_cnx;
    |             --------- help: consider changing this to be mutable: `mut redis_cnx`
...
110 |             let line: String = match redis_cnx.brpoplpush(queue_key, queue_key_tmp, 0) {
    |                                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ cannot borrow as mutable

error[E0596]: cannot borrow `redis_cnx` as mutable, as it is not declared as mutable
   --> src/flowgger/input/redis_input.rs:117:40
    |
99  |         let redis_cnx = self.redis_cnx;
    |             --------- help: consider changing this to be mutable: `mut redis_cnx`
...
117 |             let res: RedisResult<u8> = redis_cnx.lrem(queue_key_tmp as &str, 1, line as String);
    |                                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ cannot borrow as mutable

For more information about this error, try `rustc --explain E0596`.
warning: `flowgger` (lib) generated 17 warnings
error: could not compile `flowgger` due to 3 previous errors; 17 warnings emitted

Update dependencies version

Dependencies are old and needs to be updated. There are some which are going to be tricky:

OpenSSL 0.9 -> 0.10: Blocked by Kafka using 0.9 and by API change in the library
CapNProto: Change of API in the library force implementation of a new trait
serde_json 0.8 -> 1.0: Change of API in the library

ported over from amazon-archives#3

GELF decoder doesn't respect output.gelf_extra add or replace fields

Found this while writing tests

As for https://github.com/jedisct1/flowgger/wiki/GELF-Encoder the encoders should support the ability to add or replace, without duplicating tags and fields.

As the insertion of extra fields is being done before parsing the structured data in the GELF encoder, the Record fields always override the gelf_extra section of the config:

https://github.com/awslabs/flowgger/blob/master/src/flowgger/encoder/gelf_encoder.rs#L64

PR incoming

Decoder RFC 5424 does not correctly parse structured data

The decoder RFC 5424 does not correctly parse structured data. Currently, the parser parses only one structured data, which completely breaks the specification. As defined by the RFC 5424 ABF a Syslog message can contain no, one or more than one structured data.

Here is an example of a not properly parsed message:

<165>1 2003-10-11T22:14:15.003Z mymachine.example.com evntslog - ID47 [exampleSDID@32473 iut="3" eventSource= "Application" eventID="1011"][examplePriority@32473 class="high"] hello world

In this example, the message contains two structured data (but we can have a message with more than two) but the flowgger decoder only parses the first one and consider the rest a message, which is not correct too as the structured data and the message must be separated with a space.

[MAC] undefined macro: AC_DEFINE, AC_MSG_FAILURE

MAC OS X 10.10.5
$ rustc -V
rustc 1.14.0 (e8a012324 2016-12-16)
$ cargo -V
cargo 0.15.0-nightly (298a012 2016-12-20)
in the /flowgger and run "$ sudo cargo build --release -v". Get the following errors.

....
HOST = Some("x86_64-apple-darwin")
TARGET = Some("x86_64-apple-darwin")
HOST = Some("x86_64-apple-darwin")
CFLAGS_x86_64-apple-darwin = None
CFLAGS_x86_64_apple_darwin = None
HOST_CFLAGS = None
CFLAGS = None
running: "./autogen.sh"
libtoolize: putting auxiliary files in AC_CONFIG_AUX_DIR, '.'.
libtoolize: copying file './ltmain.sh'
libtoolize: putting macros in AC_CONFIG_MACRO_DIRS, 'm4'.
libtoolize: copying file 'm4/libtool.m4'
libtoolize: copying file 'm4/ltoptions.m4'
libtoolize: copying file 'm4/ltsugar.m4'
libtoolize: copying file 'm4/ltversion.m4'
libtoolize: copying file 'm4/lt~obsolete.m4'

--- stderr
configure.ac:17: installing './compile'
configure.ac:17: installing './config.guess'
configure.ac:17: installing './config.sub'
configure.ac:15: installing './install-sh'
configure.ac:15: installing './missing'
Makefile.am: installing './INSTALL'
/usr/local/share/automake-1.15/am/ltlibrary.am: warning: 'libsnappy.la': linking libtool libraries using a non-POSIX
/usr/local/share/automake-1.15/am/ltlibrary.am: archiver requires 'AM_PROG_AR' in 'configure.ac'
Makefile.am:4: while processing Libtool library 'libsnappy.la'
Makefile.am: installing './depcomp'
parallel-tests: installing './test-driver'
configure.ac:43: error: possibly undefined macro: AC_DEFINE
If this token and others are legitimate, please use m4_pattern_allow.
See the Autoconf documentation.
configure.ac:45: error: possibly undefined macro: AC_MSG_FAILURE
thread 'main' panicked at 'failed to run successfully: exit code: 1', /Users//.cargo/registry/src/github.com-1ecc6299db9ec823/snappy-sys-0.1.0/build.rs:102
note: Run with RUST_BACKTRACE=1 for a backtrace.

File output type

Perhaps I missed it, but I do not see an option to output to a file. The stdout output could be redirected to a file, I suppose, but a file output type could more easily support a "store-and-forward" scenario. Are there any plans to implement such a feature?

tls_co & tcp_co does not work anymore on flowgger 0.2.0

I use this docker image :
https://hub.docker.com/r/jedisct1/flowgger/builds/budnck6eugg9crdccv3ufsb/

Input section example :

################## 
#  Input section  #
###################
[input]
type = "tls_co"
format = "ltsv"
listen = "0.0.0.0:12201"
framing = "line"   
tls_threads = 2
tls_cert = "/etc/ssl/private/flowgger.pem"
tls_key = "/etc/ssl/private/flowgger.pem"
tls_ca_file = "/etc/ssl/private/flowgger.pem"
tls_method = "any" 

Logs:

Flowgger v0.2.0
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: Error { repr: Custom(Custom { kind: Other, error: StringError("Scheduler missing") }) }', ../src/libcore/result.rs:788
stack backtrace:   
   1:     0x7fc2b4a3984b - <unknown>
   2:     0x7fc2b4a3968e - <unknown>
   3:     0x7fc2b4a374d0 - <unknown>
   4:     0x7fc2b4a3666e - <unknown>
   5:     0x7fc2b4a3859e - <unknown>
   6:     0x7fc2b4a3815a - <unknown>
   7:     0x7fc2b4a495b8 - <unknown>
   8:     0x7fc2b49752cc - <unknown>
   9:     0x7fc2b4941bd8 - <unknown>
  10:     0x7fc2b493e98f - <unknown>
  11:     0x7fc2b4935429 - <unknown>
  12:     0x7fc2b4a45df8 - <unknown>
  13:     0x7fc2b4a460db - <unknown>
  14:     0x7fc2b493f510 - <unknown>
  15:     0x7fc2b40da82f - __libc_start_main
  16:     0x7fc2b4934ca8 - <unknown>
  17:                0x0 - <unknown>

Same with "tcp_co" :

Flowgger v0.2.0
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: Error { repr: Custom(Custom { kind: Other, error: StringError("Scheduler missing") }) }', ../src/libcore/result.rs:788
stack backtrace:
   1:     0x7f626427384b - <unknown>
   2:     0x7f626427368e - <unknown>
   3:     0x7f62642714d0 - <unknown>
   4:     0x7f626427066e - <unknown>
   5:     0x7f626427259e - <unknown>
   6:     0x7f626427215a - <unknown>
   7:     0x7f62642835b8 - <unknown>
   8:     0x7f62641af2cc - <unknown>
   9:     0x7f626417dfe5 - <unknown>
  10:     0x7f626417898f - <unknown>
  11:     0x7f626416f429 - <unknown>
  12:     0x7f626427fdf8 - <unknown>
  13:     0x7f62642800db - <unknown>
  14:     0x7f6264179510 - <unknown>
  15:     0x7f626391482f - __libc_start_main
  16:     0x7f626416eca8 - <unknown>
  17:                0x0 - <unknown>

Publish to crates.io

Is there any plan to publish this package into crates.io to allow a simple cargo install and allow a better distribution model instead of cloning / building?

Add timestamps

How can I add timestamp into each forwarded message in flowgger?

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.