Pass multiple directories to find logs about logio HOT 12 CLOSED

The harvester's env LOGS_DIRECTORIES accepts an arbitrary number of space separated " " directories.

Example:

$ docker run -d \
    -v /var/lib/docker/containers:/var/lib/docker/containers \
  -e "LOGS_DIRECTORIES=/var/lib/docker/containers /var/log" \
  --link logio:logio \
    -e "LOG_FILE_PATTERN=*-json.log" \
  --name harvester \
  --user root \
  quay.io/blacklabelops/logio harvester

Note: Logio is a master slave system. If you need different file patterns for different directories then just start more harvesters with different settings.

from logio.

For some reason when I pass space separated list of folders it only uses first in the list.

And is it possible to pass name of stream along with the list of files associated with that name, or when folders are searched for files - can we pull container name or host name as well or does it put everything into UnifiedStream only?

from logio.

Looks like a bug, I will check the container.

This Image is only file based. It crawls for files, it does not take Docker Container Ids into account. If you need such an elaborate mechanism then please take a look at the native docker log mechanism.

Inside Docker you can set the Log setting for each Container separately, please see here: https://docs.docker.com/engine/admin/logging/overview/. The log technology must be available on the Docker Host, e.g. fluentd and you can specify logging for each container separately.

from logio.

Okay fixed the bug. Multiple directories are working!

from logio.

You can define one streamname per harvester. You could start one harvester with a different streamname for each directory or logfile pattern.

If you want to attach a specific container, you need to attach the specific logfile inside the container log directory. See the container id and you have the foldername of the logfile.

Also I could implement an environment variable so you can pass your own customized harvester config file. Not sure if that would help you.

The perfect solution would be a logio logging driver inside Docker, but such a thing is currently not available.

from logio.

Awesome, thank you.

Would you take it one step further and give it an optional FILES format to provide stream name as an optional part of file pointer, ie:

Old format:
LOGIO_HARVESTER_LOGFILES=/var/log/cron /var/log/messages /var/log/nginx/site1-access.log /var/log/nginx/site2-access.log

Proposed change:
Add streamname followed by colon, preceding the path to file:
LOGIO_HARVESTER_LOGFILES=system:/var/log/cron system:/var/log/messages web:/var/log/nginx/site1-access.log web:/var/log/nginx/site2-access.log

Alternative failover support (if no streanmane specifies on a per-file basis, it could be specified on a per-directory basis) :
LOGS_DIRECTORIES=DockerAll:/var/lib/docker/containers SystemAll:/var/log

and further it would failover to the value specified in
LOGIO_HARVESTER_STREAMNAME

This would keep backwards compatibility but would allow to specify stream name to individual files or to directories

from logio.

And just an FYI - here's another approach to pulling logs from all containers, but it provides a little more data about the container and the image itself, hence opens up the possibility to include that information as part of stream name as well.

Another alternative is to have a setting per-pattern and per-directory that indicates to grab a "level-up (..)" directory name as stream name (could be useful only to us, but seems like it would make this script mostly complete for all use-cases), ie:
LOG_FILE_PATTERN=streamname_from_parent_dir:*.log

or better yet, 2 other approaches:

1. allow to include patterns for each directory, keep them as two separate options for more flexible and backward compatibility, but give an ability to specify pattern on same line
2. allow to specify entire harvester record that would go inside ( logStreams: { ... } ) and merge that in just prior to generating harvester.conf file, ie allow this - something that logio is missing - patterns in paths:

LOGIO_HARVESTER_LOGSTREAMS:
  - system:
    - /var/log/cron
    - /var/log/messages
    - /var/log/dmesg
  - webaccess:
    - /var/log/nginx/*.log
    - /var/log/php-fpm/*.log
  - weberrors:
    - /var/log/nginx/*error*.log
    - /var/log/php-fpm/*error*.log

from logio.

Very nice! I like it, please give me some time to implement it.

I could offer a fast solution that I already used inside my other containers, enumerated env variables.

Example:

LOGIO_HARVESTER_LOGSTREAMS1=/var/log/cron /var/log/messages /var/log/dmesg
LOGIO_HARVESTER_STREAMNAME1=system

LOGIO_HARVESTER_LOGSTREAMS2=var/log/nginx/.log /var/log/php-fpm/.log
LOGIO_HARVESTER_STREAMNAME2= webaccess

Could you use this? I could implement this by simply adding a for-loop around my existing logic.

from logio.

Sounds cool.. most certainly a cool work around over the docker-compose's yaml syntax and would work nicely for me, but a long-term and proper solutions seems like would be to support both ways - for docker syntax to use incremental enums and support for docer-compose too.

thank you so much for your time.

from logio.

Would love to see a solution to this issue, did you every get any further on the implementation for enumerated env variables?

from logio.

Implemented half-way.

from logio.

Finished! Example usage:

$ docker run -d \
  -e "LOGIO_HARVESTER1STREAMNAME=teststream1" \
	-e "LOGIO_HARVESTER1LOGSTREAMS=/tests" \
	-e "LOGIO_HARVESTER1FILEPATTERN=*.xml" \
	-e "LOGIO_HARVESTER2STREAMNAME=teststream2" \
	-e "LOGIO_HARVESTER2LOGSTREAMS=/tests" \
	-e "LOGIO_HARVESTER2FILEPATTERN=*.log" \
  --link logio:logio \
  --name harvester \
  blacklabelops/logio harvester

Please give feedback!

from logio.