ismaelmartinez / logger-to-kibana Goto Github PK

Is your feature request related to a problem? Please describe.
This project generates the visualisations from the logs, but not the dashboard. When you have many visualisations, gets annoying to find them all.

Describe the solution you'd like
It will be great for it to generate a dashboard with the visualisations generated. Probably the best option is to group them, at least for now, for each file. A valid initial version can be just to generate a dashboard with all the visualisations in it.

Describe alternatives you've considered
Creating the dashboards manually is a bit of a pain.

Refactor code to allow configuring the regex values, urls, index UUID and any others.

Using configparser should do the job but we should document in this task how does it work and how to overwrite those values.

Is your feature request related to a problem? Please describe.
When having many files and logs, sending the visualisations takes too long.

Describe the solution you'd like
I will like for the process take almost no time

Additional context
Each request call, waits for the previous one to finish before sending another request to the endpoint. Adding multiprocessing/threading, should considerably speed up the process.

add azure pipelines

Is your feature request related to a problem? Please describe.
It will be useful to group logs and generate multiple visualisations

Describe the solution you'd like
Been able to group logs by:

• filename
• function

Describe alternatives you've considered
Not having that generates visualisation with too many filters

Additional context
When having logs generated from code, it generates far too many filters making it not great to visualise the logs. It will be great to group the logs.

Is your feature request related to a problem? Please describe.
There is no example for other type of log filters

Describe the solution you'd like
I like to have an example for Node/JavaScript projects (console.log, etc messages)

Additional context
This project should be language agnostic. To confirm that, an example in another language is needed.

Is your feature request related to a problem? Please describe.
Having a filter for the function name seem like a good idea initially, but it is now adding some, what looks like, unnecessary complexity/limitations.

Describe the solution you'd like
I will like all the logs in all the files in the sub folder to process and put in the same visualisation

Describe alternatives you've considered
Having logic to detect the functions and grouping the logs by function works ok if you are only using functions. It is then more difficult to process multiple files and requires too much user input.
This app should be plug a play. I don't think it matter if it is dirty at the beginning.

This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.

Rate-Limited

These updates are currently rate-limited. Click on a checkbox below to force their creation now.

• Update dependency localconfig to v1.1.4
• Update dependency pyparsing to v3.1.2
• Update dependency wcwidth to v0.2.13
• Update dependency docutils to v0.21.2
• Update dependency python-dateutil to v2.9.0.post0
• Update actions/checkout action to v4
• Update dependency coverage to v7
• Update dependency flake8 to v7
• Update dependency importlib-metadata to v7
• Update dependency jmespath to v1
• Update dependency more-itertools to v10
• Update dependency packaging to v24
• Update dependency pytest to v8
• Update dependency pytest-cov to v5
• Update dependency urllib3 to v2
• Update github/codeql-action action to v3
• 🔐 Create all rate-limited PRs at once 🔐

Open

These updates have all been created already. Click a checkbox below to force a retry/rebase of any.

• Update dependency certifi to v2023 [SECURITY]
• Update dependency boto3 to v1.34.123
• Update dependency entrypoints to v0.4
• Update dependency flake8 to v3.9.2
• Update dependency mccabe to v0.7.0
• Update dependency pluggy to v1.5.0
• Update dependency s3transfer to v0.10.1
• Update dependency zipp to v3.19.2
• Update dependency attrs to v23
• Update dependency chardet to v5
• Click on this checkbox to rebase all open PRs at once

Detected dependencies

• Check this box to trigger a request for Renovate to run again on this repository

Is your feature request related to a problem? Please describe.
Add aws authentication

Describe the solution you'd like
Been able to authenticate with aws elasticSearch

Describe alternatives you've considered
Importing manually visualisations is a pain

Is your feature request related to a problem? Please describe.
Sometimes, you only need to get the visualisations generated, the files parsed, visualisations send, or everything.

Describe the solution you'd like
I would like to have the option of process, generate, send, process_and_generate and process_generate_and_send visualisations

Is your feature request related to a problem? Please describe.
Currently, in order to get most things to work with multiple src structures, you need to edit the settings.ini manually.

Describe the solution you'd like
I will like to be able to pass the configuration (ini) file. This will allow to create multiple types of configuration file for different languages (but that is another issue)

Describe alternatives you've considered
Editing the file manually, is a bit of a pain

Is your feature request related to a problem? Please describe.
Add colours to the generated visualisation object depending on the log type/object

Describe the solution you'd like
I will like that the generated visualisation has different colours for the different log messages. Specially important with error and critical.

Describe alternatives you've considered
Not having colours. It is ok but having the colours should ease pointing out issues. (aka, will be nice if not too much work)

Additional context
When looking a lot of visualisation results, you can get lost in the numbers. Adding colours could provide that extra abstraction that should ease the understanding of the visualisations.

Describe the bug

When doing a git push, the git hook does act but doesn't push the generated commit by bumpversion to git.

To Reproduce

Steps to reproduce the behavior:

1. Do a commit that changes the version number (breaking, patch, minor)
2. git push it.
3. Check your local branch an remote.

There will be a commit difference between them. Your local will have the bumpversion commit in it.

Expected behavior

The bumpversion commit will be included in the push.

Desktop (please complete the following information):

• OS: Linux
• Version 0.7.7

There is an error with this repository's Renovate configuration that needs to be fixed. As a precaution, Renovate will stop PRs until it is resolved.

Error type: undefined. Note: this is a nested preset so please contact the preset author if you are unable to fix it yourself.

Is your feature request related to a problem? Please describe.
There is only the option of having a metric visualisation. It will be great to have the option of selecting another type of visualisation (like pie, donut, histogram, etc)

Describe the solution you'd like
I would like to provide the visualisation I want to generate or for the program to select the visualisation depending on the amount of data.

Describe alternatives you've considered
Using only table visualisations. Metric supported but needed to have a filter in order for them to be usable

Additional context
There are multiple visualisations to represent the data in different ways... so is no point of only using one.

deploy to PyPI

Adding unit test framework and write the unit tests to the created function.

Add documentation about the Kibana part of the application

Is your feature request related to a problem? Please describe.
Add the filename/project to the generated visualisation

Describe the solution you'd like
Passing an argument should do.

Describe alternatives you've considered
Detecting the function name will depend of many variables (language, code structure, etc) so its probably not worth the effort.

Additional context
If no filename/project is provided, multiple projects having the same function name will result on a visualisation overwrite in kibana.

Is your feature request related to a problem? Please describe.
I like to automate the versioning as I always forget to update the setup.cfg file and that breaks the build

Describe the solution you'd like
Adding setuptools-scm should allow for automating the process

Describe alternatives you've considered
pre-hooks for git and using https://pypi.org/project/bumpversion/ might be another option, but looks like setuptools-scm is a bit nicer than the bumpversion.

Additional context
https://pypi.org/project/setuptools-scm/

Is your feature request related to a problem? Please describe.
Some log messages are better kep in variables to follow the DRY principle

Describe the solution you'd like
I will like to add extra filter to idendity variables with log messages

Describe alternatives you've considered
Forcing not to use variables to keep the log message, but that will be contraproctive

Additional context
It should be possible to add extra variable names for the log messages types. For example, if debug messages are keep in variables called debug_message, and error messages in variables called error_message, it should be possible to find them and make the code more consistent.
A bit of refactoring around the detector and filter area will be necessary, but that is always good.