pijng / moonlogs Goto Github PK

Proposal

Add the ability to specify the render order of schemes that have a tag assigned to them.

Use case

Currently, the order of issue goes in ascending order by id created tag.
It is undesirable to be bound to the order of tag creation to prioritize rendering in this case.

Proposal

Add the ability to create records in groups with a single request.

Use case

Sometimes there are situations when you need to record many logs of the same type within one process. For example, when an entity is changed, it is necessary to record the change of each attribute in a separate record.

At the moment it is necessary to send N requests to moonlogs, which causes unnecessary network costs, which can be reduced.

Perhaps it makes sense to add a separate route for group creation of logs. Route should take an array of records as input and process each of them.

In this case, synchronous logging is probably the preferred method. But it may be worth adding asynchronous support for group insertion in case we don't care about the order of records,

Proposal

Add support for simple log formatting for convenient output in the interface.

Initial candidates:

• \n
• \t
• \r

Use case

Enhance overall experience with text output

Proposal

Add the ability to extract* JSON from a specific record in the web interface.

*Extraction can be performed by one of the following methods:

• click on JSON part of the record should open a new tab with Content-type: application/json with only JSON present there. This approach can be implicitly integrated with browser extensions for easy JSON browsing, such as: JSON Formatter, JSON-handle, JSON Lite, etc;
• click on JSON part should copy this JSON payload to user's clipboard (it's a less preferable way.)

Use case

Often during problem solving, the need arises to:

1. Pass JSON to a third party, for example, when integrating with an external service;
2. Examine the payload in more detail for known problems;
3. Apply the payload to test requests/execution in the development environment to find out the details of the problem.

The current set of functionality involves only manual copying of the required text from the record. This causes the following inconveniences:

1. Inability to consider JSON payload in a convenient way with indentation and formatting;
2. Often the JSON will be enclosed within a more generalized string, e.g.: "Sending a requst to X with payload: <json_payload>". So even manual copying involves either accurately selecting the part containing JSON - which slows down the overall flow of the work - or an additional step in the form of: "copy the whole string -> paste into text editor -> copy only the JSON part".

Proposal

Add internationalization with the option to select a preferred language.

Use case

Internationalization will improve the overall experience of interacting with moonlogs. Plus, it will serve as a foundation for third-party developers to add new languages to the system.

The ability to choose a language, on the other hand, will allow you to use moonlogs in your preferred language without having to change the environment.

For example, if English is the default language in the user's browser, the entire moonlogs interface will be in English, including date formats, even if the user is a native speaker of Russian. Which, of course, is not convenient because of the drastically different perception of date formats.

The ability to select the language in the profile will allow you to switch the language of the entire interface without having to change the browser settings - thus removing unnecessary coeffect.

Preferred language selection scheme in descending order of priority:

• User-selected language in the profile
• Browser language
• English

Proposal

Currently, the record creation date is generated by moonlogs itself. This makes it easier for users to work with dates, but sometimes it is not the best solution.

Use case

In case of asynchronous logging through different workers, their order may be confused. Plus time differences between servers can create confusion on records and cause difficulty in data analysis due to different timestamps when linking logs with other data.

Proposal

Add ability to share record's request/response with unauthorized users.

Use case

Useful for transferring data to external integrators.

Proposal

Analyze insertion performance as the number of queries/growth of sql base increases.

Use case

There is a general performance hit on inserting into a standard sqlite database.

For comparison and further elaboration of this problem I will give below examples of statistics for different time intervals and rpm:

(Figure 1.1 Insert performance on the first week after release)

Approximately 1 million records were created during this time.

(Figure 1.2 Insert performance on the second week)

Approximately 3 millions records were created during this time.

We can notice that the average response time has almost doubled with a twofold increase in the number of requests to the service.
Of course, the presence of indexes in the records table is of great importance, so the decrease of insertion speed is quite expected as the number of records increases.

The above statistics reflects only a general picture of the situation, as during these periods night time is also taken into account - when requests become much less and, as a consequence, the average response time is distorted to a lesser extent.

(Figure 2.1 Insert performance during peak hours on the first week)

As far as can be seen, insertion performance during peak hours (3-7pm) on February 26 in the first week of launch averaged 54ms with a 1.1k rpm load.

(Figure 2.2 Insert performance during peak hours on the second week)

After a week and a half, insertion performance is already averaging 120ms under a 2.83k rpm load during the same peak hours.

It can be seen that the response time increased by a factor of 2.2 when the load increased by a factor of 2.5.

From this we would like to draw a premature conclusion in the linear increase of response time with the growth of the base and increase of the average rpm.

This may lead to undesirable consequences in the form of slowing down the code from which logs are sent.
Or an increase in the percentage of data loss due to more frequent timeouts.

In this connection it is necessary to analyze further performance in detail and confirm/refute this assumption.

If the assumption turns out to be false, it may be possible to get by with general tuning of sqlite database. But it should be taken into account that in the future more and more business processes will be connected to moonlogs, and some of them will be extremely explosive - for example, mass sending of push notifications via FCM.

But even then, the solution may only be temporary. On the other hand, for such business processes it is quite reasonable to set low retention_days for more frequent cleaning of large amount of data.

If the assumption turns out to be correct, you should think about adding a new base adapter. For example, ScyllaDB.

Proposal

Add the ability to enable asynchronous log creation.

Enabling this feature should generally be done in config.yaml to ensure smooth migration of functionality.
If a particular application still needs to create logs synchronously, while leaving the processing of logs from other sources in asynchronous mode, we can consider passing an additional attribute to the request, or even a separate endpoint for async actions.

Use case

At the moment, when creating a log via POST /api/logs, the entire log creation process is synchronized - from processing and validating the request to writing to the database. Thus, the sender is waiting for a response all this time, but he may not need it.

This is not a particular problem in languages that can easily perform asynchronous operations without blocking the main thread (e.g. golang's goroutines, typescript's async/await), but in languages that rely on single-threaded execution (e.g. ruby) this can be a key factor in significantly slowing down performance.

Moreover, for most tasks it should be sufficient to send a log request unilaterally.