[PERF] Journal Data is logging to redundant information in WorkflowExecutionLogRecords about elsa-core HOT 2 CLOSED

Based on the context provided, it seems you are looking to improve the performance of Elsa Workflows by addressing the issue of redundant data storage, particularly in the WorkflowExecutionLogRecords table. The SerializedPayload column in this table stores a large amount of data, which can lead to rapid database growth and potentially impact performance due to the size of the data being stored and transferred.

To address this issue, you are proposing to remove redundant data from the SerializedPayload of the WorkflowExecutionLogRecords table. This would involve not storing the entire journal data for each activity, which can include outcomes, exceptions, and other journalization data.

As an alternative, you've considered allowing users to configure what gets stored in the journal, particularly for activities that create outputs. This could be managed through the Persistence Log Record Module, where users can select which outputs to persist.

The impact of this enhancement would be significant storage savings and improved performance due to reduced data transfer to the persistence store. You've provided a benchmark showing that the size of a record could be reduced significantly, from 683 bytes to 505,923 bytes, in the case of an HTTP sample.

To proceed with this enhancement, you would need to evaluate the current implementation and identify where the redundancies occur. Then, you would need to modify the logging mechanism to either remove the redundant data or provide configuration options for users to select what data should be stored.

This change would affect any use case where activities generate outputs that are currently being stored in the SerializedPayload. Activities like Elsa.FlowSendHttpRequest, which add parsed content of HTTP calls to the journal, would be particularly affected.

In summary, the proposed enhancement aims to optimize data storage and improve performance by reducing the amount of redundant data stored in the WorkflowExecutionLogRecords table. This would involve changes to the logging mechanism and potentially adding user-configurable options for data persistence.

This comment was generated by Glime.

from elsa-core.

closed as done with #5266

from elsa-core.