masesgroup / kefcore Goto Github PK

Describe the bug
The documentation is not generated.

Additional context
See masesgroup/KNet#81 and masesgroup/JNet#35

Is your feature request related to a problem? Please describe.
The project seems enough mature for a major release, it can be used in our projects. New features or bugs will be managed with patches.

Describe the solution you'd like
Upgrade the version to 1.0.0 and create a release

Describe alternatives you've considered
N/A

Additional context
N/A

Is your feature request related to a problem? Please describe.
The types used to store information in the Kafka cluster are strings. If a key is a simple type managed from Kafka store it as native type: e.g. an integer shall be stored as integer. This can help in management of Apache Kafka Streams DSL.

Describe the solution you'd like
Remove strong definition of generic type using the constraint <string, string> at least in ProducerRecord and Producer

Describe alternatives you've considered
N/A

Additional context
Maybe KNet shall be evolved

          Create a new extension on [EntityExtractor](https://github.com/masesgroup/KEFCore/blob/62442a542a73431aeae4a3ae481685aebaa82c96/src/net/KEFCore.SerDes/KEFCoreSerDes.Helpers.cs) to reproduce this behavior with an external callback to send extracted entities

Originally posted by @masesdevelopers in #105 (comment)

Is your feature request related to a problem? Please describe.
The project miss a package template.

Describe the solution you'd like
Add something like the one in KNet

Describe alternatives you've considered
N/A

Additional context
N/A

Describe the bug
The license is not shown in the package

To Reproduce
Steps to reproduce the behavior:

1. Go to nuget.org
2. Click on on the package
3. On the right column the license is not available

Expected behavior
Report the license in the package

Screenshots

Desktop (please complete the following information):

• OS: All
• Version: All

Additional context
N/A

Is your feature request related to a problem? Please describe.
Update KEFCore each time new packages of Microsoft.EntityFrameworkCore.* will be released

Describe the solution you'd like
To be always aligned to latest Microsoft.EntityFrameworkCore features and fixes, the project must update source code and the references to last Microsoft.EntityFrameworkCore.* packages for each version

Describe alternatives you've considered
N/A

Additional context
Similar to #22

Is your feature request related to a problem? Please describe.
The tables are stored within topics using the key for Primary Key and value for all items, so it is possible to use KNetCompactedReplicator to manage the compacted topics.

Describe the solution you'd like
Add an options to choose between KNetCompactedReplicator and Apache Kafka Streams to retrieve the data from the topics.

Describe alternatives you've considered
N/A

Additional context
The types associated to KNetCompactedReplicator shall be managed like in #59

Is your feature request related to a problem? Please describe.
The name used for serializer seems to be long and the scope is directly managed using the name.

Describe the solution you'd like
Update the names used with a scoped logic like KEFCoreSerDes.[package reference].[scope].[encoder]

• Standard Json:
  • Primary Key: KEFCoreSerDes.Json.Key without [encoder] because it does not have an encoder
  • ValueContainer: KEFCoreSerDes.Json.ValueContainer without [encoder] because it does not have an encoder
• Avro:
  • Primary Key:
    • KEFCoreSerDes.Avro.Key.Binary
    • KEFCoreSerDes.Avro.Key.Json
  • ValueContainer:
    • KEFCoreSerDes.Avro.ValueContainer.Binary
    • KEFCoreSerDes.Avro.ValueContainer.Json

Describe alternatives you've considered
N/A

Additional context
N/A

Is your feature request related to a problem? Please describe.
Since main site has a new style, update this web-site to the style of main site

Describe the solution you'd like
At least shall be:

• added modern template
• added favicon of mases
• changed the footer to report info on MASES
• no more use chocolatey, instead invoke: dotnet tool update -g docfx
• review menu

Describe alternatives you've considered
N/A

Additional context
See masesgroup/NuReflector#42

Is your feature request related to a problem? Please describe.
KEFCore can track changes using Entity Framework Core feature, but if the change event is triggered from the back-end does not report anything and the modification can be visible executing queries.

Describe the solution you'd like
Add some event handler to the configuration to subscribe to the change event notification

Describe alternatives you've considered
N/A

Additional context
N/A

Is your feature request related to a problem? Please describe.
On November will be available .NET 8 and the repository projects shall be aligned

Describe the solution you'd like
Upgrade the projects

Describe alternatives you've considered
N/A

Additional context
Must wait at least new release of KNet (see masesgroup/KNet#278)

Is your feature request related to a problem? Please describe.
Currently no tests are executed.

Describe the solution you'd like
The build workflow can execute some unit tests to verify execution.
Meanwhile the test projects can be enhanced to use the Microsoft.NET.Test.Sdk and/or Microsoft.TestPlatform.TestHost

Describe alternatives you've considered
N/A

Additional context
N/A

Is your feature request related to a problem? Please describe.
Update KEFCore each time KNet updates

Describe the solution you'd like
To be always aligned to latest Apache Kafka version and KNet features, @masesdevelopers must update source code and the references to last KNet release.

Describe alternatives you've considered
N/A

Additional context
See masesgroup/KNet#88

Is your feature request related to a problem? Please describe.
When KEFCore is used with Apache Kafka Streams does not reports good performance due to its current usage.
The power of Apache Kafka Streams can come out using flows built from the EFCore queries.

Describe the solution you'd like
Find a mapping between the query requested from EFCore and Apache Kafka Streams, building processors will manage that queries.
Instead to use the DSL version of Apache Kafka Streams, a better approach is to use create specific processors.

Describe alternatives you've considered
N/A

Additional context
N/A

Is your feature request related to a problem? Please describe.
KEFCore.SerDes package is the base reference for serialization; another serialization mechanism shall be based on Protobuf to extend capabilities of the projects.

Describe the solution you'd like
Use the project in KNet as reference and create a new package in KEFCore following the Avro project.

Describe alternatives you've considered
N/A

Additional context
See #90

Describe the bug
When the application is restarted, using the same ApplicationId, previously stored data in Apache Kafka Streams can impact the behavior of the application.

To Reproduce
Steps to reproduce the behavior:

1. Change the test program adding/removing some values at the end of the current queries (e.g. add 100 items)
2. Execute the test program
3. Execute again the test program
4. The following trace does not report the current values (expected 1000), but reports something like KafkaEnumerator - ApproximateNumEntries 1100 which means the stream application had stored the latest test done

Expected behavior
The stream application shall be resetted before

Screenshots
If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

• Latest KEFCore version

Additional context
N/A

The following is the current default for a topic will be created:

However cleanup policy delete can be too much strong if the data shall remain for a longest period in the topic, so can be added a new options which helps to create the property with that option. The default will be only compact.

Is your feature request related to a problem? Please describe.
Till now the test projects use cabled values so each time it is mandatory to update and recompile the test to change the values to be used.

Describe the solution you'd like
Add an external file, preferably a JSON file, to be passed as argument with the values needed for the test to be executed.

Describe alternatives you've considered
Add as many command-line parameters as needed.

Additional context
N/A

Is your feature request related to a problem? Please describe.
The current serialization/deserialization is done using classes within the main project and external program cannot use them to retrieve the data

Describe the solution you'd like
Create a new project to support serialization/deserialization and avoid the use of MASES.KNet.Serialization.Json

Describe alternatives you've considered
N/A

Additional context
N/A

Is your feature request related to a problem? Please describe.
The pull request workflow takes too long to execute

Describe the solution you'd like
Remove the documentation generation from the workflow.
Meanwhile create a specific manually triggered workflow, detached from build one, for documentation

Describe alternatives you've considered
N/A

Additional context
N/A

Is your feature request related to a problem? Please describe.
The library is based on .NET 6 which has its own integrated JSON library that is enough for current scopes, the powerful external library Newtonsoft.Json can be removed.

Describe the solution you'd like
Remove dependency from Newtonsoft.Json and update code.

Describe alternatives you've considered
N/A

Additional context
N/A

Is your feature request related to a problem? Please describe.
Upon next release of .NET 8 and Entity Framework Core 8 the package shall distinguish functionalities based on .NET version since Entity Framework Core 8 will wok on .NET 8 only

Describe the solution you'd like
Modify the project to accept multiple frameworks and change packages based on the framework in use:

• Entity Framework Core 6 will work with .NET 6
• Entity Framework Core 7 will work with .NET 7
  and when generally available Entity Framework Core 8 will work with .NET 8

The code adaptation shall be made using compilation conditions

Describe alternatives you've considered
Create multiple packages.

Additional context
N/A

Describe the bug
Running the available test the application raises many errors

To Reproduce
Steps to reproduce the behavior:

1. Open KEFCore.Test solution
2. Run the test
3. See error

Expected behavior
The test shall run without errors

Screenshots
If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

• OS: Windows 10
• Version: 0.7.0

Additional context
Latest JNet and KNet reports some incompatibility

Describe the bug
There is a warning during build:
The set-output command is deprecated and will be disabled soon. Please upgrade to using Environment Files. For more information see: https://github.blog/changelog/2022-10-11-github-actions-deprecating-save-state-and-set-output-commands/

To Reproduce
Steps to reproduce the behavior:

1. See output of GitHub workflow

Expected behavior
Remove warning

Screenshots
If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

• OS: ALL
• Version: latest workflow images

Additional context
Add any other context about the problem here.

          Missed update of https://github.com/masesgroup/KEFCore/blob/9f795fbe2440292c3668568547ad66e55fe57ca0/src/documentation/articles/usageTemplates.md

Originally posted by @masesdevelopers in #120 (comment)

Is your feature request related to a problem? Please describe.
The Test project make some measurement, however it is not a test specifically written to make performance test

Describe the solution you'd like
Create a new project able to make measurement of performances, based on different configurations, comparing the result against other available provider, e.g. InMemory provider

Describe alternatives you've considered
N/A

Additional context
N/A

Is your feature request related to a problem? Please describe.
The current workflows push directly the documentation using github-actions user, however the branch protection avoid this behavior.

Describe the solution you'd like
When an update on documentation is needed: use a PR to request the update, this maybe avoid multiple updates on documentation on each PR approved.

Describe alternatives you've considered
N/A

Additional context
N/A

Testing the modifications introduced from this PR many times the test raises a java.lang.NullPointerException at

org.apache.kafka.clients.producer.KafkaProducer.doSend(KafkaProducer.java:996)
org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:962)
org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:847)

Originally posted by @mariomastrodicasa in #64 (comment)

          Till Apache Kafka Streams engine is not updated, the engine based on KNetCompactedReplicator shall be the default

Originally posted by @masesdevelopers in #87 (comment)

Testing the modification introduced from this PR seems the speed of LINQ query is very poor; the following code

context.Posts.Single(b => b.BlogId == 2)

takes more or less 16 seconds to be executed as reported from the following output:
Elapsed context.Posts.Single(b => b.BlogId == 2) 16299 ms. Result is PostId: 2 Title: title Content: 2 BlogId: 2

Originally posted by @mariomastrodicasa in #64 (comment)

          Add options to choose between buffered and non buffered data exchange both on key and value

Originally posted by @masesdevelopers in #231 (comment)

Is your feature request related to a problem? Please describe.
The current documentation is not complete and in many points is not clear at all.

Describe the solution you'd like
Update the documentation for public classes, add something on Internal classes like it is done in EFCore

Describe alternatives you've considered
N/A

Additional context
N/A

Describe the bug
Benchmark test under .NET 8 environment many times raises System.AccessViolationException when Org.Apache.Kafka.Streams.KeyValue<K, V> is retrieved from Org.Apache.Kafka.Streams.State.KeyValueIterator<K, V>

To Reproduce
Steps to reproduce the behavior:

1. Compile KEFCore.Benchmark.Test
2. Execute the test using .NET 8 version
3. Wait the end and, if the issue is not raised, relaunch the test
4. See an error like System.AccessViolationException: Attempted to read or write protected memory. This is often an indication that other memory is corrupt.

Expected behavior
Since in .NET 6 and .NET 7 the issue is not raised, the expectation is the same for .NET 8

Screenshots
If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

• OS: Windows 10
• .NET 8 version 8.0.1
• Version: latest release 1.1.0

Additional context
N/A

The new KEFCore.SerDes project was added as base reference; it is important to add a project, based on AVRO, to add the binary serialization used from Apache Kafka so external applications can attach to the bus and use the information retrieving the schema from Schema Registry.

Originally posted by @masesdevelopers in #86 (comment)

Is your feature request related to a problem? Please describe.
The package is published on nuget.org, while other projects publish on GitHub too.

Describe the solution you'd like
Publish packages on GitHub like JNet and KNet

Describe alternatives you've considered
N/A

Additional context
See JNet and KNet projects

Is your feature request related to a problem? Please describe.
The documentation miss cross reference with JNet and KNet

Describe the solution you'd like
Use the behavior used in masesgroup/KNet#393, fixed with masesgroup/KNet#394

Describe alternatives you've considered
N/A

Additional context
N/A

Add some extension to the serialization mechanism to retrieve CLR filled type reading the data stored in the bus. This can be helpful to analyze data externally to the main application based on EFCore.

Originally posted by @masesdevelopers in #86 (comment)

Shall be clear if it is really needed, however a serializer based on a generic array can put in place beside the one using the standard Apache Kafka serializer

Originally posted by @masesdevelopers in #95 (comment)

Describe the bug
If the test application does not recreate the Kafka topics the tests fails with Sequence contains no elements

To Reproduce
Steps to reproduce the behavior:

1. Create a JSON configuration file named KNetReplicatorNoLoadTest.json with the following content:

{
  "UseCompactedReplicator": true,
  "DeleteApplicationData": false,
  "LoadApplicationData": false,
  "BootstrapServers": "KAFKA_CLUSTER:9092"
}

2. Execute MASES.EntityFrameworkCore.KNet.Test.exe .\KNetReplicatorNoLoadTest.json
3. See error

Expected behavior
The test shall be executed without error retrieving previously data stored

Screenshots
N/A

Desktop (please complete the following information):

• OS: Windows 10
• Version: 0.8.0

Additional context
Attached the changed files in the test projects KEFCore.Test.zip

Is your feature request related to a problem? Please describe.
The content of https://github.com/masesgroup/KEFCore/blob/b11cc28bdc1e2b31078ca3a717d646173c4e6d20/src/documentation/articles/usecases.md is nonexistent.

Describe the solution you'd like
Try to fill in, even only as text, some simple use cases

Describe alternatives you've considered
N/A

Additional context
N/A

The following class, which is one of the most important, miss the documentation:

Describe the bug
Enabling the usage of KNetCompactedReplicator, the test reports Sequence contains no elements when

To Reproduce
Steps to reproduce the behavior:

1. Execute MASES.EntityFrameworkCore.KNet.Test.exe .\KNetReplicatorTest.json
2. Wait the point of exception
3. See error

Expected behavior
The code shall execute without exceptions

Screenshots
N/A

Desktop (please complete the following information):

• Current HEAD version

Additional context
N/A

Looking at the following method