masesgroup / kefcore Goto Github PK
View Code? Open in Web Editor NEWEntity Framework Core provider for Apache Kafka™
Home Page: https://kefcore.masesgroup.com/
License: Apache License 2.0
Entity Framework Core provider for Apache Kafka™
Home Page: https://kefcore.masesgroup.com/
License: Apache License 2.0
Describe the bug
The documentation is not generated.
Additional context
See masesgroup/KNet#81 and masesgroup/JNet#35
Is your feature request related to a problem? Please describe.
The project seems enough mature for a major release, it can be used in our projects. New features or bugs will be managed with patches.
Describe the solution you'd like
Upgrade the version to 1.0.0 and create a release
Describe alternatives you've considered
N/A
Additional context
N/A
Is your feature request related to a problem? Please describe.
The types used to store information in the Kafka cluster are strings. If a key is a simple type managed from Kafka store it as native type: e.g. an integer shall be stored as integer. This can help in management of Apache Kafka Streams DSL.
Describe the solution you'd like
Remove strong definition of generic type using the constraint <string, string>
at least in ProducerRecord
and Producer
Describe alternatives you've considered
N/A
Additional context
Maybe KNet shall be evolved
Create a new extension on [EntityExtractor](https://github.com/masesgroup/KEFCore/blob/62442a542a73431aeae4a3ae481685aebaa82c96/src/net/KEFCore.SerDes/KEFCoreSerDes.Helpers.cs) to reproduce this behavior with an external callback to send extracted entities
Originally posted by @masesdevelopers in #105 (comment)
Is your feature request related to a problem? Please describe.
The project miss a package template.
Describe the solution you'd like
Add something like the one in KNet
Describe alternatives you've considered
N/A
Additional context
N/A
Describe the bug
The license is not shown in the package
To Reproduce
Steps to reproduce the behavior:
Expected behavior
Report the license in the package
Screenshots
Desktop (please complete the following information):
Additional context
N/A
Is your feature request related to a problem? Please describe.
Update KEFCore each time new packages of Microsoft.EntityFrameworkCore.* will be released
Describe the solution you'd like
To be always aligned to latest Microsoft.EntityFrameworkCore features and fixes, the project must update source code and the references to last Microsoft.EntityFrameworkCore.* packages for each version
Describe alternatives you've considered
N/A
Additional context
Similar to #22
Is your feature request related to a problem? Please describe.
The tables are stored within topics using the key for Primary Key and value for all items, so it is possible to use KNetCompactedReplicator to manage the compacted topics.
Describe the solution you'd like
Add an options to choose between KNetCompactedReplicator and Apache Kafka Streams to retrieve the data from the topics.
Describe alternatives you've considered
N/A
Additional context
The types associated to KNetCompactedReplicator shall be managed like in #59
Is your feature request related to a problem? Please describe.
The name used for serializer seems to be long and the scope is directly managed using the name.
Describe the solution you'd like
Update the names used with a scoped logic like KEFCoreSerDes.[package reference].[scope].[encoder]
KEFCoreSerDes.Json.Key
without [encoder] because it does not have an encoderKEFCoreSerDes.Json.ValueContainer
without [encoder] because it does not have an encoderKEFCoreSerDes.Avro.Key.Binary
KEFCoreSerDes.Avro.Key.Json
KEFCoreSerDes.Avro.ValueContainer.Binary
KEFCoreSerDes.Avro.ValueContainer.Json
Describe alternatives you've considered
N/A
Additional context
N/A
Is your feature request related to a problem? Please describe.
Since main site has a new style, update this web-site to the style of main site
Describe the solution you'd like
At least shall be:
dotnet tool update -g docfx
Describe alternatives you've considered
N/A
Additional context
See masesgroup/NuReflector#42
Is your feature request related to a problem? Please describe.
KEFCore can track changes using Entity Framework Core feature, but if the change event is triggered from the back-end does not report anything and the modification can be visible executing queries.
Describe the solution you'd like
Add some event handler to the configuration to subscribe to the change event notification
Describe alternatives you've considered
N/A
Additional context
N/A
Is your feature request related to a problem? Please describe.
On November will be available .NET 8 and the repository projects shall be aligned
Describe the solution you'd like
Upgrade the projects
Describe alternatives you've considered
N/A
Additional context
Must wait at least new release of KNet (see masesgroup/KNet#278)
Is your feature request related to a problem? Please describe.
Currently no tests are executed.
Describe the solution you'd like
The build workflow can execute some unit tests to verify execution.
Meanwhile the test projects can be enhanced to use the Microsoft.NET.Test.Sdk and/or Microsoft.TestPlatform.TestHost
Describe alternatives you've considered
N/A
Additional context
N/A
Is your feature request related to a problem? Please describe.
Update KEFCore each time KNet updates
Describe the solution you'd like
To be always aligned to latest Apache Kafka version and KNet features, @masesdevelopers must update source code and the references to last KNet release.
Describe alternatives you've considered
N/A
Additional context
See masesgroup/KNet#88
Is your feature request related to a problem? Please describe.
When KEFCore is used with Apache Kafka Streams does not reports good performance due to its current usage.
The power of Apache Kafka Streams can come out using flows built from the EFCore queries.
Describe the solution you'd like
Find a mapping between the query requested from EFCore and Apache Kafka Streams, building processors will manage that queries.
Instead to use the DSL version of Apache Kafka Streams, a better approach is to use create specific processors.
Describe alternatives you've considered
N/A
Additional context
N/A
Is your feature request related to a problem? Please describe.
KEFCore.SerDes package is the base reference for serialization; another serialization mechanism shall be based on Protobuf to extend capabilities of the projects.
Describe the solution you'd like
Use the project in KNet as reference and create a new package in KEFCore following the Avro project.
Describe alternatives you've considered
N/A
Additional context
See #90
Describe the bug
When the application is restarted, using the same ApplicationId
, previously stored data in Apache Kafka Streams can impact the behavior of the application.
To Reproduce
Steps to reproduce the behavior:
Expected behavior
The stream application shall be resetted before
KEFCore/test/KEFCore.Test/Program.cs
Line 60 in 6c01278
Screenshots
If applicable, add screenshots to help explain your problem.
Desktop (please complete the following information):
Additional context
N/A
The following is the current default for a topic will be created:
However cleanup policy delete
can be too much strong if the data shall remain for a longest period in the topic, so can be added a new options which helps to create the property with that option. The default will be only compact
.
Is your feature request related to a problem? Please describe.
Till now the test projects use cabled values so each time it is mandatory to update and recompile the test to change the values to be used.
Describe the solution you'd like
Add an external file, preferably a JSON file, to be passed as argument with the values needed for the test to be executed.
Describe alternatives you've considered
Add as many command-line parameters as needed.
Additional context
N/A
Is your feature request related to a problem? Please describe.
The current serialization/deserialization is done using classes within the main project and external program cannot use them to retrieve the data
Describe the solution you'd like
Create a new project to support serialization/deserialization and avoid the use of MASES.KNet.Serialization.Json
Describe alternatives you've considered
N/A
Additional context
N/A
Is your feature request related to a problem? Please describe.
The pull request workflow takes too long to execute
Describe the solution you'd like
Remove the documentation generation from the workflow.
Meanwhile create a specific manually triggered workflow, detached from build one, for documentation
Describe alternatives you've considered
N/A
Additional context
N/A
Is your feature request related to a problem? Please describe.
The library is based on .NET 6 which has its own integrated JSON library that is enough for current scopes, the powerful external library Newtonsoft.Json can be removed.
Describe the solution you'd like
Remove dependency from Newtonsoft.Json and update code.
Describe alternatives you've considered
N/A
Additional context
N/A
Is your feature request related to a problem? Please describe.
Upon next release of .NET 8 and Entity Framework Core 8 the package shall distinguish functionalities based on .NET version since Entity Framework Core 8 will wok on .NET 8 only
Describe the solution you'd like
Modify the project to accept multiple frameworks and change packages based on the framework in use:
The code adaptation shall be made using compilation conditions
Describe alternatives you've considered
Create multiple packages.
Additional context
N/A
Describe the bug
Running the available test the application raises many errors
To Reproduce
Steps to reproduce the behavior:
Expected behavior
The test shall run without errors
Screenshots
If applicable, add screenshots to help explain your problem.
Desktop (please complete the following information):
Additional context
Latest JNet and KNet reports some incompatibility
Describe the bug
There is a warning during build:
The set-output
command is deprecated and will be disabled soon. Please upgrade to using Environment Files. For more information see: https://github.blog/changelog/2022-10-11-github-actions-deprecating-save-state-and-set-output-commands/
To Reproduce
Steps to reproduce the behavior:
Expected behavior
Remove warning
Screenshots
If applicable, add screenshots to help explain your problem.
Desktop (please complete the following information):
Additional context
Add any other context about the problem here.
Missed update of https://github.com/masesgroup/KEFCore/blob/9f795fbe2440292c3668568547ad66e55fe57ca0/src/documentation/articles/usageTemplates.md
Originally posted by @masesdevelopers in #120 (comment)
Is your feature request related to a problem? Please describe.
The Test project make some measurement, however it is not a test specifically written to make performance test
Describe the solution you'd like
Create a new project able to make measurement of performances, based on different configurations, comparing the result against other available provider, e.g. InMemory provider
Describe alternatives you've considered
N/A
Additional context
N/A
Is your feature request related to a problem? Please describe.
The current workflows push directly the documentation using github-actions user, however the branch protection avoid this behavior.
Describe the solution you'd like
When an update on documentation is needed: use a PR to request the update, this maybe avoid multiple updates on documentation on each PR approved.
Describe alternatives you've considered
N/A
Additional context
N/A
Testing the modifications introduced from this PR many times the test raises a java.lang.NullPointerException
at
org.apache.kafka.clients.producer.KafkaProducer.doSend(KafkaProducer.java:996)
org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:962)
org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:847)
Originally posted by @mariomastrodicasa in #64 (comment)
Till Apache Kafka Streams engine is not updated, the engine based on KNetCompactedReplicator shall be the default
Originally posted by @masesdevelopers in #87 (comment)
Testing the modification introduced from this PR seems the speed of LINQ query is very poor; the following code
context.Posts.Single(b => b.BlogId == 2)
takes more or less 16 seconds to be executed as reported from the following output:
Elapsed context.Posts.Single(b => b.BlogId == 2) 16299 ms. Result is PostId: 2 Title: title Content: 2 BlogId: 2
Originally posted by @mariomastrodicasa in #64 (comment)
Add options to choose between buffered and non buffered data exchange both on key and value
Originally posted by @masesdevelopers in #231 (comment)
Is your feature request related to a problem? Please describe.
The current documentation is not complete and in many points is not clear at all.
Describe the solution you'd like
Update the documentation for public classes, add something on Internal classes like it is done in EFCore
Describe alternatives you've considered
N/A
Additional context
N/A
Describe the bug
Benchmark test under .NET 8 environment many times raises System.AccessViolationException
when Org.Apache.Kafka.Streams.KeyValue<K, V>
is retrieved from Org.Apache.Kafka.Streams.State.KeyValueIterator<K, V>
To Reproduce
Steps to reproduce the behavior:
Expected behavior
Since in .NET 6 and .NET 7 the issue is not raised, the expectation is the same for .NET 8
Screenshots
If applicable, add screenshots to help explain your problem.
Desktop (please complete the following information):
Additional context
N/A
The new KEFCore.SerDes project was added as base reference; it is important to add a project, based on AVRO, to add the binary serialization used from Apache Kafka so external applications can attach to the bus and use the information retrieving the schema from Schema Registry.
Originally posted by @masesdevelopers in #86 (comment)
Is your feature request related to a problem? Please describe.
The package is published on nuget.org, while other projects publish on GitHub too.
Describe the solution you'd like
Publish packages on GitHub like JNet and KNet
Describe alternatives you've considered
N/A
Additional context
See JNet and KNet projects
Is your feature request related to a problem? Please describe.
The documentation miss cross reference with JNet and KNet
Describe the solution you'd like
Use the behavior used in masesgroup/KNet#393, fixed with masesgroup/KNet#394
Describe alternatives you've considered
N/A
Additional context
N/A
Add some extension to the serialization mechanism to retrieve CLR filled type reading the data stored in the bus. This can be helpful to analyze data externally to the main application based on EFCore.
Originally posted by @masesdevelopers in #86 (comment)
Shall be clear if it is really needed, however a serializer based on a generic array can put in place beside the one using the standard Apache Kafka serializer
Originally posted by @masesdevelopers in #95 (comment)
Describe the bug
If the test application does not recreate the Kafka topics the tests fails with Sequence contains no elements
To Reproduce
Steps to reproduce the behavior:
{
"UseCompactedReplicator": true,
"DeleteApplicationData": false,
"LoadApplicationData": false,
"BootstrapServers": "KAFKA_CLUSTER:9092"
}
MASES.EntityFrameworkCore.KNet.Test.exe .\KNetReplicatorNoLoadTest.json
Expected behavior
The test shall be executed without error retrieving previously data stored
Screenshots
N/A
Desktop (please complete the following information):
Additional context
Attached the changed files in the test projects KEFCore.Test.zip
Is your feature request related to a problem? Please describe.
The content of https://github.com/masesgroup/KEFCore/blob/b11cc28bdc1e2b31078ca3a717d646173c4e6d20/src/documentation/articles/usecases.md is nonexistent.
Describe the solution you'd like
Try to fill in, even only as text, some simple use cases
Describe alternatives you've considered
N/A
Additional context
N/A
The following class, which is one of the most important, miss the documentation:
KEFCore/src/net/KEFCore/KEFCore.cs
Line 23 in 59ab096
Describe the bug
Enabling the usage of KNetCompactedReplicator
, the test reports Sequence contains no elements when
KEFCore/test/KEFCore.Test/Program.cs
Line 183 in d5c8d5d
To Reproduce
Steps to reproduce the behavior:
MASES.EntityFrameworkCore.KNet.Test.exe .\KNetReplicatorTest.json
Expected behavior
The code shall execute without exceptions
Screenshots
N/A
Desktop (please complete the following information):
Additional context
N/A
Looking at the following method
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.