jtheisen / reconciler Goto Github PK

Update an entity graph in store to a given one by inserting, updating and removing the respective entities.

License: MIT License

C# 100.00%

entity-framework entity-framework-6 entity-framework-core

reconciler's Introduction

Graph Reconciler for Entity Framework 6 and Core

Warning: The EF Core variant in versions prior to 0.3 on .NET Core became buggy with a change in semantics of later EF Core releases. I don't know with exactly which release of EF Core this issue manifested, but under certain circumstances you would delete more entities than you asked for. The test suite shows this and since 0.3 this is fixed. I recommend not using the earlier versions unless you're sure your version of EF Core passes the test suite - I only know for sure that EF Core 2.1 does.

Teaser

This library allows you to write

    await context.ReconcileAsync(personSentFromClient, e => e
        .WithOne(p => p.Address)
        .WithMany(p => p.Tags, e2 => e2
            .WithShared(p => p.Tag))
        ;
    await context.SaveChangesAsync();

and it will sync the four respective tables to match the given, detached personSentFromClient entity by adding, updating and removing entities as required.

Its primary use case are updates on multiple related entities retrieved from a client through an API.

It is a replacement for the GraphDiff library.

The EF 6 and EF Core versions share the same source code as far as possible to ensure consistency.

NuGet

There is one NuGet package for each of the two frameworks:

Install-Package Reconciler.Ef6

Install-Package Reconciler.EfCore

The EF6 version also has a prerelease package that targets netstandard2.1 and EF6.3 for use with .NET Core.

Definitions

Template entities are the entities to reconcile towards (personSentFromClient in the teaser sample)
The extent is the extent of the subtree rooted in the template entity of the first parameter that is to be reconciled as defined by the second parameter to the Reconcile extension methods.

Further features

The extent definition can also contain certain extra information to help with common scenarios that would otherwise often require some convoluted manual code.

Fixing

Sometimes we need to employ certain fixes on nested parts of the graph on saving:

.OnInsertion(e => e.Id == Guid.NewGuid())
.OnInsertion(e => e.CreatedAt == DateTimeOffset.Now)
.OnUpdate(e => e.ModifiedAt == DateTimeOffset.Now)

The OnUpdate definitions does not also apply (as of version 1.2.0) to insertions.

Note the use of the equality operator, as the assignment operator can't be used in expression trees in C#.

Exclude properties

Sometimes some properties should not be updated, and sometimes they shouldn't even be passed to a client on loading:

.WithReadOnly(e => e.Unmodifiable)
.WithBlacked(e => e.Secret)

The latter implies .WithReadOnly on writing since version 1.2.0 so that saving what was previously loaded doesn't accidentally overwrite the blacked field.

Some details and caveats

The are some things to be aware of:

I'm using the library in production, but that doesn't mean it's mature. The test suite is thin and you may hit issues.
Specifying relationships on derived classes in models with inheritance is not supported.
Using entities that are part of an inheritance hierarchy in all other contexts is also untested and likely doesn't work yet.
Many-to-many relationships are not supported and will probably never be.
Key Consistency Requirement: In the EF6 version, the foreign key properties in the template entity must be set to match the respective navigational properties before the call to one of the Reconcile overloads is made. For example, it should be that person.AddressId == person.Address.Id in the unit test's sample model. For EF Core, this requirement was lifted in version 1.1.0.
The extent must represent a subtree, i.e. have no cycles, and all entities must appear only once.
The Reconcile overloads themselves access the database only for reading and thus need to be followed by a SaveChanges call. Alternatively, ReconcileAndSaveChanges does that for you.
A Reconcile call should normally be followed by a SaveChanges. Multiple Reconcile calls without saving in between will likely only work properly if the datasets involved are disjoint.
The number of reads done is the number of entities either in storage or in the template that are covered by the extent and have a non-trivial sub-extent themselves. In the above example, the address would have been fetched with the person itself and cause no further load, but there would be one load per tag-to-person bridge table row which each include the respective tag. There's room for optimization here, but that's where it's at.

Some things didn't work in the past but are supported since version 1.0.0:

Database-generated keys such as auto-increment integers should now be well-behaved.
Moving entities from one collection to another will work if those collections are on the same extent level and the move is done in a single Reconcile call: Consider for example the model Stars > Planets > Moons with three entity types, two of which have a foreign key to the object they are orbiting. You can move a moon to a different planet while reconciling on a star. However, you need the extra level of the star so that you can express this operation in a single call to Reconcile (as you need at least two planets for a move of a moon to occur).

GraphDiff

Before writing this replacement I used the GraphDiff library, but since it is no longer maintained I became motivated to write my own solution. In particular, I wanted

Entity Framework Core support,
async/await support and
really understand what's going on.

As I don't fully understand the original there are some differences beyond the names of functions: GraphDiff had OwnedEntity, OwnedCollection and AssociatedCollection as options for how to define the extent, and I don't quite know what the difference between associated and owned is.

Reconciler has WithOne, WithMany and WithShared:

WithOne and WithMany reconcile a scalar or collection navigational property, respectively, through respective addition, update and removal operations. WithShared only works on scalar navigational properties and doesn't remove a formerly related entity that is now no longer related. This is useful, for example, in the sample in the teaser where we want to insert new tag entities when they are needed but don't want to remove them as they may be shared by other entities.

Roadmap

There are a number of exciting features that would make this library much more useful. See this document for further information.

reconciler's People

Contributors

Stargazers

Watchers

Forkers

sheva-serga timsirmovics ager1912 renatmedved

reconciler's Issues

Define alternative key selector

We have tables which use an auto generated identity column as the PK, and one or more fields as another candidate key that the client knows about.

e.g.

public class SomeEntity
{
    [Key]
    public int SomeEntityId { get; set; }

    public string ClientKey1 { get; set; }

    public string ClientKey2 { get; set; }
}

In this case the key used by the client is a combination of ClientKey1 and ClientKey2, but for consistency we have an auto generated identity as the actual PK.

Could a feature be added to the ExtentBuilder to specify which properties should be used for matching?

Not able to initialise a field to default on insert and increment the field on every update

We have a UpdateCount integer field in our entity which is used to holds the count of entity updates. We need to initialise this field to 0 on insert but increment by 1 on every corresponding update. I tried to do it as shown below, but it did not work. This statement always initialized UpdateCount to 1 on insert. This might be because 'The OnUpdate definitions apply to insertions as well'.

            await _context.ReconcileAsync(entity,  e => e.WithMany(p => p.Details)              
                    .WithMany(p => p.Instructions)
                    .OnInsertion(p => p.Created == timeNow)
                    .OnInsertion(p => p.UpdateCount == 0)
                    .OnUpdate(p => p.LastUpdated == timeNow)
                    .OnUpdate(p => p.UpdateCount == p.UpdateCount + 1));

We do not any fields in our entity that could help us know if the particular entity is being updated/inserted. So, do you have any recommendations to handle this issue?

Entity state mismatch - Causing Cannot insert explicit value for identity column

I have a one to many relationship between Services (Parent Record) -----> ServiceTranslations(Child List).

When I have more than one translations and I update one child/translation, reconciler for some reason sees this Entity.State as Added instead of modified thereby causing

Cannot insert explicit value for identity column in table 'ServiceTranslation' when IDENTITY_INSERT is set to OFF.

My code is below:
await _appDbContext.ReconcileAsync(services, e => e.WithMany(m => m.Translations));

Debug is below

I was able to track it on SaveChangesAsync() override method and tracing the entity state after reconcileAsync is executed

Foreign key violation during insert

Does Reconciler support the insertion of entities with a nesting level greater than 2? In my case, for example, I have a Master -> Detail_Level_1 -> Detail_Level_2 structure, where Detail_Level_2 has a foreign key reference to Detail_Level_1, and Detail_Level_1 has a foreign key reference to Master. When I add an entry to Detail_Level_1 and Detail_Level_2 to the dbContext, Entity Framework tries to insert the Detail_Level_2 entry first generating a foreign key violation.
Thanks.

Queries and projections

We need a way to load extents based on queries, ie.

    public static async Task<E[]> LoadExtentsAsync<E>(this DbContext db, IQueryable<E> query, Action<ExtentBuilder<E>> extent)
        where E : class
    {
        ...
    }

One of the tricky parts is that really we also need this to work when E isn't an entity (called a projection because the model is projected onto a class that's not part of the model). Projections are quite common.

This likely necessitates further builder related extensions as we probably need to tell Reconciler that E isn't an entity, ie. I'm not sure we can make the following syntax work:

db.LoadExtentsAsync(db.Stuff.Select(s => new { Stuff = s, More = ... }), e => e.WithOne(s => s.Stuff))

(we may have to use a method distinct from WithOne here)

Reconciler "Move" issue

I found 1 small issue in your library while implementing a new feature.

When you trying to move some items from the collection of one parent (EF navigation property) to the collection of another parent in the scope of the same DB Context, the Reconciler just removes moved items from the DB.

I already created a fix of this issue and unit test for it, but I can't push my branch to this Repo.
Please give me permissions, then I'll be able to push changes and create pull-request (which you will be able to review).

Thanks in advance.

Re-use the same EntityEntry in modifiers

Currently EntityEntry is requested from the context in each modifier gratuitously. This is slightly cumbersome to change as EF6 instead has a DbEntityEntry wich behaves the same, but the type difference (and the fact it's a generic type) means one probably has to wrap it to use it as a parameter in the modifiers.

This would be a somewhat sensible optimization.

A higher-level interface

We want an interface to reconciler that combines context creation with the various calls to Reconcile and other functions.

In particular, a higher-level function to save data using reconciler should

call Reconcile
call SaveChanges
decycle and blacken the attached entity graph

The advantage would be that this gives write operations against the database which are

powerful enough to cover most use cases but
simple enough to be able to run in one transaction, and to be serialized to be sent over the wire

The second advantage regarding transactions is also a plus in the light of Azure SQL Databases recommending all requests to the database using a retry policy, and arbitrary combinations of database writes with contexts spread over long code paths isn't easy to retry.

Allow contexts without parameterless constructors

Unfortunately, this requirement snuck in.

Refactor the recursion steps to make the modifying step synchronous

The modifying step of the reconciling recursion could be synchronous, which would make some ugly .Waits go away and one could also make this step public.

Dropping the Key Consistency Requirement

It may be possible to drop the Key Consistency Requirement, at least in EF Core, by simply adding the template entity to a fresh context and call DetectChanges. EF Core does immediately fix foreign key properties in that moment and one could reconcile against those modified templates.

EFCore : Unable to add more than one child at a time if child's PK relies on it being autoincremented by database

First, I wanted to thank you for making this library, I really missed GraphDiff when I moved to dotnet core and appreciate your work on this.

I may be doing something wrong (as I am just trying to replicate what I do in GraphDiff), but I noticed when I try to add more than one child to to a parent in a single update I get an error indicating that the instance of what I am updating is already tracked with the same key value. This is due to me submitting two children with the same PK of 0 because the PK of the objects do not get set until they are written to the database.

This may be related to one of the items in already your ideas.md file (in which case apologies for being annoying).

Anyway, thanks again for all your work on this.

Foreign key conflict

The INSERT statement conflicted with the FOREIGN KEY constraint "FK_EventTypeContents_EventTypes_EventTypeId". The conflict occurred in database "knottfar", table.

    public class EventType
    {
        [Key]
        public int EventTypeId { get; set; }

        public string Images { get; set; }
        public DateTime EntryDate { get; set; }

        public Customer EnteredBy { get; set; }

        public List<EventTypeContent> Translations { get; set; }

        public EventType()
        {
            EntryDate = MyDateTime.Now();
            Translations = new List<EventTypeContent>();
        }
    }

    public class EventTypeContent
    {
        [Key]
        public int EventTypeContentId { get; set; }

        public string Title { get; set; }
        public string Description { get; set; }
        public string Language { get; set; }
        public DateTime EntryDate { get; set; }

        public Customer EnteredBy { get; set; }

        public int EventTypeId { get; set; }
        public EventType EventType { get; set; }
    }

EventType reconciled = await _appDbContext.ReconcileAsync(eventType, e => e.WithMany(m => m.Translations));

Everything is fine when I make changes to the one/many translations or delete a collection. But when I add a new translation to the collection. hell breaks loose.

EF Core 3.0 Preview 3 and later: Scheduling of deletion now cascades before SaveChanges

As far as I can tell new data is not inserted when using EF Core 3.0 Preview 8.

I cloned the package and all the tests passed.

I then upgraded the EF Core library and tests to .net standard 2.1 preview 8 and all the EF core dependencies to 3.0 preview 8. Now two of the tests fail TestRemoveAndAddOne and TestRemoveAndAddOneWithNestedStuffAttached.

As much as I'd love to fix it and make a pull request I'm not sure I'm up to the task. I've found the library very useful and I'm amazed at the low line count but I'm not finding it easy to see what's wrong.