Code Monkey home page Code Monkey logo

akka.persistence.extras's Introduction

Akka.Persistence.Extras

This package contains some additions to Akka.Persistence that many users commonly find to be useful.

You can read more about Akka.Persistence.Extras at: https://devops.petabridge.com/articles/msgdelivery/

You can install these tools via the Akka.Persistence.Extras NuGet Package

PS> Install-Package Akka.Persistence.Extras

AtLeastOnceDeliveryActorV2

The AtLeastOnceDeliveryActor base type in its current form is a total non-pleasure to use in production, for the following reasons:

  1. Messages that are queued for reliable delivery via the Deliver method aren't automatically persisted, therefore the actor can't fulfill its delviery guarantees unless the user manually writes code for saving the AtLeastOnceDeliverySnapshot objects;
  2. It's not clear when or how often an AtLeastOnceDeliverySnapshot should be saved - there are no clear guidelines for this and actors with a large number of outstanding messages for delivery can pay massive performance penalties, having to save their entire delivery state at once all the time for each incremental change;
  3. Saving the AtLeastOnceDeliverySnapshot object requires either continuously rewriting one snapshot object without updating its sequence number (bad practice) or some awkard combination with using the normal Persist methods BUT without those Persist methods doing anything that can interfere with the contiguous state of the delivery snapshot - TL;DR; Persist can only really be used for things not directly related to delivery state;
  4. In general, this class requires the end user to memorize a significant amount of boilerplate when it comes to actually using it correctly.

The goal of this project is to rewrite the AtLeastOnceDeliveryActor to do the following:

  1. Automatically persist all messages queued for delivery via the akka.persistence.journal - all delivery state is saved atomically at the time it is saved, rather than through continuously overwriting snapshots.
  2. Periodically take snapshots and cleanup the journal in the background, as delivery state is churned. Just do this automatically.
  3. Automatically recover all delivery state at startup.
  4. Don't intend for the user to Persist or Snapshot anything else inside this actor. Delivery state only.

We believe this will make the AtLeastOnceDeliveryActor more performant (smaller serialization / write-size impact), robust (persisted at all checkpoints), and easier to use ("it just works".)

DeDuplicatingReceiverActor

This actor is on the receiving side of an AtLeastOnceDelivery actor, and it is responsible for using a persistent strategy to ensure that messages are never processed more than once.

ExactlyOnceStronglyOrderedDeliveryActor

This actor is responsible for guaranteeing an explicit message delivery order to each individually address recipient. Messages will never be delivered in any order other than FIFO using this actor - the first un-acknowledged message MUST be delivered and ACKed before the next undelivered message will be sent.

Building this solution

To run the build script associated with this solution, execute the following:

Windows

c:\> build.cmd all

Linux / OS X

c:\> build.sh all

If you need any information on the supported commands, please execute the build.[cmd|sh] help command.

This build script is powered by FAKE; please see their API documentation should you need to make any changes to the build.fsx file.

Conventions

The attached build script will automatically do the following based on the conventions of the project names added to this project:

  • Any project name ending with .Tests will automatically be treated as a XUnit2 project and will be included during the test stages of this build script;
  • Any project name ending with .Tests will automatically be treated as a NBench project and will be included during the test stages of this build script; and
  • Any project meeting neither of these conventions will be treated as a NuGet packaging target and its .nupkg file will automatically be placed in the bin\nuget folder upon running the build.[cmd|sh] all command.

DocFx for Documentation

This solution also supports DocFx for generating both API documentation and articles to describe the behavior, output, and usages of your project.

All of the relevant articles you wish to write should be added to the /docs/articles/ folder and any API documentation you might need will also appear there.

All of the documentation will be statically generated and the output will be placed in the /docs/_site/ folder.

Previewing Documentation

To preview the documentation for this project, execute the following command at the root of this folder:

C:\> serve-docs.cmd

This will use the built-in docfx.console binary that is installed as part of the NuGet restore process from executing any of the usual build.cmd or build.sh steps to preview the fully-rendered documentation. For best results, do this immediately after calling build.cmd buildRelease.

Release Notes, Version Numbers, Etc

This project will automatically populate its release notes in all of its modules via the entries written inside RELEASE_NOTES.md and will automatically update the versions of all assemblies and NuGet packages via the metadata included inside common.props.

If you add any new projects to the solution created with this template, be sure to add the following line to each one of them in order to ensure that you can take advantage of common.props for standardization purposes:

<Import Project="..\common.props" />

Code Signing via SignService

This project uses SignService to code-sign NuGet packages prior to publication. The build.cmd and build.sh scripts will automatically download the SignClient needed to execute code signing locally on the build agent, but it's still your responsibility to set up the SignService server per the instructions at the linked repository.

Once you've gone through the ropes of setting up a code-signing server, you'll need to set a few configuration options in your project in order to use the SignClient:

  • Add your Active Directory settings to appsettings.json and
  • Pass in your signature information to the signingName, signingDescription, and signingUrl values inside build.fsx.

Whenever you're ready to run code-signing on the NuGet packages published by build.fsx, execute the following command:

C:\> build.cmd nuget SignClientSecret={your secret} SignClientUser={your username}

This will invoke the SignClient and actually execute code signing against your .nupkg files prior to NuGet publication.

If one of these two values isn't provided, the code signing stage will skip itself and simply produce unsigned NuGet code packages.

akka.persistence.extras's People

Contributors

aaronontheweb avatar arkatufus avatar dependabot-preview[bot] avatar dependabot[bot] avatar igorfedchenko avatar mrbenzu avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar

akka.persistence.extras's Issues

Akka.Persistence.TestKit implementation

Akka.Persistence.TestKit Requirements

This is an outline for an "Akka.Persistence.TestKit" NuGet package that can be used by Akka.NET developers to help test for possible errors that may come up in the course of using Akka.Persistence, such as:

  1. Rejected messages due to serialization errors;
  2. Write failures due to the journal or snapshot store being offline; and
  3. Recovery errors as a result of serialization or connectivity issues.

These fake journal and snapshot store implementations should be programmable, so if we take a typical Arrange-Act-Assert pattern and factor our "FailingJournal" or "FailingSnapshotStore" into it:

public class MyTestKitClass : Akka.TestKit.Xunit2.Testkit{
	private readonly FailingJournal _failingJournal

	public MyTestKitClass() 
		: base(FailingJournal.DefaultConfig()) // need FailingJournal HOCON 
	{ 
		// need to grab an instance of the FailingJournal for programming purposes
		_failingJournal = PersistenceExt.Get(Sys);
	}

	public async Task MyFakeUnitTest(){
		// arrange
		var myPersistentActor = Sys.ActorOf(Props.Create(() => new MyPersistActor()), "p");
		myPersistentActor.Tell("write"); // write one message to journal while it's still working
		ExpectMsg("ack"); // message gets processed

		// terminate actor
		await myPersistentActor.GracefulStop();		

		// act

		await _failingJournal.BeginFailingWrites(); // simulate journal network disconnect		

		// assert
			
		// re-create a persistent actor - should fail
		EventFilter.Exception<RecoveryException>().ExpectOne(() => {
			var myPersistentActor2 = Sys.ActorOf(Props.Create(() => new MyPersistActor()), "p");
			
			// actor should stop as a result of recovery failure
			Watch(myPersistentActor);
			ExpectTerminated(myPersistentActor);
		});
	}
}

That's the general idea - make it easy to progam the behavior the journal and shapshot store in Akka.Persistence prior to actually using it.

Lexicon and Terminology

To make it easier to understand the requirements going forward, here are some technical terms we're going to use throuhout this document:

  • FailingJournal - this is the journal that can be programmed to fail altogether or reject persistent messages.
  • FailingSnapshotStore - same idea as the FailingJournal, but for the SnapshotStore.
  • "Rejected message" - this means that the connection to the underlying journal is intact, but a specific message was rejected because it couldn't be serialized.
  • "Failed message" - this means the message couldn't be written to the journal because the underlying datastore wasn't available due to a network partition.
  • "Rejected snapshot" - same idea as "rejected message," but for the snapshot store.
  • "Failed snapshot" - same idea as the "failed message," but for the snapshot store. Should trigger a SnapshotWriteFailure message.

Functional Requirements

The Akka.Persistence.TestKit package might eventually be released as an official part of Akka.NET, but for the time being, let's release the package as part of Akka.Persistence.Extras so we can avoid the inertia ivolved in making a new Akka.NET release as we're trying to get Akka.NET v1.4.0 shipped right now.

For the time being then, this package should be shipped as Akka.Persistence.Extras.TestKit.

This project should be built against Akka.Persistence v1.3.13.

FailingJournal and FailingSnapshotStore APIs

The FailingJournal and FailingSnapshotStore classes APIs must enable the end-user to do the following:

  1. Toggle the journal/snapshot store's entire availability on and off - writes and recovers will both be rejected. If this method returns a Task, by the time that Task completes the toggle operation must have completed.
  2. Toggle only the read or the write side of the journal/snapshot store's availability on and off - writes can be rejected but reads will succeed. If this method returns a Task, by the time that Task completes the toggle operation must have completed.
  3. Set a filter on the journal/snapshot store to reject or fail messages (this should be a method parameter) on write or reead (should be a parameter) only if they match a certain predicate function specified by the end-user. That predicate will remain in-place on the journal or snapshot store until reset by the user. If this method returns a Task, by the time that Task completes the predicate must be securely in-place.
  4. Unset all filters on the journal / snapshot store, if any were previously set by the end-user. If this method returns a Task, by the time that Task completes the toggle operation must have completed.

Enabling FailingJournal and FailingSnapshotStore inside an Akka.NET TestKit Test

The FailingJournal and the FailingSnapshotStore should wrap around the underlying MemoryJournal and MemorySnapshotStore and use them for underlying storage.

But in order to activate either one of these components inside a TestKit test, the end-user must pass in the embedded HOCON built into the FailingJournal.DefaultConfig() or FailingSnapshotStore.DefaultConfig() methods into the TestKit base class constructor, like this:

public class MyTestKitClass : Akka.TestKit.Xunit2.Testkit{
	private readonly FailingJournal _failingJournal

	public MyTestKitClass() 
		: base(FailingJournal.DefaultConfig()) // need FailingJournal HOCON 
	{ 
		// need to grab an instance of the FailingJournal for programming purposes
		_failingJournal = PersistenceExt.Get(Sys);
	}

	// tests
}

Testing Requirements

Need to cover all functional requirements with unit tests, in order to make sure that the API requirements are all met by the implementation. Expand as needed along with the API surface area.

System.MissingMethodException after upgrading to Akka.NET 1.5

Apparently Akka.Persistence.Exras needs to be rebuilt with new Akka libraries. Running them with Akka.NET 1.5 causes the following error:

Method not found: 'Void Akka.Event.ILoggingAdapter.Error(System.String, System.Object[])'.

at Akka.Persistence.Extras.PersistenceSupervisor.OnTerminated(Object message)
at Akka.Persistence.Extras.PersistenceSupervisor.Receive(Object message)
at Akka.Actor.ActorBase.AroundReceive(Receive receive, Object message)
at Akka.Actor.ActorCell.ReceiveMessage(Object message)
at Akka.Actor.ActorCell.Invoke(Envelope envelope)

This is a showstopper for upgrading to Akka.NET 1.5 for those using PersistenceSupervisor.

Rewrite AtLeastOnceDeliverySemantic

Major changes needed:

  • Remove ImmutableDictionary<T> in favor of Dictionary<T> - if the semantic isn't accessed concurrently, then it's not needed.
  • Add ability to add single UnconfirmedDelivery messages to the AtLeastOnceDelivery state incrementally, so these items can be retrieved via Recover<UnconfirmedDelivery>.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.