Code Monkey home page Code Monkey logo

sourced's Introduction

Build Status

REPO (TEMPORARILY?) MOVED: Currently being maintained at https://github.com/cloudnativeentrepreneur/sourced - publishing to same npm package!

sourced

Tiny framework for building models with the event sourcing pattern (events and snapshots). Unlike Active Record where entity state is persisted on a one-model-per row database format, event sourcing stores all the changes (events) to the entity, rather than just its current state. The current state is derived by loading all events, or a latest snapshot plus subsequent events, and replaying them against the entity.

One large benefit of event sourcing: your data is your audit trail. Zero discrepancies.

For example usage, see the examples and tests.

Sourced makes no assumptions about how you store your events and snapshots. The library is small and tight with only the required functionality to define entities and their logic, enqueue and emit events, and track event state to later be persisted. To actually persist, use one of the following libraries or implement your own:

ES6 Example

const Entity = require('sourced').SourcedEntity;

class Market extends Entity {
  constructor(snapshot, events) {
    super()
    this.orders = [];
    this.price = 0;

    this.rehydrate(snapshot, events)
  }

  init(param) {
    this.id = param.id;
    this.digest('init', param);
    this.emit('initialized', param, this);
  }

  createOrder(param) {
    this.orders.push(param);
    var total = 0;
    this.orders.forEach(function (order) {
      total += order.price;
    });
    this.price = total / this.orders.length;
    this.digest('createOrder', param);
    this.emit('done', param, this);
  };
}

Reference

Classes

Entity

{Function} Entity

EntityError

{Function} EntityError

Entity

{Function} Entity

Kind: global class Requires: module:events, module:debug, module:util, module:lodash License: MIT

new Entity([snapshot], [events])

Creates an event-sourced Entity.

Param Type Description
[snapshot] Object A previously saved snapshot of an entity.
[events] Array An array of events to apply on instantiation.

entity.emit()

Wrapper around the EventEmitter.emit method that adds a condition so events are not fired during replay.

Kind: instance method of Entity

entity.enqueue()

Add events to the queue of events to emit. If called during replay, this method does nothing.

Kind: instance method of Entity

entity.digest(method, data)

Digest a command with given data.This is called whenever you want to record a command into the events for the entity. If called during replay, this method does nothing.

Kind: instance method of Entity

Param Type Description
method String the name of the method/command you want to digest.
data Object the data that should be passed to the replay.

entity.merge(snapshot)

Merge a snapshot onto the entity.

For every property passed in the snapshot, the value is deep-cloned and then merged into the instance through mergeProperty. See mergeProperty for details.

Kind: instance method of Entity See: Entity.prototype.mergeProperty

Param Type Description
snapshot Object snapshot object.

entity.mergeProperty(name, value)

Merge a property onto the instance.

Given a name and a value, mergeProperty checks first attempt to find the property in the mergeProperties map using the constructor name as key. If it is found and it is a function, the function is called. If it is NOT found we check if the property is an object. If so, we merge. If not, we simply assign the passed value to the instance.

Kind: instance method of Entity See

  • mergeProperties
  • Entity.mergeProperty
Param Type Description
name String the name of the property being merged.
value Object the value of the property being merged.

entity.replay(events)

Replay an array of events onto the instance.

The goal here is to allow application of events without emitting, enqueueing nor digesting the replayed events. This is done by setting this.replaying to true which emit, enqueue and digest check for.

If the method in the event being replayed exists in the instance, we call the mathod with the data in the event and set the version of the instance to the version of the event. If the method is not found, we attempt to parse the constructor to give a more descriptive error.

Kind: instance method of Entity

Param Type Description
events Array an array of events to be replayed.

entity.snapshot() ⇒ Object

Create a snapshot of the current state of the entity instance.

Here the instance's snapshotVersion property is set to the current version, then the instance is deep-cloned and the clone is trimmed of the internal sourced attributes using trimSnapshot and returned.

Kind: instance method of Entity

entity.trimSnapshot(snapshot)

Remove the internal sourced properties from the passed snapshot.

Snapshots are to contain only entity data properties. This trims all other properties from the snapshot.

Kind: instance method of Entity See: Entity.prototype.snapshot

Param Type Description
snapshot Object the snapshot to be trimmed.

Entity.digestMethod(type, fn)

Helper function to automatically create a method that calls digest on the param provided. Use it to add methods that automatically call digest.

Kind: static method of Entity

Param Type Description
type Object the entity class to which the method will be added.
fn function the actual function to be added.

Example

Entity.digestMethod(Car, function clearSettings (param) {

    const self = this;

    this.settings.get(param.name).forEach((name, config) => {

      config.sources.forEach((source) => {

        source.remove();

      });

    });

    return this.settings;

   });

Entity.mergeProperty(type, name, fn)

Convenience function to store references to functions that should be run when mergin a particular property.

Kind: static method of Entity See: mergeProperties

Param Type Description
type Object the entity class to which the property->fn belongs to.
name String the name of the property that holds the fn.
fn function the function to execute when merging the property.

Example

function Wheel (status) {
   this.status = status;
 }

 Wheel.prototype.go = function () {
   this.status = 'going';
 }

 function Car () {
   this.id = null;
   this.wheel = new Wheel(); // for instantiating our default wheel, when we first 'new' up a Car

   Entity.apply(this, arguments);
 }

 util.inherits(Car, Entity);

 Entity.mergeProperty(Car, 'interests', function (obj) {
   this.wheel = new Wheel(); // for instantiating our wheel from saved values in a database
 });

eventsToEmit : Array

[Description]

Kind: global variable Todo

  • discuss the use of this so it can be documented better.

newEvents : Array

[Description]

Kind: global variable Todo

  • discuss the use of this so it can be documented better.

replaying : Boolean

Boolean to prevent emit, enqueue and digest from running during replay.

Kind: global variable

snapshotVersion : Number

Holds the version of the latest snapshot for the entity.

Kind: global variable

timestamp : Number

Holds the event's timestamp in the entity.

Kind: global variable

version : Number

Holds the current version of the entity.

Kind: global variable

mergeProperties

mergeProperties holds a map of entity types to properties.

Kind: global variable See

  • Entity.mergeProperty
  • Entity.prototype.mergeProperty

EntityError

{Function} EntityError

Kind: global class

new EntityError(msg, [constr])

Extending native Error.

Param Type Default Description
msg String The error message.
[constr] Object this The constructor or instance.

sourced's People

Contributors

chrisabrams avatar lgomez avatar mateodelnorte avatar patrickleet avatar renovate-bot avatar renovate[bot] avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

sourced's Issues

How to make `id` more clear.

After fiddling a bit, I've realised that the id property on an Entity is more or less a required property when it comes to integrating it with a Repository. That's not altogether obvious from the outset.

Perhaps it would be worth adding a setId and getId method to Entity. A Repository can then replace entity.id with entity.getId() and now it's using an interface rather than being tightly coupled to a naming convention.

Thoughts?

How events work with a Service Bus.

Hi @mateodelnorte

I'm just wondering how Entity events work with a service bus? Currently I'm doing something like this for a File entity:

    serviceBus.listen('do.file.create', { ack: true }, (event: any) => {
        // TODO what happens if file already exists?
        let fileEntity = new FileEntity();

        fileEntity.create(event);

        fileRepository.commit(fileEntity, (err: Error) => {
            if (err) {
                return event.handle.reject(err);
            }

            event.handle.ack();
            serviceBus.send('on.file.created', event);
        });
    });

I'm not having the create method emit anything at this stage. Is this a reasonable approach?

Secondly, what happens if the Entity method is async, either taking a callback or returning a Promise? For example, I'm starting to build a transform method which resizes the file (in the case of an image) to maximum dimensions. It's possible that method could error, and I'd ordinarily have a method like that return a promise.

Thanks in advance.

Typedef for Typescript

Hi Matt. Just want to let you know I've done a preliminary pass on a typedef for Typescript here. Are you interested in linking that in the README for sourced (if not, that's perfectly ok)? If so, I'm happy to rig up a PR.

Missing License file.

Hi I been watching/following this project recently, and I was wondering what license this project have. Since the npmjs.com repository shows that is a BSD type license, but no written reference is found in the source code.

Regards.

Best way to handle validation?

Hi Matt.

Just want to confirm the best way to handle validation. Would you do something like this:

export class Customer exends Entity {
  public create(props: any) {
    if (!props.id) {
      let err = new Error('Customer: <id> required.');
      this.emit('error', err);
      throw err;
    }
    // ... enqueue and emit success event
  }
}

Thanks in advance.

replaying

I am confused by the need for this.replaying. Isn't the entity.replay(events) a synchronous call or am I missing something?

Catch-all events

Hi Matt,

I was trying out your library, works great. I like the simplicity!

However I wan't to catch all events using one single event handler, but it seems it currently only supports catching events on instances of Entities. Is this by design?

Many thanks,

Joey

Entity constructor arity during deserialization

If you have an Entity that has not handled 10 or more events before committing it will not be automatically snapshot with sourced-repos-mongo. When that entity is retrieved from the repository and deserialised no snapshots will be available.

function Entity (/*snapshot, evnts*/) {
  this.eventsToEmit = [];
  this.newEvents = []; 
  this.replaying = false;
  this.snapshotVersion = 0;
  this.timestamp = Date.now();
  this.version = 0;
  events.EventEmitter.call(this);
  var args = Array.prototype.slice.call(arguments);
  if (args[0]){
    var snapshot = args[0];
    this.merge(snapshot);  
  }
  if (args[1]){
    var evnts = args[1];
    this.replay(evnts);
  }
}

As a result the entity will be created using merge() rather than replay() but it will only be called with the first event in the stream.

The arity of the Entity() method should be checked to define with method should be used against the argument.

Merge of snapshot with Entity null properties fails

We ran into a problem when properties of sourced entities are declared and initialized with null.

function TestNullPropertyEntity () {
  this.property = null;
  Entity.call(this);
}

util.inherits(TestNullPropertyEntity, Entity);

This test fails:

it('should merge a complex snapshot when entity is initialized with null properties', function () {

  var snapshot = {
    property: { subProperty: true },
  };

  var test = new TestNullPropertyEntity();

  test.merge(snapshot);

  test.should.have.property('property');
  test.property.should.have.property('subProperty', true);

});

The automated release is failing 🚨

🚨 The automated release from the master branch failed. 🚨

I recommend you give this issue a high priority, so other packages depending on you could benefit from your bug fixes and new features.

You can find below the list of errors reported by semantic-release. Each one of them has to be resolved in order to automatically publish your package. I’m sure you can resolve this 💪.

Errors are usually caused by a misconfiguration or an authentication problem. With each error reported below you will find explanation and guidance to help you to resolve it.

Once all the errors are resolved, semantic-release will release your package the next time you push a commit to the master branch. You can also manually restart the failed CI job that runs semantic-release.

If you are not sure how to resolve this, here is some links that can help you:

If those don’t help, or if this issue is reporting something you think isn’t right, you can always ask the humans behind semantic-release.


The push permission to the Git repository is required.

semantic-release cannot push the version tag to the branch master on remote Git repository with URL http://[secure]@github.com/:mateodelnorte/sourced.git.

Please refer to the authentication configuration documentation to configure the Git credentials on your CI environment and make sure the repositoryUrl is configured with a valid Git URL.


Good luck with your project ✨

Your semantic-release bot 📦🚀

What to do about circular references within entities?

What will happen with entity.snapshot() is called on an entity that contains circular references within its properties? I believe clone will currently place a [Circular] placemark there instead. Will need to see what exactly is created and how that should affect de-serialization.

replay / emit

As replay doesn't emit by default, how can I replay including emit so I can update the read model?

The scenarios I'm trying to grasp are as follows:

  1. My subscriber v1 which created the read model from an event has a bug. I fixed that bug but I need the events again to fix the read model.
  2. I add a new subscriber to create another read model and I want to build the read model using existing events. How would you emit again without creating existing read models again (overwrite them)?

Dependency Dashboard

This issue provides visibility into Renovate updates and their statuses. Learn more

Rate Limited

These updates are currently rate limited. Click on a checkbox below to force their creation now.

  • chore(deps): update dependency eslint to v8

Other Branches

These updates are pending. To force PRs open, click the checkbox below.

  • chore(deps): update dependency set-value to 4.0.1 [security]

Open

These updates have all been created already. Click a checkbox below to force a retry/rebase of any.


  • Check this box to trigger a request for Renovate to run again on this repository

How to handle Entity meta?

Hi Matt. How do you normally handle entity meta like created date, last updated date, created by (user id), updated by etc?

Race condition in constructor when extending an Entity

I'm running into a bit of a roadblock when extending my Entity. It's compounded by TypeScript, but not unique to it. The problem is when extending object, TypeScript requires that the super() call (equivalent to Entity.apply(this, arguments);) comes as the first line in the constructor function. It's not possible to move the invocation such as can be found in the example.

My musing has been store the data for the entity on a state property, the TypeScript will generate this kind of code:

    function EmailTemplateEntity() {
        _super.apply(this, arguments);
        this.state = new EmailTemplate();
    }

The issue is that when integrated with sourced-repo-mongo, the Entity constructor function is doing a lot of work before it even gets to my extended constructor. Hence, my this.state is not correctly initialised before the event methods in the entity are called.

Are you open to a fix for this (I generally try to avoid doing too much in constructors for this very reason)? The fix would require removing the merging of snapshots and replaying from the Entity constructor and then changing Repository.prototype._deserialize to invoke Entity.merge and Entity.replay explicitly.

Thanks in advance.

Unhandled "Channel ended" rejection in tests

Hi Matt.

This may not be you, but I'm trying to work out where the following error is coming from in my test suite:

Potentially unhandled rejection [6] Error: Channel ended, no reply will be forthcoming

When I'm testing one file using the Service Bus as a dependency, I don't see the problem, for example the code is:

export function registerWorkflow(serviceBus: Bus, wampClient: WampClient) {
    serviceBus.listen('on.file.created', { ack: true }, (event: any) => {
        event.handle.ack();
        serviceBus.send('do.file.validate', event);
    });

    serviceBus.listen('on.file.validated', { ack: true }, (event: any) => {
        event.handle.ack();
        serviceBus.send('do.file.transform', event);
    });

    serviceBus.listen('on.file.transformed', { ack: true }, (event: any) => {
        // TODO ack should really occur after message published?
        event.handle.ack();
        wampClient.publish('on.file.ready.' + event.createdBy, [event.name], {}, {
            exclude_me: false,
            acknowledge: true
        });
    });
}

But when I run the full test suite, I get that unhandled rejection.

Before each test I do:

export function createServiceBus(config: Config): Bus {
    let serviceBus: Bus = bus(config.get('amqp'));

    serviceBus.use(retry({
        store: retry.MemoryStore()
    }));

    return serviceBus;
}

and then after each test I do:

export function closeServiceBus(serviceBus: Bus): Promise<void> {
    return new Promise<void>((resolve) => {
        setTimeout(() => {
            serviceBus.close();
            resolve();
        }, 10)

    });
}

I added the timeout because the Service Bus still seems to be doing things after my test is finished.

I'm guessing I'm closing the connection and there is a listener still working. Assuming I've given you enough information, is there any way to close down my tests more elegantly?

Thanks in advance.

Question - Inter-process events

Hello, I got to know this project through the amazing presentation given by Stefan Kutko about how they've build a solution with Event Sourcing using sourced.

I'm willing to use sourced too, but I have a question: As sourced uses EventEmitter to emit the queued events, what would be the best practice to propagate these events to other microservices? Just listen for the EventEmitter event and then re-emit it using pub/sub?

Thank you

Auth example not working on Node v 8.11.2

node examples/auth/example.js
Results in the following with Node v 8.11.2:

matt was provisioned with pw saltypass
matt was granted access to app with id 1
matt was granted access to app with id 2
matt was revoked access to app with id 1
user matt has been modified with the following operations: [{"method":"provision","data":{"username":"matt","pass":"saltypass"},"timestamp":1531567220826,"version":1},{"method":"grant","data":{"name":"sales","appId":1},"timestamp":1531567220827,"version":2},{"method":"grant","data":{"name":"marketing","appId":2},"timestamp":1531567220827,"version":3},{"method":"revoke","data":{"name":"sales","appId":1},"timestamp":1531567220827,"version":4}]
we could persist him with the following snapshot: {"apps":{"2":{"name":"marketing","appId":2}},"username":"matt","pass":"","password":"saltypass","snapshotVersion":4,"timestamp":1531567220827,"version":4,"_eventsCount":3}
/Users/dhayes/Development/Projects/sandbox/sourced/lib/entity.js:165
      this.mergeProperty(property, val);
           ^

TypeError: this.mergeProperty is not a function
    at User.merge (/Users/dhayes/Development/Projects/sandbox/sourced/lib/entity.js:165:12)
    at User.Entity (/Users/dhayes/Development/Projects/sandbox/sourced/lib/entity.js:92:10)
    at new User (/Users/dhayes/Development/Projects/sandbox/sourced/examples/auth/user.js:10:10)
    at Object.<anonymous> (/Users/dhayes/Development/Projects/sandbox/sourced/examples/auth/example.js:26:12)
    at Module._compile (module.js:652:30)
    at Object.Module._extensions..js (module.js:663:10)
    at Module.load (module.js:565:32)
    at tryModuleLoad (module.js:505:12)
    at Function.Module._load (module.js:497:3)
    at Function.Module.runMain (module.js:693:10)


This is a fresh pull from master. 

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.