Code Monkey home page Code Monkey logo

lagom-pb's Introduction

lagom-pb = lagom + protocol buffer

Build Status Codacy Badge License Maven Central Snapshot Artifacts Join the chat at https://gitter.im/super-flat/lagom-pb

Scala Developer velocity in lagom development using protocol buffer.

This library helps write lagom microservices easily by making use of protocol buffer messages to define the es/cqrs core components like api requests/responses, grpc services, events, commands and state.

Features

  • Implementation of an HTTP/Json based microservice using REST interfaces by defining api requests and responses as protobuf messages.

  • Implementation of a gRPC based microservice using protocol buffer messages. More info: gRPC.

  • ReadSide in-built battery via (Akka Projection).

  • Easy definition of aggregate root, events and command handlers.

  • Pure testable functions for events and commands handlers.

  • Easy definition of api service descriptors.

  • Easy implementation of api service either with message broker api or without.

  • Metadata adds some revision number that can help easily implement optimistic lock.

  • At every event handled a snapshot of the aggregate state with the metadata are made available for the readSide.

  • All events, snapshots and readSide offsets are persisted to Postgres SQL.

  • Encryption trait to enable events and snapshots encryption.

Documentation

Documentation is available at lagom-pb wiki at the moment.

Sample Project

There is a demo application built on top the library that can be found here Sample

License

This software is licensed under the Apache 2 license, quoted below.

Copyright © 2020 superflat

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

[http://www.apache.org/licenses/LICENSE-2.0] Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an " AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

lagom-pb's People

Contributors

codacy-badger avatar kevinsamoei avatar scala-steward avatar tochemey avatar zenyui avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar

lagom-pb's Issues

Protocol Buffer annotation for basic validation

Is your feature request related to a problem? Please describe.
Provision of a data validation at the service implementation

Describe the solution you'd like
Since we are making use of protocol buffer throughout in lagom-pb, we need to provide a protocol buffer annotations for basic validation. The solution will include:

  • protocol buffer attributes definition for validation

  • validation engine

Describe alternatives you've considered
The other alternative is to see how best to use this plugin protoc-gen-validate

Developer Checklist

  • Does not lower code coverage
  • Docstrings on all functions
  • Compiled locally

Support pluggable and stackable request handlers

Is your feature request related to a problem? Please describe.
The problem to solve is to help developers implement some pre-request handlers that will propagate some meta info in a map that can be pass it on to the command handler whenever needed.

Change encryption to yield EventWrapper

Is your feature request related to a problem? Please describe.
After playing with the encryption adapters, I've observed that the default behavior of wrapping proto's in an EncryptedProto confuses developers, especially as the default encryptor does not actually apply encryption.

Describe the solution you'd like
Since the EventWrapper and StateWrapper internally have an Any message, I propose we implement encryption inside the command/event handlers and place the EncryptedProto inside the event and state attributes directly, as they already support Any types. The only functional change would be that the metadata is not encrypted, and this does not violate our original design anyways (meta data should not be sensitive information). This also reduces complexity as we do not need to use the Event or Snapshot adapters (yet).

Describe alternatives you've considered

  • leaving as is (functional, but confusing)
  • making the adapters optional (makes the interfaces harder to control via settings)
  • eliminating encryption all together :)

Additional context
n/a

Fix gRPC code generation

Describe the bug
At the moment due to the lates release of Akka gRPC and Lagom, the existing code generator is not compatible. So we will need to wait for the next release of play-grpc to get gRPC code generation working.

To Reproduce
No

Expected behavior
Be able to generate gRpc service definitions using the play grpc code generator.

Screenshots
No

Additional context
No

Migrate to lagom-pb-protos

Is your feature request related to a problem? Please describe.
At the moment the proto definitions are inside the repo which will make it difficult for a non scala-based project to access the proto definitions.

Describe the solution you'd like
Submodule https://github.com/super-flat/lagom-pb-protos and create a separate sbt module for the runtime.

Describe alternatives you've considered
No

Additional context
No

Sample project

Is your feature request related to a problem? Please describe.
A the moment there is no sample project that shows how to get started.

Describe the solution you'd like
Add a sample project to demonstrate how easy it is to create:

  • commands, events, state and apis proto

  • commands handler and unit test it

  • events handler and unit test it

  • aggregate root

  • REST service and unit test it

  • gRPC service

Developer Checklist

  • Does not lower code coverage
  • Docstrings on all functions
  • Compiled locally

cleanup

  • Add codecov badge

  • Cleanup readme

complement issue #9

Encryption pattern for the journal

Is your feature request related to a problem? Please describe.
Need a configurable way to encrypt events in the journal and state in the snapshot store, but offer the decrypted payload to the handler service + gRPC read sides.

Describe the solution you'd like
Features:

  • An ProtoEncryption trait that offers .encrypt and .decrypt methods
  • a well known proto for encrypted payloads with:
  • a oneOf attribute for the successfully decrypted bytes or failure
  • a way to identify the key used for the ProtoEncryption implementation
  • Specific failures that distinguish deleted keys from failures
  • configurable via hocon and reflected in at boot

Behaviors:

  • chief of state should return a gRPC failure when a command is sent in for which the entity cannot be decrypted
  • gRPC read side processor should just skip (and ACK) a message which cannot be decrypted, as it means the key was revoked

Describe alternatives you've considered

  • encrypting before sending in the proto (dangerous, harder for users)
  • using the event adapter (won't handle snapshots or in-memory messages)

Additional context
n/a

Handle exception when the event handler did blow up

Describe the bug
When the event handler throws an exception, the persistence entity is stopped and a generic error message is sent back to the caller.

To Reproduce
Steps to reproduce the behavior.

Expected behavior

  • Handle exception thrown by the event handler

  • Return a validation error message or add another error type to handle event handler errors

  • Make sure to propagate the error back to the caller

Screenshots
No

Additional context
No

Cleanup documentation

Is your feature request related to a problem? Please describe.
At the moment the documentation does not reflect the latest changes. Also It needs to be hosted for easy access.

Describe the solution you'd like
Cleanup the documentation and host it either on GitHub pages or on its own site.

Describe alternatives you've considered
No

Additional context
No

Add instrumentation and tracing via Kamon

Is your feature request related to a problem? Please describe.
The kamon dependencies are already part of lagom-pb and instrumentation can easily be turned on. However there is kamon PR that needs to be merged to help cover both gRPC and REST calls:
kamon-io/Kamon#768

Describe the solution you'd like
Once the kamon PR is released then we can complete the instrumentation integration by offering few pluggable tracers and reporters out of the box

Describe alternatives you've considered
No.

Additional context
No.

Complete documentation

Is your feature request related to a problem? Please describe.
At the moment the documentation is not exhaustive enough to help understand what Lagos-pb is about and how to get started.

Describe the solution you'd like
Add more markdown files to the paradox for:

  • Read Side implementation

  • Configuration

  • gRPC based service implementation

  • HTTP-Json based service implementation

Developer Checklist

  • Does not lower code coverage
  • Docstrings on all functions
  • Compiled locally

Move from codecov to codacy

Is your feature request related to a problem? Please describe.
Change to something new like we did from drone to travis.

Describe the solution you'd like
sbt codecov support codacy as well so the transition will be smooth.

Describe alternatives you've considered
No

Additional context
No

Developer Checklist

  • Does not lower code coverage
  • Docstrings on all functions
  • Compiled locally

implement sendCommandTyped

Is your feature request related to a problem? Please describe.
latest iteration of sendCommand uses Any messages, so users have to unpack them on their own.

Describe the solution you'd like
A version of sendCommand that unpacks into the nested generated message using the ProtosRegistry.

Describe alternatives you've considered
We could also just let the user use the protosregistry directly, but it's sortof confusing.

Additional context
No

SPI for plugins architecture

Is your feature request related to a problem? Please describe.
Implementation of plugins into lagom-pb. The goal is to help anyone who wants to implement plugin to do so easily.

One of the great advantage is the choice for someone to switch from Postgres to Cassandra as persistence storage. That should be pluggable because lagom does give room to do that.

Another category of plugins that this SPI can help build will the Akka projections integration. So one can decide in between lagoon readSide or an Akka projection plugin.

Describe the solution you'd like
Provide a Service Provider Interface that helps extends lagom-pb. Then we can have a repo for each plugin that can be easily used when needed.

Describe alternatives you've considered
Not at the moment

Additional context
No

Developer Checklist

  • Does not lower code coverage
  • Docstrings on all functions
  • Compiled locally

Add more information to the readSide event

Is your feature request related to a problem? Please describe.
At the moment the readSide handlers only return the actual event which is good. However the event envelope offered by Akka Projection provide more additional details like the persistenceId the timestamp, the sequenceNumber that can help.

Describe the solution you'd like
Expose the persistenceId and the timestamp as well as the sequenceNumber.
Create a proto message for readSide event and make exportable by any application.

Describe alternatives you've considered
No

Additional context
No.

Fix REST API first request slowness

Describe the bug
At the moment the first request of the REST Api call takes a bit longer to respond while its gRPC counter part responds swiftly. The root cause is due to the protobuf serialiser and the various proto companions that load lazily.

To Reproduce
No

Expected behavior
Swift response

Screenshots
No

Additional context
No

Make it possible to run ReadSideProcessor using local Actor

Is your feature request related to a problem? Please describe.
The current implementation only run the readside processor using the Akka sharded daemon (https://doc.akka.io/docs/akka-projection/current/running.html#running-with-sharded-daemon-process).

That can become an expensive process and point of failure when running multiple instances of the same readside processor. Also since there is already an existing akka cluster offered by lagom that can become an issue. We may have to validate this theory though

Describe the solution you'd like
Add an option to the current implementation to run it within the exisiting akka cluster offered by lagom by running it with a local actor. At that point we can spawn as many as we want instances of the same actor.

Describe alternatives you've considered
NO.

Additional context
NO

Code base overall review

Is your feature request related to a problem? Please describe.
Preparation for first release

Describe the solution you'd like
Go over the whole code base as a team and see how best we can standardise it before first release.

Developer Checklist

  • Does not lower code coverage
  • Docstrings on all functions
  • Compiled locally

Better errors handling

Is your feature request related to a problem? Please describe.
At the moment the way errors are handled does not give room for the implementor to add specific error types and handle them how he/she would like to do. So far he/she is limited to three type of errors:

  • Internal Error
  • Validation Error
  • All other error default to Internal

Not all errors are validations or internal, and it's hard for us to add other well-known errors to the existing structs.

Describe the solution you'd like
In the SendCommand trait, add a base failure handler that returns throwables, like:

def handleFailureResponse(failureResponse: FailureResponse): Throwable = {
  new Exception("dummy base exception that is not handled!")
}

... then override that in the gRPC and API base services to properly map the specific errors to the proper status code for that protocol.

Describe alternatives you've considered
No

Additional context
super-flat/lagom-pb-protos#6

docs as part of build

research how scalapb / lightbend do docs, find a way to publish paradox (likely into gh pages)

Integration of Akka projection

Is your feature request related to a problem? Please describe.
This feature will help expose a readSide api that can be easily implemented without worrying about the nitty gritty of lagom readSide and akka projection. At the moment we have the normal lagom readSide. Once these implementations are done we will offer some toggling between the Akka Projection and the Lagom ReadSide at the configuration because we cannot start both at the moment as stated here Lagom Issue 1346

Describe the solution you'd like
We will support the following integration:

  • ReadSide offset in db with Slick Integration

  • ReadSide offset in db with Jdbc Integration

  • Push State to Kafka Integration

  • Push Event to Kafka Integration

Describe alternatives you've considered
No

Additional context
No

Sbt plugin for lagompb

Is your feature request related to a problem? Please describe.
At the moment there is no plugin for lagom-pb that can help bundle the necessary dependencies and plugins needed to quickly get up to speed.

Describe the solution you'd like
Get a sbt plugin package that can be used along the lagompb-core and lagompb-readside to get all that is needed to get up to speed.

Describe alternatives you've considered
No

Additional context
No

remove aggregateStateCompanion

Is your feature request related to a problem? Please describe.
remove aggregateStateCompanion, as we've moved to the generic typed methods. this also simplifies implementing classes downstream.

Describe the solution you'd like
remove it everywhere.

Describe alternatives you've considered
n/a

Additional context
n/a

Ability to write stateless service

Is your feature request related to a problem? Please describe.
At the moment the only type of services that can be provisioned with lagom-pb are the one that need Akka persistence. Even though lagom-pb aims at ease the implementation of ES/CQRS there are some scenario when the need of a stateless service is required. Example one can implement the orchestrator pattern with such a service.

Describe the solution you'd like
Rewrite the LagompbApplication class with the required components

Describe alternatives you've considered
No

Additional context
No

proto annotations for identifying entity ID

Is your feature request related to a problem? Please describe.
we deprecated the feature to invoke sendcommand and automatically identify the entity ID. we should add it back.

Describe the solution you'd like
a well-known annotation (or configuration) that allows lagom-pb to infer the entity ID based on the field descriptor / message.

Describe alternatives you've considered
n/a

Additional context
n/a

Dependencies update

Is your feature request related to a problem? Please describe.
No

Describe the solution you'd like

  • Update lagom to 1.6.3

  • Update Akka to 2.6.8

  • Update Akka gRPC to 1.0.1

Describe alternatives you've considered
No

Additional context
No

Set coverage limit

Is your feature request related to a problem? Please describe.
At the moment there is no code coverage threshold that will enforce every code to meet that standard.

Describe the solution you'd like
Turn on build failure when code coverage has not met the expected percentage

Developer Checklist

  • Does not lower code coverage
  • Docstrings on all functions
  • Compiled locally

Base trait for command / event handler without proto registry

Is your feature request related to a problem? Please describe.
We have some use cases that want to handle unmarshaling the proto manually, instead of lagom-pb doing it for you in the wrapper command/event handlers. For example, the (soon to be) open source namely/chief-of-state delegates handling of specific messages to a gRPC sidecar, and does not bundle the lagom-pb service with the event or state proto classes.

Describe the solution you'd like
Changes:

  • separate a trait for generic "CommandHandler" and "EventHandler" with handle methods that accept the Any message (or even the Any byte array) instead of the unpacked GenericMessage
  • implement an abstract class for both handler traits that final def implements the handle method, uses the local proto registry to unmarshal, and passes that GeneratedMessage into a subclass implementation of handleGenericMessage

Describe alternatives you've considered
this is the least opinionated and modular way to accomplish our goals. we could not think of another way.

Additional context
chief-of-state just forwards the Any over gRPC, and today it needlessly wraps the client event in an Event proto so that the lagom-pb registry has something to unmarshal into.

Scala formatting

Describe the bug
Currently the Scala format used is not up to the required standard.

To Reproduce
No
Expected behavior
Fix formatting

Screenshots
No

Additional context
No

Fix coverage report aggregation

Describe the bug
Currently the latest commit code coverage report is not displayed due to the fact the reports are generated for each sub project. So to solve it we will need to fix the travis.yml when uploading the coverage reports.

** Screenshot

 --> Download the codacy reporter codacy-coverage-reporter-linux... ([secure])
565######################################################################## 100.0%
5662020-05-27 12:11:55.863Z  info [ConfigurationRules] API base URL: https://api.codacy.com  - (ConfigurationRules.scala:84)
5672020-05-27 12:11:55.886Z  info [ReportRules] More than one file. Considering a partial report  - (ReportRules.scala:42)
5682020-05-27 12:11:55.886Z  info [ReportRules] Parsing coverage data from: /home/travis/build/super-flat/lagom-pb/target/scala-2.13/coverage-report/cobertura.xml ...  - (ReportRules.scala:48)
5692020-05-27 12:11:55.898Z  info [ReportRules] Generated report: /home/travis/build/super-flat/lagom-pb/target/scala-2.13/coverage-report/codacy-coverage.json (1.56 kB)  - (ReportRules.scala:224)
5702020-05-27 12:11:55.898Z  info [ReportRules] Uploading coverage data...  - (ReportRules.scala:225)
5712020-05-27 12:11:56.408Z  info [ReportRules] Coverage data uploaded. Coverage received successfully.  - (ReportRules.scala:135)
5722020-05-27 12:11:56.408Z  info [ReportRules] Parsing coverage data from: /home/travis/build/super-flat/lagom-pb/core/lagompb-core/target/scala-2.13/coverage-report/cobertura.xml ...  - (ReportRules.scala:48)
5732020-05-27 12:11:56.418Z  info [ReportRules] Generated report: /home/travis/build/super-flat/lagom-pb/core/lagompb-core/target/scala-2.13/coverage-report/codacy-coverage.json (1.56 kB)  - (ReportRules.scala:224)
5742020-05-27 12:11:56.418Z  info [ReportRules] Uploading coverage data...  - (ReportRules.scala:225)
5752020-05-27 12:11:56.576Z  info [ReportRules] Coverage data uploaded. Coverage received successfully.  - (ReportRules.scala:135)
5762020-05-27 12:11:56.576Z  info [CodacyCoverageReporter] All coverage data uploaded.  - (CodacyCoverageReporter.scala:22)

Renaming Classes, Object and co

Is your feature request related to a problem? Please describe.
A the moment most of the classes in lagom-pb start with the word Lagompb

Describe the solution you'd like
The goal is to rename all classes by removing the word Lagompb

Describe alternatives you've considered
No

Additional context
No

Fix HTTP 1 tracing

Is your feature request related to a problem? Please describe.
At the moment due to the current implementation of kamon HTTP 1 tracing needs some tweak on kamon.

Describe the solution you'd like
@klvmungai has a hack that can help get instrumentation for HTTP 1 at the moment. He will be submitting a PR after some pairing with the Kamon engineer.

Describe alternatives you've considered
No

Additional context
No

Enhance coverage for read side

Is your feature request related to a problem? Please describe.
read sides need tests prior to v1.0

Describe the solution you'd like
add tests for all read side processors.

Describe alternatives you've considered
n/a

Additional context
n/a

Make ProtosRegistry a class instead of singleton

Is your feature request related to a problem? Please describe.
ProtoRegistry is a little hard to test/deal with as a singleton. We tried making it a class, but the ApiSerializer fails with the error:

Can't generate a Lagom client for io.superflat.lagompb.data.TestService since the following abstract methods don't return service calls or topics: protosRegistry

Describe the solution you'd like
somehow convert ProtoRegistry to a class with a companion to instantiate with reflection (easier to test, mock, pass around), but in a way that does not require doing a new reflection of proto classes for each serialized response.

Describe alternatives you've considered
leaving it as it is...

Additional context
n/a

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.