jirenius / go-res Goto Github PK

Is there a way to retreive the Token information within a handler?

I would like to return items in a collection based on the current user and am a bit reluctant to pass them as query parameters. Using path parameters doesn't make much sense either in my specific usecase.

(BTW, totally digging resgate. Great work!)

Issue

The package should be migrated to use Go modules.

https://blog.golang.org/migrating-to-go-modules

I would like to have an easy way of writing tests for the go-res based services that I write. From the looks of it, a lot of this is in existence already but it isn't usable from other modules due to unexported stuff. What I'm after is something similar to net/httptest, i.e. a mock service that would allow me to send request and test responses without having to start additional services.

Issue

Package should have full support for queries, query events, and groups.

Issue

Go 1.12 is released.
Update Travis job to do test build using latest stable Go 1.12 version.
Remove Go 1.9 as it is no longer supported by NATS client

src/github.com/jirenius/go-res/service.go:564:16: cannot use nc (type *nats.Conn) as type Conn in argument to s.serve:
*nats.Conn does not implement Conn (wrong type for ChanQueueSubscribe method)
have ChanQueueSubscribe(string, string, chan *nats.Msg, ...nats.SubscribeOption) (*nats.Subscription, error)
want ChanQueueSubscribe(string, string, chan *nats.Msg) (*nats.Subscription, error)
src/github.com/jirenius/go-res/service.go:585:19: impossible type assertion:
*nats.Conn does not implement Conn (wrong type for ChanQueueSubscribe method)
have ChanQueueSubscribe(string, string, chan *nats.Msg, ...nats.SubscribeOption) (*nats.Subscription, error)
want ChanQueueSubscribe(string, string, chan *nats.Msg) (*nats.Subscription, error)

I am using go get -u on the nats-io nats-server and nats.go/ with no modules file so I have what I deserve maybe. Think of it as a heads up. :-)

Issue

When creating a service with a root handler, the service will not listen for root requests.

Example

s := res.NewService("test")
s.Handle("", res.Get(func(r res.GetRequest) {
    r.NotFound()
})
s.ListenAndServe("nats://127.0.0.1:4222")

The service will only subscribe to:

get.test.>

which will not match the requests for:

get.test

Solution

Have the service subscribe to the root as well as any resource starting with root.

Issue

There is a need for a Reset method to send custom system.reset events.

Solution

Add method:

// Reset sends a system reset for the provided resource patterns.
func (s *Service) Reset(resources []string, access []string) { /* ... */ }

Issue

The resbadger middleware is experimental, with multiple features missing.

To make it complete, the following features should be added:

IndexCollection

Create collection based on an index of a model IndexSet.

var bookIdxs = &resbadger.IndexSet{
    Indexes: []resbadger.Index{
        resbadger.Index{
            Name: "idxBook_name",,
            Key: func(v interface{}) []byte {
                book := v.(Book)
                return []byte(strings.ToLower(book.Name))
            },
        },
    },
}

resbadger.BadgerDB{DB: h.DB}.
    IndexCollection().
    WithIndex(bookIdxs, "idxBook_name")

QueryCollection and IndexCollection path params

Allow QueryCollection and IndexCollection resources to have path params to segment the collections.

Example would be having library.books.$firstletter to be an IndexCollection referring to all books with a name starting with $firstletter. Eg: library.books.b

Model mapping

Models that are based on a parent model, but marshaled into a different struct.

// A simpler version of the Customer struct
type CustomerSimple struct {
    ID      string `json:"id"`
    Name    string `json:"name"`
}

s.Handle("customer.$id.simple",
    resbadger.BadgerBD{DB: db}.
        MappedModel("customer.$id").
        WithType(CustomerSimple),
)

Allow creating a mapped model resource that exposes a model but wi

QueryCollection and IndexCollection reference mapping

Allow QueryCollections and IndexCollections to reference to mapped models,

Issue

Add support for RES Protocol's static resource type.

Issue

It should be possible to assign a wildcard handler that takes any call or auth requests not mapped by any other method.

Solution

This can be done by having the string "*" indicate the wildcard handler.

res.Call("*", wildcardHandler)

Issue

It should be possible to write middleware that can be used with go-res to handle persistance of data, synchronization between multiple instances of the services, or support queries.

Solution

The res.Handler should be expanded with methods used to apply modification caused by events:

• ApplyChange
• ApplyAdd
• ApplyRemove

Each of these handlers should return information required to revert any changes made by the event, so that, in case of synchronization or query middleware, the effect of an event may be reversed.

Notes

Reverting event

Let say we have an optimistic synchronization middleware, hooking up on eg. ApplyChange. It will send any change event "upstream" to the single-point-of-truth, while at the same time sending it to NATS/Resgate.

In case upstream returns with info that the change couldn't be applied, the middleware will use the "revert" info to undo the changes. However, when doing this, it would not want to send the reverting change event upstream, only to NATS/Resgate (and perhaps any persistance middleware).

go-res needs to support this scenario

Issue

Current version does not comply fully with RES service protocol v1.1, which will be implemented with the next release of Resgate.

Solution

Add support for v1.1, which means wrapping model change event object in a values property.

Instead of model change events like this:

{
   "foo": "bar"
}

We do this:

{
   "values": { "foo": "bar" }
}

Hi!

Today go-res subscribe to NATS using ChanSubscribe and documentation states that only one service can provide a resource.

My suggestion here is to use QueueChanSubscribe using the service name as queue name, so any number of instances of go-res services can be started and NATS will automatically load balance between them.

Issue

The service's res.Conn instance should be available through a method:

func (*Service) Conn() res.Conn

Issue

The logger interface currently used was created for Resgate, and it is not really suitable for the go-res package.

A new more simpler interface is needed, easy to create wrappers to in order to use preferred logging:

// Logger is used to write log entries.
// The interface is meant to be simple to wrap for other logger implementations.
type Logger interface {
	// Infof logs service state, such as connects and reconnects to NATS.
	Infof(format string, v ...interface{})

	// Errorf logs errors in the service, or incoming messages not complying
	// with the RES protocol.
	Errorf(format string, v ...interface{})

	// Tracef all network traffic going to and from the service.
	Tracef(format string, v ...interface{})
}

Notes

This change will be backwards incompatible for anyone who has written a wrapper for the old logger interface, or who has used the exposed Service.Logf, Service.Debugf, or Service.Tracef methods.

Issue

RES Service Protocol v1.1.1 contains the following changes:

• Query resource on non-query requests
• Invalid Query error

These two changes should be supported.

Just wanted to put this here as an Issue because its such a nice match with ResGate.

https://github.com/tobgu/qocache

NATS / ResGate is keeping the Views / Cache up to date in each MicroService.

Each MicroService can retain the data in memory and they can be queried very fast by the Client.
Retaining the "DB" in memory might be OK, considering that NATS Streaming hold the Log of everything and itself is backed by a DB (Cockroach DB is my preferred one, since its got great replication).

The language is a SQL like language.
https://github.com/tobgu/qocache/blob/master/http/http_test.go#L629

Also they are looking to integrate into the Apache Arrow system, which is very high perf.

Issue

Add support for RES Protocol's data values.

Issue

A Hello World example should be added to the examples folder.

package main

import res "github.com/jirenius/go-res"

func main() {
	s := res.NewService("example")
	s.Handle("model",
		res.Access(res.AccessGranted),
		res.GetModel(func(r res.ModelRequest) {
			r.Model(map[string]string{
				"message": "Hello, World!",
			})
		}),
	)
	s.ListenAndServe("nats://localhost:4222")
}

Issue

Update Travis job to do test build using latest stable Go 1.13 version.
Remove Go 1.10.

Issue

BadgerDB has changed the API causing the badgerDB examples not to build.

These should be updated to reflect their changes.

Issue

Service method SetReset is a vague name, and doesn't match the API of other RES Service libraries.
The method should be renamed to SetOwnedResources, and the documentation comment should be improved.

Notes

For backward compatability, the previous SetReset method should remain as an alias to SetOwnedResources, but with a Deprecated warning. The method should remain until go-res API is settled, and the package is moved to the resgateio organization.

Issue

We currently have Service.OnServe which lets you add a callback function to be called when the service has started. It should be possible to register callback functions for other events, such as:

• OnDisconnect - when the service has been disconnected from NATS server.
• OnReconnect - when the service has reconnected to NATS server and sent a system reset event.
• OnError - on errors within the service, or incoming messages not complying with the RES protocol.

Notes

The OnError callback should be called together with an error that is in additionally also logged as error message.

Issue

Update Travis job to do test build using latest stable Go 1.14 version.
Remove Go 1.11.

Issue

RES Service Protocol v1.2.0 contains the following changes:

• Resource response on query request.
• Added delete resource event.
• Added create resource event.
• Added RES client version request.
• Resource response on call and auth requests.

The above changes should be supported.

Issue

A Query Search example should be added to the examples folder.

The example should be the similar to the .NET RES Service library - Search example, with the same API and client.

Notes

The example should persist changes, using the BadgerDB middleware.

The BadgerDB middleware needs to be extended to support indexing, and QueryCollections that takes a query and uses the index to retrieve the resource ID's.

Issue

It should be possible to listen to events emitted by a resource.

Example:

s := res.NewService("test")
s.Handler("customer.$id", &customerHandler)
s.AddListener("customer.$id", func(ev *res.Event) {
    fmt.Printf("Event for %s: %s\n", ev.Resource.ResourceName(), ev.Name)
    if ev.Name == "change" {
        fmt.Printf("Changed values: %+v\n", ev.NewValues)
    }
})

Notes

It should also be possible to add listeners through a Handler:

Example:

s := res.NewService("test")
s.Handler("customer.$id", &customerHandler)
s.Handler("customers",
    res.Get(handleCustomersGetRequest)
    res.Listen("customer.$id", handleCustomersEvent)
)

Hello @jirenius!

Via twitter, one of your users requested support for NGS, the global digital dial-tone for NATS. All that is required to do this is to support a new type of NATS credential in Resgate. It's build into NATS, so no new dependencies for you.

Anyhow, we'd like to help you out.

Instructions to setup a free NGS account can be found here, and then you can use the creds file created in the signup process in your connect API. For resgate, you'd only need to support passing a creds file to your NATS connection as a type of credential.

e.g.

nc, err := Connect(url, UserCreds(“~/.nkeys/synadia/accounts/ngs/users/ngs.creds”)

Let us know if this is something you're interested in, and we'd be happy to help. If you are busy we can submit a PR. Our nats-sub example also demonstrates how to do this.

We're excited to see our users ask about Resgate!

/cc @nats-io/core

Thanks,
Colin

Issue

On lint errors, the Travis build should fail.
Additional check should be added by running staticcheck.

Issue

It should be possible to use parts of the resource name when defining worker groups.

Example
Lets say we have a resource pattern, test.user.$userId, which might have sub-resources, test.user.$userId.roles.

If the data is stored and updated together, we might want only a single worker goroutine for each user and their sub-data. But in order not to have a single worker for all users, we need to be able to use the $userId part in the group ID.

Solution

Let groups contain tags, ${tagName}. The tagName must match a parameter in the resource pattern.

Example:

s.Handle("article.$id",
    res.Access(res.AccessGranted),
    res.GetModel(func(r res.ModelRequest) {
        article := getArticle(r.PathParam("id"))
        if article == nil {
            r.NotFound()
        } else {
            r.Model(article)
        }
    }),
    res.Group("group.article.${id}"),
)

It should be possible to get the Path used when creating the Mux.

mux := NewMux("this.path")
mux.Path() // Returns "this.path"

Additionally, when mounting the Mux, there should be a FullPath that returns the path including that of the parent and the mount path.

mux := NewMux("this.path")
sub := NewMux("")
mux.Mount("sub", sub)
sub.FullPath() // Returns "this.path.sub"

Issue

The service package should support making call requests and getting resources and events from other RES services connected to NATS.

// Example of what the API might be like

// Connect client to NATS
client := res.NewClient("nats://127.0.0.1")

// Sets token used in call requests
client.SetToken(nil)

 // Get resource from another 
books, err := client.Get("library.books")

// Get resource type; res.TypeModel, res.TypeCollection
typ := books.Type()

// Returns a map[string]Value. Panics if books.Type() is not res.TypeModel
m := books.Model()

// Returns a []Value. Panics if books.Type() is not res.TypeCollection
c := books.Collection()

// Unmarshals the resource into the provided interface value
var b []Book
err := books.Unmarshal(&b)

// Subscribe to events
sub := book.Subscribe(func(r res.Resource, ev res.Event) {
    // Handle event
})

// Unsubscribe to events
sub.Unsubscribe()

Issue

The Value type in the store package should support RES protocol v1.2.1 values:

• Data values
• Soft resource references

Issue

Full wild cards, that listens handles any resource with a certain prefix, should be supported.

Solution

Use the greather than ( > ) as wild card character at the end of a resource pattern:

s.Handle("data.>", handler) // Will match "data.foo", "data.foo.bar", etc.

Since wild card handling may be mixed with both Models and Collections, a generic Get handler should be introduced as well:

s.Handle("model.>", res.GetResource(func(r res.GetRequest) { /* ... */ }))

This also means that we should soften the checks for calling methods like ChangeEvent, AddEvent and RemoveEvent, to only be invalid incase it is called on a resource with an explicit (and wrong) resource type. The methods should not panic when called on resources handled by a GetRequest handler, or when no Get handler is defined.

Issue

When creating a child Mux with PathParams:

m.Handle("sub.$id", handlerOpt);

Or with groups

m.Handle("sub.$id", handlerOpt, res.Group("my${id}group"));

The idx will be set relative to the subpath, which becomes wrong when mounted to a root.
This results in the wrong PathParams and the wrong Group name.

Issue

When a resprot.SendRequest call gets the timeout extended, it should be possible to get a notification by callback.

Issue

A restest subpackage for unit testing should be added, based on the code used for tests of go-res.

The existing tests should be refactored to use the new restest package.

See discussion #68

Issue

Support for soft references should be implemented.

Notes

The feature requires Resgate supporting RES protocol > v1.2.0

Issue

It is often required for a handler to access data from an underlying database storage.

So far, each implementation has been forced to create their own ways of fetching data, and making sure the proper events are propagated from the storage layer to the handlers.

By defining a storage interface both for simple CRUD operations, as well as queries, the creation of handlers to interact with the database layer would become much simpler.

Scope

This issue both includes:

• creating the interface
• creating the generic handlers that access these interfaces
• creating badgerDB implementation for the interfaces

Interface

The interface, as suggested:

// Predefined errors
var (
    ErrNotFound  = res.ErrNotFound
    ErrDuplicate = &res.Error{Code: res.CodeInvalidParams, Message: "Duplicate resource"}
)

// Store is a CRUD interface for storing resources of a specific type. The
// resources are identified by a unique ID string.
//
// The value type returned by ReadTxn.Value, and passed to the OnChange
// callback, is determined by the Store, and remains the same for all calls. The
// Store expects that the same type is also passed as values to the
// WriteTxn.Create and WriteTxn.Update methods.
type Store interface {
    // Read will return a read transaction once there are no open write
    // transaction for the same resource. The transaction will last until Close
    // is called.
    Read(id string) ReadTxn

    // Write will return a write transaction once there are no open read or
    // write transactions for the same resource. The transaction will last until
    // Close is called.
    //
    // If the Store implementation does not support the caller generating its
    // own ID for resource creation, the implementation's Write method may
    // accept an empty ID string. In such case, any call to WriteTxn.Value,
    // WriteTxn.Update, and WriteTxn.Delete returns ErrNotFound (or an error
    // that wraps ErrNotFound), until WriteTxn.Create has been called. After
    // Create is called, the ID method should returns the new ID. If the Store
    // implementation does not support generating new IDs, a call to
    // WriteTxn.Create with an empty ID returns an error.
    Write(id string) WriteTxn

    // OnChange registers a callback that is called whenever a resource has been
    // modified. The callback parameters describes the ID of the modified
    // resource, and the value before and after modification.
    //
    // If the before-value is nil, the resource was created. If the after-value
    // is nil, the resource was deleted.
    OnChange(func(id string, before, after interface{}))
}

// ReadTxn represents a read transaction.
type ReadTxn interface {
    // ID returns the ID string of the resource.
    ID() string

    // Close closes the transaction, rendering it unusable for any subsequent
    // calls. Close will return an error if it has already been called.
    Close() error

    // Exists returns true if the value exists, or false on read error or if the
    // resource does not exist.
    Exists() bool

    // Value returns the stored value. Value returns ErrNotFound (or an error
    // that wraps ErrNotFound), if a resource with the provided ID does not
    // exist in the store.
    Value() (interface{}, error)
}

// WriteTxn represents a write transaction.
type WriteTxn interface {
    ReadTxn

    // Create adds a new value to the store.
    //
    // If a resource with the same ID already exists in the store, or if a
    // unique index is violated, Create returns ErrDuplicate (or an error that
    // wraps ErrDuplicate).
    //
    // If the value is successfully created, the Store OnChange callbacks will
    // be triggered on the calling goroutine with the before-value set to nil.
    Create(interface{}) error

    // Update replaces an existing value in the store.
    //
    // If the value does not exist, Update returns ErrNotFound (or an error that
    // wraps ErrNotFound).
    //
    // If the value is successfully updated, the Store OnChange callbacks will
    // be triggered on the calling goroutine.
    Update(interface{}) error

    // Delete deletes an existing value from the store.
    //
    // If the value does not exist, Delete returns ErrNotFound (or an error that
    // wraps ErrNotFound).
    //
    // If the value is successfully deleted, the Store OnChange callbacks will
    // be triggered on the calling goroutine with the after-value set to nil.
    Delete() error
}

// QueryStore is an interface for quering the resource in a store.
type QueryStore interface {
    // Query returns a result based on the provided query values.
    //
    // The result type is determined by the QueryStore implementation, and must
    // remain the same for all calls regardless of query values. If error is
    // non-nil the returned interface{} is nil.
    Query(query url.Values) (interface{}, error)

    // OnQueryChange registers a callback that is called whenever a change to a
    // reasource has occurred that may affect the results returned by Query.
    OnQueryChange(func(QueryChange))
}

// QueryChange represents a change to a resource that may affects queries.
type QueryChange interface {
    // ID returns the ID of the changed resource triggering the event.
    ID() string

    // Before returns the resource value before the change. The value type is
    // defined by the underlying store. If the resource was created, Before will
    // return nil.
    Before() interface{}

    // After returns the resource value after the change. The value type is
    // defined by the underlying store. If the resource was deleted, After will
    // return nil.
    After() interface{}

    // Events returns a list of events that describes mutations of the results,
    // caused by the change, for a given query.
    //
    // If the query result is a collection, where the change caused a value to
    // move position, the "remove" event should come prior to the "add" event.
    //
    // The QueryStore implementation may return zero or nil events, even if the
    // query may be affected by the change, but must then have the returned
    // reset flag set to true.
    Events(q url.Values) (events []ResultEvent, reset bool, err error)
}

// ResultEvent represents an event on a query result.
//
// See: https://resgate.io/docs/specification/res-service-protocol/#events
type ResultEvent struct {
    // Name of the event.
    Name string

    // Index position where the resource is added or removed from the query
    // result.
    //
    // Only valid for "add" and "remove" events.
    Idx int

    // ID of resource being added or removed from the query result.
    //
    // Only valid for "add" and "remove" events.
    Value interface{}

    // Changed property values for the model emitting the event.
    //
    // Only valid for "change" events.
    Changed map[string]interface{}
}

Issue

GitHub funding support should be added to the project.

Notes

Done by adding the file .github/FUNDING.yml with the content:

# These are supported funding model platforms
github: [jirenius]
custom: ["https://www.paypal.me/jirenius"]

Issue

It should be possible to run on a resource worker goroutine, using the Resource context instance, instead of using With which would require do a handle lookup.

Example

r, _ := service.Resource("test.model") // This preforms a lookup
/* ... */
service.WithResource(r, cb)

Note

This would also include adding a Group method to Resource that returns the group which the resource belongs to (or the resource name, in case no group is set).

Issue

In case a res.Handler does not contain a handler method for a given request, it should bubble up to the next best res.Handler match.

Example

s := res.Service("test")
s.Handle("model",
   res.GetModel(func(r res.ModelRequest) {
      r.Model(json.RawMessage(`{"foo":"bar"}`)
   }),
)
s.Handle(">", res.Access(res.AccessGranted))

An access request, access.test.model, should be handled by the handler for the wildcard ">" as the "model" handler has does not have an access handler method.

Issue

Incoming requests that doesn't have any payload currently gives a json unmarshal error, but should instead be handled as if it was a null.

This should be valid for all types of requests.

Notes

While Resgate always sends a json object that sometimes can be empty ({}), inter-service requests should still be allowed to send empty requests.

Issue

It should be optional if a handler uses ChanQueueSubscribe instead of ChanSubscribe.

The queue group should also be configurable, or else resolve to the resource pattern.

Discussion

This feature has been requested and discussed in issue #12

Notes

If ChanQueueSubscribe is used for a resource, mutating events (ChangeEvent, AddEvent, and RemoveEvent) should not be allowed for that resource, as it would cause races with other services belonging to the same queue group.

I think next thing will be to make a cli using cobra to test interactions.

A 2nd bash shell can have a CMD to get he subscriptions being updated.

That gets the get and set side of things working and it's easy to extend the cli.

I guess this client would be calling the resgate over http and so then calling the Microservice using NATS.
It's a good test harness I think and much quicker and testable for regressions than web stuff.

What ya think ?

Issue

The res package should have a sub package, resprot, with low level structs and methods for making requests to RES Services connected to NATS server.

The resprot package would be used for making requests and parsing responses. The package should not include logic about event handling with real-time updates, as that will be covered by the broader scope of issue #66 .

Issue

If two overlapping full wildcard handlers are registered to a Mux, GetHandler will fetch the less specific one.

Example

m := res.NewMux("")
m.Handle("model.>", h1)
m.Handle(">", h2)
m.GetHandler("model.foo") // returns h2 instead of h1

Issue

When there is a panic within a With callback, the service will currently crash. This means using RequireValue, which panics if the Get handler returns an error, will be very dangerous within a With callback.

Example

s.With("library.book.1337", func(r res.Resource) {
    // Since RequireValue will panic if the Get handler
    // returns a NotFound error, the entire service will crash.
    book := r.RequireValue.(*Book)
})

Solution

Any panic that happens within the With callback should be recovered and logged as an error.

Since it the callback is (most likely) running on a different goroutine than the caller of With, it will not be possible to return any recovered panic as an error on the With call.

Issue

Resgate and ResClient has moved from github.com/jirenius to github.com/resgateio

And link should be updated.