Motivation

Our documentation contains lots of useful information about contributing but since we use couple different repositories, we gathered required information in one place, till we get them ported to AeroGear website over time.

Introduction

AeroGear is set of the mobile focused services that help developers to write their mobile and backend software. For more general information about AeroGear please check our website:

http://aerogear.org

Voyager (this repository) is one of the services that are currently under active development. Voyager is a set of GraphQL extensions that help mobile developers to adapt GraphQL technology.

Early preview of documentation is available here:
https://deploy-preview-613--aerogear-docs-preview.netlify.com/aerogear/latest/data-sync/

Please mind that this document is still incomplete and individual repositories should have much more information.

How I can start with GraphQL/Voyager

GraphQL is a data query standard that is gaining lots of attention recently over classical REST API approach.
There are numerous tutorials related to GraphQL and Apollo GraphQL.
Best way to start is to learn the basics of GraphQL
https://graphql.org/graphql-js/

Next step is to get yourself familiar with Apollo GraphQL technology.
I personally found robinwieruch.com tutorials one of the best and most up to date resources:

For the client:
https://www.robinwieruch.de/graphql-apollo-client-tutorial/
For the server:
https://www.robinwieruch.de/graphql-apollo-server-tutorial/

Voyager Getting Started

Voyager is currently maintained in 3 different repositories:

Our Example app that contains server and client integrations for voyager and other AeroGear extensions:

https://github.com/aerogear/ionic-showcase

Server:
https://github.com/aerogear/voyager-server/

Client:
https://github.com/aerogear/aerogear-js-sdk/tree/master/packages/sync

Please check individual README.md files and CONTRIBUTING.md files for more information.

Working on the tasks

To start working on one of the ideas please leave a a comment on one issue:
https://github.com/aerogear/voyager-server/issues?q=is%3Aissue+is%3Aopen+label%3Aideas

Mentors will reach out to you and explain the details of the tasks and possible steps.

Ask us anything

If you found any information unclear feel free to ask question below.

ping @aerogear/voyager team.

Motivation

Currently our solution is using schema first aproach when users type schema.
From top graphql predictions it looks like community is moving towards resolver first aproach:
https://medium.com/graphqlconf/top-5-graphql-predictions-for-2019-6f281f16fac1

Tasks

Investigate using resolver first aproach in voyager server using one of the resolver first generators like https://github.com/graphql-nexus/nexus

We need to figure out how nexus generated schema can be supported.
Alternatively we can build generator for nexus to wrap typescript schema into the GraphQL types.

Currently project consist of the 3 different repositories that

• server (current repository)
• js-sdk for client
• ionic-showcase that contains example server and client

As result we having couple different problems:

• Server examples cannot be easily tested on the client
• Integration tests on client now will include example server
• client has different release schedule than other js-sdk packages.

All this contributes to larger number of artifacts that will need to be maintained later.

What are your thoughts on integrating client and server into mono repository?

Motivation

Our conflict package now provides basic information about the conflict.
We know the current state of the server and current state of the client.
However we cannot determine base of the both changes - thus we cannot provide exact diff to users.
This limits options that are available for viable conflict resolution engine.

Tasks

Investigate how conflict resolution can be done. Some investigation were already performed in #46
One of the option will be to reuse already existing conflict resolution package: automerge.
Automerge can give us really solid production ready conflict resolution mechanism:
https://github.com/automerge/automerge

Technical details

This requires some changes in the way we are working with GraphQL diffs.
Automerge can be reused on both client and server however we do not want to make GraphQL less flexible in order to integrate with Automerge.

Getting started

Evaluate current conflict resolution mechanism.
Check for limitations in current mechanism.

Investigate how to work with GraphQL schema with ability to send additional diff data.
This can involve:

• Using GraphQL extensions to send/receive data
• Having JSON like scalar on top of other queries that will contain diff
• Generating fields without required ! modifier
• Generating schema with diff capabilities (separate mutations)
• Extending GraphQL spec (some progress has been made)

Once solution for diffing will be made we can provide GraphQL wrapper library that will work with Automerge.

Tasks

• Implement hello graphql server
• Integrate with automerge
• Note challenges around diff
• Work on resolving this challenges using proposed options above or suggesting new one.
• Finish POC with end to end conflict resolution example

Possible inspirations

Logux: https://github.com/logux/client

• Module: apollo-voyager-metrics
• Version: all
• Node.js / npm versions: all

Right now our test coverage of apollo-voyager-metrics is really poor. We need

• To test that things like resolver timing metrics are actually exposed
• That the middleware methods behave the way we want

There is an error with this repository's Renovate configuration that needs to be fixed. As a precaution, Renovate will stop PRs until it is resolved.

File: renovate.json
Error type: Duplicate keys in JSON
Message: "Syntax error: duplicated keys \"ignoreDeps\" near Deps\": [\"g"

Add ability to perform network state aware data sync capabilities.

aerogear/offix#49

Create production ready datasync application that work with graphback and offix

Motivation

Note: This task is more generic version of #63

Currently Voyager will require us to deploy GraphQL server to a single lambda.
If we want to deploy individual resolvers (or potentially groups of resolvers) to separate lambdas, allowing to scale them individually we would need to swap server with lambda implementation.

Task

Investigate modular architecture using reusable https://graphql-modules.com modules that provide extensions for Auth/Conflict/Binary etc.

For example for authentication we can provide following schema out of the box

type Query {
    me: User
}

type Mutation {
    login(): User
}

extend type User {
    username: String!
}

Then we can wrap schema together with custom resolvers and directives for auth.

GraphBack can generate backends in just 2 minutes from very very complex schema.
Target for this is to gather usage patterns and requirements for GraphBack in order to be more generic solution. Thanks to templates we can also integrate with various Red Hat technologies - making it great tool for kickstarting node.js based projects using modern stack:

• Postgress
• Express
• GraphQL
• React

Ability to use voyager-server modules like keycloak or conflicts in graphback will help with prototyping and generating sample backends.

Motivation

Provide conflict resolution mechanism that provides all the data required to resolve conflict.
Conflicts should be resolved to enable merging objects together without losing any changes in the process. I In case when changes are made inside the same field appropriate strategy should define how overwrite should happen.

Approaches

Conflicts should be resolved on client and server depending on the developer choice.

Provide diff mechanism on the server by persisting previous state

Persist previous state of the data so server side version can be compared against it giving diff.
When conflict will happen server or client will have access to base object that will allow users to perform 3 way merge using any strategy that is possible. This way we can also provide diffs for client that can also provide strategies or even resolve the client visually (same as when doing git merge)

Alternatively to storing entire objects we can persist diffs, however this will make comparison process more complex

When resolving conflicts changes are compared against base giving nice diff for server and client.
This aproach is already implemented, however we require users handle persistence for previous states,
which is additional overhead.

Pros

• Enables tracking changes and auditing them
• Enables full flexibility in terms of conflict resolution

Cons

• Requires separate storage (however we can provide helpers to handle those later)

Force client to operate on diff data

Clients can send only updates that were actually happening.
For example for mutation:
updateTask(id:ID!,name: String, description: String): Task

Developer can send only name field and ignore description that was not changed.
This will give us ability to know what was changed as only changed fields are sent to mutations
Clients could send to server only the data that was actually changed. This will imply that mutations should not have required fields as this will prevent this mechanism from happening.
This also means that clients will need to perform diffing logic to determine changesets,
which can be tricky.

With this aproach changes that are happening on the server are ignored. Only client diff is applied
on top of the server making it impossible to employ more complex strategies and use cases.

Pros

• Works almost out of the box on server
• Handling logic is very simple to understand and implement for developers

Cons

• Requires developers to change the way they work with GraphQL - to do the diff
• Limits how schema can be build
• Ability to only apply the client side changeset (no server/base comparision)

Motivation

Voyager has TypeScript native implementation however we do not benefit from it because GraphQL resolvers are not properly typed. Prisma GraphQL gen provides really good developer experience giving developers more insights about objects that are processed. More info: https://github.com/prisma/graphqlgen

Tasks

• Investigate how graphqlgen can be included into the voyager-server
• Provide prototype and example app that will generate code.
• Document process for developers and share it with community.

The code coverage we have right now is really low. The last time I checked it was around 55% which is fairly unacceptable. There are a couple of modules (example: metrics) that are totally lacking in tests.

Motivation

AeroGear Voyager offers authentication and Authorization using Keycloak project.
We integrate with keycloak in keycloak package:
https://github.com/aerogear/voyager-server/tree/master/packages/voyager-keycloak

Current integration requires from developers to have some manual steps and model their own user data.
Currently developers need to specify directives in their schema in order to support authorization:
https://github.com/aerogear/voyager-server/blob/master/examples/keycloak/server.js#L18

Authentication also exposes helpers to resolver context that users can use.
GraphQL modules can provide better user support by providing entire abstraction to implement authentication in GraphQL including additional schema and resolvers.

For Authentication we can provide basic user type:

type User {
 firstName: String!
 lastName: String!
 userName: String!
 email: String!
}

and set of resolvers for it.
Implementation could be keycloak specific or even allow to plug other authentication providers like passport.

Getting Started

Please follow keycloak example to check how authentication works:
https://github.com/aerogear/voyager-server/tree/master/examples/keycloak

Please inspect how keycloak package is implemented
https://github.com/aerogear/voyager-server/tree/master/packages/voyager-keycloak

Experiment and try to build the first GraphQL package inside keycloak example.

Technical resources

https://www.keycloak.org/docs/3.0/securing_apps/topics/oidc/javascript-adapter.html
https://graphql-modules.com/
https://github.com/Urigo/graphql-modules

Current graphql dashboard has very prominent ux issues and has really bad user experience.
https://github.com/imolorhe/altair is quickly gaining popularity in graphql scene and it has much more functionalities.

We've had reports that MQTT based subscriptions are delivering messages that happened while a client was not connected after that client reconnects. It's not entirely clear how this is happening yet but my suspicious is that either

• the MQTTPubSub asyncIterator implementation is producing some strange logic with how it manages connections
• The SubscriptionServer and/or its lower level libraries e.g. ws or net are buffering messages in some way.

• Module:
• Version:
• Node.js / npm versions:

Because of this code here: https://github.com/aerogear/datasync/blob/master/packages/voyager-server/src/VoyagerServer.ts#L21-L23

All top level resolvers are getting wrapped inside another resolver that does metrics and audit logging

This breaks custom scalars because the scalar resolver function also gets wrapped

Resolver wrapper logic in server and tools package is really raw and may cause many issues under different circumstances. We should look to refactor this

ping @darahayes

Run the basic example from the showcase. But use NODE_ENV=production. You will see GET query missing. when you open localhost:4000/graphql in the browser.

This is probably a feature that disables playground in production cases, but I want to make sure ...later

Roadmap for DataSync 1.0

DataSync 1.0 will focus on aggregating all libraries available in the ecosystem to work together in form of fully functional examples and boilerplates. Together with Offix and Graphback we can provide end to end data synchronization system

Tasks

• DataSync boilerplate #233
• Subscriptions with diff capabilities #232
• Integration with Graphback #231
• Documentation and website refinement

This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.

Edited/Blocked

These updates have been manually edited so Renovate will no longer make changes. To discard all commits and start over, click on a checkbox.

• fix(deps): update dependency ws to v7.4.6 [security]
• chore(deps): update dependency lerna to v3.22.1
• chore(deps): update dependency jest to v26.6.3

Open

These updates have all been created already. Click a checkbox below to force a retry/rebase of any.

• chore(deps): update dependency node-fetch to v2.6.7 [security]
• fix(deps): update dependency apollo-server to v2.25.4 [security]
• fix(deps): update dependency keycloak-connect to v21 [security]
• chore(deps): update all (@types/express, @types/express-session, @types/node, @types/node-fetch, @types/pino, @types/ws, coveralls, express-session, keycloak-connect-graphql, offix-client-boost, offix-conflicts-server, subscriptions-transport-ws, ts-jest, typedoc, typescript)
• fix(deps): update dependency apollo-server-express to v2.26.1
• chore(deps): update all (major) (@types/joi, @types/node, @types/pino, @types/ws, docker, standard, ts-node, typescript)
• chore(deps): update dependency lerna to v6
• chore(deps): update jest monorepo to v29 (major) (@types/jest, jest, ts-jest)
• fix(deps): update dependency apollo-server-express to v3
• fix(deps): update dependency graphql-tools to v8
• Click on this checkbox to rebase all open PRs at once

Ignored or Blocked

These are blocked by an existing closed PR and will not be recreated unless you click a checkbox below.

• fix(deps): update dependency graphql-tools to v4.0.8

Detected dependencies

• Check this box to trigger a request for Renovate to run again on this repository

#246

With the new version of the ava we can finally fix some of the issues that were blocking dependency updates.

CC @kingsleyzissou @darahayes

See: https://github.com/RedHatWorkshops/dayinthelife-streaming/blob/master/docs/labs/02-Change-Data-Capture/walkthrough.adoc#lab-2---change-data-capture

The goal of this project is to create a Node.js library that can be used to implement GraphQL subscriptions using Red Hat AMQ as the underlying Pub Sub technology. (Upstream project is Apache ActiveMQ Artemis 2.0.0)

There is a huge amount of scope here for the student to learn about GraphQL, Microservices, Asynchronous Messaging, Event Driven Architectures and Red Hat technologies. Because the deliverable (A Node.js library) would be quite small, The student is also encouraged to write tests, fully featured documentation, and to take ownership of the publishing/release lifecycle (on npm).

As a stretch goal, the student could also build an example application e.g. a real time messaging application that uses their new node.js library and the apollo-voyager-server framework as a proof of concept.

In GraphQL, subscriptions is a concept that allows clients to subscribe to events happening on the server. The basic idea is that an event happens on the server and the server has some way to notify any clients that are interested in that event. For example, in a group chat application, all of the clients could subscribe to a new message event. The most common implementation of subscriptions uses websockets between the server and client. This allows the server to send back events to the clients in real time.

In a production environment, there will be several instances of the backend application running at the same time. The backend servers need a Publish/Subscribe (Pub Sub) mechanism to share events back and forth between each other.

Publish/Subscribe is a is a form of asynchronous service-to-service communication used in serverless and microservices architectures. In a pub/sub model, any message published to a topic is immediately received by all of the subscribers to the topic.

For example, Using Pub Sub, Server 2 can know about an event that happened on server 1 and server 2 can notify its connected clients about the event.
There are many technologies that can be used for Pub Sub. (Examples: RabbitMQ, Apache Kafka, MQTT) and there are many Node.js libraries that allow us to use Pub Sub technologies in GraphQL applications. They would be a great starting point for trying to implement a new one. https://github.com/apollographql/graphql-subscriptions#pubsub-implementations

Red Hat has a messaging product called Red Hat AMQ which can be used to implement a production ready Publish/Subscribe architecture. The goal of this project is to create a node.js library that can be used to implement GraphQL subscriptions using Red Hat AMQ as the underlying Pub Sub technology.

GraphQL-shield is the very popular library that can be used for authorization purposes.

Task

Integrate server auth module with https://github.com/maticzav/graphql-shield

I setup the publishing process for this project.

Last step is npm run publish which tells lerna to publish each package.

All packages are published nicely, but examples is also a module handled by lerna and it is marked as private and can't be published.

See https://circleci.com/gh/aerogear/apollo-voyager-server/142

Easiest solution is to publish that package as well.

Alternative is somehow configure lerna to not publish this package. I haven't looked into this right now.

If you check https://circleci.com/gh/aerogear/apollo-voyager-server/142, you will see it as red, but in fact all necessary packages are published nicely.

Don't want to work on this right now and just creating an issue here.