expressgateway / express-gateway Goto Github PK

Add unique ID for each entity to refer the entity through the EG API

There are public endpoints, and then there are APIs, we need to validate the logical model that an API is a set of public endpoints.

Roman's design within the Express gateway has the most granular level listed within the configuration - public endpoint URLs (non-grouped) and private endpoints that the EG proxies to. Need to think about introducing API entity as a set of public endpoints.

Add API Consumer to configure the EG for finer granularity of control. at Quality of service, ACLs, and etc.

Extend the express-gateway-admin to dynamically manipulate the EG config

Create EG Admin API to manipulate the JSON config parameters.

Investigating options with and without LoopBack.

https://docs.google.com/document/d/1AxrFidpb92Y_pnMdPrCVLiWmIMnQnMX60S9J_kWblzI/edit

Everyone feel free to add and edit

For the authentication that is NOT handled by the EG, needs to be re-directed an external authorization URL so that the URL can be redirected to when a user accesses the EG.

This should be specified in authorization_url

Extend LB workspace to deploy the new EG created in LB workspace.

Ability to create a pipeline with key based processor to provide authentication through a set of EG generated keys for each API Consumer

An API Consumer should be able to have a set of key credentials associated with their identity.

Have consistent entities between EG UX and LB UX.
-change EG Readme processors to policies
-check and update the reference name in codes accordingly.

For OAuth2, that can potentially be extended for others authentication later

Add environment variable override based on the 12-factor app config for the EG config.

Example description

publicEndpoints: # specify pipeline per domain and path
cats:
host: ".cats.com" # regexp support
path: / # optional defaults to /
pipeline: cats_pipeline
mycats:
host: ".mycats.com"
path: /cats
pipeline: cats_pipeline
big-dogs:
host: big-dogs.com
pipeline: dogs_pipeline
animals-cats:
host: "animals.com"
path: /cats
pipeline: cats_pipeline
animals-dogs:
host: "animals.com"
path: /dogs
pipeline: dogs_pipeline

Need to discuss to breakdown to smaller tasks

Create a Design/Architectural Document that describes how and what the EG admin API be built, so that all dependencies from the EG OSS project AND LB commercial product are identified with the right decisions.

Breakdown config file to more linear structure

####Public Endpoints
host, path, pipeline_name

####Private Endpoints
url, port, path

http server configs
port

https server configs
port and certificates

pipelines

APIs - group of public endpoint (Does not include consumer management)

YAML example:
publicEndpoints: # specify pipeline per domain and path
cats:
host: ".cats.com" # regexp support
path: / # optional defaults to /
pipeline: cats_pipeline
mycats:
host: ".mycats.com"
path: /cats
pipeline: cats_pipeline
big-dogs:
host: big-dogs.com
pipeline: dogs_pipeline
animals-cats:
host: "animals.com"
path: /cats
pipeline: cats_pipeline
animals-dogs:
host: "animals.com"
path: /dogs
pipeline: dogs_pipeline

http: # if not specified EG will not start http server
port: 9080

https: # if not specified EG will not start https server
port: 9443
certificates: # will use SNI callback to provide proper certificate for domain
"*.lunchbadger.io":
key: "example/keys/lunchbadger.io.key.pem"
cert: "example/keys/lunchbadger.io.cert.pem"
"api.lunchbadger.com":
key: "example/keys/lunchbadger.com.key.pem"
cert: "example/keys/lunchbadger.com.cert.pem"
"default":
key: "example/keys/lunchbadger.io.key.pem"
cert: "example/keys/lunchbadger.io.cert.pem"

privateEndpoints: # urls to downstream services
cats_service:
url: "http://localhost"
port: 3000
path: / # optional defaults to /
dogs_service:
url: http://localhost
port: 4000

pipelines:
cats_pipeline:
title: Cats Pipeline
policies:
- action:
name: proxy # proxy to cats_service downstream service
privateEndpoint: cats_service

dogs_pipeline:
title: dogs Pipeline
policies:
- condition:
name: always
# action:
# name: oauth2-basic
# params:
# - action:
# name: oauth2-bearer
# - action:
# name: ensure-autenticated # ensure user has logged in 403 otherwise
# - action:
# name: oauth2-scope
# allow-scopes:
# - '/dogs'
action:
name: proxy # proxy to dogs_service downstream service
privateEndpoint: dogs_service

apis:
animals:
name: animals
plan: free
public_endpoints:
- cats
- mycats
- dogs

Create application via CLI

Allow a user to specify OAuth2 scope as OAuth2 scope tags within a pipeline to match public endpoint URLs

Example:
{
"condition": ["pathExact", "/protected/resourceA"],
"action": "tag",
"params": {
"tags": ["oauth2", "scope:resourceA"]
}
}

Allow TLS and SNI support for https to be configurable externally so that multiple certs can be used for multiple domains.

A developer familiar with Express middleware should be able to take any middleware that extracts the parameters that are necessary to construct a configured instance of it to be passed dynamically when instantiating that middleware.

Those parameters should go into EG config files in a standardized format.

EG App data would be more key-value and or rows and tables and not have hierarchical nor document like structure like the config data. Because this can live in distributed and single data stores, the lowest common denominator would be something like keys and values that can be transposed into rows/columnar based store and grouped into tables or namespaces.

To ensure EG will be capable of playing both roles as the authorization and resource server in the OAuth2 use case

The EG runtime configuration needs associate API consumers with the APIs (and their endpoints) through the configured pipelines. Although it's not necesary to explicity do so, granularity could be specified to link APIs with specific instances of pipelines where a processor may have specific parameter values for a given API consumer.

Example:
Customer API is linked to a pipeline where a rate limiter is specified and Consumer Bob and Consumer Bill exist
Consumer Bob may have rate limiting parameters: 1000 req/minute
Consumer Bill may have rate limiting parameters: 2000 req/minute
The SAME API backed by the SAME pipeline have two different parameter values for Bob and Bill.

There is EG app data (such as consumer records) that is kept separate from the config data so that I can separate configuration from master/app data

EG should support the ability to separate app data from config data. Config data would be data that is used to configure Express Gateway and be considered metadata. App data would be data that a USER of EG would generate using EG, for example - creating API consumers in the consumer management system

Example:
Consumer Bob uses OAuth2 credentials to authenticate and authorize externally into the EG, a set of internal credentials can be used further downstream into internal systems and central IAM systems like LDAP through SAML.

https://docs.google.com/document/d/1o7aXdcb8TskjjNFtDi29x10wnjF0TwSPHNcN-OkVy6s/edit

Everyone feel free to add and edit

Store default config file in /etc/expressgateway/gateway.conf

Use Redis connector from loopback. Support for both standalone and distributed environment.

Create policy to handle different OAuth2 strategy

Extend express-gateway-admin to deploy EG Admin API.

The LB architecture has an API called a workspace API. The workspace API manipulates LoopBack projects. The same workspace API needs to be extended to manipulate Express Gateways. The big question here is - how do we surface an API for Express Gateway? Do we do this without LoopBack?

1. take LoopBack, which already has Express in it - and pare down to barebone REST (strong-remoting) and marry Roman's EG capabilities into it
2. take Express Gateway - examine LoopBack, identify REST capabilities and extract and put into Express Gateway
3. take Express Gateway - extend with REST capabilities w/o LoopBack (controversial - we want to avoid)

Ability to passing in EG custom headers downstream so that use cases like path tracing, audit, request/response time introspection and metadata can be realized.

EG should be able to inject, modify and preserve upstream and downstream headers as needed for its own instrumentation as well as preserving system metadata for consumers and backend producers.

Allow a user to specify the OAuth2 tag within a pipeline's path to match all endpoints URLs

Example:
condition: ["always"]
or by a particular path
condition: ["pathExact", "/protected/resourceA"]
this would trigger the policy.

EG has APIs as containers for public endpoints. EG has private endpoints as well. We should think about the parallel that we have already within LunchBadger. A microservice is a unit of deployment of functionality that has dependencies and externalization in the form of private enpoints. In LB - a model has a built in private endpoint. If you proxy in front of a microservice, the microservice can have N number of private endpoints. Therefore a microservice is a container of private endpoints.

• Spec (if applicable)
• Implementation
• Tests
• Code Review
• Demo
• Docs (if applicable)

Need to have CLI for key-gen that accepts keys as user inputs or makes new ones

As an EG admin, I would want to be able to configure multiple EG instances that are all driven off central configuration, so that I have scale and high availability.

An EG admin should be able to create mutiple gateway instances that are assigned different configurations but also share centralized information.

Example:
Gateway 1: has 2 pipelines and fronts /catalog and /cart public endpoints
Gateway 2: has 1 pipeline and fronts /user endpoint
Both Gateway 1 and 2 should share the same set of API consumer data for access and assignment

Investigate the OAuth2 design per Roman to see if it can do both authorization and resource server in OAuth2 use case.
https://docs.google.com/document/d/1CKQEAvnAgG5E7twGIUZdCZqqdhoM7Zeri9CN9uTLFgA/edit

The authorization system would need to provide a set of scopes that the user is authorizing to be provided to the client application. We need API endpoints with OAuth2 scopes

Ability to create a pipeline with a Basic Auth processor to provide basic authentication for API Consumers.

An API Consumer should be able to have basic auth credentials associated with their identity.

For reference, review and discussion

Create consumer via CLI

As an LB Developer, I want to be able to deploy the changed EG config without reloading (hot deploy).

Investigate:
expressjs/express#2596