Code Monkey home page Code Monkey logo

3scale-istio-adapter's Introduction

3scale Istio Mixer gRPC Adapter

An out of process gRPC Adapter which integrates 3scale with Istio

Overview

When running Istio in a Kubernetes or OpenShift cluster, this adapter allows a user to label their service running within the mesh, and have that service integrated with the 3scale Api Management solution.

Prerequisites

  1. Istio version 1.1 with policies enabled
  2. A working 3scale account (SaaS or On-Premises)
  3. kubectl or oc command line tool

Enabling Policies

As of Istio 1.1.0 GA, policies are now disabled by default. This impacts the 3scale adapter and policies need to be re-enabled in order for the adapter to receive traffic.

Follow these instructions to enable policies.

Deploy the adapter

A Kubernetes deployment and service manifest are located in the deploy directory. To deploy the adapter to a Kubernetes/OpenShift cluster, run:

kubectl create -f deploy/

Configuring the adapter

See the adapter configuration options to understand the default behaviour of the adapter, and how to modify it.

Create the required resources

The required CustomResources are located in the istio directory.

Create the required resources in the desired istio namespace. By default, this is istio-system, however, in a multi-tenant environment, this could be different.

kubectl create -f istio/authorization-template.yaml
kubectl creae -f istio/threescale-adapter.yaml

Generating and creating configuration

The adapter embeds a tool which allows generation of the handler,instance and rule CR's. More detail can be found in the tools documentation.

To generate these manifests from a deployed adapter, assuming it is deployed in the istio-system namespace, run the following:

export NS="istio-system" URL="https://replaceme-admin.3scale.net:443" NAME="name" TOKEN="token"
oc exec -n ${NS} $(oc get po -n ${NS} -o jsonpath='{.items[?(@.metadata.labels.app=="3scale-istio-adapter")].metadata.name}') \
-it -- ./3scale-config-gen \
--url ${URL} --name ${NAME} --token ${TOKEN} -n ${NS}

This will produce some sample output to the terminal. Edit these samples if required and create the objects using oc create command.

Update the workload (target service deployment's Pod Spec) with the required annotations:

export CREDENTIALS_NAME="replace-me"
export SERVICE_ID="replace-me"
export DEPLOYMENT="replace-me"
patch="$(oc get deployment "${DEPLOYMENT}" --template='{"spec":{"template":{"metadata":{"labels":{ {{ range $k,$v := .spec.template.metadata.labels }}"{{ $k }}":"{{ $v }}",{{ end }}"service-mesh.3scale.net/service-id":"'"${SERVICE_ID}"'","service-mesh.3scale.net/credentials":"'"${CREDENTIALS_NAME}"'"}}}}}' )"
oc patch deployment "${DEPLOYMENT}" --patch ''"${patch}"''

Sample manifest can be found in the testdata directory which can be used as a base if you prefer not to leverage the tool.

Routing service traffic through the adapter

The following statements assume that configuration has been generated using the tool provided. Custom modifications to the match condition in the rule are possible, but explanation of any or all combinations is outside of the scope of this document. If interested, check out the upstream documentation

In order to drive traffic for your service through the adapter and be managed by 3scale, we need to match the rule destination.labels["service-mesh.3scale.net/credentials"] == "threescale" we previously created in the configuration, in the kind: rule resource.

Integration of a service requires that the above label be added to PodTemplateSpec on the Deployment of the target workload. The value, threescale, refers to the name of the generated handler. This handler will store the access token required to call 3scale.

When the request reaches the adapter, the adapter needs to know the how the service maps to the an API on 3scale. This can be provided in one of two ways:

  1. As a label on the workload (recommended)
  2. Hardcoded in the handler as service_id

To pass the service ID to the adapter via the instance at request time, add the following label to the workload:

destination.labels["service-mesh.3scale.net/service-id"] == "replace-me"

Your 3scale administrator should be able to provide you with both the required credentials name and the service ID.

Authenticating requests

Now that the we have configured the service to be managed by 3scale we can decide how requests should be authenticated. Currently there are two supported mechanisms:

  1. The API Key authentication pattern
  2. The Application ID, Application Key (optional) pair authentication pattern

You can read more detailed information about these patterns and their behaviour in the 3scale documentation.

Applying Patterns

When you have decided what pattern best fits your needs, you can modify the instance CustomResource to configure this behaviour. You can also decide if authentication credentials should be read from headers or query parameters, or allow both.

It is important to note that when specifying values from headers, Istio expects they key to be lower case. So for example if you want to send a header as User-Key, this must be referenced in the configuration as request.headers["user-key"].

API Key Pattern

To use the API Key authentication pattern, you should use the user value on the subject field like so:

apiVersion: "config.istio.io/v1alpha2"
kind: instance
metadata:
  name: threescale-authorization
spec:
  template: threescale-authorization
  params:
    subject:
      user: request.query_params["user_key"] | request.headers["user_key"] | ""
    action:
      path: request.url_path
      method: request.method | "get"
      service: destination.labels["service-mesh.3scale.net/service-id"] | ""

This configuration will examine the user_key query parameter, followed by the user-key header in search of the api key. As mentioned, this can be restricted to one or the other by removing that particular attribute. The order can be changed to determine precedence.

If you would like for the adapter to examine a different, for example query parameter than user_key, you would simply change [user_key] to [foo]. The same pattern applies to the headers.

Application ID Pattern

To use the Application ID authentication pattern, you should use the properties value on the subject field to set app_id, and optionally app_key.

Manipulation of this object can be done in using the methods described previously. An example configuration is shown below.

apiVersion: "config.istio.io/v1alpha2"
kind: instance
metadata:
  name: threescale-authorization
spec:
  template: threescale-authorization
  params:
    subject:
      properties:
        app_id: request.query_params["app_id"] | request.headers["app_id"] | ""
        app_key: request.query_params["app_key"] | request.headers["app_key"] | ""
    action:
      path: request.url_path
      method: request.method | "get"
      service: destination.labels["service-mesh.3scale.net/service-id"] | ""

OpenID Connect Pattern

To use the OpenID Connect authentication pattern, you should use the properties value on the subject field to set client_id, and optionally app_key.

Manipulation of this object can be done in using the methods described previously. An example configuration is shown below. Here the client identifier (application ID) is parsed from the JWT under the label "azp". Modify this as desired.

apiVersion: "config.istio.io/v1alpha2"
kind: instance
metadata:
  name: threescale-authorization 
spec:
  template: threescale-authorization
  params:
    subject:
      properties:
        app_key: request.query_params["app_key"] | request.headers["app_key"] | ""
        client_id: request.auth.claims["azp"] | ""
    action:
      path: request.url_path
      method: request.method | "get"
      service: destination.labels["service-mesh.3scale.net/service-id"] | ""

For this integration to work correctly. OpenID configuration must still be done in 3scale in order for the client to be created in the IdP. For the service the user wants to protect, they should create Request authorization in the same namespace as that service. The JWT should then be passed in the Authorization header of the request.

In the sample RequestAuthentication defined below, replace issuer, jwksUri and selector as appropriate.

- apiVersion: security.istio.io/v1beta1
  kind: RequestAuthentication
  metadata:
    name: jwt-example
    namespace: bookinfo
  spec:
    selector:
      matchLabels:
        app: productpage
    jwtRules:
    - issuer: >-
        http://keycloak-keycloak.34.242.107.254.nip.io/auth/realms/3scale-keycloak
      jwksUri: >-
        http://keycloak-keycloak.34.242.107.254.nip.io/auth/realms/3scale-keycloak/protocol/openid-connect/certs

Hybrid Pattern

Finally, you may decide to not enforce a particular authentication method but accept any valid credentials for either pattern. In that case, you can do a hybrid configuration where the user key pattern will be preferred if both are provided:

apiVersion: "config.istio.io/v1alpha2"
kind: instance
metadata:
  name: threescale-authorization
spec:
  template: threescale-authorization
  params:
    subject:
      user: request.query_params["user_key"] | request.headers["user_key"] | ""
      properties:
        app_id: request.query_params["app_id"] | request.headers["app_id"] | ""
        app_key: request.query_params["app_key"] | request.headers["app_key"] | ""
    action:
      path: request.url_path
      method: request.method | "get"
      service: destination.labels["service-mesh.3scale.net/service-id"] | ""

Adapter metrics

The adapter, by default reports various Prometheus metrics which are exposed on port 8080 at the /metrics endpoint. These allow some insight into how the interactions between the adapter and 3scale are performing. The service gets labelled and automatically discovered and scraped by Prometheus.

Development and contributing

Check the DEVEL.md for more info on how to hack/test this adapter.

3scale-istio-adapter's People

Contributors

jmprusi avatar krishvoor avatar leathersole avatar pehala avatar philipgough avatar unleashed avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar

3scale-istio-adapter's Issues

Generated UID thorugh CLI tool can be longer than 63 characters

If you will use --url parameter with url that is around 63 characters long (without the scheme), the generated CRD will use UID which will be longer than that, since it also appends the url, and it will be rejected as a valid label for service.

Example:

  • oc exec -n istio-system $(oc get po -n istio-system -o jsonpath='{.items[?(@.metadata.labels.app=="3scale-istio-adapter")].metadata.name}') -it -- ./3scale-config-gen --url="https://3scale-admin.extremely.long.url.with.sixtythree.characters.com" --service="10" --token="access-token"
  • This will generate UID like this
    3scale-admin.extremely.long.url.with.sixtythree.characters.com-10, which is longer than 63 characters and wont be accepted as a valid label.

Adapter does not respect last? attribute in mapping rules

Overview

According to our product documentation https://access.redhat.com/documentation/en-us/red_hat_3scale_api_management/2.8/html/getting_started/first-steps#defining-mapping-rules

If the request hits mapping rule with last? it should stop all the processing. Right now adapter continues with processing even after this rule was hit.

How to replicate

  • Create new service and application
  • Add a new metric to the service called metric_once
  • Add two mapping rules to the service:
    • Pattern: /anything/bar/, metric: metric_once, last: true
    • Pattern: /anything/bar/123, metric: metric_once, last: false
  • Check that the mapping rules have correct order (the one with last, should be before the other one)
  • Do request to /anything/bar/123
  • Check analytics, metric_once should increase by two, which means the second rule was hit as well.

Undesired caching behaviour

There is an issue in relation to the caching at the Envoy layer which appears to be reproducible with latest code on any cluster.

Steps to reproduce:

  1. Deploy the bookinfo example app
  2. Deploy 3scale adapter and rules contained in the istio directory
  3. Integrate with a valid 3scale account and credentials
  4. Authenticate the call to productpage with a valid api_key query param

This should result in an authenticated request. However once a single successful request has been authenticated, modifying to an invalid api_key or removing the query param altogether, still enables a certain amount of requests (that should not be authenticated) through the gateway.

I've observed that the requests above still result in mixer calling out to the adapter. The following is the debug logs generated in the adapter after one of these successful requests that should have been denied:

{"level":"debug","time":"2018-11-23T10:28:44.248764Z","msg":"Got instance &InstanceMsg{Subject:&SubjectMsg{User:,Groups:,Properties:map[string]*istio_policy_v1beta11.Value{},},Action:&ActionMsg{Namespace:,Service:,Method:GET,Path:/details/0,Properties:map[string]*istio_policy_v1beta11.Value{},},Name:threescale-authorization.instance.istio-system,}"}
{"level":"debug","time":"2018-11-23T10:28:44.255724Z","msg":"Got instance &InstanceMsg{Subject:&SubjectMsg{User:,Groups:,Properties:map[string]*istio_policy_v1beta11.Value{},},Action:&ActionMsg{Namespace:,Service:,Method:GET,Path:/reviews/0,Properties:map[string]*istio_policy_v1beta11.Value{},},Name:threescale-authorization.instance.istio-system,}"}
{"level":"debug","time":"2018-11-23T10:28:44.261362Z","msg":"Got instance &InstanceMsg{Subject:&SubjectMsg{User:,Groups:,Properties:map[string]*istio_policy_v1beta11.Value{},},Action:&ActionMsg{Namespace:,Service:,Method:GET,Path:/reviews/0,Properties:map[string]*istio_policy_v1beta11.Value{},},Name:threescale-authorization.instance.istio-system,}"}
{"level":"error","time":"2018-11-23T10:28:45.250692Z","msg":"error reporting usage - missing required parameters"}
{"level":"error","time":"2018-11-23T10:28:45.257040Z","msg":"error reporting usage - missing required parameters"}
{"level":"error","time":"2018-11-23T10:28:45.263897Z","msg":"error reporting usage - missing required parameters"}
{"level":"error","time":"2018-11-23T10:28:45.265293Z","msg":"error reporting usage - missing required parameters"}

Config generation tools flags are being appended to by a dependency

The config generation tool provids a --help flag to display the available options and requirements to the tool.

It now appears that since we import some kubernetes packages for template generation, glog which is supposed to be a library, is running an init function (linked) and appending its options to our help command.

While this doesn't break our tool in any way, it is unsightly, and confusing and we should avoid this somehow.

Support wildcard mapping rule patterns

Apicast supports patterns like /anything/bar/{id} which acts like a wildcard and hits both /anything/bar/123 and /anything/bar/test.

Should this feature be added also into the adapter?

Investigate moving to a single handler for multiple services - improve UX for end user

Right now we create a handler, rule and instance per 3scale API. We could look at adding labels on the workloads to expose this info to a generic handler.

This might simplify the user experience if we could get this working. We would need to adapt the instance to read the system_url service_id and access_token from the destination service.

The user would be responsible for adding these labels to the workload. My main concern would be the security implications of exposing the access token on the request, but perhaps this is not a big problem.

@unleashed Do you see any downsides to this or think it might not be feasible for any reason?

Create e2e walkthrough for the adapter

Using the booking example, provide documentation to walk the end user through deploying and configuring the adapter and custom resources and setting up on 3scale, testing requests ...

The "3scale-config-gen" command fails when the 3scale admin url contains a port

The "3scale-config-gen" command fails when the 3scale admin url contains a port. It works if I remove the port.

Current behavior

$ oc rsh -n istio-system $(oc get pod -n istio-system -o name -l app=3scale-istio-adapter) ./3scale-config-gen --url="https://nmasse-redhat-admin.3scale.net:443" --service="2555417791085" --token="[redacted]" 
error. DNS 1123 label value net:443-2555417791085 contains invalid characters at positions 3
error. DNS 1123 label value net:443-2555417791085 does not pass upstream k8s validation:
a DNS-1123 label must consist of lower case alphanumeric characters or '-', and must start and end with an alphanumeric character (e.g. 'my-name',  or '123-abc', regex used for validation is '[a-z0-9]([-a-z0-9]*[a-z0-9])?')
error. Resource name nmasse-redhat-admin.3scale.net:443-2555417791085 does not pass upstream k8s validation:
a DNS-1123 subdomain must consist of lower case alphanumeric characters, '-' or '.', and must start and end with an alphanumeric character (e.g. 'example.com', regex used for validation is '[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*')
command terminated with exit code 1

Expected behavior

The previous command succeeds.

Review configuration

With the addition of the backend cache, we need to review the configuration in the adapter and allow the operator to set cache config.

It might also be worth taking a look at using a ConfigMap to make life easier. Will need a follow up docs ticket.

Enable authorization caching and async metrics report

Right now the adapter only implements the Authorization template. To allow reporting async metrics to 3scale we need to add a new template: Metric.

This way, we can cache (hits and/or time) the authorization but still report async metrics to the 3scale control plane.

Update 19/10/2018

If we want to cache the auth for X hits and/or X seconds we need to define what identifies a user

Our authorization template looks like this now:

apiVersion: "config.istio.io/v1alpha2"
kind: instance
metadata:
 name: threescaleauthorizationtemplate
 namespace: istio-system
spec:
 template: authorization
 params:
   action:
     namespace: destination.namespace | ""
     service: destination.service | ""
     path: request.path | ""
     method: request.method | ""
     properties:
       user-key: request.headers["user-key"] | ""

Istio uses attributes to define the values:
https://istio.io/docs/reference/config/policy-and-telemetry/attribute-vocabulary/

Checking those, we can see there's a request.api_key attribute.. by default, that attribute points to the param api_key.

Ideally our authorization could look like this:

apiVersion: "config.istio.io/v1alpha2"
kind: authorization
metadata:
  name: threescaleauthorizationtemplate
  namespace: istio-system
spec:
 subject:
   user: request.api_key | request.headers["api-key"] | ""

Just caching based on the api_key.

To do that, we need to default the Istio integration to "api_key" instead of "user_key".

Of course, there's another way:

Defining a HTTPAPISpec and a HTTPAPISpecBinding:

apiVersion: config.istio.io/v1alpha2
kind: HTTPAPISpec
metadata:
  name: myapi
  namespace: myproject
spec:
  api_keys:
  - query: user-key
  attributes:
    attributes:
      api.service:
        stringValue: myapi
      api.version:
        stringValue: 1.0.0
  patterns:
  - attributes:
      attributes:
        api.operation:
          stringValue: getcatchall
    httpMethod: GET
    uriTemplate: /
----
apiVersion: config.istio.io/v1alpha2
kind: HTTPAPISpecBinding
metadata:
  name: myapi-binding
spec:
  api_specs:
  - name: myapi
    namespace: myproject
  services:
  - name: productpage
    namespace: myproject

Here you can see the extra complexity that this adds. But we can define api_key to be a query param called user-key:

  api_keys:
  - query: user-key

But you need to "bind" the API spec with a service. This is something we can easily automate on the API operator / Ostia project.

Seg Fault when API Manager is using self-signed cert

Hello.
Appears that 3scale istio adapter breaks when API Manager is secured using a self-signed cert.
This currently affects training scenarios where a student learning about the 3scale istio adapter is assigned their own API Manager (that uses a self-signed cert).
This problem does not occur when the API Manager is using a cert signed by a legitimate CA.
Logs are as follows: https://gist.github.com/jbride/d60b96758b6d61143db38b85def7afcc

Currently using : registry.redhat.io/openshift-istio-tech-preview/3scale-istio-adapter:0.4.1

I have an environment that I can reproduce / demonstrate the problem if someone is interested in jumping on a bluejeans. Just contact me: jbride at redhat.com.

thanks!

OIDC doesn't warn about app_key not being supplied

If you use authentication by user_key or app_id/app_key it warns you with error message:
UNAUTHENTICATED:httpbin.handler.service-mesh:no auth credentials provided or provided in invalid location, but if you don't supply app_key with OIDC it just throws PERMISSION_DENIED.

Integration between istio and 3scale doesn't work: X509 certificate problem

Describe the bug
When I access the productpage via the 3scale, here is the error message :
INTERNAL:performing check operation failed: 1 error occurred:%0A%0A* rpc error: code = Unknown desc = currently unable to fetch required data from 3scale system - Get https://3scale-admin.openshift.intranet:443/admin/api/services/4/proxy/configs/production/latest.json?access_token=6fbde48bad6068e440c01eb19b97a72025bf334d3f1147930d8320072545bbb4: x509: certificate is valid for *.router.default.svc.cluster.local, router.default.svc.cluster.local, not 3scale-admin.openshift.intranet

Expected behavior
the access to the productpage of bookinfo is permitted if you use the right user_key defined at the level of the portal 3scale
change the environment variable at the level of the 3scale-adapter to accept no secure connection doesn't work because 3scale redirect to https.

Steps to reproduce the bug
in the browser, just access the page productpage on the bookinfo application
The configuration of the environment is based on the following video: https://www.youtube.com/watch?v=L8UCMlmij4E

Version
okd 3.11
maistra 0.12
3scale open source version 2.4

Installation
installation okd 3.11 : download source and execute oc cluster up --base-dir=/opt/config --
installation maistra 0.12:
installation 3scale 2.4.0 open source: oc new-app --param WILDCARD_DOMAIN="$(minishift ip).nip.io" --param AMP_RELEASE=2.4.0 -f https://raw.githubusercontent.com/3scale/3scale-operator/master/pkg/3scale/amp/auto-generated-templates/amp/amp-eval.yml
debug activated on 3scale-adapter

handler:
apiVersion: config.istio.io/v1alpha2
kind: handler
metadata:
creationTimestamp: null
name: threescale
namespace: istio-system
spec:
adapter: threescale
connection:
address: dns:///threescale-istio-adapter:3333
params:
access_token: f7c1d9a5947b2522b3b3b2cb690f7cb426ee45e32e127ebc43018407040bcad3
service_id: "4"
system_url: https://3scale-admin.openshift.intranet

Instance:
apiVersion: config.istio.io/v1alpha2
kind: instance
metadata:
creationTimestamp: null
name: threescale
namespace: istio-system
spec:
params:
action:
method: request.method | "get"
path: request.url_path
service: destination.labels["service-mesh.3scale.net/service-id"] | ""
subject:
user: request.query_params["user_key"] | request.headers["x-user-key"] | ""
template: threescale-authorization

rule:
apiVersion: config.istio.io/v1alpha2
kind: rule
metadata:
creationTimestamp: null
name: threescale
namespace: istio-system
spec:
actions:

  • handler: threescale.handler.istio-system
    instances:
    • threescale.instance.istio-system
      match: |-
      context.reporter.kind == "inbound" &&
      destination.labels["service-mesh.3scale.net/credentials"] == "threescale" &&
      destination.labels["service-mesh.3scale.net/authentication-method"] == ""

Could you please help me which certificate do I need to generate? is it not possible to bypass the certificate check seeing that this is self signed?

thanks !
Olivier

Support multi tenancy in the config generation tool

Now that multi tenancy is due to land in Maistra as of TP11, we should update the config generation tool to support this.

Right now, the tool will generate resources with the istio-system namespace hardcoded. This can be the logical default since a standard installation will preserve this naming convention. We should allow it to be overridden via a flag however to support alternative namespaces.

As part of this, we should also review the docs and update as required.

Adapter can panic on interface conversion

When the adapter fetches the proxy config from 3scale, it uses the client to do so which returns a custom error type of ApiErr. This line of code is susceptible to causing the adapter to panic.

We should use a type switch in this instance to attempt to parse the status code, and default to returning the error message from any other type.

The case that flagged this for me was an invalid certificate on the system route (self-signed). @unleashed do we want to support allowing the deployer of the adapter to skip certificate verification?

Allow Backend URL to be overridden (use internal DNS)

We have identified an area where we can slightly improve performance in certain situations.
In deployments where 3scale exists in the same cluster as istio, we should provide the option for a user to specify URL to leverage the clusters internal DNS.

The adapter currently expects the system admin portal URL to be provided by the user via yaml. This is expected to be the publicly reachable address with https as the scheme.

Currently, we have implemented caching for system configuration, so the overhead in calling the public route is low. All cache refreshes are done as a background task causing zero impact thereafter to user request latencies.

However the adapter then reads the backend URL from the system configuration which is also a publicly addressable endpoint. We want to allow this to be configurable and use internal network if available.

Some investigation re impact produced the following:

Initial request to public route:

    time_namelookup:  0.050793
       time_connect:  0.051067
    time_appconnect:  0.061728
   time_pretransfer:  0.061759
      time_redirect:  0.000000
 time_starttransfer:  0.068366
                    ----------
         time_total:  0.068438

Lowest time_total within the preceding requests within TTL period -> time_total: 0.019566

This compares to a consistent request time on the internal network of:

    time_namelookup:  0.001532
       time_connect:  0.001629
    time_appconnect:  0.000000
   time_pretransfer:  0.001677
      time_redirect:  0.000000
 time_starttransfer:  0.007879
                    ----------
         time_total:  0.007947

templating package can generate invalid dns names

The templating package used to generate yamls for istio specific custom resources relies on a hostname from system and the service id to make them unique and support multiple services.

However the name it generates must adhere to DNS-1035 label, which is validated against the following regular expression: [a-z]([-a-z0-9]*[a-z0-9])?

The length of the string must be less than 63 characters.

Providing a system url such as https://3scale.net will cause an error and the templates will not be generated.

To fix this we should:

  • Strip characters that start with numeric values
  • Validate and reduce if required the length of the formatted string to 62 chars
  • Allow a --uid= flag so users can provide their own id and work around any unforeseen issues or use their own vanity id's etc

Inconsistent mapping rules status code

If you have only mapping rule [GET] /ip, and you try to request [POST] /ip it returns 403, but if you try [GET] /incorrect it returns 404.

I think those cases should return the same status code.

Default values for header names in Instance differs from Apicast

The adapter uses x-user-key, app-id, app-key while Apicast (or System) by default uses user_key, app_id and app_key. It can be easily changed in instance object so it is a very minor issue, but I think it would be a good idea for the adapter to use the same ones as Apicast.

Investigate current metrics situation

In the previous release, we documented and exposed some Prometheus metrics related to the behaviour of the connected 3scale(s).

With the extracting of the system cache and now the merger of the backend cache, we have likely broken some of these metrics.

This issue is to investigate what has been broken/changed and is also a good opportunity to think about how they can be improved.

We have exposed, in the client refactoring some callback functions which can be used to get some info about req/res from 3scale that could be used.

Bug: Adapter panics when user_key param not provided

When running the latest adapter deployed within a service mesh and using the bookinfo example app, the following request causes the adapter to panic :

curl http://istio-ingressgateway-istio-system.176.34.139.81.nip.io/productpage

This appears to be related to the following line in the authz handler: userKey := request.Subject.User. Full stack trace provided below.

panic: runtime error: invalid memory address or nil pointer dereference
--
  | [signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0x9c24e4]
  |  
  | goroutine 45 [running]:
  | github.com/3scale/istio-integration/3scaleAdapter/pkg/threescale.(*Threescale).isAuthorized(0xc4201bf920, 0xc4200aeaa0, 0xc420284300, 0x35, 0x0, 0xc4200aea00, 0xc4205244a0, 0x0, 0x0, 0x0, ...)
  | /go/src/github.com/3scale/istio-integration/3scaleAdapter/pkg/threescale/threescale.go:237 +0x84
  | github.com/3scale/istio-integration/3scaleAdapter/pkg/threescale.(*Threescale).HandleAuthorization(0xc4201bf920, 0xc7b880, 0xc42029ac90, 0xc4201234a0, 0xc4201bf920, 0xc42029ac00, 0xb10420)
  | /go/src/github.com/3scale/istio-integration/3scaleAdapter/pkg/threescale/threescale.go:209 +0x645
  | github.com/3scale/istio-integration/3scaleAdapter/vendor/istio.io/istio/mixer/template/authorization._HandleAuthorizationService_HandleAuthorization_Handler(0xb8c280, 0xc4201bf920, 0xc7b880, 0xc42029ac90, 0xc4200ae9b0, 0x0, 0x0, 0x0, 0xc4200b8840, 0x145)
  | /go/src/github.com/3scale/istio-integration/3scaleAdapter/vendor/istio.io/istio/mixer/template/authorization/template_handler_service.pb.go:317 +0x241
  | github.com/3scale/istio-integration/3scaleAdapter/vendor/google.golang.org/grpc.(*Server).processUnaryRPC(0xc42017a300, 0xc80160, 0xc42017af00, 0xc4201bc400, 0xc4201bfd70, 0x1077c20, 0x0, 0x0, 0x0)
  | /go/src/github.com/3scale/istio-integration/3scaleAdapter/vendor/google.golang.org/grpc/server.go:935 +0x455
  | github.com/3scale/istio-integration/3scaleAdapter/vendor/google.golang.org/grpc.(*Server).handleStream(0xc42017a300, 0xc80160, 0xc42017af00, 0xc4201bc400, 0x0)
  | /go/src/github.com/3scale/istio-integration/3scaleAdapter/vendor/google.golang.org/grpc/server.go:1139 +0xd69
  | github.com/3scale/istio-integration/3scaleAdapter/vendor/google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc420040900, 0xc42017a300, 0xc80160, 0xc42017af00, 0xc4201bc400)
  | /go/src/github.com/3scale/istio-integration/3scaleAdapter/vendor/google.golang.org/grpc/server.go:683 +0x9f
  | created by github.com/3scale/istio-integration/3scaleAdapter/vendor/google.golang.org/grpc.(*Server).serveStreams.func1
  | /go/src/github.com/3scale/istio-integration/3scaleAdapter/vendor/google.golang.org/grpc/server.go:681 +0xa1

Document what can be configured and how to do so

In the adapter, we have a number of setting thats can be tweaked via environment variables. These settings and their default values should be documented in the readme. We should also expose the env vars to the deployment template.

Enable ppc64le on 3Scale-istio-adapter

Hi,

Thanks for the wonderful readme/documentation, I could build 3scale-istio-adapter from the source almost instantly, run the tests outlined successfully on ppc64le architecture.

In case, if you are wondering what ppc64le architecture is?
It is a RISC Architecture and is IBM's prodigy, IBM has recently made its ISA (Instruction Set Architecture) opensource,
doing so they have contributed back to the opensource world. Many of the pioneers of banking, HPC industries today run on ppc64le architecture.

As an ongoing effort to enable open-source projects to which ppc64le architecture can add value, we are trying to enable this project. We have seen absolutely no deviation in build/test instructions. I will soon submit a PR to enable build/testing on ppc64le architecture as well.

So, what would it take from us to enable ppc64le in the CI/CD pipeline?
Currently we are enabled/certified on Travis & Jenkins.

Empty authentication messages

When you supply for example wrong user_key this message is returned PERMISSION_DENIED:httpbin.handler.service-mesh:. This message also appears when you use OIDC and you don't supply app_key.
Adapter version: latest master

3scale Mixer Adapter doesn't report any adapter metrics

In the "Mixer Dashboard" in Grafana we can see several adapters reporting some metrics (dispatch count, and success/error for dispatches). We should investigate why the 3scale Mixer Adapter doesn't show any kind of metric, and try to fix that.

Screenshot:
image

----- Update 17/10/2018

Seems clear from Prometheus targets that Istio Mixer is exposing a prometheus /metrics endpoint:

image

Checking the source code, we can see the implementation here:
https://github.com/istio/istio/blob/master/mixer/pkg/server/monitoring.go

------ update 18/10/2018

After asking Istio devels, looks like the mixer dispatching metrics for our adapter should be reported... trying to get more info there. (Bug in our code or mixer?)

------ Update 22/10/2018

We need to check if mixer reports for dynamic adapters the dispatch metrics. Those metrics are not enough for monitoring the actual state of the adapter, we should expose a Prometheus endpoint with internal adapter stats.

Related to #18

Improve handling of unavailable backend

If the backend is unavailable, it currently returns 500 with {"level":"error","time":"2020-05-25T08:08:22.016045Z","msg":"request authorization failed - XML syntax error on line 107: element <meta> closed by </head>"} this error.

I think returning backend unavailable message would be better

Investigate OIDC authentication

The adapter should support OpenID Connect authentication.

We need to spend some time investigating and documenting how this might work, what changes are required to our config and especially, what we can do as the Istio level to support this feature.

There is support for jwt passed as attributes in the form of request.auth.principal, request.auth.audiences, request.auth.presenter, request.auth.claims.

@vramosp has shared some links, this one I found most likely relevant and I think investigation should begin here: https://istio.io/help/ops/security/end-user-auth/

Adapter panics if name parameter is longer than 63 characters

If you specify --name parameter that is longer than 63 characters, adapter will panic and throw error like this:

panic: error creating config generator provided name aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa fails Kubernetes validation: must be no more than 63 characters

goroutine 1 [running]:
main.execute(0x0, 0x0)
/go/src/github.com/3scale/3scale-istio-adapter/cmd/cli/main.go:130 +0xdd1
main.main()
/go/src/github.com/3scale/3scale-istio-adapter/cmd/cli/main.go:158 +0x1a6

Adapter can use incorrect request path when contacting 3scale

I have httpbin deployed in service mesh with Virtual service rewrite rule /httpbin => / and I am requesting URL [POST] <INGRESS>/httpbin/post, which is correctly identified by service mesh and is forwarded to httpbin as [POST] <INGRESS>/post, but when I have mapping rule in 3scale only to /post it is rejected with PERMISSION_DENIED:threescale-httpbin.handler.istio-system:no matching mapping rule for request with method POST and path //post.

I have attached the httpbin templates used for the service mesh service.
templates.zip

Implement caching between adapter and 3scale backend

We need to be able to reduce request/response time between Mixer and our adapter.

The adapter currently calls out to 3scale backend to authenticate each request. We should be able to fetch limits from 3scale initial and make authorisation decisions within the adapter to reduce a network hop and load on 3scale platform.

We need to create a caching implementation that makes this possible. We should make the cache configurable so that it can support additional use cases (eg remote shared cache) going forward.

  • Bring in 3scale backend client which supports rate limit extensions as a dependency
  • Abstract out any istio dependencies from the current backend integration to make the component reusable
  • Benchmark some candidates that we can use as data structure #66
  • Implement cache as required (add a simple design doc for this) #59
  • Split out auth and reporting functionality to allow us to report in batches

Requests not load balanced between pods when the adapter is scaled past one

Based on some feedback, we investigated how the adapter was behaving when scaled to multiple pods.

It was expected that requests from mixer to the adapter would be load-balanced across each instance of the adapter but this was not the case. This is due to the way mixer establishes a connection with the kubernetes service as a one time operation and maintains the connection information.

A change has been made to mixer here providing us a way to workaround this (available as of Maistra TP10). In order to leverage this we need to do two things:

  1. Migrate our adapters service resource to a headless service
  2. Add some keep alive params to the gRPC server when initiating. See here

Keepalive can be set as follows:

grpc.KeepaliveParams(keepalive.ServerParameters{
                MaxConnectionAge: 1 * time.Minute, 
        })

Ability to match on backend host

I would like to see the ability to breakdown microservices and match to backend api's against the service host or some other mapping. This would allow me to have each service broken down in the UI instead of all my mappings at the product level.

Instrument the adapter for Prometheus metrics

As per #9, the adapter should be generating some dispatcher metrics that are scraped by Prometheus by Mixer.

It would also be beneficial however, to expose our own metrics internal to the adapter to give insight and observability as to how the adapter and its integration with 3scale system and backend its behaving.

  • Add and a expose a metrics endpoint which can be scraped directly from the adapter
  • Discuss and implement the metrics that should be gathered
  • Build a dashboard for Grafana

Deprecate the standalone `service_id` in handler config

Previously, we introduced the service discovery mechanism in #93. However some of the docs still point as embedding the service_id inside the handler and referencing the previous 1:1 relationship.

Since the CLI tool generates in the format of #93 and this is our preferred configuration method, we should deprecate the old 1:1 method.

Evaluate using RouteDirectives to leverage async reporting in batches

Envoy provides Mixer with batches of "Report" requests which in turn are batched to the appropriate adapter handler. It might benefit us to use this feature in the adapter and report in batches to 3scale backend.

As of istio 1.1.x, there is a RouteDirective feature. Essentially we might be able to add some arbitrary header and have a "Check" request in to the adapter generate a async request, to be batched to the "Report" handler.

This is documented as an alpha feature and so is open to change but could be beneficial to evaluate the flow and see if we can leverage it going forward. This would work for cached requests also.

Config generator doesn't accept URLs starting with number

It will fail on this command.

oc exec -n istio-system $(oc get po -n istio-system -o jsonpath='{.items[?(@.metadata.labels.app=="3scale-istio-adapter")].metadata.name}') \
-it -- ./3scale-config-gen \
--url="https://3scale-admin.3scale.net:443" --service="3" --token="token"

with this error message

error - generating configuration - UID generation failed: error. Generated UID does not conform to requirements. a DNS-1035 label must consist of lower case alphanumeric characters or '-', start with an alphabetic character, and end with an alphanumeric character (e.g. 'my-name', or 'abc-123', regex used for validation is '[a-z]([-a-z0-9]*[a-z0-9])?') . UID 3scale-admin-3scale-net-443-3

Support passing logging implementation to libraries

In the backend cache, we have many to-do comments based around logging since we didn't want to force a specific logging library.

This task is to create a logging interface and allow the passing of a custom logger to our libs

Use labels to match expression in rule

We can use the attribute destination.labels to allow service creators to have requests for their service be handled by the 3scale adapter.

This can be done by adding a map to the properties of our instance configs, for example

      properties:
        dest_labels: destination.labels["example"] | "unknown"

The rule would then be updated with the match expression:

spec:
  match: destination.labels["example"] == "some-value"

@unleashed if we can decide on a naming convention I will go ahead and update the rule and the documentation and test that this works as expected.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.