orchestracities / boost Goto Github PK

Refactor Go code in orionadapter a bit to make it slightly more maintainable and easier to understand. Even if a heavy hammer would be welcome, it's kinda pointless to go overboard just now (remember #29?!) so do just as little as possible to make the code a bit more decent.

Deploy Orion in a K8s/Istio mesh in the cloud with our adapter. Modulo some (hopefully!) minor changes we should be able to replicate the local Minikube set up we used for the demo preview in Dec 2019.

Configuration entry names for DAPS related keys/certs are confusing and not well documented.

Gernot's proposal:

connector_private_key... (as i.e. xxxx.key)...
connector_certificate...(as i.e. xxx.cert)...
and daps_certificate... (as i.e. dapsaisec.cert) ...
to have a reference to the files provided...

And if it is only a part of the whole file to cut out, we should describe that.

and as we're at it, we should also find a better name for idsa_public_key

So I've had endless fun debugging Mixer calls to find out why with the latest changes and our caching in place the Mixer hits the adapter twice for each incoming HTTP request. Thanks to our caching though we basically pay no price for this as the second call returns in about 0.2ms on average which is peanuts but still annoying. I don't think it's a good idea to waste any more dev cycles on this, so will park the issue here for posterity.

A) Incoming requests:

1. We have already extracted the header-Header from the message and you have all enclosed data at hand.

2. We have extracted and validated the DAPS JWT
   -> New: We have to compare the issuer ID from the header-Header with the issuer from the DAPS JWT

3. We already have the Verb and Path from the request

4. New: We now need to extract and secret-validate the data from the Authorization JWT.
   (Secret must be configured, Domain and AppID now come from the Authorization JWT

5. Update the XACML request (I will create a temp
   -> with shortest exp Date

Outgoing Reponse:
The response does not need an Authorization token
The header-Object Header should be created and attached as it is now, until the next version.

B) Outgoing requests (Notifications):
The header-Object Header should be created and attached as it is now, until the next version.
No Generation/Attachment of an Authorization header (might be already added by Orion)

Incoming Response:

1. We have already extracted the header-Header from the message and you have all enclosed data at hand.
2. We have extracted and validated the DAPS JWT
   -> We have to compare the issuer from the header-Header with the issuer from the DAPS JWT

Need to be able to output an attribute containing the server token to be added to Orion responses.

At the moment both pub and private keys sit in the adapter config. While it's okay for the pub key to be there, we should find a better home for the private key---think raising the bar for security. Figure out if Istio Citadel could help here or should we rather use K8s secrets/tmpfs volumes? Any other options?

The advantage of adapter config storage is access speed and caching. We should consider these two factors too when scratching around for a solution.

This Connector will issue and visualize the calls Query-Calls to the provider connector. It helps to understand creating of tokens and visualizes the respsones

Text from IDS Communication Guide

b) The second more secure option is to not only add IDS header part from the multipart-messages as an additional header to the original HTTP but also a signature following the W3C specifications for signing HTTP messages (https://tools.ietf.org/html/draft-cavage-http-signatures-11 - section 2) (the content of the header-Header is matching the example from 4.4.2 2. QueryMessage):

A digital signature helps to ensure that the message was not tampered with during transit. It calculates a signature from a constructed string following a defined schema.

It considers sequential HTTP message headers as mentioned in the (exemplary) additional headers-HTTP-Header and also the message payload if available. The procedure how to construct the concatenated signature string is described here (https://tools.ietf.org/html/draft-cavage-http-signatures-11 - section 2.3). The creation of a signature is described here (https://tools.ietf.org/html/draft-cavage-http-signatures-11 - section 2.4) and the verification of a signature is described here (https://tools.ietf.org/html/draft-cavage-http-signatures-11 - section 2.5).

The created-HTTP-Header expresses when the signature was created. The value of the keyId-HTTP-Header is providing an opaque string that the server can use to look up the component they need to validate the signature and it is depending on the content of the algorithm-HTTP-Header describing the algorithm that is being used to create the signature.

The final signature string is then added as the value to the signature-HTTP-Header.

Remarks:

• change to "header" from "dth-header" ... this is due to alignment with the standard
• value of header must be a quoted string with all doublequote escaped
• no spaces in the quoted string

Message-Type for Requests is "QueryMessage"
Message-Type for Responses is "ResultMessage"

Caching is an important aspect of the Istio architecture:

• https://istio.io/blog/2017/mixer-spof-myth/

From what I've gathered up to this point through debugging and code inspection is that the ValidDuration and ValidUseCount fields returned in adapter responses get used by the Mixer/Envoy to cache those responses.

This affects both token validation and generation. In fact, if a positive validation response gets cached for a time period C, then we've got to be sure

now + C < expiry_date(token)

otherwise security flies out of the window.

If you don't set the ValidDuration and ValidUseCount fields explicitly in the adapter response, it looks like the Mixer fills them in for you with default values. I ran a test where the adapter just ignores the fields and then I could see the Mixer client outputting some values instead:

$ sh scripts/send-token.sh my.fat.jwt
...
Check RPC completed successfully. Check status was OK
Valid use count: 10000, valid duration: 1m0s

So it looks like if we want security, we'll have to deal with caching explicitly to make sure the above invariant holds true at all times. Ditto for the response attribute we use to store the generated server token.

Hear ye, hear ye! His Majesty Istio I.V commands all our loving subjects whatsoever to discover and apprehend Sir Adapter Boost, Baronet, to carry him before the Chief Magistrate of the country, town or place where he shall be apprehended. Sir Adapter Boost shall be hanged, drawn and quartered!
God save the King!

Joking aside, Istio 1.5 got released yesterday (5 Mar 2020) and apparently they're going to ditch the whole Mixer/Adapter ecosystem:

Mixer is deprecated. The functionality provided by Mixer is being moved into the Envoy proxies. Use of Mixer with Istio will only be supported through the 1.7 release of Istio.

See deprecation warnings on:

• https://istio.io/docs/reference/config/policy-and-telemetry/
• https://istio.io/docs/reference/config/policy-and-telemetry/adapters/

Also the adapter dev guide hasn't been updated with a migration path or anything related to deprecation:

• https://github.com/istio/istio/wiki/Mixer-Out-Of-Process-Adapter-Dev-Guide

Given the fast-paced release cycle, 1.7 is likely to be released before the end of the year. Where does that leave us? Are we going to deliver something that in practice will never be used?

One option would be to still use an Istio mesh but deploy our own sidecar instead, but we need to discuss a way forward.

We cache adapter Istio config to be able to request an ID token to add to HTTP requests originating from the mesh. This is a consequence of #25 which implemented a workaround to #24. However if there's a cache miss, we won't be able to add the ID token to the outbound HTTP request at hand.

Notice the Mixer passes the current config to the adapter on each mesh inbound HTTP call and the adapter caches it every time, replacing any old cached config with the most recent one. So a cache miss can only happen if

1. The adapter process starts.
2. An outbound HTTP request is intercepted by the Envoy filter---think Orion notification.
3. The filter calls the adapter to get an ID token to add to the request.

and no mesh inbound HTTP request gets intercepted before (3). My gut tells me this isn't very likely to happen in practice, especially for high-volume sites and also considering that client requests will typically hit the mesh before a mesh service tries to call an external service---e.g. a subscription/entity update has to happen before Orion sends out a notification. However no matter how likely, the issue is there, so we might fail to inject ID tokens into outbound Orion notifications occasionally.

We should look for an easy way to solve this, e.g. in principle we should be able request current configuration from Galley or the Mixer itself so that if there's a cache miss, we would have a fallback.

Replace dummy HttpBin with Orion. We're using HttpBin in our local Minikube deployment, we have to swap it out for Orion. @chicco785 has developed a number of Helm charts for Orchestra we can possibly use out of the box or just tweak a little.

@gboege Need to decide what FiWare components to bring in exactly.

Install a public instance of KeyRock/AuthZForce to lighten Andrea's system a bit from laod

See: istio/istio#20602

I spent hours chasing non-existing bugs. Why?! I'm still building, testing & deploying manually---the horror!---so it's easy to wind up with an inconsistent env, e.g. outdated files, etc.

So we're reaching the threshold where introducing some build automation will pay off. I'm holding the fire for now though, procrastinating until this really hurts...

@gboege needs refinement.

Adding an Istio sidecar to Orion's pod makes the service unreachable. Also it makes it real hard to do any service initialisation through K8s init containers. Sob, sob.

After so much grief trying to figure out why Envoy doesn't play well with Orion, I've decided to park this issue here for now and deal with it some other time to avoid dev paralysis. What's happening is that, with a sidecar, any HTTP call to Orion from another host, be that inside or outside the mesh, fails with a 503. Requests actually hit the sidecar but then something goes wrong when forwarding to Orion which never actually gets the request and ultimately Envoy returns a 503 to the client. Notice that local HTTP calls actually work, i.e. if you log in to the Orion pod and e.g.

$ curl -v http://localhost:1026/v2

you'll get the expected response from Orion.

One thing to consider is that deploying Orion without a sidecar may actually be an option. In fact:

• The adapter can still intercept requests to Orion, extract the IDS header, do token validation, and ditch or authorise the request as the case may be.
• The Envoy filter won't work since there's no sidecar. So when Orion makes a request to a host external to the mesh, the IDS header can't be added to the request. But if we find a solution to #24, then the adapter will be able to intercept outgoing requests too, so problem solved. And we can ditch the Envoy filter too which is a bit of a hack anyway.
• We can use a simple init container to create DB indexes, etc. With a sidecar in between, we'd need to come up with a real clever init process instead...

PR #39 implemented pretty much everything we need for AuthZ support except for authentication with AuthZ, i.e. the adapter should authenticate with AuthZ beforehand. As far as I understand this is not strictly needed for the beta release and is only relevant if AuthZ gets deployed outside of the mesh?

Below is @gboege's note about it, taken from the doc detailing AuthZ requirements for the beta release---like I said, #39 implemented all of that stuff, except for the below

QUESTION:
One thing, that just came to my mind is: We might need authorization for AuthZForce.
Please have a look at the function "authenticate" https://github.com/ging/fiware-pep-proxy/blob/master/lib/idm.js.
Straight forward code, plus also a bit caching and configuration.

The README is a disaster area. Sort it out! Most of its content belongs somewhere else. It's time we started documenting software architecture---e.g. functional, process, deployment, implementation views, etc. At the moment just a few diagrams and paragraphs for each topic will do I reckon, i.e. no need for anything fancy, but having a compass will definitely help navigate through the code!

We should be able to decode/encode correctly any HTTP header value as specified by section 3.2.6. We need this for processing the JSON containing the IDS token which gets encoded as an HTTP quoted string in the IDS header---see this example.

Ideally we'd use an HTTP lib to do this, but I couldn't manage to get it right with textproto, http & friends---i.e. Go's standard approach to handling HTTP messages. So at the moment we have a stopgap solution in place to decode quoted values which is okay-ish for a prototype but not really what you'd like to have for a production grade product.

Here's a possible way forward:

• Scrap current poor man's implementation of parseIdsHeaderValue in sec/tokenrep.go
• See if you can implement it using Go's built in libs---see notes below.
• Failing that, comb the interwebs to find a lib that implements robust handling of HTTP headers as spec'd by RFC 7230.

Another way to look at it is that we could perhaps change the way the JSON header value gets encoded. For example, wouldn't it be easier to encode the JSON payload in Base 64?

@gboege needs refinement.

What we will need for the future (not now), is a description of how to add/change the headers that will go into the signature process. But this is step 2

@gboege I reckon if it's just about choosing the header names to use, that's a no-brainer. It can be done with Istio operator config when mapping incoming requests to Mixer attributes as in:

https://github.com/orchestracities/boost/blob/dev/deployment/sample_operator_cfg.yaml#L27

@gboege kinda rings a bell, but need to refresh my memory...

gb: It is what we have spoken about:
Consumer: Send out request, receiving response
Provider: Receiving request, sending response
FIWARE CIM Connector has to be able to handle both, because reqests are incoming, notifications outgoing.
So there is no only Provider or Consumer Connector

Then we have to check the work for creating microservices for 1) a self-declaration endpoint; 2)
a broker managing endpoint

@gboege needs refinement; should we create GH issues out of this card?

gb: I do build it this week in Ballerina

We should get rid of the following params in the adapter config:

• idsa_public_key
• idsa_private_key

In fact, idsa_public_key is the pub key the adapter uses to validate tokens in incoming requests from another IDS connector whereas idsa_private_key is the corresponding private key. Since we expect incoming requests will contain an ID token the client connector got from the same DAPS server we use to get out ID token, then:

idsa_public_key = daps.server_certificate

We'd like to be able to ditch the request IDS-DTH header with the client token before passing on the request to Orion. It looks like that isn't the easiest thing to do with Istio if our adapter should be able to access the header's value---which it should since we've got to validate the token!

As things stand now, if we remove the header through Istio configuration then the header value (token) the adapter gets to see is an empty string. Hence token validation would always fail (even for valid tokens!) if we removed the IDS-DTH header from the request.

Since we've got to validate tokens, at the moment we are not removing the header from the client request and that header will also be in the request the Envoy forwards to Orion. Is there any way to have both token validation and header removal?

Validation works for the most part but there are some gotchas. Specifically, to jwt-go (the lib we use under the bonnet for validation) a field with a 0 value is the same as the field not being there! So e.g. this token (pseudo code):

{ alg: RS256 }.{ exp: 0 }.valid-rs256-signature

passes validation even though it expired at the beginning of the epoch!

Instead of delegating validation to jwt-go, consider rolling out your own. Specifically, for the following JWT registered claims: exp ("expires at"), iat ("issued at"), nbf ("not before").