Question/documentation: GRPC-JSON transcoder and grpc ext_authz filter (envoy control plane ext_authz ) integration about envoy HOT 6 OPEN

cc @nareddyt

from envoy.

gRPC transcoder with ext_authz should work well, I believe I personally ran a similar configuration in the past.

Your config LGTM. I noticed the following:

• The router is only configured with gRPC routes and not HTTP routes. I believe this is OK, as the transcoder doesn't read route config and does it's own path matching on the HTTP request headers.
• The transcoder comes before ext_authz. This is also good, you should transcode requests before doing authz to prevent authz bypass.

To help further debug, please provide the following details:

1. What is the HTTP request your client is sending? I see POST /http_reuest_url/xyz/abc?a=3&b=4 with 0 body bytes, which I am suspicious about. gRPC transcoder might be expecting a GET request. Please share the full HTTP request (curl command would be nice) and the HTTP annotation on your gRPC service definition.
2. Run envoy with --log-level debug and share the application logs. gRPC transcoder outputs many logs which would help debug here.
3. Share your full Envoy config (including clusters) as text please, it's hard to parse the VS Code image.

2.a). Trying not clearing the the route cache, clear_route_cache

FYI gRPC transcoder will automatically clear the route cache once it translates the HTTP request unless match_incoming_request_route=true. Let's debug with the default behavior where match_incoming_request_route=false. Your route config looks correct for this case.

from envoy.

@nareddyt Thanks for looking out, from long time we got stuck on this.
if possible please suggest further.

1. Attaching the client details

curl request here: curl --location 'local.api.com/sms_grpc/v1/sms/v1/api/v1/org/sendsms?message=hello&senderId=fg&number=6778&templateId=123'
--header 'X-API-KEY: abcdefgh'.

which is able to match the envoy grpc service and without envoy ext_auth and it gives response.

grpc service anotation: service SmsApiService {
rpc SendSmsExternalGETV1(SendSmsExternalGETV1Request) returns (SendSmsExternalGETV1Response){
option (google.api.http) = {
get: "/sms_grpc/v1/sms/v1/api/v1/org/sendsms"
};
}
}
client response in postman:

2 envoy debug log and screen shot where we are able to get the details its making POST API call from inside of ext_auth_filter,
envoy_debuglog.txt

3. i am running envoy with dynamic configuration sharing listener and cluster config:
   cds.txt
   lds.txt
   bootstrap_envoy_dynamic.txt

from envoy.

Thanks for providing the logs and configuration files, that gave me everything I needed to debug the issue. I'll walk you through it.

Logs show the request is following the request path correctly:

Client --> grpc transcoder --> ext_authz --> upstream cluster new_api_grpc_cluster

In fact, the HTTP 501 is from your own backend. Notice the response has header x-envoy-upstream-service-time which indicates it tried reaching out to your backend (upstream).

[2024-04-25 22:21:48.961][15][debug][http] [source/common/http/conn_manager_impl.cc:1838] [Tags: "ConnectionId":"0","StreamId":"18191014628947526340"] encoding headers via codec (end_stream=false):
':status', '501'
'content-type', 'application/json'
'x-envoy-upstream-service-time', '1'
'content-length', '167'
'date', 'Thu, 25 Apr 2024 22:21:48 GMT'
'server', 'envoy'
'x-request-id', '3241b8e0-c1eb-46d8-8f1d-fce751fbad69'

Obviously I don't have your backend to verify this. But you can try making the same request Envoy makes directly to your backend (host.docker.internal:8897, assuming you expose that container port) and observe the response. Request:

[2024-04-25 22:21:48.960][15][debug][router] [source/common/router/router.cc:738] [Tags: "ConnectionId":"0","StreamId":"18191014628947526340"] router decoding headers:
':scheme', 'http'
':method', 'POST'
':path', '/edugo.api.services.sms.v1.SmsApiService/SendSmsExternalGETV1?org_id=5b00abbab9'
':authority', 'local.api.com'
'x-api-key', 'abcdefgh'
'x-org-id', '5b00abbab9'
'x-forwarded-proto', 'http'
'postman-token', 'ec105acf-c79b-4ffd-930d-4a07d9480a79'
'te', 'trailers'
'x-envoy-auth-partial-body', 'false'
'x-envoy-original-method', 'GET'
'x-envoy-original-path', '/sms_grpc/v1/sms/v1/api/v1/org/sendsms?message=hello&senderId=fg&number=6778&templateId=123'
'x-request-id', '3241b8e0-c1eb-46d8-8f1d-fce751fbad69'
'content-type', 'application/grpc'
'x-envoy-expected-rq-timeout-ms', '60000'

I do have a guess for why your backend is sending back HTTP 501. Take a look at the HTTP path that envoy sends to your backend:

':path', '/edugo.api.services.sms.v1.SmsApiService/SendSmsExternalGETV1?org_id=5b00abbab9'

IIUC the query parameter ?org_id=5b00abbab9 in the path is not compliant with the gRPC over HTTP2 specification. All metadata you wish to forward to gRPC server should be as gRPC metadata, i.e. lowercase HTTP headers like x-api-key and x-org-id.

Why is Envoy sending this query parameter to the backend? It is because of the ext_authz response from your ext_authz server:

status {
}
ok_response {
  headers {
    header {
      key: "X-ORG-ID"
      value: "5b00abbab9"
    }
  }
  headers {
    header {
      key: ":scheme"
      value: "http"
    }
    keep_empty_value: true
  }
  headers {
    header {
      key: ":method"
      value: "POST"
    }
    keep_empty_value: true
  }
  headers {
    header {
      key: ":path"
      value: "/edugo.api.services.sms.v1.SmsApiService/SendSmsExternalGETV1"
    }
    keep_empty_value: true
  }
  headers {
    header {
      key: "x-forwarded-proto"
      value: "http"
    }
    keep_empty_value: true
  }
  headers {
    header {
      key: "postman-token"
      value: "ec105acf-c79b-4ffd-930d-4a07d9480a79"
    }
    keep_empty_value: true
  }
  headers {
    header {
      key: "te"
      value: "trailers"
    }
    keep_empty_value: true
  }
  headers {
    header {
      key: "x-envoy-auth-partial-body"
      value: "false"
    }
    keep_empty_value: true
  }
  headers {
    header {
      key: ":authority"
      value: "local.api.com"
    }
    keep_empty_value: true
  }
  headers {
    header {
      key: "x-envoy-original-method"
      value: "GET"
    }
    keep_empty_value: true
  }
  headers {
    header {
      key: "x-envoy-original-path"
      value: "/sms_grpc/v1/sms/v1/api/v1/org/sendsms?message=hello&senderId=fg&number=6778&templateId=123"
    }
    keep_empty_value: true
  }
  headers {
    header {
      key: "x-request-id"
      value: "3241b8e0-c1eb-46d8-8f1d-fce751fbad69"
    }
    keep_empty_value: true
  }
  headers {
    header {
      key: "content-type"
      value: "application/grpc"
    }
    keep_empty_value: true
  }
  headers_to_remove: "apikey"
  query_parameters_to_set {
    key: "org_id"
    value: "5b00abbab9"
  }
}
dynamic_metadata {
  fields {
    key: "x-edu-org-id"
    value {
      string_value: "5b00abbab9"
    }
  }
}

The following part should be removed, as it is not compliant with gRPC over HTTP2:

  query_parameters_to_set {
    key: "org_id"
    value: "5b00abbab9"
  }

In general, i don't recommend you rewrite the entire request via the ext_authz response. Just add on the headers you need, like x-org-id, and omit the rest of the fields.

from envoy.

TLDR - there is no issue with gRPC JSON transcoder or ext_authz filter integration. It is due to malformed ext_authz response causing the user's backend to respond with 501.

from envoy.

i have removed query_parameters_to_set and make path as grpc compliant and its working fine. Thanks a lot, for giving the details insight and suggestion.

We have used query params to use this feature of grpc-gateway: https://github.com/googleapis/googleapis/blob/master/google/api/http.proto#L223
Means if we pass some field as query params and they are mentioned/member of in body proto struct, it will be filled with passed query data.

from envoy.