Comments (11)
I have done some further digging and found what seems to be a solution for configuration esp + ingress + grpc on GKE
There are 2 issues at hand which were not clear to me in the beginning. Nginx does not seem to support http/2 without ssl so you have to configure SSL on esp. Another problem is that you need to enable to the esp health check to answer on the root path (/)
Let's break down the issue in 2
1. SSL
If we look at how ESP is configured we can see that if you choose ssl (using the --ssl_port
flag) instead of --http2_port
, it will configure SSL with http2
This of course means you need to mount the certificates in the esp container as per the the esp documentation
You should be able to mount the same secret used by the ingress itself
2. Health Check
I gave up trying to get the readiness probes to work and looked for a way to configure esp to respond on / with 200 while proxying everything else to the backend.
Turns out that this is possible. ESP has a -z
(also --healthz
) flag to supply which path to respond to health checks on. The issue here is the template looks like this
% if healthz:
location = /${healthz} {
return 200;
access_log off;
}
% endif
So we cannot supply /
as the path in the argument because this would give us location = // {
which is not valid.
The trick here is to supply a whitespace as the argument for the health check path.
conclusion
This is how I configured esp on my pod to take care of both issues above
- name: endpoints-proxy
image: gcr.io/endpoints-release/endpoints-runtime:1
args: [
"--ssl_port=8080",
"--backend=grpc://127.0.0.1:50051",
"--service=myendpoint.endpoints.myproject-17272.cloud.goog",
"--rollout_strategy=managed",
"--service_account_key=/etc/nginx/creds/endpoints-credentials.json",
"-z",
" "
]
ports:
- containerPort: 8080
protocol: TCP
volumeMounts:
- name: ingress-tls
subPath: tls.crt
mountPath: /etc/nginx/ssl/nginx.crt
readOnly: true
- name: ingress-tls
subPath: tls.key
mountPath: /etc/nginx/ssl/nginx.key
readOnly: true
This worked for me though I have not yet performed extensive testing. I point the service to port 8080
above and ingress to whatever port the service exposes. Ingress then configures the load balancer, it stays healthy and I was able to perform grpc calls on it using the google endpoints domain (once I configured the endpoints to point to the Ingress load balancer IP).
Now if anybody could tell me how to configure GRPC-WEB through google endpoints, that would complete the package
from endpoints-samples.
@unludo sure
The configurations above are based on the fictional config I posted in my previous post (so, for example, it should be obvious the secret name pointed by ingress is the same secret mounted in the pod)
My service looks like this (assuming the pod it points to has the label app: myApp
)
apiVersion: v1
kind: Service
metadata:
name: myService
annotations:
cloud.google.com/app-protocols: '{"http2":"HTTP2"}'
spec:
type: NodePort
ports:
- name: http2
port: 8080
targetPort: 8080
protocol: TCP
selector:
app: myApp
And my Ingress looks like this (Assuming use of let's encrypt through cert manager
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: myService-ingress-endpoints
annotations:
kubernetes.io/tls-acme: "true"
kubernetes.io/ingress.class: "gce"
certmanager.k8s.io/acme-http01-edit-in-place: "true"
certmanager.k8s.io/cluster-issuer: "letsencrypt-prod"
certmanager.k8s.io/acme-challenge-type: "http01"
spec:
tls:
- hosts:
- "myapp.endpoints.myprojectID.cloud.goog"
secretName: ingress-tls
backend:
serviceName: myService
servicePort: 8080
from endpoints-samples.
Hi,
same issue here.
As ESP does not allow root path (https://cloud.google.com/endpoints/docs/openapi/openapi-limitations#operations_on_url_root_path) and Ingress GLB does a health check on /
(https://cloud.google.com/kubernetes-engine/docs/tutorials/http-balancer), it seems impossible to configure the GKE Ingress + ESP + gRPC stack.
UPDATE :
- To make it working, I've updated manually the health check endpoint in Compute Engine > Health checks section
UPDATE 2 :
According to this link https://github.com/kubernetes/kubernetes/tree/master/cluster/addons/cluster-loadbalancing/glbc, Ingress scans for readiness probes and configures automatically its health check endpoint.
It's working for me now
from endpoints-samples.
@paoesco Can you post how you got Grpc + Esp to work on GKE? I have been struggling finding detailed information and I seem to have hit a wall on the matter.
My health check is always yellow. I have tried configuring readiness probes but it did not help
from endpoints-samples.
And here it is :
apiVersion: v1
kind: Service
metadata:
name: myapp-svc
spec:
# NodePort is mandatory for Ingress to perform load balancer
type: NodePort
ports:
- port: 80
protocol: TCP
targetPort: 8081
selector:
app: myapp-app
sessionAffinity: None
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: myapp-app
spec:
progressDeadlineSeconds: 600
replicas: 1
revisionHistoryLimit: 10
selector:
matchLabels:
app: myapp-app
template:
metadata:
labels:
app: myapp-app
spec:
containers:
# ESP container for Google Cloud Endpoints
# Every requests are caught by ESP first to perform
# validation/monitoring/authentication and then redirects
# to our app
- name: esp
image: gcr.io/endpoints-release/endpoints-runtime:1
args: [
"--http_port", "8081",
"--backend", "127.0.0.1:8080",
"--service", "myapp.endpoints.PROJECT-XXX.cloud.goog",
"--rollout_strategy=managed",
"-z", "healthz"
]
# HEALTH CHECK START
# Used by ingress to perform health check
readinessProbe:
httpGet:
path: /healthz
port: 8081
ports:
- containerPort: 8081
# HEALTH CHECK END
# Our app container
- name: myapp-app
image: gcr.io/PROJECT-XXX/myapp-app-image:latest
imagePullPolicy: Always
ports:
- containerPort: 8080
protocol: TCP
resources: {}
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
dnsPolicy: ClusterFirst
restartPolicy: Always
schedulerName: default-scheduler
securityContext: {}
terminationGracePeriodSeconds: 30
from endpoints-samples.
Thank you very much but is that gRPC? The esp is configured with http and not http2.
Do you not need to set the esp backend to be grpc://127.0.0.1:8080?
from endpoints-samples.
I was also never able to get the readiness probe to behave correctly. Perhaps the key is making sure containerPort
is set? (kubernetes/ingress-gce#241 (comment)). On ingress-gce@master
the docs
directory looks very much in flux and the examples
directory has been removed. Here's a link to the latest tag of the health checks doc: https://github.com/kubernetes/ingress-gce/blob/v1.4.1/examples/health-checks/README.md. Perhaps that will help? Based on the discussion in kubernetes/ingress-gce#42, I'm still not convinced readiness probes always work.
from endpoints-samples.
I did configure containerPort
and I never saw my load balancer configuration point on the cloud console point to any other port than the http2 port
from endpoints-samples.
Did you ever make progress on getting grpc-web support through google endpoints?
from endpoints-samples.
@jcramb Yes I did.
You can find more details on the grpc-web PR on the esp code.
It works but only if you use the binary grpc-web wire format which does not support streaming
from endpoints-samples.
Hello @alethenorio,
If it's not too much trouble, could you please post the rest of your configuration for grpc endpoint with ssl (service, ingress)? I don't manage to make it work and can't find what I am missing.
thanks
from endpoints-samples.
Related Issues (12)
- nginx: [emerg] unknown "endpoints_api_userinfo" variable HOT 2
- add liveness, readiness and resources in k8s examples HOT 1
- GCE L7 Loadbalancer and readiness HOT 2
- Examples for k8s are wrong HOT 2
- References lead to 404
- kubernetes SSL sample not working HOT 3
- ESP =? Extensible Service Proxy HOT 1
- [k8s] The configuration file for the custom nginx.conf example in GKE (with gRPC) is invalid.
- clients: fix references to bookstore and non-existent files HOT 1
- Fix duplicate ports in kubernetes/echo.yaml HOT 1
- Typo in start args for kubernetes ESPv2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from endpoints-samples.