googlecloudplatform / endpoints-samples Goto Github PK

I had a tough time figuring out how to get an Ingress on GKE to behave with ESP, with HTTPS end-to-end. I believe the underlying issues have to do with health checks and readiness probes and how ingress-gke handles those. I commented on the following issue in that project: kubernetes/ingress-gce#18 (comment)

Once health checks can be customized, perhaps an update can be made to esp_echo_gke_ingress.yaml.

The two references you mention at the end of README lead to 404. I couldn't find those references any more in python-docs-samples.

Following the steps from here ran into an error in the start_proxy.py from pod logs

start_proxy.py: error: unrecognized arguments: --non_gpc

I believe this should be --non_gcp

Followed instruction from this readme:
https://github.com/GoogleCloudPlatform/endpoints-samples/tree/master/k8s/dns-ssl-kube-lego

CLUSTER_NAME=endpoints-dns-sample
CLUSTER_ZONE="us-west1-a"
gcloud services enable container.googleapis.com
gcloud container clusters create ${CLUSTER_NAME} --zone=${CLUSTER_ZONE} --num-nodes=3
gcloud container clusters get-credentials ${CLUSTER_NAME} --zone=${CLUSTER_ZONE}

git clone https://github.com/GoogleCloudPlatform/endpoints-samples.git
cd endpoints-samples/k8s/dns-ssl-kube-lego
deploy.sh

Enter your email address (for Let's Encrypt to send certificate expiration notifications): [email protected]
Deploying KubeLego in the kube-lego namespace
namespace "kube-lego" created
configmap "kube-lego" created
deployment "kube-lego" created
Enter the project ID: my-project-name
Deploying Kubernetes Ingress
namespace "echo" created
ingress "echo-ingress" created
Waiting for Ingress public IP address...
NAME HOSTS ADDRESS PORTS AGE
echo-ingress echo.endpoints.my-project-name.cloud.goog 80, 443 1s
NAME HOSTS ADDRESS PORTS AGE
echo-ingress echo.endpoints.my-project-name.cloud.goog 80, 443 8s
NAME HOSTS ADDRESS PORTS AGE
echo-ingress echo.endpoints.my-project-name.cloud.goog 80, 443 15s
NAME HOSTS ADDRESS PORTS AGE
echo-ingress echo.endpoints.my-project-name.cloud.goog 80, 443 22s
NAME HOSTS ADDRESS PORTS AGE
echo-ingress echo.endpoints.my-project-name.cloud.goog 80, 443 29s
NAME HOSTS ADDRESS PORTS AGE
echo-ingress echo.endpoints.my-project-name.cloud.goog 80, 443 36s
NAME HOSTS ADDRESS PORTS AGE
echo-ingress echo.endpoints.my-project-name.cloud.goog 80, 443 43s
NAME HOSTS ADDRESS PORTS AGE
echo-ingress echo.endpoints.my-project-name.cloud.goog 35.227.214.24 80, 443 49s
Ingress IP address : 35.227.214.24
Deploying Endpoints service configuration
Waiting for async operation operations/serviceConfigs.echo.endpoints.my-project-name.cloud.goog:381266e9-c515-44f1-9019-dfa6f9c834c5 to complete...
Operation finished successfully. The following command can describe the Operation details:
gcloud endpoints operations describe operations/serviceConfigs.echo.endpoints.my-project-name.cloud.goog:381266e9-c515-44f1-9019-dfa6f9c834c5
Waiting for async operation operations/rollouts.echo.endpoints.my-project-name.cloud.goog:961f109c-11e5-47e5-b347-40b68b6af17c to complete...
Operation finished successfully. The following command can describe the Operation details:
gcloud endpoints operations describe operations/rollouts.echo.endpoints.my-project-name.cloud.goog:961f109c-11e5-47e5-b347-40b68b6af17c
Service Configuration [2018-05-25r4] uploaded for service [echo.endpoints.my-project-name.cloud.goog]
To manage your API, go to: https://console.cloud.google.com/endpoints/api/echo.endpoints.my-project-name.cloud.goog/overview?project=my-project-name
Deploying the echo backend
service "echo-service" created
deployment "echo-backend" created
In a few minutes, https://echo.endpoints.my-project-name.cloud.goog should be provisioned with a Let's Encrypt certificate

Waited fro 30mins

The HTTP endpoint is working but HTTPS is not

dig echo.endpoints.${PROJECT_ID}.cloud.goog

;; ANSWER SECTION:
echo.endpoints.my-project-name.cloud.goog. 59 IN A xxx.yyy.zzz.aaa

curl http://echo.endpoints.${PROJECT_ID}.cloud.goog/.well-known/acme-challenge/_selftest

{
"code": 5,
"message": "Method does not exist.",
"details": [
{
"@type": "type.googleapis.com/google.rpc.DebugInfo",
"stackEntries": [],
"detail": "service_control"
}
]
}

kubectl --namespace=echo get secret echo-tls

Error from server (NotFound): secrets "echo-tls" not found

kubectl logs kube-lego-7dcvc89cb5-fbm5k --namespace kube-lego

E0525 16:56:28.842756 1 reflector.go:201] github.com/jetstack/kube-lego/pkg/kubelego/watch.go:112: Failed to list *v1beta1.Ingress: ingresses.extensions is forbidden: User "system:serv
iceaccount:kube-lego:default" cannot list ingresses.extensions at the cluster scope: Unknown user "system:serviceaccount:kube-lego:default"

The current document provides a nginx.conf file that does not support gRPC. (And endpoints-samples/k8s/ does not contains such .conf)

nginx.conf that supports gRPC should look something like this (auto-generated by start_esp):

# Auto-generated by start_esp
# Copyright 2017 Google Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

daemon off;

user nginx nginx;

pid /var/run/nginx.pid;

# Worker/connection processing limits
worker_processes 1;
worker_rlimit_nofile 10240;
events { worker_connections 10240; }

# Logging to stderr enables better integration with Docker and GKE/Kubernetes.
error_log stderr warn;

http {
  include /etc/nginx/mime.types;
  server_tokens off;
  client_max_body_size 32m;
  client_body_buffer_size 128k;

  # HTTP subrequests
  endpoints_resolver 8.8.8.8;
  endpoints_certificates /etc/nginx/trusted-ca-certificates.crt;


  set_real_ip_from  0.0.0.0/0;
  set_real_ip_from  0::/0;
  real_ip_header    X-Forwarded-For;
  real_ip_recursive on;


  server {
    server_name "";

    listen 9000 http2 backlog=16384;

    access_log /dev/stdout;




    location / {
      # Begin Endpoints v2 Support
      endpoints {
        on;
        server_config /etc/nginx/server_config.pb.txt;
        metadata_server http://169.254.169.254;
      }
      # End Endpoints v2 Support



      # WARNING: only first backend is used
      grpc_pass 127.0.0.1:8000 override;
    }

    include /var/lib/nginx/extra/*.conf;
  }

  server {
    # expose /nginx_status and /endpoints_status but on a different port to
    # avoid external visibility / conflicts with the app.
    listen 8090;
    location /nginx_status {
      stub_status on;
      access_log off;
    }
    location /endpoints_status {
      endpoints_status;
      access_log off;
    }
    location /healthz {
      return 200;
      access_log off;
    }
    location / {
      root /dev/null;
    }
  }
}

So I suggest adding the above .conf file.

Various example yaml does not reflect the right way to setup on k8s.

One such example is the args for this file https://github.com/GoogleCloudPlatform/endpoints-samples/blob/master/k8s/esp_echo_http.yaml

It is the same thing on the google documentation on cloud endpoints, arguments passed should be -p -a -s -v etc.

Link to the google documentation that does not comply with the code in ESP: https://cloud.google.com/endpoints/docs/openapi/get-started-kubernetes

For the b.gcr.io/endpoints/endpoints-runtime:0.3 container is there an endpoint to test for liveness and/or readiness?

And what resources does it need?

https://github.com/GoogleCloudPlatform/endpoints-samples/blob/master/gce/README.md#L28

I think there is some confusion here, as ESP usually refers to Extensible Service Proxy, and not "Endpoints Server Proxy" as suggested by the README. There are other instances using this naming.

Maybe we are areferring to different things, but if ESP means Extensible Service Proxy, we sould unify the naming conventions to avoid confusion.

This section looks pretty wrong:

• ports defined twice for ESP container.
• volumeMounts under ports, should be top-level for ESP container.

@qiwzhang: Which tutorial is this file used in? It seems like we did not catch the error in our testing (thought we haven't all finished testing yet).

Hi,

While trying out the examples for k8s, we tried all four esp_echo_*.yaml examples. When trying out the last example (esp_echo_custom_config_gke.yaml) we got the following error in the esp container:

nginx: [warn] Failed to open trusted CA certificates file: /etc/nginx/custom/trusted-ca-certificates.crt. Outgoing HTTPS requests from Endpoints will not check server certificates.
nginx: [emerg] unknown "endpoints_api_userinfo" variable

For some reason the endpoints_api_userinfo variable is not set. Not sure why.

When commenting out the proxy header like this:

# Begin Endpoints v2 Support
# proxy_set_header X-Endpoint-API-UserInfo $endpoints_api_userinfo;
# End Endpoints v2 Support

it works as expected (http/https) but obviously the X-Endpoint-API-UserInfo is not set.

I've been testing with the following command:

curl -d '{"message":"hello world"}' -H "content-type:application/json" -k http://146.148.30.62/echo\?key\=A....

Any idea what might go wrong? The service definition obviously works and authentication as well; just the userinfo is not getting set for some reason.

(we've not yet tested OAuth, perhaps X-Endpoint-API-UserInfo only works then?)

Kind regards,
Niels

https://github.com/GoogleCloudPlatform/endpoints-samples/tree/master/k8s#using-gce-l7-load-balancer-with-esp
documents how to implement the health-checking required for the loadbalancer. Can you also link to the api implementation? Since the grpc is running on 8080, I don't see how one would actually serve the /health requests or is that done transparently via nginx?

The echo sample code/docker image are dead links (404).

examples/swagger/bookstore/swagger-firebase.json

These should point toward the Echo samples' swagger.yaml