aws-containers / eks-app-mesh-polyglot-demo Goto Github PK

The problem is here: https://github.com/aws-containers/eks-app-mesh-polyglot-demo/blob/master/workshop/apps/product_catalog/app_efs.py

When using this app with EFS persistent storage, if you insert a new product into the product catalogue with a space character in the product name, then the app will break.

On line 137, the new product is written to the file on EFS: f.write('{} {}'.format(id, request.json['name']))
So if e.g. id = "121", and request.json['name'] contains "Brown shoes", the result written to file is "121 Brown shoes"

When reading from the storage, on line 71 it reads the contents of the file line by line with: (key, val) = line.split()
But in Python, this will thrown and error if the line contains more than 2 values: "ValueError: too many values to unpack (expected 2)"

Seems that this error is not handled properly in the application and the application will crash/hang due to this.

One option is to replace:

(key, val) = line.split()

with:

(key, val) = line.split(" ",1)

To make sure it splits the line into exactly 2 parts. But I haven't verified if this causes any further problems upstream when val returned is "Brown shoes" (including a space). Hopefully someone has the time and interest to test & investigate this further?

1. Virtual gateway service is missing annotations to create internet facing NLB.
2. The NLB target type is defaulted to instance.

Autoscaling - HPA - This chapter has this line : You can go to Cloudwatch -> Metrics -> All Metrics -> and select the namespace **workshop** and find the metric pod_cpu_utilization_over_pod_limit. You can also click this [cloudwatch metrics ] to see CPU Utilization (Over Limit). Here the workshop refers to cloudwatch metric namespace and not kubernetes namespace as cloudwtach namespace with name workshop is never created. This should refer to the ContainerInsights namespace.

StatefulSet using AWS EBS - Deploy StatefulSet - mysql-0 and mysql-1 are not in sync.

This is for mysql-1 which has the table with data

 kubectl -n workshop run mysql-client --image=mysql:5.7 -it --rm --restart=Never --  mysql -h mysql-1.mysql -e "SELECT * FROM dev.product"                                                             
+--------+---------------+
| prodId | prodName      |
+--------+---------------+
| 999    | Mountain Bike |
| 1      | blog-db       |
+--------+---------------+

This is for mysql-0 which does not have either the table or the data, it is also unclear which one is leader and which one is follower.

kubectl -n workshop run mysql-client --image=mysql:5.7 -it --rm --restart=Never --  mysql -h mysql-0.mysql -e "SELECT * FROM dev.product"                                                             
ERROR 1146 (42S02) at line 1: Table 'dev.product' doesn't exist

The additional replicas if scaled has the synced data.

StatefulSet using AWS EBS - Test MySQL - Application Testing section has an helm upgrade command helm upgrade --reuse-values -f ~/environment/eks-app-mesh-polyglot-demo/workshop/helm-chart/values-ebs.yaml workshop workshop/helm-chart/ this refers to the helm chart workshop/helm-chart/ but does not have instructions to cd to the direct which has the workshop folder. Without the cd users might not understand why the command is failing.

There should be an addition command such as cd ~/environment/eks-app-mesh-polyglot-demo

Hi, I'm following the tutorial and got stuck at creating GatewayRoute resource using manifest.

Versions

Kubernetes version 1.20
Platform version eks.1
kubectl client v1.16.8
kubectl server v1.20.4-eks-6b7464
helm v3.6.2
appmesh-controller v1.4.0
I have the following for creating the VGW and GatewayRoute and getting failed to find matching virtualGateway for gatewayRoute error.

VGW gets created successfully but somehow GatewayRoute cannot locate the VGW within the same namespace. I tried different combinations but wasn't able to create the GatewayRoute using the manifest. I can create one in the console without any issue.

I found some help from this link but still couldn't get past the error.

I found some help from this link https://github.com/aws/aws-app-mesh-controller-for-k8s/blob/master/docs/reference/vgw.md but still couldn't get past the error.

Error Message

Error from server (failed to find matching virtualGateway for gatewayRoute: gateway-route, expecting 1 but found 0): error when creating "appmesh/9_virtualgateway.yaml": admission webhook "mgatewayroute.appmesh.k8s.aws" denied the request: failed to find matching virtualGateway for gatewayRoute: gateway-route, expecting 1 but found 0

Manifests

apiVersion: appmesh.k8s.aws/v1beta2
kind: VirtualGateway
metadata:
  name: ingress-gw
  namespace: my-apps
spec:
  namespaceSelector:
    matchLabels:
      gateway: ingress-gw
  gatewayRouteSelector:
    matchLabels:
      gateway: ingress-gw
  podSelector:
    matchLabels:
      app: ingress-gw
  listeners:
    - portMapping:
        port: 8088
        protocol: http
  logging:
    accessLog:
      file:
        path: /dev/stdout
---
apiVersion: appmesh.k8s.aws/v1beta2
kind: GatewayRoute
metadata:
  name: gateway-route
  namespace: my-apps
  labels:
    gateway: ingress-gw
spec:
  httpRoute:
    match:
      prefix: "/"
    action:
      target:
        virtualService:
          virtualServiceRef:
            name: my-service-a

StatefulSet using AWS EBS - Test Scaling - Incorrect Data shown, this is a cosmetic issue where after scaling the replicas to 3 there is a line which says You will see 5 servers are running. Since the replicas are only scaled to 3 not sure how the servers could be 5.

Secondly after this line It will show the same data that leader has. it shows the following data that the table might have:

+--------+----------+
| prodId | prodName |
+--------+----------+
| 1      | Desk1    |
+--------+----------+

All the examples have been talking about the data of 999 | Mountain Bike and since suddenly we are saying it could have 1 | Desk1 the users might be confused.

Trying lab from this tutorial but get errors with docker entrypoint
https://www.eksworkshop.com/advanced/330_servicemesh_using_appmesh/deploy_applications/create_the_app/

Observability - Logs, Traces and Metric - Xray - chapter where it says to View Traces, does not really fetch any traces. This happens even after the deployed application is accessed. Tried this twice still did not work

catalog_deployment.yaml file has an additional . for mountPath: {{ .Values.catalog.volume.path }}. on line 26. This causes the application to not function correctly as it creates a mount inside the container with value /product. The trailing . should be removed from catalog_deployment.yaml

I deployed the app using the steps mentioned in the guide. Initial deployment works well and I was able to browse product catalog UI through ELB endpoint without any issues. But, after adding efs storage, while attempting to add a product, product catalog UI loads forever and throws 400 Badrequest error. Below is the error log from the frontend pod. Am I missing any steps?

 ............
     url: '',
      method: null,
      statusCode: 400,
      statusMessage: 'BAD REQUEST',
      client: [Socket],
      _consuming: false,
      _dumped: false,
      req: [Circular *1],
      responseUrl: 'http://prodcatalog.workshop:5000/products/',
      redirects: [],
      
    }
  },
  response: {
    status: 400,
    statusText: 'BAD REQUEST',
    headers: {
      'content-type': 'application/json',
      'content-length': '142',
      'x-amzn-trace-id': 'Root=1-6621ab25-b3af8bb3c543e15333aee057',
      'access-control-allow-origin': '*',
      server: 'Werkzeug/1.0.1 Python/3.9.6',
      date: 'Thu, 18 Apr 2024 23:22:13 GMT'
    },
    config: {
      url: 'http://prodcatalog.workshop:5000/products/',
      method: 'get',
      headers: [Object],
      transformRequest: [Array],
      socket: [Socket],
      _header: 'GET /products/ HTTP/1.1\r\n' +
        'Accept: application/json, text/plain, */*\r\n' +
        'User-Agent: axios/0.21.1\r\n' +
        'X-Amzn-Trace-Id: Root=1-6621ab26-8352871e19ebe2e6c7766f67;Parent=f09b2f2e6d7042dd;Sampled=0\r\n' +
        'Host: prodcatalog.workshop:5000\r\n' +
        'Connection: close\r\n' +
        '\r\n',
      _keepAliveTimeout: 0,
      _onPendingData: [Function: noopPendingOutput],
      agent: [Agent],
      socketPath: undefined,
      method: 'GET',
      maxHeaderSize: undefined,
      insecureHTTPParser: undefined,
      path: '/products/',
      _ended: true,
      res: [IncomingMessage],
There was error in Product Catalog get call from frontend
 ............

Reproduction Steps

1. Deploy the app using steps mentioned here
2. Add the EFS storage using the steps mentioned here. It is basically using the
3. After efs is added, the guide says, we could add the products in the product catalog UI. Issue is after pressing add button, the page loads forever.

Now lets add a product Mike with ID as 1 to the Product Catalog from the UI, it will show below product added.

I deployed the pods in EKS, frontend-node and proddetail seems running fine, but prodcatalog pod goes in crashloopbackoff.

Here is the pod log error.

ec2-user:~/environment/eks-app-mesh-polyglot-demo/deployment (master) $ kubectl logs -f prodcatalog-5b85b57487-r4qp7 -n prodcatalog-ns
Traceback (most recent call last):
  File "/usr/local/bin/flask", line 5, in <module>
    from flask.cli import main
  File "/usr/local/lib/python3.9/site-packages/flask/__init__.py", line 21, in <module>
    from .app import Flask, Request, Response
  File "/usr/local/lib/python3.9/site-packages/flask/app.py", line 26, in <module>
    from . import cli, json
  File "/usr/local/lib/python3.9/site-packages/flask/json/__init__.py", line 21, in <module>
    from itsdangerous import json as _json
ImportError: cannot import name 'json' from 'itsdangerous' (/usr/local/lib/python3.9/site-packages/itsdangerous/__init__.py)

I think it's the issue of python docker image version of the way we are using pip ? Any suggestion or fixes?

Produt_catalog pod doesn't run and the docker file entry point needs to have the first argument with a "sh" to successfully run the image.

Doing a walkthrough of the blog https://aws.amazon.com/blogs/containers/using-iam-database-authentication-with-workloads-running-on-amazon-eks/

Getting an error after step #9 in the blog.

pymysql.err.OperationalError: (2003, "Can't connect to MySQL server on 'irsa-test-instance.cluster-c1oewrerw6um3jc.us-east-1.rds.amazonaws.com' ([SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate

Downloading the new us-east-1 cert from this page https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/UsingWithRDS.SSL.html#UsingWithRDS.SSL.RegionCertificates

using link:
https://truststore.pki.rds.amazonaws.com/us-east-1/us-east-1-bundle.pem

Updating the prodcatalog pod with this cert help resolve the issue.

Suggestion:
Update the repo with new certificate

I grabbed workshop/otel-collector-config.yaml from master and installed per instructions in the official workshop. It looks like there is some configuration missing.

Logs from observability-collector:

2023/05/15 23:00:39 AWS OTel Collector version: v0.17.0
2023/05/15 23:00:39 found no extra config, skip it, err: open /opt/aws/aws-otel-collector/etc/extracfg.txt: no such file or directory
Error: failed to get config: cannot unmarshal the configuration: unknown extensions type "sigv4auth" for "sigv4auth" (valid values: [health_check pprof zpages memory_ballast awsproxy ecs_observer])
2023/05/15 23:00:39 application run finished with error: failed to get config: cannot unmarshal the configuration: unknown extensions type "sigv4auth" for "sigv4auth" (valid values: [health_check pprof zpages memory_ballast awsproxy ecs_observer])