aws / amazon-ecs-service-connect-agent Goto Github PK

View Code? Open in Web Editor NEW

28.0 10.0 10.0 8.58 MB

Amazon ECS Service Connect Agent

License: Apache License 2.0

Makefile 0.40% Go 99.60%

amazon-ecs-service-connect-agent's Issues

[Feature Request] X-Ray support on Service Connect

Summary

X-Ray support on Service Connect

Description

I'd like to have the ability to configure X-Ray when using Service Connect, with these env vars:

ENABLE_ENVOY_XRAY_TRACING

AWS_XRAY_DAEMON_ADDRESS
XRAY_DAEMON_PORT
XRAY_SAMPLING_RATE
XRAY_SAMPLING_RULE_MANIFEST
XRAY_SEGMENT_NAME

Bug: nofile soft limit on EKS Fargate causes connection limits and crashes

Summary

We have workloads running on EKS Fargate with an aws-appmesh-envoy sidecar injected by AWS App Mesh Controller.
The appnet agent process (PID 1) has a nofile soft limit of 65535, while the forked envoy process has a nofile soft limit of 1024 only.

kubectl exec -i -t -n default example-5ff7dbfc5d-strcr -c envoy -- sh
sh-4.2$ cat /proc/1/cmdline; echo
/usr/bin/agent
sh-4.2$ grep open /proc/1/limits
Max open files            65535                65535                files
sh-4.2$ cat /proc/31/cmdline; echo
/usr/bin/envoy-c/tmp/envoy-config-459706937.yaml-linfo--drain-time-s20
sh-4.2$ grep open /proc/31/limits
Max open files            1024                 65535                files

This imposes a limits of max. ~480 possible TCP connections, since a file handle is created for each ingress/egress.
Reaching the limit causes the envoy process to crash and being restarted by the appnet agent (#181), which causes outage.

Steps to Reproduce

Please refer to support case 170713370901828 for this.

Are you currently working around this issue?

We are unable to workaround this issue, because the appnet agent seems to be closed source.

[BUG] gRPC load balancer does not work

Summary

Service Connect failed to distribute traffic between the two services correctly when using gRPC

Description

We utilize gRPC for communication between two services: Service A (client) and Service B (server), with Service B running multiple instances for scalability.
Under AppMesh, load balancing across these instances was effective. However, since migrating to Service Connect, all traffic from Service A is directed to a single instance of Service B, bypassing the expected load distribution.

Expected Behavior

Service Connect should distribute traffic across all instances of Service B similarly to how AppMesh did.

Observed Behavior

All traffic is routed to only one instance of Service B, leading to potential performance bottlenecks.

Environment Details

We are using AWS ECS with fargate.

[BUG]

Summary

ECS Service connect agent becomes unhealthy.

Description

I have ECS cluster with several Services running under EC2 launch type (2-3 t2.micro) and some of them use ServiceConnect feature of AWS ECS.
However, from time to time services that are in bridge or awsvpc network are just restarted due to ecs-service-connect-agent:interface-v1 sidecar container becomes UNHEALTHY. All subsequent deployments of Services to that instance are failing due to ecs service connect agent healthcheck. After that I have to restart EC2 instance (maybe docker restart would also help?).

Backend:

"networkMode": "bridge",
"portMappings": [
                {
                    "name": "backend-8080-tcp",
                    "containerPort": 8080,
                    "hostPort": 80,
                    "protocol": "tcp"
                }
            ],
"ServiceConnect": enabled, client only

Redis:

"networkMode": "awsvpc", 
"portMappings": [
                {
                    "name": "redis-6379-tcp",
                    "containerPort": 6379,
                    "hostPort": 6379,
                    "protocol": "tcp"
                }
            ],
"ServiceConnect": enabled, server & client

Frontend:

 "portMappings": [
                {
                    "containerPort": 3000,
                    "hostPort": 81,
                    "protocol": "tcp"
                }
            ],
Network: default,
ServiceConnect: disabled.

Expected Behavior

Container Image: ecs-service-connect-agent:interface-v1
Logs on success deployment:

time="2023-08-15T08:04:18Z" level=info msg="Envoy Environment Variables: [ENVOY_ADMIN_MODE=UDS ENVOY_CONCURRENCY=2 ENVOY_ENABLE_IAM_AUTH_FOR_XDS=0]"
time="2023-08-15T08:04:18Z" level=info msg="Agent Environment Variables: [APPNET_AGENT_ADMIN_MODE=uds APPNET_AGENT_ADMIN_UDS_PATH=/var/run/ecs/appnet_admin.sock APPNET_ENVOY_RESTART_COUNT=3 APPNET_LISTENER_PORT_MAPPING={\"egress\":38221,\"ingress-redis\":42106}]"
[2023-08-15 08:04:18.557][1][info] [AppNet Agent] Server started, /var/run/ecs/appnet_admin.sock
[2023-08-15 08:04:18.557][1][info] [AppNet Agent] Executing command: [/usr/bin/envoy -c /tmp/envoy-config-594589294.yaml -l info --concurrency 2 --drain-time-s 20]
[2023-08-15 08:04:18.669][15][info][main] [source/server/server.cc:404] initializing epoch 0 (base id=0, hot restart version=11.104)
[2023-08-15 08:04:18.669][15][info][main] [source/server/server.cc:406] statically linked extensions:
[2023-08-15 08:04:18.669][15][info][main] [source/server/server.cc:408]   network.connection.client: default, envoy_internal
[2023-08-15 08:04:18.669][15][info][main] [source/server/server.cc:408]   envoy.retry_priorities: envoy.retry_priorities.previous_priorities
[2023-08-15 08:04:18.669][15][info][main] [source/server/server.cc:408]   envoy.matching.action: envoy.matching.actions.format_string, filter-chain-name
[2023-08-15 08:04:18.670][15][info][main] [source/server/server.cc:408]   quic.http_server_connection: quic.http_server_connection.default
[2023-08-15 08:04:18.670][15][info][main] [source/server/server.cc:408]   envoy.thrift_proxy.filters: envoy.filters.thrift.header_to_metadata, envoy.filters.thrift.payload_to_metadata, envoy.filters.thrift.rate_limit, envoy.filters.thrift.router
[2023-08-15 08:04:18.670][15][info][main] [source/server/server.cc:408]   envoy.tracers: envoy.dynamic.ot, envoy.tracers.datadog, envoy.tracers.dynamic_ot, envoy.tracers.opencensus, envoy.tracers.opentelemetry, envoy.tracers.skywalking, envoy.tracers.xray, envoy.tracers.zipkin, envoy.zipkin

...

[2023-08-15 08:04:18.678][15][info][main] [source/server/server.cc:459]   response trailer map: 144 bytes: grpc-message,grpc-status
[2023-08-15 08:04:18.760][15][info][main] [source/server/server.cc:819] runtime: layers:
  - name: static_layer_0
    static_layer:
      envoy.reloadable_features.sanitize_original_path: true
      envoy.reloadable_features.http_set_tracing_decision_in_request_id: true
      envoy.reloadable_features.tcp_pool_idle_timeout: true
      envoy.reloadable_features.no_extension_lookup_by_name: true
      envoy.features.enable_all_deprecated_features: true
      re2.max_program_size.error_level: 1000
  - name: admin_layer
    admin_layer:
      {}
[2023-08-15 08:04:18.761][15][info][admin] [source/server/admin/admin.cc:67] admin address: /tmp/envoy_admin.sock
[2023-08-15 08:04:18.762][15][info][config] [source/server/configuration_impl.cc:131] loading tracing configuration
[2023-08-15 08:04:18.762][15][info][config] [source/server/configuration_impl.cc:91] loading 0 static secret(s)
[2023-08-15 08:04:18.762][15][info][config] [source/server/configuration_impl.cc:97] loading 0 cluster(s)
[2023-08-15 08:04:18.763][15][info][config] [source/server/configuration_impl.cc:101] loading 0 listener(s)
[2023-08-15 08:04:18.763][15][info][config] [source/server/configuration_impl.cc:113] loading stats configuration
[2023-08-15 08:04:18.764][15][info][runtime] [source/common/runtime/runtime_impl.cc:463] RTDS has finished initialization
[2023-08-15 08:04:18.764][15][info][upstream] [source/common/upstream/cluster_manager_impl.cc:222] cm init: initializing cds
[2023-08-15 08:04:18.764][15][warning][main] [source/server/server.cc:794] there is no configured limit to the number of allowed active connections. Set a limit via the runtime key overload.global_downstream_max_connections
[2023-08-15 08:04:18.764][15][info][main] [source/server/server.cc:915] starting main dispatch loop
[2023-08-15 08:04:18.804][15][info][upstream] [source/common/upstream/cds_api_helper.cc:35] cds: add 2 cluster(s), remove 0 cluster(s)
[2023-08-15 08:04:18.809][15][info][upstream] [source/common/upstream/cds_api_helper.cc:72] cds: added/updated 2 cluster(s), skipped 0 unmodified cluster(s)
[2023-08-15 08:04:18.809][15][info][upstream] [source/common/upstream/cluster_manager_impl.cc:196] cm init: initializing secondary clusters
[2023-08-15 08:04:24.811][15][info][upstream] [source/common/upstream/cluster_manager_impl.cc:226] cm init: all clusters initialized
[2023-08-15 08:04:24.811][15][info][main] [source/server/server.cc:896] all clusters initialized. initializing init manager
[2023-08-15 08:04:25.872][15][info][upstream] [source/extensions/listener_managers/listener_manager/lds_api.cc:82] lds: add/update listener 'ingress-redis'
[2023-08-15 08:04:25.957][15][info][upstream] [source/extensions/listener_managers/listener_manager/lds_api.cc:82] lds: add/update listener 'egress'
[2023-08-15 08:04:25.958][15][info][config] [source/extensions/listener_managers/listener_manager/listener_manager_impl.cc:852] all dependencies initialized. starting workers

Observed Behavior



time="2023-08-15T07:17:31Z" level=info msg="App Mesh Environment Variables: [APPMESH_RESOURCE_ARN=arn:aws:ecs:us-east-1:639434372228:task-set/prod-ECSCluster-V6MzUj4NX7pS/prod-redis-3ju4Jrm839kMd/ecs-svc/6456803592997768987 APPMESH_XDS_ENDPOINT=unix:///var/run/ecs/relay/envoy_xds.sock APPMESH_METRIC_EXTENSION_VERSION=1]"
--
time="2023-08-15T07:17:31Z" level=info msg="Envoy Environment Variables: [ENVOY_ENABLE_IAM_AUTH_FOR_XDS=0 ENVOY_ADMIN_MODE=UDS ENVOY_CONCURRENCY=2]"
time="2023-08-15T07:17:31Z" level=info msg="Agent Environment Variables: [APPNET_AGENT_ADMIN_MODE=uds APPNET_ENVOY_RESTART_COUNT=3 APPNET_AGENT_ADMIN_UDS_PATH=/var/run/ecs/appnet_admin.sock APPNET_LISTENER_PORT_MAPPING={\"egress\":38431,\"ingress-redis\":34415}]"
[2023-08-15 07:17:31.634][1][info] [AppNet Agent] Server started, /var/run/ecs/appnet_admin.sock
[2023-08-15 07:17:31.635][1][info] [AppNet Agent] Executing command: [/usr/bin/envoy -c /tmp/envoy-config-2133285547.yaml -l info --concurrency 2 --drain-time-s 20]
[2023-08-15 07:17:32.024][14][info][main] [source/server/server.cc:404] initializing epoch 0 (base id=0, hot restart version=11.104)
[2023-08-15 07:17:32.024][14][info][main] [source/server/server.cc:406] statically linked extensions:
[2023-08-15 07:17:32.024][14][info][main] [source/server/server.cc:408]   envoy.wasm.runtime: envoy.wasm.runtime.null, envoy.wasm.runtime.v8
[2023-08-15 07:17:32.024][14][info][main] [source/server/server.cc:408]   envoy.http.early_header_mutation: envoy.http.early_header_mutation.header_mutation
[2023-08-15 07:17:32.025][14][info][main] [source/server/server.cc:408]   envoy.quic.server.crypto_stream: envoy.quic.crypto_stream.server.quiche
[2023-08-15 07:17:32.025][14][info][main] [source/server/server.cc:408]   envoy.matching.input_matchers: envoy.matching.matchers.consistent_hashing, envoy.matching.matchers.ip
[2023-08-15 07:17:32.025][14][info][main] [source/server/server.cc:408]   envoy.upstreams: envoy.filters.connection_pools.tcp.generic
[2023-08-15 07:17:32.025][14][info][main] [source/server/server.cc:408]   envoy.http.original_ip_detection: envoy.http.original_ip_detection.custom_header, envoy.http.original_ip_detection.xff
[2023-08-15 07:17:32.025][14][info][main] [source/server/server.cc:408]   envoy.transport_sockets.upstream: envoy.transport_sockets.alts, envoy.transport_sockets.http_11_proxy, envoy.transport_sockets.internal_upstream, envoy.transport_sockets.quic, envoy.transport_sockets.raw_buffer, envoy.transport_sockets.starttls, envoy.transport_sockets.tap, envoy.transport_sockets.tcp_stats, envoy.transport_sockets.tls, envoy.transport_sockets.upstream_proxy_protocol, raw_buffer, starttls, tls
[2023-08-15 07:17:32.025][14][info][main] [source/server/server.cc:408]   envoy.matching.http.custom_matchers: envoy.matching.custom_matchers.trie_matcher
[2023-08-15 07:17:32.025][14][info][main] [source/server/server.cc:408]   envoy.network.dns_resolver: envoy.network.dns_resolver.cares, envoy.network.dns_resolver.getaddrinfo
[2023-08-15 07:17:32.025][14][info][main] [source/server/server.cc:408]   envoy.route.early_data_policy: envoy.route.early_data_policy.default
[2023-08-15 07:17:32.025][14][info][main] [source/server/server.cc:408]   envoy.filters.udp_listener: envoy.filters.udp.dns_filter, envoy.filters.udp_listener.udp_proxy
[2023-08-15 07:17:32.025][14][info][main] [source/server/server.cc:408]   envoy.http.stateful_header_formatters: envoy.http.stateful_header_formatters.preserve_case, preserve_case
[2023-08-15 07:17:32.025][14][info][main] [source/server/server.cc:408]   envoy.dubbo_proxy.filters: envoy.filters.dubbo.router
[2023-08-15 07:17:32.026][14][info][main] [source/server/server.cc:408]   envoy.resolvers: envoy.ip
[2023-08-15 07:17:32.026][14][info][main] [source/server/server.cc:408]   envoy.quic.connection_id_generator: envoy.quic.deterministic_connection_id_generator
[2023-08-15 07:17:32.026][14][info][main] [source/server/server.cc:408]   envoy.health_checkers: envoy.health_checkers.redis, envoy.health_checkers.thrift
[2023-08-15 07:17:32.026][14][info][main] [source/server/server.cc:408]   envoy.http.custom_response: envoy.extensions.http.custom_response.local_response_policy, envoy.extensions.http.custom_response.redirect_policy
[2023-08-15 07:17:32.026][14][info][main] [source/server/server.cc:408]   envoy.path.rewrite: envoy.path.rewrite.uri_template.uri_template_rewriter
[2023-08-15 07:17:32.026][14][info][main] [source/server/server.cc:408]   envoy.regex_engines: envoy.regex_engines.google_re2
[2023-08-15 07:17:32.026][14][info][main] [source/server/server.cc:408]   envoy.resource_monitors: envoy.resource_monitors.fixed_heap, envoy.resource_monitors.injected_resource
[2023-08-15 07:17:32.026][14][info][main] [source/server/server.cc:408]   envoy.rbac.matchers: envoy.rbac.matchers.upstream_ip_port
[2023-08-15 07:17:32.026][14][info][main] [source/server/server.cc:408]   envoy.clusters: envoy.cluster.eds, envoy.cluster.logical_dns, envoy.cluster.original_dst, envoy.cluster.static, envoy.cluster.strict_dns, envoy.clusters.aggregate, envoy.clusters.dynamic_forward_proxy, envoy.clusters.redis
[2023-08-15 07:17:32.026][14][info][main] [source/server/server.cc:408]   envoy.matching.http.input: envoy.matching.inputs.destination_ip, envoy.matching.inputs.destination_port, envoy.matching.inputs.direct_source_ip, envoy.matching.inputs.dns_san, envoy.matching.inputs.request_headers, envoy.matching.inputs.request_trailers, envoy.matching.inputs.response_headers, envoy.matching.inputs.response_trailers, envoy.matching.inputs.server_name, envoy.matching.inputs.source_ip, envoy.matching.inputs.source_port, envoy.matching.inputs.source_type, envoy.matching.inputs.status_code_class_input, envoy.matching.inputs.status_code_input, envoy.matching.inputs.subject, envoy.matching.inputs.uri_san
[2023-08-15 07:17:32.026][14][info][main] [source/server/server.cc:408]   envoy.compression.decompressor: envoy.compression.brotli.decompressor, envoy.compression.gzip.decompressor, envoy.compression.zstd.decompressor
[2023-08-15 07:17:32.026][14][info][main] [source/server/server.cc:408]   envoy.access_loggers: envoy.access_loggers.file, envoy.access_loggers.http_grpc, envoy.access_loggers.open_telemetry, envoy.access_loggers.stderr, envoy.access_loggers.stdout, envoy.access_loggers.tcp_grpc, envoy.access_loggers.wasm, envoy.file_access_log, envoy.http_grpc_access_log, envoy.open_telemetry_access_log, envoy.stderr_access_log, envoy.stdout_access_log, envoy.tcp_grpc_access_log, envoy.wasm_access_log
[2023-08-15 07:17:32.027][14][info][main] [source/server/server.cc:408]   envoy.retry_host_predicates: envoy.retry_host_predicates.omit_canary_hosts, envoy.retry_host_predicates.omit_host_metadata, envoy.retry_host_predicates.previous_hosts
[2023-08-15 07:17:32.027][14][info][main] [source/server/server.cc:408]   envoy.common.key_value: envoy.key_value.file_based
[2023-08-15 07:17:32.027][14][info][main] [source/server/server.cc:408]   envoy.filters.http: envoy.bandwidth_limit, envoy.buffer, envoy.cors, envoy.csrf, envoy.ext_authz, envoy.ext_proc, envoy.fault, envoy.filters.http.adaptive_concurrency, envoy.filters.http.admission_control, envoy.filters.http.alternate_protocols_cache, envoy.filters.http.aws_lambda, envoy.filters.http.aws_request_signing, envoy.filters.http.bandwidth_limit, envoy.filters.http.buffer, envoy.filters.http.cache, envoy.filters.http.cdn_loop, envoy.filters.http.composite, envoy.filters.http.compressor, envoy.filters.http.cors, envoy.filters.http.csrf, envoy.filters.http.custom_response, envoy.filters.http.decompressor, envoy.filters.http.dynamic_forward_proxy, envoy.filters.http.ext_authz, envoy.filters.http.ext_proc, envoy.filters.http.fault, envoy.filters.http.file_system_buffer, envoy.filters.http.gcp_authn, envoy.filters.http.grpc_http1_bridge, envoy.filters.http.grpc_http1_reverse_bridge, envoy.filters.http.grpc_json_transcoder, envoy.filters.http.grpc_stats, envoy.filters.http.grpc_web, envoy.filters.http.header_to_metadata, envoy.filters.http.health_check, envoy.filters.http.ip_tagging, envoy.filters.http.jwt_authn, envoy.filters.http.local_ratelimit, envoy.filters.http.lua, envoy.filters.http.match_delegate, envoy.filters.http.oauth2, envoy.filters.http.on_demand, envoy.filters.http.original_src, envoy.filters.http.rate_limit_quota, envoy.filters.http.ratelimit, envoy.filters.http.rbac, envoy.filters.http.router, envoy.filters.http.set_metadata, envoy.filters.http.stateful_session, envoy.filters.http.tap, envoy.filters.http.wasm, envoy.grpc_http1_bridge, envoy.grpc_json_transcoder, envoy.grpc_web, envoy.health_check, envoy.ip_tagging, envoy.local_rate_limit, envoy.lua, envoy.rate_limit, envoy.router
[2023-08-15 07:17:32.027][14][info][main] [source/server/server.cc:408]   envoy.http.header_validators: envoy.http.header_validators.envoy_default
[2023-08-15 07:17:32.027][14][info][main] [source/server/server.cc:408]   envoy.internal_redirect_predicates: envoy.internal_redirect_predicates.allow_listed_routes, envoy.internal_redirect_predicates.previous_routes, envoy.internal_redirect_predicates.safe_cross_scheme
[2023-08-15 07:17:32.027][14][info][main] [source/server/server.cc:408]   envoy.grpc_credentials: envoy.grpc_credentials.aws_iam, envoy.grpc_credentials.default, envoy.grpc_credentials.file_based_metadata
[2023-08-15 07:17:32.027][14][info][main] [source/server/server.cc:408]   envoy.thrift_proxy.protocols: auto, binary, binary/non-strict, compact, twitter
[2023-08-15 07:17:32.027][14][info][main] [source/server/server.cc:408]   envoy.transport_sockets.downstream: envoy.transport_sockets.alts, envoy.transport_sockets.quic, envoy.transport_sockets.raw_buffer, envoy.transport_sockets.starttls, envoy.transport_sockets.tap, envoy.transport_sockets.tcp_stats, envoy.transport_sockets.tls, raw_buffer, starttls, tls
[2023-08-15 07:17:32.027][14][info][main] [source/server/server.cc:408]   envoy.compression.compressor: envoy.compression.brotli.compressor, envoy.compression.gzip.compressor, envoy.compression.zstd.compressor
[2023-08-15 07:17:32.027][14][info][main] [source/server/server.cc:408]   envoy.listener_manager_impl: envoy.listener_manager_impl.default
[2023-08-15 07:17:32.027][14][info][main] [source/server/server.cc:408]   envoy.thrift_proxy.transports: auto, framed, header, unframed
[2023-08-15 07:17:32.027][14][info][main] [source/server/server.cc:408]   envoy.matching.network.custom_matchers: envoy.matching.custom_matchers.trie_matcher
[2023-08-15 07:17:32.027][14][info][main] [source/server/server.cc:408]   envoy.connection_handler: envoy.connection_handler.default
[2023-08-15 07:17:32.027][14][info][main] [source/server/server.cc:408]   envoy.dubbo_proxy.protocols: dubbo
[2023-08-15 07:17:32.027][14][info][main] [source/server/server.cc:408]   envoy.access_loggers.extension_filters: envoy.access_loggers.extension_filters.cel
[2023-08-15 07:17:32.027][14][info][main] [source/server/server.cc:408]   envoy.quic.proof_source: envoy.quic.proof_source.filter_chain
[2023-08-15 07:17:32.027][14][info][main] [source/server/server.cc:408]   envoy.matching.common_inputs: envoy.matching.common_inputs.environment_variable
[2023-08-15 07:17:32.027][14][info][main] [source/server/server.cc:408]   quic.http_server_connection: quic.http_server_connection.default
[2023-08-15 07:17:32.027][14][info][main] [source/server/server.cc:408]   envoy.thrift_proxy.filters: envoy.filters.thrift.header_to_metadata, envoy.filters.thrift.payload_to_metadata, envoy.filters.thrift.rate_limit, envoy.filters.thrift.router
[2023-08-15 07:17:32.027][14][info][main] [source/server/server.cc:408]   envoy.dubbo_proxy.serializers: dubbo.hessian2
[2023-08-15 07:17:32.027][14][info][main] [source/server/server.cc:408]   envoy.filters.http.upstream: envoy.buffer, envoy.filters.http.admission_control, envoy.filters.http.buffer, envoy.filters.http.upstream_codec
[2023-08-15 07:17:32.027][14][info][main] [source/server/server.cc:408]   envoy.matching.action: envoy.matching.actions.format_string, filter-chain-name
[2023-08-15 07:17:32.027][14][info][main] [source/server/server.cc:408]   envoy.tracers: envoy.dynamic.ot, envoy.tracers.datadog, envoy.tracers.dynamic_ot, envoy.tracers.opencensus, envoy.tracers.opentelemetry, envoy.tracers.skywalking, envoy.tracers.xray, envoy.tracers.zipkin, envoy.zipkin
[2023-08-15 07:17:32.027][14][info][main] [source/server/server.cc:408]   envoy.guarddog_actions: envoy.watchdog.abort_action, envoy.watchdog.profile_action
[2023-08-15 07:17:32.027][14][info][main] [source/server/server.cc:408]   envoy.stats_sinks: envoy.dog_statsd, envoy.graphite_statsd, envoy.metrics_service, envoy.stat_sinks.dog_statsd, envoy.stat_sinks.graphite_statsd, envoy.stat_sinks.hystrix, envoy.stat_sinks.metrics_service, envoy.stat_sinks.statsd, envoy.stat_sinks.wasm, envoy.statsd
[2023-08-15 07:17:32.027][14][info][main] [source/server/server.cc:408]   envoy.load_balancing_policies: envoy.load_balancing_policies.least_request, envoy.load_balancing_policies.random, envoy.load_balancing_policies.round_robin
[2023-08-15 07:17:32.027][14][info][main] [source/server/server.cc:408]   envoy.http.stateful_session: envoy.http.stateful_session.cookie, envoy.http.stateful_session.header
[2023-08-15 07:17:32.027][14][info][main] [source/server/server.cc:408]   envoy.http.cache: envoy.extensions.http.cache.file_system_http_cache, envoy.extensions.http.cache.simple
[2023-08-15 07:17:32.027][14][info][main] [source/server/server.cc:408]   network.connection.client: default, envoy_internal
[2023-08-15 07:17:32.027][14][info][main] [source/server/server.cc:408]   envoy.request_id: envoy.request_id.uuid
[2023-08-15 07:17:32.027][14][info][main] [source/server/server.cc:408]   envoy.udp_packet_writer: envoy.udp_packet_writer.default, envoy.udp_packet_writer.gso
[2023-08-15 07:17:32.027][14][info][main] [source/server/server.cc:408]   envoy.matching.network.input: envoy.matching.inputs.application_protocol, envoy.matching.inputs.destination_ip, envoy.matching.inputs.destination_port, envoy.matching.inputs.direct_source_ip, envoy.matching.inputs.dns_san, envoy.matching.inputs.server_name, envoy.matching.inputs.source_ip, envoy.matching.inputs.source_port, envoy.matching.inputs.source_type, envoy.matching.inputs.subject, envoy.matching.inputs.transport_protocol, envoy.matching.inputs.uri_san
[2023-08-15 07:17:32.027][14][info][main] [source/server/server.cc:408]   envoy.tls.cert_validator: envoy.tls.cert_validator.default, envoy.tls.cert_validator.spiffe
[2023-08-15 07:17:32.027][14][info][main] [source/server/server.cc:408]   envoy.filters.listener: envoy.filters.listener.http_inspector, envoy.filters.listener.original_dst, envoy.filters.listener.original_src, envoy.filters.listener.proxy_protocol, envoy.filters.listener.tls_inspector, envoy.listener.http_inspector, envoy.listener.original_dst, envoy.listener.original_src, envoy.listener.proxy_protocol, envoy.listener.tls_inspector
[2023-08-15 07:17:32.027][14][info][main] [source/server/server.cc:408]   envoy.bootstrap: envoy.bootstrap.internal_listener, envoy.bootstrap.wasm, envoy.extensions.network.socket_interface.default_socket_interface
[2023-08-15 07:17:32.027][14][info][main] [source/server/server.cc:408]   envoy.upstream_options: envoy.extensions.upstreams.http.v3.HttpProtocolOptions, envoy.extensions.upstreams.tcp.v3.TcpProtocolOptions, envoy.upstreams.http.http_protocol_options, envoy.upstreams.tcp.tcp_protocol_options
[2023-08-15 07:17:32.027][14][info][main] [source/server/server.cc:408]   envoy.path.match: envoy.path.match.uri_template.uri_template_matcher
[2023-08-15 07:17:32.027][14][info][main] [source/server/server.cc:408]   envoy.config.validators: envoy.config.validators.minimum_clusters, envoy.config.validators.minimum_clusters_validator
[2023-08-15 07:17:32.027][14][info][main] [source/server/server.cc:408]   envoy.retry_priorities: envoy.retry_priorities.previous_priorities
[2023-08-15 07:17:32.027][14][info][main] [source/server/server.cc:408]   envoy.rate_limit_descriptors: envoy.rate_limit_descriptors.expr
[2023-08-15 07:17:32.027][14][info][main] [source/server/server.cc:408]   envoy.formatter: envoy.formatter.metadata, envoy.formatter.req_without_query
[2023-08-15 07:17:32.027][14][info][main] [source/server/server.cc:408]   envoy.filters.network: envoy.echo, envoy.ext_authz, envoy.filters.network.connection_limit, envoy.filters.network.direct_response, envoy.filters.network.dubbo_proxy, envoy.filters.network.echo, envoy.filters.network.ext_authz, envoy.filters.network.http_connection_manager, envoy.filters.network.local_ratelimit, envoy.filters.network.mongo_proxy, envoy.filters.network.ratelimit, envoy.filters.network.rbac, envoy.filters.network.redis_proxy, envoy.filters.network.sni_cluster, envoy.filters.network.sni_dynamic_forward_proxy, envoy.filters.network.tcp_proxy, envoy.filters.network.thrift_proxy, envoy.filters.network.wasm, envoy.filters.network.zookeeper_proxy, envoy.http_connection_manager, envoy.mongo_proxy, envoy.ratelimit, envoy.redis_proxy, envoy.tcp_proxy
[2023-08-15 07:17:32.032][14][info][main] [source/server/server.cc:456] HTTP header map info:
[2023-08-15 07:17:32.033][14][info][main] [source/server/server.cc:459]   request header map: 672 bytes: :authority,:method,:path,:protocol,:scheme,accept,accept-encoding,access-control-request-headers,access-control-request-method,access-control-request-private-network,authentication,authorization,cache-control,cdn-loop,connection,content-encoding,content-length,content-type,expect,grpc-accept-encoding,grpc-timeout,if-match,if-modified-since,if-none-match,if-range,if-unmodified-since,keep-alive,origin,pragma,proxy-connection,proxy-status,referer,te,transfer-encoding,upgrade,user-agent,via,x-client-trace-id,x-envoy-attempt-count,x-envoy-decorator-operation,x-envoy-downstream-service-cluster,x-envoy-downstream-service-node,x-envoy-expected-rq-timeout-ms,x-envoy-external-address,x-envoy-force-trace,x-envoy-hedge-on-per-try-timeout,x-envoy-internal,x-envoy-ip-tags,x-envoy-is-timeout-retry,x-envoy-max-retries,x-envoy-original-path,x-envoy-original-url,x-envoy-retriable-header-names,x-envoy-retriable-status-codes,x-envoy-retry-grpc-on,x-envoy-retry-on,x-envoy-upstream-alt-stat-name,x-envoy-upstream-rq-per-try-timeout-ms,x-envoy-upstream-rq-timeout-alt-response,x-envoy-upstream-rq-timeout-ms,x-envoy-upstream-stream-duration-ms,x-forwarded-client-cert,x-forwarded-for,x-forwarded-host,x-forwarded-port,x-forwarded-proto,x-ot-span-context,x-request-id
[2023-08-15 07:17:32.034][14][info][main] [source/server/server.cc:459]   request trailer map: 120 bytes:
[2023-08-15 07:17:32.034][14][info][main] [source/server/server.cc:459]   response header map: 432 bytes: :status,access-control-allow-credentials,access-control-allow-headers,access-control-allow-methods,access-control-allow-origin,access-control-allow-private-network,access-control-expose-headers,access-control-max-age,age,cache-control,connection,content-encoding,content-length,content-type,date,etag,expires,grpc-message,grpc-status,keep-alive,last-modified,location,proxy-connection,proxy-status,server,transfer-encoding,upgrade,vary,via,x-envoy-attempt-count,x-envoy-decorator-operation,x-envoy-degraded,x-envoy-immediate-health-check-fail,x-envoy-ratelimited,x-envoy-upstream-canary,x-envoy-upstream-healthchecked-cluster,x-envoy-upstream-service-time,x-request-id
[2023-08-15 07:17:32.034][14][info][main] [source/server/server.cc:459]   response trailer map: 144 bytes: grpc-message,grpc-status
[2023-08-15 07:17:32.128][14][info][main] [source/server/server.cc:819] runtime: layers:
- name: static_layer_0
static_layer:
envoy.features.enable_all_deprecated_features: true
envoy.reloadable_features.http_set_tracing_decision_in_request_id: true
envoy.reloadable_features.tcp_pool_idle_timeout: true
envoy.reloadable_features.sanitize_original_path: true
re2.max_program_size.error_level: 1000
envoy.reloadable_features.no_extension_lookup_by_name: true
- name: admin_layer
admin_layer:
{}
[2023-08-15 07:17:32.129][14][info][admin] [source/server/admin/admin.cc:67] admin address: /tmp/envoy_admin.sock
[2023-08-15 07:17:32.129][14][info][config] [source/server/configuration_impl.cc:131] loading tracing configuration
[2023-08-15 07:17:32.129][14][info][config] [source/server/configuration_impl.cc:91] loading 0 static secret(s)
[2023-08-15 07:17:32.130][14][info][config] [source/server/configuration_impl.cc:97] loading 0 cluster(s)
[2023-08-15 07:17:32.131][14][info][config] [source/server/configuration_impl.cc:101] loading 0 listener(s)
[2023-08-15 07:17:32.131][14][info][config] [source/server/configuration_impl.cc:113] loading stats configuration
[2023-08-15 07:17:32.131][14][info][runtime] [source/common/runtime/runtime_impl.cc:463] RTDS has finished initialization
[2023-08-15 07:17:32.131][14][info][upstream] [source/common/upstream/cluster_manager_impl.cc:222] cm init: initializing cds
[2023-08-15 07:17:32.132][14][warning][main] [source/server/server.cc:794] there is no configured limit to the number of allowed active connections. Set a limit via the runtime key overload.global_downstream_max_connections
[2023-08-15 07:17:32.132][14][info][main] [source/server/server.cc:915] starting main dispatch loop
[2023-08-15 07:17:32.132][14][warning][config] [./source/common/config/grpc_stream.h:163] StreamAggregatedResources gRPC config stream to unix:///var/run/ecs/relay/envoy_xds.sock closed: 13,
[2023-08-15 07:17:32.319][14][warning][config] [./source/common/config/grpc_stream.h:163] StreamAggregatedResources gRPC config stream to unix:///var/run/ecs/relay/envoy_xds.sock closed: 13,
[2023-08-15 07:17:32.914][14][warning][config] [./source/common/config/grpc_stream.h:163] StreamAggregatedResources gRPC config stream to unix:///var/run/ecs/relay/envoy_xds.sock closed: 13,
[2023-08-15 07:17:34.741][14][warning][config] [./source/common/config/grpc_stream.h:163] StreamAggregatedResources gRPC config stream to unix:///var/run/ecs/relay/envoy_xds.sock closed: 13,

Environment Details

docker info
Client:
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc., 0.0.0+unknown)

Server:
 Containers: 14
  Running: 7
  Paused: 0
  Stopped: 7
 Images: 5
 Server Version: 20.10.23
 Storage Driver: overlay2
  Backing Filesystem: xfs
  Supports d_type: true
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 1e1ea6e986c6c86565bc33d52e34b81b3e2bc71f
 runc version: f19387a6bec4944c770f7668ab51c4348d9c2f38
 init version: de40ad0
 Security Options:
  seccomp
   Profile: default
 Kernel Version: 4.14.320-242.534.amzn2.x86_64
 Operating System: Amazon Linux 2
 OSType: linux
 Architecture: x86_64
 CPUs: 1
 Total Memory: 970.2MiB
 Name: ip-172-14-3-154.ec2.internal
 ID: CWLP:UUAF:NIN3:WZHK:SEHB:YBDL:HUHO:PS57:2QBS:JPMW:4XE2:J2L7
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

Is Blue/green deployment (powered by AWS CodeDeploy) supported?

[BUG] Envoy Does Not Respect ENVOY_LOG_LEVEL Setting from Service Connect Agent in AWS App Mesh

Summary

In the AWS App Mesh scenario, the Service Connect Agent is utilized to generate static bootstrap configurations for Envoy proxies. Despite this setup, there is an observed issue where Envoy logs do not respect the ENVOY_LOG_LEVEL setting specified by the Service Connect Agent. This behavior is unexpected.

Description

When setting the ENVOY_LOG_LEVEL to off via the Service Connect Agent, Envoy sidecar containers continue to output logs at the info level. Despite correctly setting this environment variable here, it appears that Envoy is not respecting the specified log level.

Expected Behavior

Envoy should not produce any logs when ENVOY_LOG_LEVEL is set to off, as per the configuration defined by the Service Connect Agent.

Observed Behavior

Envoy continues to output logs at the info level, disregarding the ENVOY_LOG_LEVEL set to off. More detailed in this ticket

Environment Details

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.