googlecloudplatform / openjdk-runtime Goto Github PK

Verify that an application can read/write from memcached.

https://cloud.google.com/appengine/docs/flexible/java/caching-application-data

openjdk:9 installs wget, but it may not actually be necessary.

See: https://github.com/GoogleCloudPlatform/openjdk-runtime/blob/master/openjdk8/src/main/docker/Dockerfile#L35

# workaround for https://bugs.debian.org/775775
RUN /var/lib/dpkg/info/ca-certificates-java.postinst configure

See #121 for more details where this came up.

In the "local emulation" of the cloud build a python script is pulled in from the python runtime - which is not an ideal solution necessarily - especially that the old version (the one I switched back to) worked, the new version doesn't, as it got modularized in this commit .

Let's find a better way to emulate cloud build locally!

Options might be:

1. "curl" all of the dependent scripts in at runtime (ugly)
2. write our own fork of these scripts
3. lift these "cloud-scripts" into a separate projects which could be reused across runtime projects?

We should document it like this: https://github.com/GoogleCloudPlatform/nodejs-docker#container-engine--other-docker-hosts

setup-env.bash has this block of code to set up the Stackdriver Debugger agent. The line that I'm confused about is DBG_AGENT="$( RUNTIME_DIR=$JETTY_BASE /opt/cdbg/format-env-appengine-vm.sh )". Why are we referencing $JETTY_BASE here? It wouldn't be defined for a vanilla Java app.

DBG_AGENT=
if isTrue "${DBG_ENABLE:=$( if [[ -z ${CDBG_DISABLE} ]] ; then echo true; else echo false ; fi )}" ; then
  if [[ "$GAE_PARTITION" = "dev" ]]; then
    echo "Running locally and DBG_ENABLE is set, enabling standard Java debugger agent"
    DBG_AGENT="-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=${DBG_PORT:=5005}"
  else
    unset CDBG_DISABLE
    DBG_AGENT="$( RUNTIME_DIR=$JETTY_BASE /opt/cdbg/format-env-appengine-vm.sh )"
  fi
fi

Some googling indicates that the openjdk-debuginfo package might be missing.

root@37d7e4efd82b:/# jmap -heap 1
Attaching to process ID 1, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 25.121-b13
using thread-local object allocation.
Garbage-First (G1) GC with 1 thread(s)
Heap Configuration:
   MinHeapFreeRatio         = 40
   MaxHeapFreeRatio         = 70
   MaxHeapSize              = 515899392 (492.0MB)
   NewSize                  = 1363144 (1.2999954223632812MB)
   MaxNewSize               = 309329920 (295.0MB)
   OldSize                  = 5452592 (5.1999969482421875MB)
   NewRatio                 = 2
   SurvivorRatio            = 8
   MetaspaceSize            = 21807104 (20.796875MB)
   CompressedClassSpaceSize = 1073741824 (1024.0MB)
   MaxMetaspaceSize         = 17592186044415 MB
   G1HeapRegionSize         = 1048576 (1.0MB)
Heap Usage:
Exception in thread "main" java.lang.reflect.InvocationTargetException
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:
43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at sun.tools.jmap.JMap.runTool(JMap.java:201)
        at sun.tools.jmap.JMap.main(JMap.java:130)
Caused by: java.lang.RuntimeException: unknown CollectedHeap type : class sun.jvm.hotspot.gc_
interface.CollectedHeap
        at sun.jvm.hotspot.tools.HeapSummary.run(HeapSummary.java:144)
        at sun.jvm.hotspot.tools.Tool.startInternal(Tool.java:260)
        at sun.jvm.hotspot.tools.Tool.start(Tool.java:223)
        at sun.jvm.hotspot.tools.Tool.execute(Tool.java:118)
        at sun.jvm.hotspot.tools.HeapSummary.main(HeapSummary.java:49)
        ... 6 more

If this is still needed, maybe it belongs in the base image.

# Upgrade to OpenSSL 1.0.2 (via sid)
ADD sid.list /etc/apt/sources.list.d/
RUN apt-get -y update \
 && apt-get -y -q --no-install-recommends install \
    libssl1.0.2 \
    openssl \
# Cleanup sid references
 && rm /etc/apt/sources.list.d/sid.list \
 && apt-get -y update \
# Cleanup apt-get temporary files
 && apt-get -y -q upgrade \
 && apt-get -y -q autoremove

The monitoring integration test is flaky.
The following are the main reasons:

1. The driver can't find the custom metric

BadRequest: 400 The provided filter doesn't refer to any known metric. (GET https://monitoring.googleapis.com/v3/projects/java-runtime-test/timeSeries/?filter=metric.type+%3D+%22custom.googleapis.com%2FUzLlwfHpNOCsZSuu%22&interval.endTime=2017-08-03T21%3A13%3A00.000000Z&interval.startTime=2017-08-03T21%3A11%3A00.000000Z)

Solution: reproduce the error and identify a workaround (work out timings, retries, double check if the metric is stored, etc.)

2. The gRPC call fails while storing the metric

exit code: 500, text: {"timestamp":1501793413295,"status":500,"error":"Internal Server Error","exception":"com.google.api.gax.grpc.ApiException","message":"io.grpc.StatusRuntimeException: INTERNAL: HTTP/2 error code: INTERNAL_ERROR\nReceived Rst Stream","path":"/monitoring"}

Solution: file a bug with the Stackdriver team and hunt down the root cause, and in the mean time catch the exception and retry (maybe double check if the metric was stored)

The docker layer for the apt-get install libssl will continue to use the old libssl version, even if there was a new version released.

We should change the build (somehow) to always rebuild that layer.

The GC log is printed to standard out by default.
By default it should be turned off or redirected to a separate file.

Something like:
-XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles= -XX:GCLogFileSize=

When the GKE integration script runs, it creates a new cluster, if one is not already present. The creation of a new cluster leads to a delay in the deployment of the application witch may cause some tests to fail.

Use the one from DockerHub?

The openjdk-8-jre-headless seems like a good fit for a docker image (which obviously has no framebuffer or UI).

Tests should verify that the contract with App Engine flex is obeyed - logs sent by an application to stderr and stdout should be viewable in the cloud logging API, or via gcloud.

Implement the /monitoring endpoint of the test application and implement the monitoring part of the test spec. See https://github.com/GoogleCloudPlatform/runtimes-common/tree/master/integration_tests#monitoring

See: gcloud app deploy line in ae_integration_test.sh.
Passing a static app version can enable faster deploy cycle.

We should add an integration test for the deployment to Google Container Engine (based on Integration Frameworks)

Once GoogleCloudPlatform/jetty-runtime#68 is complete, there are no users of the JSON log formatter in the logging module, so it should be removed.

At this line: https://github.com/GoogleCloudPlatform/openjdk-runtime/blob/master/openjdk8/src/main/docker/docker-entrypoint.bash#L21

Maybe we change it to exec env JAVA_OPTS="$JAVA_OPTS" "$@" 2>&1? (I'm not sure if 2>&1 is needed. Suggestion welcomed).

I was using Scala Playframework, and JAVA_OPTS wasn't set properly. I have to make my own docker-entrypoint.bash.

Making this change would allow me to get rid of my own docker-entrypoint.bash.

Actually 9.1 was released recently https://www.debian.org/releases/stretch/

The default HEAP_SIZE configuration in setup-env.bash should be set as follows:
If GAE_MEMORY_MB is present:
heap_size = GAE_MEMORY_MB * 0.8
Otherwise
heap_size = (/proc/meminfo - 400MB) * 0.8

Via running the gcr.io/gcp-runtimes/integration_test image locally (same functionality as in integration_test.yaml), we could actually iterate much faster when implementing changes on the test-app.
Currently the local_integration_test.sh is only testing whether the app starts and verifies the shutdown logs.

As @aslo pointed out, with the approach on docker-library, we could avoid duplications like this between our dockerfiles.

We could go as easy as providing a separate folder where the generated Dockerfile is visible for understandability, or as far as (I assume that's the goal in docker-library) the generated Dockerfile sits in the full folder structure, thus you can build the image using only docker.

I'd argue, this is a worthwile exercise to do, so that you don't need maven to build the image. You'd need maven only to generate the "image source", what is published to github as an artifact itself.

$mvn install
...
[ERROR] Failed to execute goal com.spotify:docker-maven-plugin:0.4.13:build (build) on project openjdk8: Exception caught: The command '/bin/sh -c echo 'deb http://httpredir.debian.org/debian jessie-backports main' > /etc/apt/sources.list.d/jessie-backports.list  && apt-get -q update  && apt-get -y -q --no-install-recommends install     ca-certificates     openjdk-8-jre=8u102'-*'     netbase     wget     unzip  && apt-get clean  && rm /var/lib/apt/lists/*_*' returned a non-zero code: 100 -> [Help 1]

More detailed:

$/bin/sh -c echo 'deb http://httpredir.debian.org/debian jessie-backports main' > /etc/apt/sources.list.d/jessie-backports.list  && apt-get -q update  && apt-get -y -q --no-install-recommends install     ca-certificates     openjdk-8-jre=8u102'-*'     netbase     wget     unzip  && apt-get clean  && rm /var/lib/apt/lists/*_*

Get:1 http://security.debian.org jessie/updates InRelease [63.1 kB]
Ign http://httpredir.debian.org jessie InRelease
Get:2 http://httpredir.debian.org jessie-updates InRelease [145 kB]
Get:3 http://httpredir.debian.org jessie Release.gpg [2373 B]
Get:4 http://security.debian.org jessie/updates/main amd64 Packages [407 kB]
Get:5 http://httpredir.debian.org jessie Release [148 kB]
Get:6 http://httpredir.debian.org jessie-updates/main amd64 Packages [17.6 kB]
Get:7 http://httpredir.debian.org jessie/main amd64 Packages [9064 kB]
Fetched 9847 kB in 2s (3552 kB/s)
Reading package lists...
Reading package lists...
Building dependency tree...
E: Unable to locate package openjdk-8-jre

I see a setting to enable/disable the debugger, but is there anything else in this image for stackdriver integration?

GAE_MEMORY_MB is not available on GKE, but the Kubernetes Downward API may be used to compute it. See: http://kubernetes.io/docs/user-guide/downward-api/

Related issue: #3

The test could be something simple like

• deploy a sample app
• set a snapshot (aka a breakpoint) via the API or gcloud
• exercise the breakpoint
• verify that the breakpoint has been hit

Currently, we leave the app running after executing the integration tests. It's a problem because we need to then manually shutdown the app/cluster to stop accumulating GCP costs.

@aslo @cassand

The image passes:
--cdbg_extra_class_path=/webapps/root/WEB-INF/classes:/webapps/root/WEB-INF/lib

as a fragment of:

java -showversion -agentpath:/opt/cdbg/cdbg_java_agent.so=--log_dir=/var/log/app_engine,--logtostderr=false,--cdbg_extra_class_path=/webapps/root/WEB-INF/classes:/webapps/root/WEB-
INF/lib -Xms491M -Xmx491M -XX:+UseG1GC -XX:+ParallelRefProcEnabled -XX:+PrintCommandLineFlags -jar /app/spring-boot-simple-5.0.jar

which causes the /var/log/app_engine/cdbg_java_agent.INFO to contain:

cdbg_java_agent.8c8c13cfad57.invalid-user.log.INFO.20170228-155954.1  cdbg_java_agent.8c8c13cfad57.invalid-user.log.WARNING.20170228-155955.1  cdbg_java_agent.INFO  cdbg_java_agent.WARNING  custom_logs
root@8c8c13cfad57:/# cat /var/log/app_engine/cdbg_java_agent.INFO
Log file created at: 2017/02/28 15:59:54
Running on machine: 8c8c13cfad57
Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
I0228 15:59:54.725625    15 jvmti_globals.cc:336] Build time: Jan  3 2017 09:10:03
I0228 15:59:54.725790    15 jvmti_agent.cc:157] Java debuglet initialization started
I0228 15:59:54.726048    15 jvmti_agent.cc:191] Java debuglet initialization completed
I0228 15:59:54.804965    15 jvmti_agent.cc:202] Java VM started
I0228 15:59:54.809967    15 jvmti_agent.cc:212] JvmtiAgent::JvmtiOnVMInit initialization time: 5007 microseconds
I0228 15:59:54.810044    26 jvmti_agent_thread.cc:99] Agent thread started: CloudDebugger_main_worker_thread
I0228 15:59:54.810212    26 jvm_internals.cc:376] Loading internals from /opt/cdbg/cdbg_java_agent_internals.jar
I0228 15:59:55.224081    26 jni_logger.cc:31] Initializing ClassPathLookup, default classpath: true, extra classpath: [/webapps/root/WEB-INF/classes, /webapps/root/WEB-INF/lib], config: null
I0228 15:59:55.286089    26 jni_logger.cc:31] Total size of indexed resources database: 1432 bytes
I0228 15:59:55.329637    26 jvm_internals.cc:132] ClassPathLookup constructor time: 118480 microseconds
W0228 15:59:55.777546    26 jni_logger.cc:46] Failed to compute hash of application file /webapps/root/WEB-INF/classes
java.nio.file.NoSuchFileException: /webapps/root/WEB-INF/classes
        at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
        at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
        at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
        at sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:214)
        at java.nio.file.Files.newByteChannel(Files.java:361)
        at java.nio.file.Files.newByteChannel(Files.java:407)
        at java.nio.file.spi.FileSystemProvider.newInputStream(FileSystemProvider.java:384)
        at java.nio.file.Files.newInputStream(Files.java:152)
        at com.google.devtools.cdbg.debuglets.java.UniquifierComputer.appendBinaryFile(Unknown Source)
        at com.google.devtools.cdbg.debuglets.java.UniquifierComputer.append(Unknown Source)
        at com.google.devtools.cdbg.debuglets.java.UniquifierComputer.<init>(Unknown Source)
        at com.google.devtools.cdbg.debuglets.java.ClassPathLookup.computeDebuggeeUniquifier(Unknown Source)
        at com.google.devtools.cdbg.debuglets.java.GcpHubClient.getDebuggeeInfo(Unknown Source)
        at com.google.devtools.cdbg.debuglets.java.GcpHubClient.registerDebuggee(Unknown Source)
W0228 15:59:55.777968    26 jni_logger.cc:46] Failed to compute hash of application file /webapps/root/WEB-INF/lib
java.nio.file.NoSuchFileException: /webapps/root/WEB-INF/lib
        at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
        at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
        at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
        at sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:214)
        at java.nio.file.Files.newByteChannel(Files.java:361)
        at java.nio.file.Files.newByteChannel(Files.java:407)
        at java.nio.file.spi.FileSystemProvider.newInputStream(FileSystemProvider.java:384)
        at java.nio.file.Files.newInputStream(Files.java:152)
        at com.google.devtools.cdbg.debuglets.java.UniquifierComputer.appendBinaryFile(Unknown Source)
        at com.google.devtools.cdbg.debuglets.java.UniquifierComputer.append(Unknown Source)
        at com.google.devtools.cdbg.debuglets.java.UniquifierComputer.<init>(Unknown Source)
        at com.google.devtools.cdbg.debuglets.java.ClassPathLookup.computeDebuggeeUniquifier(Unknown Source)
        at com.google.devtools.cdbg.debuglets.java.GcpHubClient.getDebuggeeInfo(Unknown Source)
        at com.google.devtools.cdbg.debuglets.java.GcpHubClient.registerDebuggee(Unknown Source)
I0228 15:59:57.652278    26 jni_logger.cc:31] Debuggee gcp:1031550152456:8402ef630c825e73 registered: {"debuggee":{"id":"gcp:1031550152456:8402ef630c825e73","project":"1031550152456","uniquifier":"8DA069F1CA0B61EAFC6587E57F9FEE9411FF23E4","descr
iption":"paflynn-demo-project","agentVersion":"google.com/java-gcp/@2"}}, agent version: 2.8
I0228 15:59:57.652299    26 jvmti_agent.cc:414] Attaching Java debuglet
I0228 15:59:57.652760    26 rate_limit.cc:122] CPU count: 1
I0228 15:59:57.652768    26 debugger.cc:97] Initializing Java debuglet

It's unclear what impact this has, as CDB appears broken for Java Flex for other reasons.

It might be more convenient for users to set the heap size as percentage of the container memory.

For example through an environment variable.

The version 121 of openjdk8 have been remove from the debian repository.
The new available version is 131.

https://packages.debian.org/jessie-backports/java/openjdk-8-jdk-headless

Explore and implement in the openjdk-9 image the useful bits from the new JVM logging features:

https://docs.oracle.com/javase/9/whatsnew/toc.htm#JSNEW-GUID-BA9D8AF6-E706-4327-8909-F6747B8F35C5

GC logging seems interesting for debugging: http://openjdk.java.net/jeps/271

TL;DR: On a VM shutdown, allow logging the application’s thread dump and heap info to Cloud Logging.

On App Engine, sometimes autohealer decides to kill unhealthy VMs. In that case we don't have sufficient logging done from the VM to debug if it was an issue with the app or the VM. This can happen when the app code gets stuck (like deadlock) and stopped returning requests, including health checks.

Because it is possible that the JVM is ailing, it might not have enough resources to log to Cloud Logging directly. So we will be using stdout + fluentd to capture app container logs.

Explicit user intent is required for potentially sensitive information. The following environment variables will be used to enable/disable app container shutdown reporting (must be set to true or false):

SHUTDOWN_LOGGING_THREAD_DUMP - save thread traces
SHUTDOWN_LOGGING_HEAP_INFO - save heap information

The runtime will capture SIGTERM, and run debugging tools on the JVM to emit the thread traces and heap information to stdout.

Further, for customers with a small number of VMs we default to reporting shutdown logs for all VMs. For larger customers running hundreds or thousands of instances, we would allow them to be able to sample logs. This will be done by using the environment variable SHUTDOWN_LOGGING_SAMPLE_THRESHOLD which is an int between 0 and 100. 0 means no VMs report logs, 100 means all VMs report logs. (If this env var is not set, we default to reporting for all VMs). The runtime will then generate a random number between 0 (inclusive) and 100 (exclusive) and if it’s less than the threshold set, it should do the trap-and-log logic described above.

@karan

Hello.

Not sure if i should file the issue here or where, however, in my pet project I cannot seem to be able to make networking calls outside of the docker container once deployed to the GAE.

Here is the stack trace:

18:32:06.000 2017-01-14 02:32:06.670 [vert.x-eventloop-thread-1] ERROR io.vertx.ext.web.authentication.GoogleUserDataRetriever - Connection error:Network is unreachable: www.googleapis.com/2607:f8b0:4001:c0d:0:0:0:5f:443
18:32:06.000 io.netty.channel.AbstractChannel$AnnotatedSocketException: Network is unreachable: www.googleapis.com/2607:f8b0:4001:c0d:0:0:0:5f:443

My Dockerfile is the following:

FROM gcr.io/google_appengine/openjdk8
RUN useradd -ms /bin/bash vertx
RUN mkdir /app
WORKDIR /app
COPY build/distributions/server.zip /app
RUN unzip /app/server.zip
RUN mv /app/server/* /app
RUN rm -rf /app/server
RUN chown -R vertx /app
USER vertx
EXPOSE 8080
ENV APP_HOME=/app
ENTRYPOINT /app/bin/server

And my app.yaml is this:

runtime: custom
env: flex

handlers:
- url: /.*
  script: this field is required, but ignored
  secure: always

# [START env_variables]
env_variables:
  USE_GAE_MEMCACHE: 1
# [END env_variables]
# [END appyaml]

https://github.com/GoogleCloudPlatform/runtimes-common/tree/master/structure_tests

It has been discussed as part of GoogleCloudPlatform/jetty-runtime#68 (and it's associated PR GoogleCloudPlatform/jetty-runtime#81) that container independent parts of the logging configuration should eventually be moved from the jetty-runtime to the openjdk-runtime.

This would potentially involve: google-cloud-java jars and their dependencies being placed in the image; a configuration file that would be able to be turned on via an environment variable.

However, on consideration, I am not sure if doing the above is all that valuable. With modern build tools, assembling the jars necessary for an application is simple. It is actually more difficult to merge in a classpath of image-provided jars with an application, as there may be duplicates and version differences that need to be resolved (exactly the task that application build tools are best at).

Furthermore, turning on logging with a simple env variable will seldom be sufficient. Users will often wish to configure their logging with a custom logging.properties file, so a provided one is of less value.

We are providing stackdriver logging in the jetty-runtime because we are providing the container and we are responsible for the containers logs. However, we do hide the stackdriving classes from the application and it is free to provide it's own different mechanisms for logging and their dependencies. If the users of openjdk-runtime are treated like the users of the jetty-runtime, then they are responsible for their own application logging. Of course google-cloud-logging is easily available and should be well documented to make logging to stackdriver simple if the application chooses to do so.

Thus I'm in favour of leaving stackdriver logging only in the jetty-runtime.

There's some issue with memory calculation.

Successfully tagged openjdk-local-integration:latest
Starting app container...
Shutdown logging threshold of 100% satisfied with sample 46.
Start command: /shutdown/shutdown-wrapper.bash java -showversion -Xms0M -Xmx0M -XX:+UseG1GC -XX:+ParallelRefProcEnabled -XX:+PrintCommandLineFlags -jar app.jar
Invalid maximum heap size: -Xmx0M
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.

The integration test on GKE need kubectl to be installed in order to deploy the test application.
The kubectl executable is currently not available during CI build and need to be installed in the cloud-init script.

We're installing ca-certificates-java as a requirement for the openjdk, but it seems to depend on JRE 7. We need to make sure that we're not polluting the image with JRE 7.

The cloud debugger (/opt/cdbg/format-env-appengine-vm.sh) assumes that the directory /var/log/app_engine exists, so this needs to be created in the Dockerfile