googlecloudplatform / jetty-runtime Goto Github PK

The use of chown -R in Dockerfiles is currently used to ensure the files are correctly owned. However as pointed out in moby/moby#6119, this can result in significant bloat in the size of the resulting docker images.

We should consider new docker features to avoid this issue moby/moby#10775, or at least to a manual unpack of the tgz files that will allow user and group to be set correctly initially

Merge from here: GoogleCloudPlatform/appengine-java-vm-runtime#295

@joakime Can you pick this up?

Rename HttpURLUtil to HttpUrlUtil

The documentation for using a Dockerfile at https://cloud.google.com/appengine/docs/flexible/java/dev-jetty9 includes the modules to enable jmx and the hawtio console. This should not be deployed by default as it has no authentication and allows anybody to access and control the server via JMX.

note also that the documentation is out of date and it does not indicate a docker director is needed.

Update plugin version to 1.0.0, via property in parent pom.

If gcloud deployment allow for native / JNI executions then we should create a test webapp to ensure that works.

This could include a combination of using pre-existing binaries on the docker images, and also using libs included in the webapp.

Should be preceded by #41 that enables clustered session management using Datastore.
This feature should be optionally turned on using an environment variable.

As noted in /pull/17 a new version of the appengine-maven-plugin version has been released.

We should start to use it.

Create a /tests/test-war-quickstart that tests the quickstart facility of jetty.

The sample project should be sufficiently complex. Perhaps either the spring petclinic, or the cdi/weld petclinic.

Jersey is a common toolkit seen with Jetty.

We should create a webapp showing setup and proper behavior of Jersey.

• Should include problematic annotations (like @Resource)

Tests should verify that jetty logs and application logs make their way to cloud logging. (Logs should be viewable via the cloud logging API or gcloud).

@janbartel @gregw @joakime @jmcc0nn3ll
A customer has asked this question. Do you guys have any recommendations for this?

The various features of jetty are activated and configured by our module mechanism, which has so far been done within the docker file. There is a discussion about using environment variables to control some of these features, so this issue is to discuss exactly how that can be done.

Firstly some background on the jetty module mechanism. An example is to activate gcloud sessions we need to run a command (currently from a Dockerfile RUN command) like:

java -jar $JETTY_HOME/start.jar --add-to-start=session-store-gcloud,jcl

The --add-to-start command enables both the session-store-gcloud module and it's dependencies, which primarily is the sessions module. Note that we also add the jcl module to pick one of our available implementations of java commons logging, which is a dependency of the gcloud library used (if there was only one, we'd pick it automatically).

To add memcache, you would need to run the command (or add it to the previous command):

java -jar $JETTY_HOME/start.jar --add-to-start=session-store-cache

This is easy enough to do in custom Dockerfiles, but the suggestion is that such features need to be selectable at run time via env variables. So we need to consider how best to do that, so let's see what our --add-to-start command does.

When adding a thirdparty module like gcloud session, the --add-to-start command downloads any dependencies not part of the normal jetty distribution, which for gcloud, this is quiet a few jars, so I don't think we'd want to be doing this for every instance we start up as it increases the chance of a failure. The solution to this is that in our image Dockerfile for jetty-runtime we can enable all the modules that might be enabled by env variables, so the library downloads are done, and then disable them by removing the related start.d/*.ini files.

Next the default command line is modified to include activation of the module, which is simply adding something like: --module=session-store-gcloud to the command line. This is what could be done by a start script that checks the env variables.

Finally, the --add-to-start command creates a start.d/session-store-gcloud.ini file with parameters that can be configured for the module. e.g.:

[] cat start.d/session-store-gcloud.ini 
# --------------------------------------- 
# Module: session-store-gcloud
# Enables GCloudDatastore session management.
# --------------------------------------- 
--module=session-store-gcloud


## GCloudDatastore Session config
#jetty.session.gcloud.maxRetries=5
#jetty.session.gcloud.backoffMs=1000



[] cat start.d/sessions.ini 
# --------------------------------------- 
# Module: sessions
# The session management. By enabling this module, it allows
# session management to be configured via the ini templates
# created or by enabling other session-cache or session-store
# modules.  Without this module enabled, the server may still
# use sessions, but their management cannot be configured.
# --------------------------------------- 
--module=sessions

## The name to uniquely identify this server instance
#jetty.sessionIdManager.workerName=node1

## Period between runs of the session scavenger (in seconds)
#jetty.sessionScavengeInterval.seconds=60

[] cat start.d/session-store-cache.ini 
# --------------------------------------- 
# Module: session-store-cache
# Enables caching of SessionData in front of a SessionDataStore.
# --------------------------------------- 
--module=session-store-cache


## Session Data Cache type: xmemcached
session-data-cache=xmemcached
#jetty.session.memcached.host=localhost
#jetty.session.memcached.port=11211
#jetty.session.memcached.expirySec=

As you can see, almost all the parameters are commented out, so the default settings are normally good. This means that we can access the default module configuration simply by a start script that adds the appropriate --module= args to the command line based on the environment variables.

However, as soon as a user wants to configure any of these features, they are going to have to re-create the ini files and edit the parameters in them in their own custom Dockerfile. This then creates a duality where a module might be enabled in either a start.d/*.ini file and/or enabled by the start script adding a command line argument based on an environment variable.

I don't think this duality is good and it will confuse many that the mechanism changes the moment they go from a default image to a custom image.

So another idea is that our Dockerfile can enable all the possible modules and then move their start.d/*.ini files to a gae.d/ directory. The start script would then check the env variables and if a feature is turned on, it would move the ini file(s) from gae.d/ back to start.d/. To configure these features they could then use a custom Dockerfile simply to replace/edit the gae.d/*.ini files that the start script would use. Still not perfect but better.

Perhaps a better way would be for the start script to first check if a feature is already enabled, and if so ignore the env variables. Note that jetty will throw an exception if you try to configure two contradictory modules (eg two non compatible session data stores), so if a user enables one session store and the env variable is indicating another, we need to either error or ignore the env variable?

To make more progress on deciding how to do this, we need to know more about what the environment variables will look like. In #41 @janbartel will soon enumerated the various session configurations that we could deploy. @meltsufin can you look at that issue and when you indicate which of those you want available on jetty-runtime can you also indicate how you imagine the environment variable will look like? Will it be several booleans? an enumeration? will there be configuration env variables such as ports etc.?

Followup on #60.

The docker label prefix mechanism is no longer needed. This was used to add a branch prefix to docker labels, but there is no longer an async branch.

The /tests/gcloud-testing-core/ uses the gcloud CLI to access the remote logs.

Migrate to the google cloud logging API / Library directly.

Identified behaviors to look for

• Add individual WEB-INF/lib/*.jar for each bytecode support level 1.1 to 1.8
• Add known common bad bytecode - (eg: icu4j-2.6.1.jar and the com/ibm/icu/impl/data/LocaleElements_zh__PINYIN.class entry)
• Add known slow to scan jars - (eg: org.webjars:extjs:5.1.0.jar)
• Include java 1.8 new bytecode (lamdas, etc)
• Add webapp usage of an old asm.jar (to ensure that the server side will not fail with this setup)

Identified behaviors

• Normal AsyncProxyServlet
• Normal ConnectHandler
• AsyncMiddleManServlet

Continuous integration testing needs to setup with:

• a snapshot build for every commit to this repository against a fixed version of openjdk-runtime
• a cascade build for every commit to openjdk-runtime against the new image

Some identified webapps

• Servlet 2.3
• Servlet 2.4
• Servlet 2.5
• Servlet 3.0 (w/ annotations, and async context)
• Servlet 3.1 (w/ async I/O)
  • WEB-INF/lib resource manifests
  • WEB-INF/lib fragments

Jetty request logging should either be completely turned off because request logging is already done by nginx proxy, or log rotation must be setup to limit the log file size on disk.

The Google Cloud Logging API or the stdout and stderr should be used to send logs to Stackdriver Logging.

Should quickstart be supported by default, or documented as a desirable feature.
Testing done in #8 using quickstart and/or annotations

Session clustering should be optionally enabled using an environment variable. If not enabled, local in-memory session management would be used.

Useful links:

• Jetty Session Clustering using Memcached and Datastore
• Jetty GCloudMemcachedSessionManager
• AccessingMemcached Service on GCP

This will provide access to:

• greatly improved session manager and gcloud/memcache integration
• more flexible request customizer
• improved performance

Do we want to have a test webapp showing scala behavior and usage with jetty-runtime?

The spring petclinic should be a good general usecase of a bigger webapp to integration test with

We'll have to build our own version, as the existing spring-petclinic is currently broken due to a Dandelion bug.

See: dandelion/dandelion#113

Identified behaviors

• Native Jetty WebSocket
  • Server
  • Client
  • Servlet Based
  • Filter Based
  • Annotation Based
  • Listener Based
    • WebSocketConnectionListener
    • WebSocketFrameListener
    • WebSocketListener
    • WebSocketPartialListener
    • WebSocketPingPongListener
• JSR356 (javax.websocket)
  • Server
    • @PathParam's
    • ServerApplicationConfig
    • ServerEndpointConfig (w/ HttpSession capture)
  • Client
    • ClientEndpointConfig
  • Custom Encoders / Decoders
  • Annotation Based
  • Endpoint Based

There are a number of weld maintained examples at

https://github.com/weld/core/tree/master/examples

I propose we use weld's jsf/permalink example

regular builds should be tested against deployments of several common webapps such as spring pet clinic, to both demonstrate how to deploy those webapplication into gcloud flex and to test for regressions

openjdk-runtime is using 0.4.13, this runtime should update from 0.3.2 to the new version

Jetty is fully instrumented with JMX mbeans that allow monitoring, control and debugging of the server. The JVM also provides useful information via JMX.

Should we provide modules and/or documentation to demonstrate how to enable JMX in the jetty-runtime and how to access it remotely.

Requests to the runtime will have the following headers:

• x-forwarded-for is a list of upstream proxy IP addresses
• x-cloud-trace-context is a unique identifier for the request used for traces and logging
• x-forwarded-proto shows HTTP or HTTPS based on the origin request protocol

Test that user servlets see the correct headers to comply with the servlet spec.

How does JMX fit into the picture of testcases?

• test that a user application can read/write directly to memcached
• test that jetty session storage via memcached works

https://cloud.google.com/appengine/docs/flexible/java/caching-application-data

There's a few scenarios where Jetty is in use, but wouldn't use the jetty-runtime (or jetty-distribution) to execute.

A few common examples:

• Solr
• Spring-Boot
• Dropwizard

Proposal to add the following to the repository:

• /third_party/dandelion/ - which will be an initial import of HEAD of the master branch from https://github.com/dandelion/dandelion followed by the patch from PR dandelion/dandelion#116
• /third_party/spring_petclinic/ - which will be an initial import of HEAD of the master branch from https://github.com/spring-projects/spring-petclinic followed by a patch to use the patched dandelion above

These 2 projects will be wired up into the maven build.

What should the groupId and artifactId be for those two new maven modules?

https://firebase.google.com/docs/auth/

Session clustering backed by Datastore should be optionally enabled using an environment variable. If not enabled, local in-memory session management would be used.

See #5 for previous discussion on this topic.

Identified sample test webapps:

• Sessions using HashSessionManager
• Sessions using jetty-gcloud-session-manager
  • include positive results with a sufficient cluster
  • include negative results when using insufficient cluster size

Regular load testing should be conducted against deployed webapplication. Several fixed levels of load should be offered and the latency/qos measured and logged.

This will allow tracking of performance changes over releases.

As there can be a delay between deploy and the webapp being available for traffic, the HttpURLUtil should have retry logic to keep retrying (for a little while) in case the startup is slow.

There are situations where the webapp is accidentally built with SCM contents included.

We can build a test webapp to ensure that those SCM contents are not being accessed.

Not sure how important this style of validation is for gcloud though.

The test could be something simple like

• deploy a sample app
• set a snapshot (aka a breakpoint) via the API or gcloud
• exercise the breakpoint
• verify that the breakpoint has been hit

Identified webapps

• JSP 2.0 very old usage
• JSP 2.1 older usages
• JSP 2.2 (seen in Jetty 8)
• JSP 2.3 (seen in Jetty 9)
• Custom Taglibs
• JSTL / EL usages
• WEB-INF/lib/*.jar inclusions that can cause problems
  • extra javax.el (duplicates or more than 1 versions) api libs
  • extra javax.el (duplicates or more than 1 versions) impl libs
  • extra jstl (duplicates or more than 1 versions) api libs
  • extra jstl (duplicates or more than 1 versions) impl llibs

The following scenarios are identified:

• log4j on server (application logging separate)
• logback on server (application logging separate)
• log4j on server (application forced to use server log4j)
• consolidated server logging (application forced to use server logging implementations and configurations for: commons-logging, java.util.logging, slf4j, log4j, logkit, and juli)

Identified webapp

• FastCGI + Wordpress

The following combinations are being done.

• commons_logging_1.0.3 (using Log4JLogger)
• commons_logging_1.1 (using Jdk14Logger)
• java_util_logging (direct usage, reset of root, configured output to console)
• log4j_1.1.3 (direct usage and configured output)
• log4j_1.2.15 (direct usage and configured output)
• log4j2 (direct usage and configured output. more in future)
• slf4j_1.2 (direct usage, log4j output)
• slf4j_1.5.6 (configured to capture log4j + commons-logging, output to java.util.logging)
• slf4j_1.6.1 (configured to capture java.util.logging, output to log4j)
• slf4j_1.6.6 (w/ org.slf4j.spi.LocationAwareLogger usage to log4j)
• slf4j_1.7.2 (direct usage, log4j output)
• apache juli (direct usage and from apache jsp serviceloader)
• logback (direct usage, slf4j usage, capture java.util.logging, capture log4j, capture commons-logging, w/logback access, output by logback)
• multiple logging libs (log4j direct + configured output, java.util.logging direct + configured output, and commons-logging discovered, slf4j-api -> logback + configured output)