JVM never receives SIGTERM (shutdownhook never called) about of-watchdog HOT 12 OPEN

Hi thanks for the question. You get a graceful shutdown period built-in. This gives any HTTP requests a grace period to shut down (you should be able to read about this in the readme)

The period is connected to write_timeout which I see you may not be specifying correctly at present. A Golang duration is needed i.e. 20s.

from of-watchdog.

Derek add label: question

from of-watchdog.

What I'm thinking about is a hook whenever a container is taken out of commission (e.g. scale to 0, function removed, etc.). Having a warm JVM means that collecting/releasing resources on each invocation seems a bit silly, so it would be nice to know when the resources MUST be released.

from of-watchdog.

Are you saying it does indeed forward SIGTERM to the fprocess?

from of-watchdog.

Hi @hejfelix I would be curious as to what Lambda, Google Functions or Azure Functions do in this scenario. If they do pass on SIGTERM then we should investigate it.

Alex

from of-watchdog.

Kubernetes has a rather complicated shutdown procedure regarding health-checks which does not make things easy. cc @LucasRoesler @stefanprodan

from of-watchdog.

My experience with AWS Lambda was that there was no shutdown hook. This essentially meant that I wasted a lot of time acquiring and releasing resources on every invocation. This defeats the purpose of having a warm function.

from of-watchdog.

There are a couple of things that happen during shutdown in a k8s pod

1. the Pod is marked as terminated/deleted, this also removes it from various Endpoints and Services in k8s
2. the preStopHook is executed, this allows the Pod to respond to and customize the stop behavior. But ... this is not something we support configuration of
3. SIGTERM is sent to the Pod
4. when the grace period expires, SIGKILL is sent. The grace period defaults to 30 seconds and can be customized on the Pod spec via terminationGracePeriodSeconds

By far the simplest thing we could do is to listen for and send the SIGTERM to the child process. Which should already be happening (this is the relevant chunk of code that sends the TERM to the fprocess in http mode)

In short, a quick review of the code looks like it is trying to send the TERM signal to the fprocess, in this case the JVM.

from of-watchdog.

Interesting -- I am running on docker swarm in all my tests so far. Is that supposed to behave the same way? I guess I'm struggling to find a way to verify that my shutdown hook is run since nothing appears in docker service logs after node is taken down. Any ideas?

from of-watchdog.

Probably the simplest way to verify it is to send a message somewhere, e.g. to RequestBin . If you are certain that your shutdown hook runs on SIGTERM and If you don't get a message, then that means you didn't get a SIGTERM

from of-watchdog.

Right, so I'm not getting any shutdown hook. This is my scala code:

  Runtime.getRuntime().addShutdownHook(new Thread(){
    override def run(): Unit = Source.fromURL(s"http://requestbin.fullcontact.com/wrh63owr/SHUTDOWN_HOOK_SUCK_IT_LAMBDA_${Random.nextInt}").toList
  } )

If I run this line e.g. in the REPL, it works:

Source.fromURL(s"http://requestbin.fullcontact.com/wrh63owr/SHUTDOWN_HOOK_SUCK_IT_LAMBDA_${Random.nextInt}").toList

from of-watchdog.

Im starting to believe the problem occurs because my web framework is not reacting to sigterm. Will investigate now and return.

from of-watchdog.