Upstream HTTP request error about of-watchdog HOT 11 CLOSED

I experienced this just today as well. This is an issue with Go and how the connections are handled. Sorry I can't provide a solution off-hand, but know you're not the only one!

In my case, I am using the go-http template as the process function where this function calls out to 3 others to complete a workflow. I receive that error on the last of the 3 calls. The call succeeds, but the error is still present.

Here's what I was able to find on StackOverflow
To summarize:

The problem here occurs when you initiate another request before the readLoop actually detects the close, and the EOF that it reads get interpreted as an error for that new request rather than a close that occurred prior to the request.

from of-watchdog.

@burtonr were you able to resolve this?

It looks like the request is occuring here rather than in my handler code:
https://github.com/openfaas-incubator/of-watchdog/blob/85505a7210cf413e455f8a03d74ba94d9a9fcd30/executor/http_runner.go#L119-L134

from of-watchdog.

I was not able to solve this. Since it's the last function call, and the call still succeeds, I moved on. Most of the team on this project are in the European time zones, so I was going to ask when they were available.

Based on that StackOverflow answer, I would say that you're probably right. Adding request.Close = true there may fix it. I'll give it a try and see what happens!

from of-watchdog.

I have added request.Close = true below line 124 and that did not resolve the problem.

from of-watchdog.

@burtonr After a bunch of debugging it appears the issue is occurring here:

https://github.com/openfaas-incubator/golang-http-template/blob/4783533d117fcd65d16aa8557aff38e164a889e0/template/golang-http/main.go#L63-L64

ReadTimeout:  3 * time.Second,
WriteTimeout: 3 * time.Second,

That value is currently hard coded, but when I increase it I get this:

2019/02/20 05:20:52 POST / - 200 OK - ContentLength: 33

instead of this:

2019/02/20 05:15:59 Upstream HTTP request error: Post http://127.0.0.1:8081/: EOF

Perhaps this will help you.

I am not sure why that connection has to persist throughout the function execution. It would seem the of-watchdog process should just create a new connection when it needs to send data.

from of-watchdog.

Interesting. What did you set those values to?

Those values should be configurable via environment variables read_timeout and write_timeout

I'll wait until @alexellis comes back online to check with him. I'm sure he can provide better input.

I know this template is one of the more popular ones, and I haven't heard of this issue before. It's possible either we're doing something not quite right, or nobody's used this template in a long running function yet (which is quite possible since it's probably the fastest at the moment)

from of-watchdog.

With this test function, I tested with a 20 sec value:

func Handle(req handler.Request) (handler.Response, error) {
	var err error

	time.Sleep(7 * time.Second)
	fmt.Printf("Current time %v", time.Now().Unix())
	message := fmt.Sprintf("Hello world, input was: %s", string(req.Body))

	return handler.Response{
		Body:       []byte(message),
		StatusCode: http.StatusOK,
	}, err
}

For my app, I am using 14400s.

from of-watchdog.

These values need to be made configurable in the specific template that you've chosen. a proposal for this on a specific repository would be welcome.

What you might want to do in the interim is use the standard go template which already supports a configurable time out value.

from of-watchdog.

@alexellis I have made a PR here:
openfaas/golang-http-template#20
I did this all in the GitHub web ui so there is no signoff option.
But I am mostly looking for feedback to see if the specific approach makes sense.

from of-watchdog.

Looks like this was already raised in October 2018, but nobody from the community volunteered to work on it.

openfaas/golang-http-template#9

Alex

from of-watchdog.

@brandonkal did you delete your comment? I just got an email from GitHub

from of-watchdog.