datastax / burnell Goto Github PK

When running with burnell enabled as part of the Pulsar Proxy, I am seeing the following error:

2022/04/26 05:49:39 error GET brokers https://pulsar-broker.default.svc.cluster.local:8443/admin/v2/brokers/pulsar error Get "https://pulsar-broker.default.svc.cluster.local:8443/admin/v2/brokers/pulsar": x509: certificate signed by unknown authority app=broker stats cache

Based on a quick look through the source code, there is no support for configuring the root CA when using self signed certs. This is already support using the TrustStore config for the pulsar+ssl protocol.

The same appears to be true for the websocket proxying done in burnell, although I am not sure if this feature is needed any more.

Problem

When deploying burnell into a pulsar cluster that does not (and has never) had the function worker running, burnell can become unresponsive when trying to use it to proxy calls. It cannot proxy calls because it cannot open connections. For reference, after burnell had been running for a little bit of time, I ran netstat -natp | wc -l within the container and found that there were about 28252 connections. It's also relevant to point out that the pulsar broker logs were filled with this log over and over again:

19:54:05.966 [ForkJoinPool.commonPool-worker-3] WARN org.apache.pulsar.broker.web.PulsarWebResource - Policies not found for public/functions namespace

And burnells logs are filled with the following block:

2021/08/24 19:54:23 error function listener terminated app=pulsar-function-listener
time="2021-08-24T19:54:23Z" level=info msg="Connecting to broker" remote_addr="pulsar://pulsar-broker:6650"
time="2021-08-24T19:54:23Z" level=info msg="TCP connection established" local_addr="10.244.2.4:52079" remote_addr="pulsar://pulsar-broker:6650"
time="2021-08-24T19:54:23Z" level=info msg="Connection is ready" local_addr="10.244.2.4:52079" remote_addr="pulsar://pulsar-broker:6650"
time="2021-08-24T19:54:23Z" level=warning msg="Failed to lookup topic: persistent://public/functions/metadata, error msg: MetadataError"
time="2021-08-24T19:54:23Z" level=warning msg="Failed to lookup topic" consumerID=1 error="failed to lookup topic: MetadataError" name= subscription=reader-xtxox topic="persistent://public/functions/metadata"
time="2021-08-24T19:54:23Z" level=error msg="Failed to create consumer" error="failed to lookup topic: MetadataError"
2021/08/24 19:54:23 error pulsar.CreateReader failed to lookup topic: MetadataError app=pulsar-function-listener

It seems to me that burnell must be making this lookup call in a loop without any type of backoff.

Proposed Solution

I think burnell should first check to see that the function namespace exists. If it does exist, proceed. If it doesn't exist, do some type of backoff to prevent near constant checking.