External metric not working for Eventhubs about azure-k8s-metrics-adapter HOT 11 OPEN

The error reported above "Error from server (ServiceUnavailable): the server is currently unable to handle the request" was the exact same error message as #84 which was resolved by making sure everything was properly namespaced.

Happy to help resolve if it is not related. Please be mindful of our contributing guidelines and code of conduct.

Could you provide the logs of the metric adapter (specifically after a request for the eventhubs metric was queried) and the following info:

Kubernetes version (kubectl version):

• Running on AKS

Logs (kubectl logs <metric adapter pod id>)

Additional context
kubectl describe hpa

Adapter version:

from azure-k8s-metrics-adapter.

I was able to track down the issue. It appears that IncomingMessages doesn't return any values through the Metric API at the Eventhub level which is where you are filtering with filter: EntityName eq 'customname'. I am looking into why that metric is not being reported at the EventHub level. It is reported at the Namespace level.

The reason you are getting the Error from server (ServiceUnavailable): the server is currently unable to handle the request message is because the code for the Metric API checks for null but not length so end up with the following exception:

    1 monitor_client.go:50] resource uri: /subscriptions/b9d9436a-0c07-4fe8-b779-2c1030bd7997/resourceGroups/sb-external-example/providers/Microsoft.EventHub/namespaces/samplejjs
E0310 20:36:05.708112       1 runtime.go:71] Observed a panic: runtime error: index out of range
goroutine 62133 [running]:
github.com/Azure/azure-k8s-metrics-adapter/vendor/k8s.io/apiserver/pkg/server/filters.(*timeoutHandler).ServeHTTP.func1.1(0xc0004db560)
        /go/src/github.com/Azure/azure-k8s-metrics-adapter/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:106 +0xda
panic(0x1701480, 0x2bdaa40)
        /usr/local/go/src/runtime/panic.go:522 +0x1b5
github.com/Azure/azure-k8s-metrics-adapter/pkg/azure/externalmetrics.extractValue(0xc0004f7f30, 0x10, 0xc000580bd0, 0x24, 0xc0004f7f40, 0xc, 0xc0004f7f13, 0x9, 0xc000667900, 0x12, ...)
        /go/src/github.com/Azure/azure-k8s-metrics-adapter/pkg/azure/externalmetrics/monitor_client.go:81 +0xd56
github.com/Azure/azure-k8s-metrics-adapter/pkg/azure/externalmetrics.(*monitorClient).GetAzureMetric(0xc000530460, 0xc0004f7f30, 0x10, 0xc000580bd0, 0x24, 0xc0004f7f40, 0xc, 0xc0004f7f13, 0x9, 0xc000667900, ...)
        /go/src/github.com/Azure/azure-k8s-metrics-adapter/pkg/azure/externalmetrics/monitor_client.go:60 +0x4a0
github.com/Azure/azure-k8s-metrics-adapter/pkg/provider.(*AzureProvider).GetExternalMetric(0xc00052e600, 0xc0000d6491, 0x7, 0x1c15360, 0xc000530320, 0xc0000d6499, 0xf, 0x3860f5559f5, 0xc00cfdb8d9, 0xcfdb8d900000000)
        /go/src/github.com/Azure/azure-k8s-metrics-adapter/pkg/provider/provider_external.go:41 +0x4f1
github.com/Azure/azure-k8s-metrics-adapter/vendor/github.com/kubernetes-incubator/custom-metrics-apiserver/pkg/registry/external_metrics.(*REST).List(0xc000593ab0, 0x1c04140, 0xc0001aac30, 0xc00037b580, 0x1918386, 0x2, 0x1bd9340, 0xc00037b580)
        /go/src/github.com/Azure/azure-k8s-metrics-adapter/vendor/github.com/kubernetes-incubator/custom-metrics-apiserver/pkg/registry/external_metrics/reststorage.go:79 +0x209
github.com/Azure/azure-k8s-metrics-adapter/vendor/k8s.io/apiserver/pkg/endpoints/handlers.ListResource.func1(0x1bf68c0, 0xc00000e368, 0xc000931400)
        /go/src/github.com/Azure/azure-k8s-metrics-adapter/vendor/k8s.io/apiserver/pkg/endpoints/handlers/get.go:268 +0xc3e
github.com/Azure/azure-k8s-metrics-adapter/vendor/github.com/kubernetes-incubator/custom-metrics-apiserver/pkg/apiserver/installer.restfulListResource.func1(0xc0001aab40, 0xc0004fac60)
        /go/src/github.com/Azure/azure-k8s-metrics-adapter/vendor/github.com/kubernetes-incubator/custom-metrics-apiserver/pkg/apiserver/installer/installer.go:288 +0x8f
github.com/Azure/azure-k8s-metrics-adapter/vendor/k8s.io/apiserver/pkg/endpoints/metrics.InstrumentRouteFunc.func1(0xc0001aab40, 0xc0004fac60)
        /go/src/github.com/Azure/azure-k8s-metrics-adapter/vendor/k8s.io/apiserver/pkg/endpoints/metrics/metrics.go:272 +0x254
github.com/Azure/azure-k8s-metrics-adapter/vendor/github.com/emicklei/go-restful.(*Container).dispatch(0xc000320480, 0x7fbf4ccb44c8, 0xc00000e348, 0xc000931400)
        /go/src/github.com/Azure/azure-k8s-metrics-adapter/vendor/github.com/emicklei/go-restful/container.go:277 +0x985
github.com/Azure/azure-k8s-metrics-adapter/vendor/github.com/emicklei/go-restful.(*Container).Dispatch(...)
        /go/src/github.com/Azure/azure-k8s-metrics-adapter/vendor/github.com/emicklei/go-restful/container.go:199
github.com/Azure/azure-k8s-metrics-adapter/vendor/k8s.io/apiserver/pkg/server.director.ServeHTTP(0x192efce, 0x16, 0xc000320480, 0xc000299880, 0x7fbf4ccb44c8, 0xc00000e348, 0xc000931400)
        /go/src/github.com/Azure/azure-k8s-metrics-adapter/vendor/k8s.io/apiserver/pkg/server/handler.go:146 +0x4e4
github.com/Azure/azure-k8s-metrics-adapter/vendor/k8s.io/apiserver/pkg/endpoints/filters.WithAuthorization.func1(0x7fbf4ccb44c8, 0xc00000e348, 0xc000931400)
        /go/src/github.com/Azure/azure-k8s-metrics-adapter/vendor/k8s.io/apiserver/pkg/endpoints/filters/authorization.go:64 +0x4fa
net/http.HandlerFunc.ServeHTTP(0xc00007b2c0, 0x7fbf4ccb44c8, 0xc00000e348, 0xc000931400)
        /usr/local/go/src/net/http/server.go:1995 +0x44
github.com/Azure/azure-k8s-metrics-adapter/vendor/k8s.io/apiserver/pkg/server/filters.WithMaxInFlightLimit.func1(0x7fbf4ccb44c8, 0xc00000e348, 0xc000931400)
        /go/src/github.com/Azure/azure-k8s-metrics-adapter/vendor/k8s.io/apiserver/pkg/server/filters/maxinflight.go:160 +0x5c7
net/http.HandlerFunc.ServeHTTP(0xc0003b07e0, 0x7fbf4ccb44c8, 0xc00000e348, 0xc000931400)
        /usr/local/go/src/net/http/server.go:1995 +0x44
github.com/Azure/azure-k8s-metrics-adapter/vendor/k8s.io/apiserver/pkg/endpoints/filters.WithImpersonation.func1(0x7fbf4ccb44c8, 0xc00000e348, 0xc000931400)
        /go/src/github.com/Azure/azure-k8s-metrics-adapter/vendor/k8s.io/apiserver/pkg/endpoints/filters/impersonation.go:50 +0x1ec3
net/http.HandlerFunc.ServeHTTP(0xc00007b300, 0x7fbf4ccb44c8, 0xc00000e348, 0xc000931400)
        /usr/local/go/src/net/http/server.go:1995 +0x44
github.com/Azure/azure-k8s-metrics-adapter/vendor/k8s.io/apiserver/pkg/endpoints/filters.WithAuthentication.func1(0x7fbf4ccb44c8, 0xc00000e348, 0xc000931300)
        /go/src/github.com/Azure/azure-k8s-metrics-adapter/vendor/k8s.io/apiserver/pkg/endpoints/filters/authentication.go:81 +0x527
net/http.HandlerFunc.ServeHTTP(0xc00037fb30, 0x7fbf4ccb44c8, 0xc00000e348, 0xc000931300)
        /usr/local/go/src/net/http/server.go:1995 +0x44
github.com/Azure/azure-k8s-metrics-adapter/vendor/k8s.io/apiserver/pkg/server/filters.(*timeoutHandler).ServeHTTP.func1(0xc0004db560, 0xc000215a40, 0x1c04480, 0xc00000e348, 0xc000931300)
        /go/src/github.com/Azure/azure-k8s-metrics-adapter/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:111 +0xb3
created by github.com/Azure/azure-k8s-metrics-adapter/vendor/k8s.io/apiserver/pkg/server/filters.(*timeoutHandler).ServeHTTP
        /go/src/github.com/Azure/azure-k8s-metrics-adapter/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:98 +0x1b1

At a more meta level, I am not sure you would want to scale on just the number of messages incoming. I would think you would either want scale based on the lag of the messages being processed or number of unprocessed messages. It doesn't look like either of those are captured via the Azure Metrics api. KEDA does have support for all the metrics you are currenlty scaling on, including supporting the number of unprocessed messages in eventhub.

from azure-k8s-metrics-adapter.

related to #84

from azure-k8s-metrics-adapter.

related to #84

May I know how is this related to #84 ?? Kindly read my description, I've setup external metrics for other services and they are working fine. Now in the same cluster using the same adapter I'm unable to implement using event hubs. As per your suggestion if launching in different namespace was the problem nothing should have worked for me. I would like to have this issue reopened as I feel there has been a misunderstanding. I've even raised a case with Azure support regarding this and even they were not sure what the problem is. My case is still open there too

from azure-k8s-metrics-adapter.

@jsturtevant
I've corrected my last reply according to the code of conduct. Sorry for that. Below are the details you asked for.

Adapter Version: 0.5.0
Kubernetes Version: 1.14.8
Yes, The cluster is running on AKS

Logs from metric-adapter Pod:

I0303 06:05:07.749429 1 controller.go:149] adding item to queue for 'mynamespace/extmetric03' with kind 'ExternalMetric'
I0303 06:05:07.754483 1 handler.go:82] processing item 'extmetric02' in namespace 'mynamespace'
I0303 06:05:07.754607 1 handler.go:111] adding to cache item 'extmetric02' in namespace 'mynamespace'
I0303 06:05:07.754625 1 controller.go:134] succesfully proccessed item '{mynamespace/extmetric02 ExternalMetric}'
I0303 06:05:07.754634 1 controller.go:91] processing next item
I0303 06:05:07.754640 1 controller.go:98] processing item
I0303 06:05:07.754579 1 handler.go:82] processing item 'extmetric01-eventhub' in namespace 'mynamespace'
I0303 06:05:07.754729 1 handler.go:111] adding to cache item 'extmetric01-eventhub' in namespace 'mynamespace'
I0303 06:05:07.754737 1 controller.go:134] succesfully proccessed item '{mynamespace/extmetric01-eventhub ExternalMetric}'

In the logs, the external metric "extmetric01-eventhub" is the one using event-hubs. You can safely ignore others as they use other Azure services and they are working fine too.

When I enter the command kubectl get hpa I'm getting the same error "Error from server (ServiceUnavailable): the server is currently unable to handle the request"

from azure-k8s-metrics-adapter.

So it looks like the HPA is not able to find the metric adapter. It is interesting that it is working for the other metrics in the cluster.

A few thoughts:

• Is this HPA and External metric in a specific namespace? Is extmetric02 working as expected? it looks like it is in the same namespace.
• Is it possible there is RBAC or networking policy is blocking access to the api server?
• Can you share the HPA definition that is pointing at the external metric for event hub?

Can you hit the metric adapter directly using the following command:

kubectl  get --raw "/apis/external.metrics.k8s.io/v1beta1/namespaces/mynamespace/extmetric01-eventhub" 

from azure-k8s-metrics-adapter.

Another thing to check is that the metric adapter is registered:

kubectl describe apiservices v1beta1.external.metrics.k8s.io

from azure-k8s-metrics-adapter.

@jsturtevant
Following is the output when I enter the command
kubectl get --raw "/apis/external.metrics.k8s.io/v1beta1/namespaces/mynamespace/extmetric01-eventhub"
Output:

Error from server (ServiceUnavailable): the server is currently unable to handle the request

kubectl describe apiservices v1beta1.external.metrics.k8s.io
Output:

Name: v1beta1.external.metrics.k8s.io
Namespace:
Labels:
Annotations:
API Version: apiregistration.k8s.io/v1
Kind: APIService
Metadata:
Creation Timestamp: 2020-02-13T13:10:23Z
Resource Version: 15729871
Self Link: /apis/apiregistration.k8s.io/v1/apiservices/v1beta1.external.metrics.k8s.io
UID: 31955796-4e62-11ea-ad67-1a32613f77cc
Spec:
Group: external.metrics.k8s.io
Group Priority Minimum: 100
Insecure Skip TLS Verify: true
Service:
Name: azureadapter-azure-k8s-metrics-adapter
Namespace: custom-metrics
Version: v1beta1
Version Priority: 100
Status:
Conditions:
Last Transition Time: 2020-03-03T23:50:49Z
Message: all checks passed
Reason: Passed
Status: True
Type: Available
Events:

Coming to your point "It appears that IncomingMessages doesn't return any values through the Metric API at the Eventhub level which is where you are filtering with filter: EntityName eq 'customname'" I'm able to see metric graph for the metric "IncomingMessages" from the portal at eventhub level too. Below is a screenshot for your reference.

Let me know if I've got anything wrong here and coming to KEDA I cannot do that since it supports only if I enable capture, as it can currently get from storage account only. This won't work for me for many reasons and cost is also one of them.

Your efforts on this are to be appreciated!

from azure-k8s-metrics-adapter.

It seems when there are no messages for a period of time the result metric api for IncomingMessage returns an empty time series. After putting load on it for awhile a while I was getting a correct result back but after letting it sit for awhile I am getting the error again. The initial time for that value to come back was over several hours though.

Although I understand your current cost constraint, the delay in the initial metrics showing up initially causes me to hesitate that this metric would be correct metric to scale on. As well, this metric might not work well as it will return the current Messages coming in which might peak then drop zero even though you haven't made a dent in the queue. Using this metric, you could be in a situation where there are no/low number of Incoming Messages which would cause the HPA to scale down but the processing pods and yet you could have a really high quantity in the event queue.

KEDA does support the Azure Monitor metrics so you could use it to scale on this same metric there if you wanted thought I think you will run into the same challenges with this particular metric.

from azure-k8s-metrics-adapter.

@jsturtevant Yeah saw that KEDA has launched 1.3.0 recently with the Azure monitor scaler. Gonna give it a try now. Will update you how that goes, Also with respect to the metric behavior I think that is how it is supposed to behave as there are checkpoints and other stuff. Anyway I'll discuss with my team and update you.

from azure-k8s-metrics-adapter.

@jsturtevant Just now saw the exact same error in KEDA too. They closed your issue but it seems like they haven't fixed it. I've requested to reopen your issue again there.

from azure-k8s-metrics-adapter.