[FEATURE] Expose Pulsar-Client Metrics with Prometheus about pulsar-spark HOT 11 OPEN

@nlu90, checking in on this; when we last spoke about this, you had indicated June as a potential month for this release. Do you still feel confident that is the case? Please let me know if there is anything we can do to help.

from pulsar-spark.

I spoke with Neng and the work on this will be delayed.

from pulsar-spark.

@mboyanna-sn @nlu90 can you please advise if this is still something that will be done given the change we talked about?

from pulsar-spark.

@martijngonlag for any feature requests like this, going forward please create issue in a feature-request repository instead.

from pulsar-spark.

@martijngonlag aah I see this is engineering lead feature that's why it's here - for engineering to track it down. I think the right thing to do was for it to have a counterpart in feature-request (@sara-hannigan could you help us get organized here to ensure there's the counterpart in feature-request repo).

from pulsar-spark.

@nlu90 given this is in pulsar-spark repo did you refer to this feature in engineering OKRs as the Spark metrics tracking?

from pulsar-spark.

@martijngonlag You can answer Yes to the customer, this is part of Q3, I just confirmed with Neng.

from pulsar-spark.

@mboyanna-sn Checking in to see if this is still scheduled for Q3

from pulsar-spark.

Key metrics I am interested in seeing (some are directly pulsar and some are more application oriented)

1. How big a backlog there is (delta between the current "pointer" and the "head" of the topic)
2. Any systemic failures e.g. errors connecting to Pulsar or the pointer no longer refers to a position in the topic (e.g. due to retention policy which is too low and when we recreated the subscription the data was garbage collected)
3. Error counts / retries when processing

from pulsar-spark.

@frankjkelly Thanks for your comments. I was looking for a way to expose these metrics.

from pulsar-spark.

FYI we have ended our usage of spark with Pulsar so this is no longer a priority for us. Thanks!

from pulsar-spark.