robusta-dev / kubewatch Goto Github PK

For organizations running multiple clusters, right now if they were to implement kubewatch, they would likely need to open unique channels for each cluster and configure each individually so that they could discern the where the message is coming from.

For example, if you have dev, test and prod clusters, each with the same namespaces, and you get an alert saying that a secret was launched in the default namespace, how could you tell which cluster had the new secret launched.

By allowing custom tags, the user could add a field like:

customTags:
  cluster: prod

which would result in a message being sent that reads as:

kubewatch
A deployment in namespace `kubewatch` in the `my-cluster` cluster has been created:
kubewatch/kubewatch

tags:
- cluster: prod

This may be considered a duplicate of #22, but this could also be used to write more custom information to the slack message.

I'd be happy to add this functionality, or alternatively the functionality outlined in the above related issue, I am making this issue to open a discussion of the best way to move forward.

ideally, helm charts that have sensitive information uses secretKeyRef to decode base64 k8s secret and takes the name of the secret, this improves the usage of helm charts in public gitops based repos which use helm charts.

Can this be added to helm charts?

The best example of it is given here which allows both plain information or secret key reference
https://github.com/minio/minio/blob/master/helm/minio/values.yaml#L344-L348

Tried setting up using kubectl but alerts are not generating in kubectl, whereas in helm its working, If possible could you please resolve this or tell me how to proceed.

I deployed kubewatch via the Helm chart. I have it set to watch most resources and all namespaces. I see a bunch of Processing add to Event and Processing update to Event entries in the kubewatch pod log. However, I'm not receiving any messages in the Slack channel I specified in the configuration. And there are no Slack related messages in the pod log. How can I get more information about what's going on and/or troubleshoot this problem?

How possible would it be to add custom headers to webook and CloudEvent handlers?

A couple of the services we use require Authentication headers to process webhooks

When starting up the container I get the error below.

E0111 13:41:59.199162       1 reflector.go:123] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:96: Failed to list *v1.Event: Unauthorized

Any Idea how to resove this ?

Hi, the option to ignore "Event" resources doesn't seem to work.

I tried to specify in the configuration similarly to other modules:

resourcesToWatch:
  event: false

or

resourcesToWatch:
  events: false

This does not help, I see in the logs that the corresponding module starts despite the configuration:

time="2023-01-27T10:09:55Z" level=info msg="Starting kubewatch controller" pkg=kubewatch-Event

On our large cluster, this gives a lot of noise in notifications to Slack. Thank you in advance!

I have a namespace that is very noisy, i.e. jobs and pods created, updated, and deleted every minute. This namespace is on every cluster is the same and I'd like to have one configuration that works on all clusters. I don't want to have to include every namespace but this one from every cluster in a namespaceToWatch list.

It would be nice to have something like a clusterIdentity parameter that I could specify in the configuration and have that clusterIdentity included in the message sent to Slack or elsewhere.

For example, if I specified clusterIdentity: "my-cluster" in the configuration, the message in Slack would look something like this:

kubewatch
A deployment in namespace `kubewatch` in the `my-cluster` cluster has been created:
kubewatch/kubewatch

This way I can use the same Slack channel for multiple clusters and still distinguish between them.

I'm getting 10-20 of those info logs every second in my cluster (I'm viewing all logs with stern). These info logs are quite spammy. I would like to suppress those logs in the robusta helm chart, e.g. by env variables.

monitoring robusta-forwarder-5fcb56d9d-4p2nn kubewatch time="2023-02-05T13:14:42Z" level=info msg="Processing update to ConfigMap: kube-system/cluster-autoscaler-status" pkg=kubewatch-ConfigMap
monitoring robusta-forwarder-5fcb56d9d-4p2nn kubewatch 2023/02/05 13:14:42 Message successfully sent to [http://robusta-runner:80/api/handle](http://robusta-runner/api/handle) at 2023-02-05 13:14:42.869867159 +0000 UTC m=+11939.538309659

Is there any reason why Teams is not listed as supported notifications target?

The Teams support landed long time ago

• 4cc8a76
• d121cde

and some docs have been updated about it too

• 63f81f3

Interestingly, not the README.

Is the Teams support status unofficial, unreleased, etc.?

Currently it would sending to other location.
Just wonder if it is possible to save it locally.

The Helm chart doesn't allow to declare a SlackWebHook configuration (configMap only include static declarations).
It could be good to let it flexible with an extraHandlers.

Thanks for maintaining this. I'm trying to set this up in my cluster so I can receive a Slack notification whenever a warning or error level event occurs in the cluster. For example, an image pull failure or a container exists with non-zero exit code.

I do not need reporting for every event that happens. Is there a way to do this?

Thank you.

need to filter out k8s "Normal" events so they are NOT forwarded to my Slack channel. looking for a correct syntax in configmap.yaml for section "resourcesToWatch:" or "resource:".

my configmap example:
resource:
clusterrole: false
clusterrolebinding: false
configmap: false
daemonset: false
deployment: false
event: true

my helmrelease example:
resourcesToWatch:
events: true

Hi, I see in the config.json example that a "reason" configuration was planned, but it looks like it's not yet supported.
I saw in the config.go that the reason section in the Config is commented.

I understand that the goal of this section is to be able to configure which type of events to watch, or restrict which ones we want to capture. Am I right?

Is this planned to the near future?

Hi Team,

We have few custom resources running in our cluster, is it possible to generate notifications for them as well

Hi Team,

Is it possible to get notified with the events happening in the pods? I see events are not notified. PFA for the screenshot having events which are not notified.

Go runtime metrics could be useful to view on kubewatch

https://github.com/prometheus/client_golang/blob/main/prometheus/go_collector.go

It would be good to expose these from kubewatch

AWS customers would benefit from adding Amazon EventBridge as a supported destination for notifications.

Amazon EventBridge is a scalable, highly-performant, and low-cost serverless event bus. It's the default event bus provider for many AWS services, including EC2, Auto Scaling, ECS, and many more. Events can be matched using a JSON-based pattern matching, and can be delivered to many supported destinations, including Amazon SNS, AWS Lambda, Amazon API Gateway, Amazon SQS, and many more.

A typical use case would be to match a Kubernetes resource event, forward the event to EventBridge, match the event and invoke an SNS topic, then deliver the notification to the topic's email subscriber(s). Alternatively, EventBridge could deliver the event to CloudWatch Logs, then a pattern matcher could notify someone after N matched events delivered within a bounded timeframe.

Hi there,

thank you for providing and maintaining that great software.

As I am using kubewatch with cloudevents, I investigated some unknown Message encoding errors in combination with the reference Golang SDK and detected the “by hand” implementation in this project.

In order to be and stay more compatible with that standard, an implementation using this SDK would be great.

Kind regards
Florian

Hi,

Accessing ingresses through extensions/v1beta1 is no longer possible since 1.22. Deprecation notice

This means that the following code has to start using networking.k8s.io/v1

However, by doing so we are breaking ingress support for people running 1.18 and lower.

Since people should have moved on from 1.18 a long time ago, I think it's fair enough to proceed.

Let me know and I can open a pull request pretty quickly

Hi ,
The latest kubewatch helm package is not indexed in bitnami.Is there any new link for this helm install kubewatch bitnami/kubewatch --set='rbac.create=true,slack.channel=#YOUR_CHANNEL,slack.token=xoxb-YOUR_TOKEN,resourcesToWatch.pod=true,resourcesToWatch.daemonset=true'

Notice the `` in the example update message below.

kubewatch
A pod in namespace `` has been updated:
vmware-system-tmc/agentupdater-workload-1674792840-kk7b4

However, it is a bit redundant as the resource name includes the namespace.

Notice the example create and delete messages below include the namespace in the message.

kubewatch
A job in namespace vmware-system-tmc has been created:
agentupdater-workload-1674792900

kubewatch
A job in namespace vmware-system-tmc has been deleted:
vmware-system-tmc/agentupdater-workload-1674792780

Hi,

First, thanks for this great project. Showing what's happeing on a k8s cluster is very useful.
I started testing kubewatch, and I have two example about sent messages which is not so informative.

A Node in namespace `` has been Updated:
It would be great if we have more info about that.

A Event in namespace graylog has been Created:
graylog-mongodb-0.173e2cfeb153ed44

This is better, so I can see the event id.

kubectl get event graylog-mongodb-0.173e2cfeb153ed44 -n graylog
LAST SEEN TYPE REASON OBJECT MESSAGE
5m15s Warning Unhealthy pod/graylog-mongodb-0 Readiness probe failed: command "/opt/scripts/readinessprobe" timed out

I think it would be even better if kubewatch send the event contents. Or it is configurable and I'm missing something?

Hello 👋

Firstly, thank you for taking up this project and helping maintain it. It's very much appreciated.

Secondly, I wanted to ask a few questions regarding kubewatch:

1. Is there a version support policy that kubewatch will be following moving forward? (For eg: Is the latest major.minor supported? Or is the latest major.minor.patch supported? Or is there an n and n-1 policy? etc).
2. Are there any compatibility concerns with any currently supported versions of kubernetes?
3. If someone is still using the (now deprecated) bitnami image for kubewatch, what will be your recommend method of upgrading to this one? Will it be to uninstall and then reinstall using the provided method of installation in README?

Your guidance and help will be very much appreciated!

Thank you 🙂