Hi Team,

Is it possible to get notified with the events happening in the pods? I see events are not notified. PFA for the screenshot having events which are not notified.

User may want to watch their own crd objects, like advanced-statefulset in openkruise. Current object types is too few.\

Actually, if user config kube-watch controller rbac correcly, we can use k8s.io/client-go/dynamic/dynamicinformer to watch any unstructured object if we know it's gvk info (resources can be found by discovery client)\

By this implementation, pkg/controller/controller.go::Start() function will be more short and clear. We can remove so many if conf.Resource.XXX branch.

Is there any reason why Teams is not listed as supported notifications target?

The Teams support landed long time ago

• 4cc8a76
• d121cde

and some docs have been updated about it too

• 63f81f3

Interestingly, not the README.

Is the Teams support status unofficial, unreleased, etc.?

It would be nice to have something like a clusterIdentity parameter that I could specify in the configuration and have that clusterIdentity included in the message sent to Slack or elsewhere.

For example, if I specified clusterIdentity: "my-cluster" in the configuration, the message in Slack would look something like this:

kubewatch
A deployment in namespace `kubewatch` in the `my-cluster` cluster has been created:
kubewatch/kubewatch

This way I can use the same Slack channel for multiple clusters and still distinguish between them.

rbac:
create: true
resourcesToWatch:
deployment: true
replicationcontroller: true
replicaset: true
daemonset: true
services: true
pod: true
job: true
node: true
clusterrole: true
clusterrolebinding: true
serviceaccount: true
persistentvolume: true
namespace: true
secret: true
configmap: true
ingress: true
coreevent: true
event: true
slack:
channel: '#general'
token: 'xoxb-################################'
webhook:
enabled: true
url: "https://webhook.site/da1ca748-042d-408b-8805-b273821e1f0b"

How possible would it be to add custom headers to webook and CloudEvent handlers?

A couple of the services we use require Authentication headers to process webhooks

something like this

Hi ,
Is multiple namespace supported ?
I can see option to add either on or leave empty to watch all namespaces.

I have deployed kubewatch using helm. And i want resources which are namespace based to be watched from few namespace only.

I'm getting 10-20 of those info logs every second in my cluster (I'm viewing all logs with stern). These info logs are quite spammy. I would like to suppress those logs in the robusta helm chart, e.g. by env variables.

monitoring robusta-forwarder-5fcb56d9d-4p2nn kubewatch time="2023-02-05T13:14:42Z" level=info msg="Processing update to ConfigMap: kube-system/cluster-autoscaler-status" pkg=kubewatch-ConfigMap
monitoring robusta-forwarder-5fcb56d9d-4p2nn kubewatch 2023/02/05 13:14:42 Message successfully sent to [http://robusta-runner:80/api/handle](http://robusta-runner/api/handle) at 2023-02-05 13:14:42.869867159 +0000 UTC m=+11939.538309659

When starting up the container I get the error below.

E0111 13:41:59.199162       1 reflector.go:123] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:96: Failed to list *v1.Event: Unauthorized

Any Idea how to resove this ?

Currently it would sending to other location.
Just wonder if it is possible to save it locally.

For organizations running multiple clusters, right now if they were to implement kubewatch, they would likely need to open unique channels for each cluster and configure each individually so that they could discern the where the message is coming from.

For example, if you have dev, test and prod clusters, each with the same namespaces, and you get an alert saying that a secret was launched in the default namespace, how could you tell which cluster had the new secret launched.

By allowing custom tags, the user could add a field like:

customTags:
  cluster: prod

which would result in a message being sent that reads as:

kubewatch
A deployment in namespace `kubewatch` in the `my-cluster` cluster has been created:
kubewatch/kubewatch

tags:
- cluster: prod

This may be considered a duplicate of #22, but this could also be used to write more custom information to the slack message.

I'd be happy to add this functionality, or alternatively the functionality outlined in the above related issue, I am making this issue to open a discussion of the best way to move forward.

AWS customers would benefit from adding Amazon EventBridge as a supported destination for notifications.

Amazon EventBridge is a scalable, highly-performant, and low-cost serverless event bus. It's the default event bus provider for many AWS services, including EC2, Auto Scaling, ECS, and many more. Events can be matched using a JSON-based pattern matching, and can be delivered to many supported destinations, including Amazon SNS, AWS Lambda, Amazon API Gateway, Amazon SQS, and many more.

A typical use case would be to match a Kubernetes resource event, forward the event to EventBridge, match the event and invoke an SNS topic, then deliver the notification to the topic's email subscriber(s). Alternatively, EventBridge could deliver the event to CloudWatch Logs, then a pattern matcher could notify someone after N matched events delivered within a bounded timeframe.

Good afternoon .
Okay, I'm using the project, congratulations for the same.
I just configured it and I'm testing something I saw and it generates 3 identical messages for a change. Do you know how I can fix it?

A Pod in namespace test has been Created:
test
14h38
A Pod in namespace test has been Updated:
test
14h38
A Pod in namespace test has been Updated:
test
14h38
A Pod in namespace est has been Updated:
test

Hello 👋

Firstly, thank you for taking up this project and helping maintain it. It's very much appreciated.

Secondly, I wanted to ask a few questions regarding kubewatch:

1. Is there a version support policy that kubewatch will be following moving forward? (For eg: Is the latest major.minor supported? Or is the latest major.minor.patch supported? Or is there an n and n-1 policy? etc).
2. Are there any compatibility concerns with any currently supported versions of kubernetes?
3. If someone is still using the (now deprecated) bitnami image for kubewatch, what will be your recommend method of upgrading to this one? Will it be to uninstall and then reinstall using the provided method of installation in README?

Your guidance and help will be very much appreciated!

Thank you 🙂

Hi Team,

We have few custom resources running in our cluster, is it possible to generate notifications for them as well

need to filter out k8s "Normal" events so they are NOT forwarded to my Slack channel. looking for a correct syntax in configmap.yaml for section "resourcesToWatch:" or "resource:".

my configmap example:
resource:
clusterrole: false
clusterrolebinding: false
configmap: false
daemonset: false
deployment: false
event: true

my helmrelease example:
resourcesToWatch:
events: true

The Helm chart doesn't allow to declare a SlackWebHook configuration (configMap only include static declarations).
It could be good to let it flexible with an extraHandlers.

Hi ,
The latest kubewatch helm package is not indexed in bitnami.Is there any new link for this helm install kubewatch bitnami/kubewatch --set='rbac.create=true,slack.channel=#YOUR_CHANNEL,slack.token=xoxb-YOUR_TOKEN,resourcesToWatch.pod=true,resourcesToWatch.daemonset=true'

Hello, collegues
Is it possible to get notified what I preferred only messages?
For example, I want to see if pods status Evicted only.

ideally, helm charts that have sensitive information uses secretKeyRef to decode base64 k8s secret and takes the name of the secret, this improves the usage of helm charts in public gitops based repos which use helm charts.

Can this be added to helm charts?

The best example of it is given here which allows both plain information or secret key reference
https://github.com/minio/minio/blob/master/helm/minio/values.yaml#L344-L348

Hi, I see in the config.json example that a "reason" configuration was planned, but it looks like it's not yet supported.
I saw in the config.go that the reason section in the Config is commented.

I understand that the goal of this section is to be able to configure which type of events to watch, or restrict which ones we want to capture. Am I right?

Is this planned to the near future?

I deployed kubewatch via the Helm chart. I have it set to watch most resources and all namespaces. I see a bunch of Processing add to Event and Processing update to Event entries in the kubewatch pod log. However, I'm not receiving any messages in the Slack channel I specified in the configuration. And there are no Slack related messages in the pod log. How can I get more information about what's going on and/or troubleshoot this problem?

Notice the `` in the example update message below.

kubewatch
A pod in namespace `` has been updated:
vmware-system-tmc/agentupdater-workload-1674792840-kk7b4

However, it is a bit redundant as the resource name includes the namespace.

Notice the example create and delete messages below include the namespace in the message.

kubewatch
A job in namespace vmware-system-tmc has been created:
agentupdater-workload-1674792900

kubewatch
A job in namespace vmware-system-tmc has been deleted:
vmware-system-tmc/agentupdater-workload-1674792780

Thanks for maintaining this. I'm trying to set this up in my cluster so I can receive a Slack notification whenever a warning or error level event occurs in the cluster. For example, an image pull failure or a container exists with non-zero exit code.

I do not need reporting for every event that happens. Is there a way to do this?

Thank you.

Hi there,

thank you for providing and maintaining that great software.

As I am using kubewatch with cloudevents, I investigated some unknown Message encoding errors in combination with the reference Golang SDK and detected the “by hand” implementation in this project.

In order to be and stay more compatible with that standard, an implementation using this SDK would be great.

Kind regards
Florian

Hi,

Accessing ingresses through extensions/v1beta1 is no longer possible since 1.22. Deprecation notice

This means that the following code has to start using networking.k8s.io/v1

However, by doing so we are breaking ingress support for people running 1.18 and lower.

Since people should have moved on from 1.18 a long time ago, I think it's fair enough to proceed.

Let me know and I can open a pull request pretty quickly

Hi,

First, thanks for this great project. Showing what's happeing on a k8s cluster is very useful.
I started testing kubewatch, and I have two example about sent messages which is not so informative.

A Node in namespace `` has been Updated:
It would be great if we have more info about that.

A Event in namespace graylog has been Created:
graylog-mongodb-0.173e2cfeb153ed44

This is better, so I can see the event id.

kubectl get event graylog-mongodb-0.173e2cfeb153ed44 -n graylog
LAST SEEN TYPE REASON OBJECT MESSAGE
5m15s Warning Unhealthy pod/graylog-mongodb-0 Readiness probe failed: command "/opt/scripts/readinessprobe" timed out

I think it would be even better if kubewatch send the event contents. Or it is configurable and I'm missing something?

I have a namespace that is very noisy, i.e. jobs and pods created, updated, and deleted every minute. This namespace is on every cluster is the same and I'd like to have one configuration that works on all clusters. I don't want to have to include every namespace but this one from every cluster in a namespaceToWatch list.

Hi Team,
Adding a feature to Kubewatch that enables sending alerts only for specific pod states (e.g., Killed, Pending) would greatly assist in efficiently managing alerts, preventing unnecessary stacking

Hello, I'm trying to use kubewatch to send everything to a google-chat webhook.
I configured everything, and the test seems fine:

root@kubewatch-64577c4cc7-v5thc:/# kubewatch config test
INFO[0000] Custom log formatter: json                   
INFO[0000] Setting custom log formatter to: json        
Testing Handler configs from .kubewatch.yaml
{"level":"info","msg":"No webhook cert is given","time":"2024-01-26T11:42:54Z"}
{"level":"info","msg":"Message successfully sent to https://chat.googleapis.com/v1/spaces/CENSORED at 2024-01-26 11:42:54.488587515 +0000 UTC m=+0.251134753 ","time":"2024-01-26T11:42:54Z"}

But nothing is actually received in the group chat mentioned.
As a reference, something simple as this in python actually sends the message:

from httplib2 import Http
from json import dumps
import sys

def main():
    url = 'https://chat.googleapis.com/v1/spaces/CENSORED'
    bot_message = {
        'text' : sys.argv[1]}

    message_headers = { 'Content-Type': 'application/json; charset=UTF-8'}

    http_obj = Http()

    response = http_obj.request(
        uri=url,
        method='POST',
        headers=message_headers,
        body=dumps(bot_message),
    )

    print(response)

if __name__ == '__main__':
    main()

What could be the issue? Am I missing something?
Thanks.

Go runtime metrics could be useful to view on kubewatch

https://github.com/prometheus/client_golang/blob/main/prometheus/go_collector.go

It would be good to expose these from kubewatch

Tried setting up using kubectl but alerts are not generating in kubectl, whereas in helm its working, If possible could you please resolve this or tell me how to proceed.

Hi guys.
May you add https://element.io/ webhook support?

Would you accept a PR to allow a user to add the cluster name to be added to each of the events.

This is handy when you have many clusters and don't want multiple channels. If this is something you would accept I can work in the PR.

Thanks

Hello,
latest helm chart kubewatch-3.3.10 with kubewatch:v2.4, please add new helm chart with kubewatch:v2.5 and new watch events

Hi, the option to ignore "Event" resources doesn't seem to work.

I tried to specify in the configuration similarly to other modules:

resourcesToWatch:
  event: false

or

resourcesToWatch:
  events: false

This does not help, I see in the logs that the corresponding module starts despite the configuration:

time="2023-01-27T10:09:55Z" level=info msg="Starting kubewatch controller" pkg=kubewatch-Event

On our large cluster, this gives a lot of noise in notifications to Slack. Thank you in advance!