Steps to reproduce:

1. Start Nsm
2. Create alpine.yaml like this one

kind: Pod
metadata:
  name: alpine-1
  namespace: default
  annotations:
    networkservicemesh.io: "kernel://my-vl3-network/nsm-1"
spec:
  containers:
  - image: python:3-alpine
    command:
      - /bin/sh
      - "-c"
      - "sleep 60m"
    imagePullPolicy: IfNotPresent
    name: alpine
  restartPolicy: Always

3. Apply it.

You should get an error:
Error from server (InternalError): error when creating "alpine.yaml": Internal error occurred: failed calling webhook "[admission-webhook-k8s-898d775b5-h588g.networkservicemesh.io](http://admission-webhook-k8s-898d775b5-h588g.networkservicemesh.io/)": failed to call webhook: an error on the server ("unknown") has prevented the request from succeeding

Hi,
currently, I can configure images of the injected initContainers and the containers.
but what if I also want to configure the entry point itself?
(I have a use case where I want to load env variables from a file that is being created by an initContainer that determines the NSURL)

currently, I need to create additional dockerfile and images with nsc and nsc-init as their base image and override their entrypoints.
this process is a litte excessive, as I could've just add the command property.

another approach is to not use this mutating webhook, and define the containers & initContainers for the nsc myself and specify the command property then.

Steps to reproduce

1. Deploy alpine with NSM annotation
2. Deploy webhook

Actual: Alpie did not get connectivity.
Expected: Alpine gets NSM patch right after webhook has started.

Currently the app doesn't specify any limits to matching resources when it registers the webhook, which results in a webhook with scope set to *, which means it matches any resources, regardless of labels, annotations, etc.

If the app is unavailable for any reason and k8s can't successfully call it when creating or updating a resource, the create/update action will fail, even if the resource doesn't have anything to do with NSM.

In our environment we require user namespaces to specify requests and limits for containers.
We enforce this with policy in the cluster.

When the webhook injects the NSM init container into a PodSpec no resources and limits are specified.
See: https://github.com/networkservicemesh/cmd-admission-webhook-k8s/blob/main/main.go#L254

We would like to be able to configure the requests and limits on the init container so we can set them in a way that complies with our policy requirements.

Description

Add the ability to use a spire as a source of certs instead of an automatically generated to improve security and safety of NSM webhook.

Description

Logs:

{"level":"error","ts":1705506855.7787008,"logger":"admissionWebhookServer","caller":"build/main.go:97","msg":"failed to get namespace by name: client rate limiter Wait returned an error: rate: Wait(n=1) would exceed context deadline","stacktrace":"main.(*admissionWebhookServer).Review\n\t/build/main.go:97\nmain.main.func3\n\t/build/main.go:482\ngithub.com/labstack/echo/v4.(*Echo).add.func1\n\t/go/pkg/mod/github.com/labstack/echo/[email protected]/echo.go:582\ngithub.com/labstack/echo/v4/middleware.RecoverWithConfig.func1.1\n\t/go/pkg/mod/github.com/labstack/echo/[email protected]/middleware/recover.go:131\ngithub.com/labstack/echo/v4/middleware.LoggerWithConfig.func2.1\n\t/go/pkg/mod/github.com/labstack/echo/[email protected]/middleware/logger.go:126\ngithub.com/labstack/echo/v4.(*Echo).ServeHTTP\n\t/go/pkg/mod/github.com/labstack/echo/[email protected]/echo.go:669\nnet/http.serverHandler.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2936\nnet/http.(*conn).serve\n\t/usr/local/go/src/net/http/server.go:1995"}

Build: https://github.com/networkservicemesh/integration-k8s-kind/actions/runs/7558182942/job/20579105121

This issue appears because k8s does not support annotation filtering in webhooks.

Now we have a configuration file for mutating webhook: https://github.com/networkservicemesh/deployments-k8s/pull/10139/files#diff-44daa96e3cdba7c4414e87f6831e06bd63a9984e606efc8a5cd7e4d7829d6850
These mutations will be applied to all resources by default. The idea of ​​the new functionality is to add labels to all resources that should be affected by the webhook mutation, and configure filtering by labels in the webhook configuration file. This problem was described here: #282 but it looks like the "scope" field is not applicable for this purpose. Most likely using objectSelector and namespaceSelector with matchLabel may lead to necessary resource filtering, something similar is done in the Kuma webhook configuration: https://github.com/kumahq/kuma/blob/6f86847aee5d50fd912aeaaa8f5470427f127676/app/kumactl/cmd/install/testdata/install-control-plane.with-helm-values.yaml#L610
https://github.com/kumahq/kuma/blob/6f86847aee5d50fd912aeaaa8f5470427f127676/app/kumactl/cmd/install/testdata/install-control-plane.with-helm-values.yaml#L588

We already have labelling support for namespaces so let's add labelling support for all other resources

Label example: networkservicemesh.io: "nsm-1://kernel/value"

Description

We have a many NSM envs that allow us to configure the NSC in the best possible way:
https://github.com/networkservicemesh/cmd-nsc/blob/main/internal/config/config.go#L31-L43

	Name             string        `default:"nsc" desc:"Name of Network Service Client"`
	ConnectTo        url.URL       `default:"unix:///var/lib/networkservicemesh/nsm.io.sock" desc:"url to connect to NSM" split_words:"true"`
	DialTimeout      time.Duration `default:"5s" desc:"timeout to dial NSMgr" split_words:"true"`
	RequestTimeout   time.Duration `default:"15s" desc:"timeout to request NSE" split_words:"true"`
	MaxTokenLifetime time.Duration `default:"10m" desc:"maximum lifetime of tokens" split_words:"true"`

	Labels    []string `default:"" desc:"A list of client labels with format key1=val1,key2=val2, will be used a primary list for network services" split_words:"true"`
	Mechanism string   `default:"kernel" desc:"Default Mechanism to use, supported values: kernel, vfio" split_words:"true"`

	NetworkServices       []url.URL               `default:"" desc:"A list of Network Service Requests" split_words:"true"`
	AwarenessGroups       awarenessgroups.Decoder `defailt:"" desc:"Awareness groups for mutually aware NSEs" split_words:"true"`
	LogLevel              string                  `default:"INFO" desc:"Log level" split_words:"true"`
	OpenTelemetryEndpoint string                  `default:"otel-collector.observability.svc.cluster.local:4317" desc:"OpenTelemetry Collector Endpoint"`

And we definitely can do it when we use a bare NSC, without any actual client application.

But this is not the main use case for NSM - users can add networkservicemesh annotations:

...
annotations:
    networkservicemesh.io: "kernel://my-networkservice-1/nsm-1"
...

NetworkServices or Mechanism - we can take it from the annotation. But what about others? What if the user wants to add AwarenessGroups and so on?

Possible solutions:

1. Use NSM_ prefixed envs that are passed by user and inject them into NSM-containers
2. Expand the current networkservicemesh.io annotation and add new parameters

Description

Auto registration creates a MutatingWebhookConfiguration with no Namespace filters.
The mutation applies to all Namespaces.

The Webhook watching the system namespaces can create circular dependencies. Where system pods are required for Nodes to be made Ready to have the Webhook scheduled onto them.

We would like to configure this selector to exclude some Namespaces which should never be modified.
For example we could configure the webhook to exclude the namespaces: nsm-system kube-system and spire.

Example of current webhook config:

kind: MutatingWebhookConfiguration
metadata:
  name: nsm-admission-webhook-k8s-5dfd78487d-26n5b
webhooks:
- admissionReviewVersions:
  - v1
  clientConfig:
    caBundle: <redacted>
    service:
      name: admission-webhook-svc
      namespace: nsm-system
      path: /mutate
      port: 443
  failurePolicy: Fail
  matchPolicy: Equivalent
  name: nsm-admission-webhook-k8s-5dfd78487d-26n5b.networkservicemesh.io
  namespaceSelector: {}
  objectSelector: {}
  reinvocationPolicy: Never
  rules:
  - apiGroups:
    - ""
    apiVersions:
    - v1
    operations:
    - CREATE
    - UPDATE
    resources:
    - pods
    scope: '*'
  - apiGroups:
    - apps
    apiVersions:
    - v1
    operations:
    - CREATE
    - UPDATE
    resources:
    - deployments
    - statefulsets
    - daemonsets
    - replicasets
    scope: '*'
  sideEffects: None
  timeoutSeconds: 10

Example of desired webhook config:

kind: MutatingWebhookConfiguration
metadata:
  name: nsm-admission-webhook-k8s-5dfd78487d-26n5b
webhooks:
- admissionReviewVersions:
  - v1
  clientConfig:
    caBundle: <redacted>
    service:
      name: admission-webhook-svc
      namespace: nsm-system
      path: /mutate
      port: 443
  failurePolicy: Fail
  matchPolicy: Equivalent
  name: nsm-admission-webhook-k8s-5dfd78487d-26n5b.networkservicemesh.io
  namespaceSelector:
    matchExpressions:
    - key: kubernetes.io/metadata.name
      operator: NotIn
      values:
      - nsm-system
      - kube-system
      - spire
  objectSelector: {}
  reinvocationPolicy: Never
  rules:
  - apiGroups:
    - ""
    apiVersions:
    - v1
    operations:
    - CREATE
    - UPDATE
    resources:
    - pods
    scope: '*'
  - apiGroups:
    - apps
    apiVersions:
    - v1
    operations:
    - CREATE
    - UPDATE
    resources:
    - deployments
    - statefulsets
    - daemonsets
    - replicasets
    scope: '*'
  sideEffects: None
  timeoutSeconds: 10

The volume mounted in the NSC is hardcoded in the webhook, so if the CONNECT_TO environment variable is set to another directory than /var/lib/networkservicemesh, the NSC containers injected will not be able to communicate with the NSM API.

Steps to reproduce

1. Add the CONNECT_TO variable to the ENVS in the environment variables of the webhook

- name: NSM_ENVS
  value: NSM_CONNECT_TO=unix:///new-path/nsm.io.sock

2. Set the CONNECT_TO (or LISTEN_ON for nsmgr) environment variable for the other NSM components
3. Deploy NSM + webhook
4. Deploy an NSM application (NSC + NSE)

new-path is not mounted, so the NSC will not start.

There are already presented packages for webhook functionality, for example:
https://pkg.go.dev/sigs.k8s.io/[email protected]/pkg/webhook
https://pkg.go.dev/sigs.k8s.io/[email protected]/pkg/webhook/admission

I suppose it is worth checking whether we can use them in our code.

Context

#21

Motivation

Hardcoded strings can be replaced to const from sdk-sriov
https://github.com/networkservicemesh/cmd-admission-webhook-k8s/blob/main/main.go#L186-L187

Implementation details

1. Export constant from sdk-sriov by separate PR.
2. Replace hardcoded strings to the constant.

add support for sriovToken label in webhook. in case of sriov use case, service request happens with sriovToken label, in this case webhook has to inject resource specified in the label so that init container can create service request with right token id for VF attachment.

For example, pod annotation is added like this

annotations:
    networkservicemesh.io: kernel://icmp-responder/nsm-1?sriovToken=worker.domain/10G

Steps to reproduce:

1. Apply basic NSM setup: https://github.com/networkservicemesh/deployments-k8s/tree/main/examples/basic
2. Make sure webhook started and self-registered: check k1 get mutatingwebhookconfigurations.admissionregistration.k8s.io
3. Delete NSM from cluster
4. Check k1 get mutatingwebhookconfigurations.admissionregistration.k8s.io again

Expected result:

List doesn't contain admission-webhook-* instance

Actual result:

Mutating webhook list still contains NSM webhook.

Additional info:

It seems like webhook tries to unregister the entry but it doesn't happen for some reason:

Also, in the deployments repo in the cleanup section we delete the webhook manually, so it must be a known behavior? But surely this must be just a half-documented bug.
https://github.com/networkservicemesh/deployments-k8s/tree/d43747cbd24515b152ef11aec6bc4f61ea2bdc88/examples/basic

ReplicaSet kind doesn't supported by webhook: https://github.com/networkservicemesh/cmd-admission-webhook-k8s/blob/main/main.go#L106

nsm webhook could appned nsm nsc sidecar on kubectl annotate

Description

Since NSM is able to provide vl3 networks, it could be useful to add support for labled namespaces. All pods within the labled namespace could be handled by the webhook that injects nsm client based on the NSM label in each POD.

Write an admissions webhook similar to https://github.com/networkservicemesh/networkservicemesh/tree/master/k8s/cmd/admission-webhook

Please do not bring across the "github.com/networkservicemesh/networkservicemesh/*" dependencies. Many of them do not need to be replicated (like probes). Keep things simple :)

@Bolodya1997 do we need to update the NSM URL schema compared to https://github.com/networkservicemesh/networkservicemesh/blob/756a5e48592b29174765657cf3a4cb79c3491240/pkg/tools/tools.go#L195 ? Do we have a lib in sdk for that currently? I recall you did some smart things for sriov in terms of updating the URL.. but don't remember them precisely. Pointers welcome :)

Description

Currently, we cannot inject labels via webhook.
So, spiffe.io/spiffe-id label can be added only manually to yaml files.
See the context for more info.

Context

networkservicemesh/deployments-k8s#3211 (comment)