Unexpected HELM validation manifests behaviour about helm HOT 7 OPEN

I've also encountered this issue and managed to track down the version where this behaviour change was introduced: 3.9.0

Using helm create chart-test and modifying the service as @fradee described yields different results based on the version (tested against Kubernetes v1.28.8+k3s1):

With version 3.8.1 chart install gets rejected due to the unknown field:

$ helm version
version.BuildInfo{Version:"v3.8.1", GitCommit:"5cb9af4b1b271d11d7a97a71df3ac337dd94ad37", GitTreeState:"", GoVersion:"go1.17.9"}
$ helm install --generate-name ./chart-test/
Error: INSTALLATION FAILED: unable to build kubernetes objects from release manifest: error validating "": error validating data: ValidationError(Service.spec): unknown field "someFake" in io.k8s.api.core.v1.ServiceSpec

Updating to helm 3.9.0. Now the install succeeds and only produces a warning:

$ helm version
version.BuildInfo{Version:"v3.9.0", GitCommit:"7ceeda6c585217a19a1131663d8cd1f7d641b2a7", GitTreeState:"", GoVersion:"go1.17.11"}
$ helm install --generate-name ./chart-test/
W0702 08:03:40.093323  503060 warnings.go:70] unknown field "spec.someFake"
NAME: chart-test-1719900219
LAST DEPLOYED: Tue Jul  2 08:03:40 2024
NAMESPACE: default
STATUS: deployed
REVISION: 1

from helm.

We are also facing the same issue. @mattfarina has this behaviour changed in newer helm versions ?

from helm.

Hello everyone. I was looking for a way to contribute to Helm and this ticket looked interesting so I started diving into it.

I can confirm what @gjenkins8 stated about this being related to the switch in Kubectl to support server-side validation. I used git bisect to narrow down the commit where this was introduced (bd7c97f), which bumped the kubectl version and updated client.go to utilize the new Kubectl Factory interface.

From here, I dove into what changed in Kubectl that broke client-side validation in Helm. I traced it down to the new Schema Validator function used. Specifically, https://github.com/kubernetes/kubectl/blob/b315eb8455a7d5c11ed788d1592b4afeca85771d/pkg/validation/schema.go#L140 appears to only run client-side validation if it detects that server-side validation is not supported for the resource. This can be verified by going back to a Kubernetes version before server-side validation was introduced (<1.25), where Helm commands error as expected when manifests contain invalid fields.

I tried to figure out a way to resolve this on the Helm side, but I came up short. Admittedly, my Go skills need some work. I was not able to come up with a clean way of resolving this that didn't involve updates on the Kubectl side. Perhaps this is worth a ticket with kubectl to add better support for explicitly running client-side validation?

from helm.

thanks @james-mchugh for the investigation and find

from helm.

I can confirm, and I'm not sure why the client-side validation implementation has stopped working:
https://github.com/helm/helm/blob/main/pkg/kube/client.go#L344-L352

What I think happened, is Kubernetes introduced service side validation. And somehow (speculating, I didn't track down how) the openapi validation done client side is no longer reporting errors. Server-side validation blog:
https://kubernetes.io/blog/2023/04/24/openapi-v3-field-validation-ga/

Helm also should to update to use server side validation, but likely compatibity concerns need to be considered. Updating is simplish implementation-wise., just needs to be determined how to pass the validation flag through to the Kubernetes API object helper. Like e.g. kubectl:

WithFieldValidation(o.ValidationDirective)

https://github.com/kubernetes/kubectl/blob/master/pkg/cmd/apply/apply.go#L568

Debugging why the client-side validation is not working should also be done (at least for the standard Service object. Karpenter nodeclass being a custom resource, it might not be possible).

This was also noticed back in this issue: #12470 . But I think work from here, as that issue started from a different question. And the PR there has languished.

from helm.

The PR to introduce server-side validation looks something like: #13123

I don't think this can easily be implemented in Helm v3, due to the way Helm manages this Kubernetes client (schema validation is "per-operation". This client is constructed early and shared across operations (in the SDK)

Onwards to figuring out why the client side validation has failed I guess.

from helm.

I created a branch for kubectl that has a potential solution for this. It adds a preferredStrategy argument to the Validator function and the newParamVerifyingSchema struct. It's still a WIP, but I wanted to see if it would actually resolve the issue and it does.

I opened an issue with kubectl in case this is something that should be resolved on their end.

from helm.