Update verify-conformance-tests to have better xml parsing about k8s-conformance HOT 6 CLOSED

"what sonobuoy generates" for the e2e junit is actually "what the Kubernetes e2e binary generates".

It seems wrong to submit results that are not from the Kubernetes e2e binary.

from k8s-conformance.

/cc @smarterclayton what did you use to generate your results? Is there a use case here?

from k8s-conformance.

@smarterclayton would you like to keep this open?

from k8s-conformance.

/close
Seem to only be an edge case scenario

from k8s-conformance.

classify as flake

from k8s-conformance.

It seems wrong to submit results that are not from the Kubernetes e2e binary.

This is summary of results generated by multiple executions of the kube e2e binary. To do that, you have to merge the files somehow from those executions. The tool I used was a simple merge tool, but you can absolutely run the kube e2e suite in multiple chunks and then have to merge the data afterwards. I don't think conformance should be based only on "the kind of junit output created by a tool wrapped by another tool" - but instead junit (which has a spec, and makes no assumptions that the top level is a single suite). That being said, it's fairly easy for me to work around this for now. I just wanted to register that (in my official Kube conformance hat) I'm extremely leery of assumptions that overly bind conformant distributions from automating their pipelines and this was on the border for me.

from k8s-conformance.