running upstream project tests via kola? about coreos-assembler HOT 6 CLOSED

One approach would be for us to reuse kola's provisioning/teardown bits, and have it write out the ssh connection information (as a ssh-config file e.g.) and then something higher level can reuse that to execute other test suites.

kola spawn could possibly be adapted for this. I like the idea of writing out an ssh config file.

from coreos-assembler.

Oh I forgot to add...I had short discussions with @jlebon about finding a generic way for running these kinds of upstream tests via kola. I'm not convinced about the usefulness of such an effort, since there don't seem to be a lot of projects that we work with that execute their tests the same way (i.e. Fedora STI).

from coreos-assembler.

Do we want/need these to be added to kola? I'm not familiar with vmcheck, if it's meant more to test rpm-ostree it might make sense to just add a subset of native tests that target core functionality to make sure that what we have packaged in the image is functional rather than testing all of rpm-ostree.

Here's some miscellaneous thoughts I've had on the subject of external tests in kola that don't necessarily directly target any of your points:

Ideally I'd like to avoid adding more vendor bloat to mantle for additional tests unless directly necessary.

Another thing for running non-native kola tests is that we don't have a great way of discovering them and providing accurate information about the test (just stdout, stderr, and exit status). In the case of vmcheck where there's a series of tests it would need it's own test runner that kola just runs.

My personal opinion (if we want to add these into kola) would learn more towards either building subsets of the tests natively (and dealing with the extra maintenance burden of duplicating existing relevant tests in golang) or some sort of hybrid where we accept that the tests will require a git clone or some form of download (could just be pulling a container) to get the relevant test bits and the kola test essentially handles creating the machine, downloading the relevant source bits it needs, and executing it in a simple way (ideally just executing a binary/script that will exit 0 or 1). In the latter case it might be worthwhile to add some sort of flag to enable/disable said tests.

Definitely open to hearing other alternatives.

cc @bgilbert for other opinions

from coreos-assembler.

We haven't generally had kola tests which wrap other test suites. We probably shouldn't build many tests of that kind, but it might make sense to do so in some cases. Pulling and running a container seems like the obvious way to implement them. I agree that vendoring tests directly into mantle doesn't seem great.

from coreos-assembler.

if it's meant more to test rpm-ostree it might make sense to just add a subset of native tests that target core functionality to make sure that what we have packaged in the image is functional rather than testing all of rpm-ostree.

Yeah, regardless we should probably at least have a basic invocation of e.g. rpm-ostree --status.

One high level thing to keep in mind here is that since we now have two streams, that makes everything at least...four times more complicated =) Since we have to think about versioning a lot more.

Additional complexity here is that rpm-ostree is used by other projects in Fedora too.

So.....

One approach would be for us to reuse kola's provisioning/teardown bits, and have it write out the ssh connection information (as a ssh-config file e.g.) and then something higher level can reuse that to execute other test suites.

from coreos-assembler.

I think this is fixed by #1252

from coreos-assembler.