Add feature kernelci-try about kernelci-core HOT 34 OPEN

the current PoC of the kci-dev tool
https://github.com/aliceinwire/kci-dev

from kernelci-core.

Following this issue
I propose to create a tool called kci-dev

$ kci-dev --help
kci-dev -- tool for developer for test local tree with local changes against a enabled KernelCI server

where:
	--connect=[ssh,api] define the way to connect to KernelCI server
	--repository        define the kernel upstream repository where to test local changes
	--branch            define the repository branch
	--publish           define if the test results will be published
	--dir               define the directory of the local tree with local changes

This file will also accept a .kci-dev.yaml configuration file for define password and api keys
The changes can also be sent from a own repository fork, in such case the upstream repository will be the fork owned.

from kernelci-core.

Hi! Thats exactly what we are working on this milestone.
We implemented already resubmission for different kernel branch commit, i will implement also lab test retry logic and then we will move on discussing how to implement this tool. I will update issue with follow up on design, before implementing it.

from kernelci-core.

@nuclearcat which milestone are you talking about, there is any way I can help out?

from kernelci-core.

@aliceinwire The milestone @nuclearcat mentioned is a public roadmap/development plan, e.g. M3 document

Kernel developers provided of a API key can push request for builds to the KernelCI current running environment from the command line.

This should be already possible for monitored trees/branches by submitting custom Node to the API

Also add the possibility of pushing local kernel source changes.

That's also supported using patchset Node type. Current CLI might not support it yet, though (so there's a room for improvement)

from kernelci-core.

what about the publish feature that permit to decide to publish or don't publish results

	--publish           define if the test results will be published

from kernelci-core.

This should be already possible for monitored trees/branches by submitting custom Node to the API

do you mean by using the KernelCI-api ? I tried doing it with the help of @nuclearcat but looks like it was not a feature that KernelCI-api can provide.

That's also supported using patchset Node type. Current CLI might not support it yet, though (so there's a room for improvement)

?

from kernelci-core.

@nuclearcat which milestone are you talking about, there is any way I can help out?
We're currently planning our work for the coming weeks and prioritizing new tasks, including:

• Issue 403
• Issue 509
• Issue 334
• Issue 664

We may add more issues based on logical and demanded features (including this one). Additional ideas will be gathered during our upcoming weekly meeting.

Once planning is complete, we'll publish a Milestone document outlining the defined tasks. We'll strive to complete these tasks within the milestone, but some may extend beyond due to external dependencies or underestimated complexity.

from kernelci-core.

That's also supported using patchset Node type. Current CLI might not support it yet, though (so there's a room for improvement)

?

API supports Nodes with patches stored as artifacts. Kernel sources that given Node references together with these patches are later packaged into tarball for further processing.

By CLI I meant kci tool which should be extended to cover patchset-related use cases.

from kernelci-core.

Draft of the API calls i am designing (that this tool will use):

Design of tree maintainers API (KISS principle):

This is a simple API that will allow to interact with kernelci pipeline in a more flexible way.
All requests are authenticated with JWT token that is passed in Authorization header.

P1)Custom checkout for bisection

Specify full specification (url, branch, kbuildname, testname) OR use failing test node as reference (it already contains all necessary information)

So request can be as following:
POST /api/checkout

{
    "node": "<failing_test_node_id>",
    "commit": "<commit_id>",
}

Will return custom checkout node id that can be followed by traditional API calls or on dashboard.

Purpose: Kernel developers can do bisection on kernelci infrastructure. As git bisect guides with commit, based on test results, developer can
answer to bisect good/bad.

P2)Custom test run retry

POST /api/retry

{
    "node": "<failing_test_node_id>"
}

Not sure yet, we cannot return node id in this case, but i can return kbuild node id that will have new "child" node with test run results.

Purpose: Sometimes hardware or tests are "flakey" (unreliable), so it is useful to retry test several times.

P3) Test fix(patch)

POST /api/testfix

{
    "node": "<failing_test_node_id>",
    "patch": "<patch_content_base64?>"
}

It will fetch same sources as test failed, but apply patch before build. Then after building patched kernel it will run test.
It will return as in p1 custom checkout node id that can be followed by traditional API calls or on dashboard.

Purpose: Kernel developers can test patches on kernelci infrastructure, if they fix failing test/regression.

P4) Testing new patches

POST /api/patch

{
    "patch": "<patch_content_base64?>",
    "treeurl": "<tree_url>",
    "branch": "<branch>",
    "commit": "<commit_id>",
    "kbuildname": "<kbuildname>",
    "testname": "<testname>"
}

Fields kbuildname and testname are optional, if not specified it will run all tests and all builds, as configured in kernelci pipeline.

It will fetch sources from treeurl, checkout to branch, then to commit, apply patch.
Then it will build kernel and run tests. If kbuildname and testname are specified, it will restrict to run only this build and test.

Purpose: Kernel developers can test new patches on kernelci infrastructure, making sure they are not breaking anything.

from kernelci-core.

I updated draft

from kernelci-core.

@nuclearcat thanks for the draft of the API, as discussed on the weekly web meeting this is a good start.
my further suggestion was:

• Get the results as json file or at least get the build id number for getting the results from other sources (like KCIDB)
• Add a optional email option for send the results by email
• Add a privacy option for don't publish the results to KCIDB or KernelCI dashboard

from kernelci-core.

Following this issue I propose to create a tool called kci-dev

$ kci-dev --help
kci-dev -- tool for developer for test local tree with local changes against a enabled KernelCI server

where:
	--connect=[ssh,api] define the way to connect to KernelCI server
	--repository        define the kernel upstream repository where to test local changes
	--branch            define the repository branch
	--publish           define if the test results will be published
	--dir               define the directory of the local tree with local changes

This file will also accept a .kci-dev.yaml configuration file for define password and api keys The changes can also be sent from a own repository fork, in such case the upstream repository will be the fork owned.

I believe it should be mandatory to the repository enabled in KernelCI already. I think that is safer.

I also wonder why we would not publish the results up to KCIDB. If you are sending to KernelCI, it is public code.

from kernelci-core.

Following this issue I propose to create a tool called kci-dev

$ kci-dev --help
kci-dev -- tool for developer for test local tree with local changes against a enabled KernelCI server

where:
	--connect=[ssh,api] define the way to connect to KernelCI server
	--repository        define the kernel upstream repository where to test local changes
	--branch            define the repository branch
	--publish           define if the test results will be published
	--dir               define the directory of the local tree with local changes

This file will also accept a .kci-dev.yaml configuration file for define password and api keys The changes can also be sent from a own repository fork, in such case the upstream repository will be the fork owned.

I believe it should be mandatory to the repository enabled in KernelCI already. I think that is safer.

currently yes also because we are sending only a diff, not all the code and we need to match the diff with a full repository tree
the --repository option is for know to which repository tree we are matching against the diff patches

I also wonder why we would not publish the results up to KCIDB. If you are sending to KernelCI, it is public code.

As we discussed on the weekly meeting we thought of some possibility:

• In the case of embargoed security related code (but after talking with Greg K-H looks like there is no need for this case)
• for development testing
• for preventing to show fake failure on stable tree

from kernelci-core.

P4) Testing new patches

POST /api/patch

{
    "patch": "<patch_content_base64?>",
    "treeurl": "<tree_url>",
    "branch": "<branch>",
    "commit": "<commit_id>",
    "kbuildname": "<kbuildname>",
    "testname": "<testname>"
}

Fields kbuildname and testname are optional, if not specified it will run all tests and all builds, as configured in kernelci pipeline.

It will fetch sources from treeurl, checkout to branch, then to commit, apply patch. Then it will build kernel and run tests. If kbuildname and testname are specified, it will restrict to run only this build and test.

Purpose: Kernel developers can test new patches on kernelci infrastructure, making sure they are not breaking anything.

would be useful to have also a laboratory field like we had in Jenkins, for chose to which laboratory run the tests, default to everyone.

from kernelci-core.

Draft of the API calls i am designing (that this tool will use):

Design of tree maintainers API (KISS principle):

This is a simple API that will allow to interact with kernelci pipeline in a more flexible way. All requests are authenticated with JWT token that is passed in Authorization header.

P1)Custom checkout for bisection

Specify full specification (url, branch, kbuildname, testname) OR use failing test node as reference (it already contains all necessary information)

So request can be as following: POST /api/checkout

{
    "node": "<failing_test_node_id>",
    "commit": "<commit_id>",
}

Will return custom checkout node id that can be followed by traditional API calls or on dashboard.

Purpose: Kernel developers can do bisection on kernelci infrastructure. As git bisect guides with commit, based on test results, developer can answer to bisect good/bad.

P2)Custom test run retry

POST /api/retry

{
    "node": "<failing_test_node_id>"
}

Not sure yet, we cannot return node id in this case, but i can return kbuild node id that will have new "child" node with test run results.

Purpose: Sometimes hardware or tests are "flakey" (unreliable), so it is useful to retry test several times.

P3) Test fix(patch)

POST /api/testfix

{
    "node": "<failing_test_node_id>",
    "patch": "<patch_content_base64?>"
}

It will fetch same sources as test failed, but apply patch before build. Then after building patched kernel it will run test. It will return as in p1 custom checkout node id that can be followed by traditional API calls or on dashboard.

Purpose: Kernel developers can test patches on kernelci infrastructure, if they fix failing test/regression.

where do you get the failing test node id? is this some public information and easy to retrieve?
also retry the same test I feel like is more something for KCI developers than kernel developers. retry the same tests on the same kernel build with same code is supposed to give same results.

from kernelci-core.

Draft of the API calls i am designing (that this tool will use):

Design of tree maintainers API (KISS principle):

This is a simple API that will allow to interact with kernelci pipeline in a more flexible way. All requests are authenticated with JWT token that is passed in Authorization header.

P1)Custom checkout for bisection

Specify full specification (url, branch, kbuildname, testname) OR use failing test node as reference (it already contains all necessary information)
So request can be as following: POST /api/checkout

{
    "node": "<failing_test_node_id>",
    "commit": "<commit_id>",
}

Will return custom checkout node id that can be followed by traditional API calls or on dashboard.
Purpose: Kernel developers can do bisection on kernelci infrastructure. As git bisect guides with commit, based on test results, developer can answer to bisect good/bad.

Kernel developers are not suppose to ssh into KernelCI for doing bisection on KernelCI infrastructure

from kernelci-core.

where do you get the failing test node id? is this some public information and easy to retrieve? also retry the same test I feel like is more something for KCI developers than kernel developers. retry the same tests on the same kernel build with same code is supposed to give same results.

Yes, it is available in Grafana (which is available now) and in public dashboard. It is something like failed test ID.
Additionally, i just added recently function to specify parameters manually (or retrieve them semi-automatically if you point to git repo) kernelci/kernelci-pipeline#739, such as github repo URL, branch, commit id, but then developer will have to specify in cli options kernel build and test he wants to run.
Today or tomorrow i will add to PoC tool support of that too.

from kernelci-core.

P4) Testing new patches

POST /api/patch

{
    "patch": "<patch_content_base64?>",
    "treeurl": "<tree_url>",
    "branch": "<branch>",
    "commit": "<commit_id>",
    "kbuildname": "<kbuildname>",
    "testname": "<testname>"
}

Fields kbuildname and testname are optional, if not specified it will run all tests and all builds, as configured in kernelci pipeline.
It will fetch sources from treeurl, checkout to branch, then to commit, apply patch. Then it will build kernel and run tests. If kbuildname and testname are specified, it will restrict to run only this build and test.
Purpose: Kernel developers can test new patches on kernelci infrastructure, making sure they are not breaking anything.

would be useful to have also a laboratory field like we had in Jenkins, for chose to which laboratory run the tests, default to everyone.

Yes, thats planned. Right now i have filters for job (to limit test to specific build and test name), but need to add also lab AND also possible filter to limit test to specific device type as well.

from kernelci-core.

P4) Testing new patches

POST /api/patch

{
    "patch": "<patch_content_base64?>",
    "treeurl": "<tree_url>",
    "branch": "<branch>",
    "commit": "<commit_id>",
    "kbuildname": "<kbuildname>",
    "testname": "<testname>"
}

Fields kbuildname and testname are optional, if not specified it will run all tests and all builds, as configured in kernelci pipeline.
It will fetch sources from treeurl, checkout to branch, then to commit, apply patch. Then it will build kernel and run tests. If kbuildname and testname are specified, it will restrict to run only this build and test.
Purpose: Kernel developers can test new patches on kernelci infrastructure, making sure they are not breaking anything.

would be useful to have also a laboratory field like we had in Jenkins, for chose to which laboratory run the tests, default to everyone.

Yes, thats planned. Right now i have filters for job (to limit test to specific build and test name), but need to add also lab AND also possible filter to limit test to specific device type as well.

We don't need the commit id when the change is only local,
commit_id should be optional too

from kernelci-core.

I can make it optional, but that means it will just fetch current "tip of tree" commit id and fill it "automagically" (and create some uncertainity).
If we specify tree, it is not necessary that your local repository is updated and have same tip of tree, this is why i need commit id.
For example developer started to develop on Aug 1, and tip of tree at that moment was commit XXXX, but when he finished patch and he want to test it, tip of tree moved forward and it is commit YYYY and patch might not apply. So there is always required commit id to know at which place of tree we apply patch.
Also if we want to review results later, to reproduce build/tests we will need to know commit id, as after few days again tip of tree will change.

from kernelci-core.

I can make it optional, but that means it will just fetch current "tip of tree" commit id and fill it "automagically" (and create some uncertainity). If we specify tree, it is not necessary that your local repository is updated and have same tip of tree, this is why i need commit id. For example developer started to develop on Aug 1, and tip of tree at that moment was commit XXXX, but when he finished patch and he want to test it, tip of tree moved forward and it is commit YYYY and patch might not apply. So there is always required commit id to know at which place of tree we apply patch. Also if we want to review results later, to reproduce build/tests we will need to know commit id, as after few days again tip of tree will change.

I see, that works for me.
I will try to add the commit id from the commit, if there is one, in case there isn't a commit id drop to add it automatically on the server side works for me. so please leave it as optional.

from kernelci-core.

two other parameters that I would like to see are:

{
    "patch": "<patch_content_base64?>",
    "treeurl": "<tree_url>",
    "branch": "<branch>",
    "commit": "<commit_id>",
    "kbuildname": "<kbuildname>",
    "testname": "<testname>"
   "laboratory": <laboratory names>
    "tests": <tests name>
}

laboratories is for decide in which laboratory or laboratories the tests will be executed, default to all laboraries
tests is for decide which tests will be executed, default to all tests available for such build

from kernelci-core.

kbuild and testname is already in jobfilter (i will need to explain how they works)
i will add also labfilter and dutfilter (filter by lab and device type)
this is also within plans

from kernelci-core.

poetry run kci-dev --help
Usage: kci-dev [OPTIONS] COMMAND [ARGS]...

  Stand alone tool for Linux Kernel developers and maintainers that can test
  local Linux Kernel changes on a enabled KernelCI server

Options:
  --version  Show the version and exit.
  --help     Show this message and exit.

Commands:
  commit  Test commits from a local Kernel repository
  patch   Test a patch or a mbox file

from kernelci-core.

poetry run kci-dev commit --help
Usage: kci-dev commit [OPTIONS]

  Test commits from a local Kernel repository

Options:
  --repository TEXT  define the kernel upstream repository where to test local
                     changes
  --branch TEXT      define the repository branch
  --private          define if the test results will be published
  --path TEXT        define the directory of the local tree with local changes
  --settings TEXT    path of toml setting file
  --help             Show this message and exit.

from kernelci-core.

poetry run kci-dev patch --help
Usage: kci-dev patch [OPTIONS]

  Test a patch or a mbox file

Options:
  --repository TEXT  define the kernel upstream repository where to test local
                     changes
  --branch TEXT      define the repository branch
  --private          define if the test results will be published
  --patch TEXT       mbox or patch file path  [required]
  --settings TEXT    path of toml setting file
  --help             Show this message and exit.

from kernelci-core.

the JWT token and api host address are managed by the .kci-dev.toml.example setting file

from kernelci-core.

I will finish implementing today initial patchset features (after discovering how this subsystem works, and will publish updates), and will propose API for it.
Then i will take a look at tool. I guess we might need some repository where can publish such tools, or even individual repository for kci-dev. Probably this is good topic to discuss during upcoming weekly.

from kernelci-core.

My intention is to move the tool to kernelci organization, can you give me the admin on the repo after I moved it?
Also the tool can be published to the pip repository so that it can be used also by downloading it from pip

from kernelci-core.

Definitely, great idea, personally i am all for it and it should be possible.
I think to create repo and give admin privileges we need to vote at TSC maillist (we can do it before meeting).

from kernelci-core.

Is not creating the repo is a transfer
I already checked that I can transfer the repo even now, I just want to be sure to don't loose my admin rights after the transfer

from kernelci-core.

Yes, but i think according the rules we need to discuss with TSC adding or transferring repo to kernelci organization, but i am not sure about that.
We voted also for creating new repo, but TSC rules doesn't require that.
Also we need to update https://docs.kernelci.org/org/maintainers/#software-maintainers if we move repo and we need to set you as maintainer, and describe project.

from kernelci-core.

sure I was just specifying that is not a creation but a transfer

from kernelci-core.