At the moment a ruruku binary cannot tell its own version - this is no good. The ruruku binary should be able to tell its own version, as well as the one of the server it's connected to.

use case

1. In my app, testding is done by visually comparison of generated output. So I generate the output and an HTML-File which shows "reference / diff / current". I would like to enhance this file such that a teste can click on ok / ? / fail and enter some comment. This should then be sent to ruruku - server.

2. one could build a webapp with a test-mode which which would reveal a verdict button, so the user could report finding directly while investigating the web app. These buttons would be removed for a production build.

request

need an easy to use javascript - API to send verdicts to ruruku.

Ruruku should eat it's own dogfood and have its own testplan, specifically for features for which no automated tests exist.

If one configures a minTesterCount for a test, this value is not used in the testcase status.

Once a user has submitted a testcase run it can no longer be edited. This is inconvenient. Users should be able to edit their testcase run after submission.

At the moment session.go mixes the session management and websocket handling. If we ever want other interfaces (e.g. a REST interface) to claim/complete tests, those two concerns need to be separated.

Passing the min-tester-count as command line flag to ruruku init testcase has no effect. It should set for the testcase however.

In Ruruku I constantly get error messages of this kind:

Error: Authorization token is not valid

Occasionally I have to log in again.

A reload and re-login helps.

UseCase

I play around with ruruku to monitor tests of zupfnoter (bwl21/zupfnoter#245).

As a developer I create testcases before I make changes in Zupfnoter. As a "release manager" I am using a scripting environment (rspec) to generate diff of all testcase-pdfs and messages produced by the app. But I need a quick way to investigate these diffs and collect findings. This is where ruruku comes in.

Solution approach

• let the testsuite (rspec) generate a ruruku testplan. In the steps section of a particular test refer to or even embed the aforehead mentioned diff files.
• go through all tests in ruruku and do the investigation. Thereby show the test results. It could look like this:

I have created this screenshot by a testcase like tis

- description: "./test-source/083_Der_Mirjambrunnen_Bernhard.abc"
  group: ''
  id: "./test-source/083"
  minTesterCount: 0
  mustPass: false
  name: mal schaun ./test-source/083_Der_Mirjambrunnen_Bernhard.abc
  steps: "\n\n## investigate ./test-source/083_Der_Mirjambrunnen_Bernhard.abc\n\n<img
    src=\"http://localhost:3000/test-output/083_Der_Mirjambrunnen_Bernhard_-test_a3.png\"
    alt=\"drawing\" width=\"100%\"/>\n<img src=\"http://localhost:3000/test-diff/083_Der_Mirjambrunnen_Bernhard.diff.png\"
    alt=\"drawing\" width=\"100%\"/>\n\n  "
- description: "./test-source/1003_Jumplines.abc"

As you see I had to setup another webserver to serve the outputfiles of my testsuite. It would be great if ruruku could serve these files with an url resolved relative to the position of the testplan.yaml. It would even be better if testplan.yaml could specify the root of the server files.

At the moment a lot of things are in flux and there's little point in creating individual releases for each bugfix. We should have nightly builds so that users can enjoy the latest fixes.

The column headers are clickable, but its content is not sorted accordingly.

Ruruku is a CLI heavy application. It should sport auto-complete in Bash and ZSH. This is easily added using https://github.com/spf13/cobra/blob/master/bash_completions.md

ruruku init testcase -f testsuite.yaml -y --group foo --id bar --name Foobar still interactively asks for a description despite -y being set.

1. it appears that testplan.yaml is read only once, and then is persisted in ruruku.db, So I delete ruruku.db before every start

2. rururku should give messages when validateing testplan.yaml. For example it loads if "group" is an empty string. But it does not show testcases

3. ruruku should not stop if there are duplicate ids. It would be enough to raise a warning and ignore the the newer duplcates

One does not always want to test all testcases in a single testsuite. Sometimes one only wants to run a very basic test or run all tests for a specific feature. Ruruku should support something like a testset which is selection of testcases from a testsuite. During the creation of a session one should be able to select a testset (or a subset of testcases directly).

For this feature we'd need to introduce three new concepts:

1. testcase annotations: annotations are key/value pairs attached to a testcase (e.g. feature=signup, concurrentUsers=5 or dependsOnBrowser=true. The values of annotations are either symbols (strings without spaces) or numeric values.
2. selection expressions: based on the annotations we want to select testcases. An expression evaluates to a boolean value for each testcase depending on if the expression matches the testcase's annotations or not. For example: feature == signup and (concurentUsers > 2 or dependsOnBrowser == true) The expression language is as follows:

SYMBOL = /\w[\w\d]+/
NUMBER = /\d+(\.\d+)?/
OPERATOR = '==' | '!=' | '>' | '>=' | '<' | '<='
VALUE = SYMBOL | NUMBER
SINGLE_EXPR = SYMBOL OPERATOR VALUE
CONJUNCTION = 'and' | 'or'
MULTI_EXPR = SINGLE_EXPR (CONJUNCTION SINGLE_EXPR)*
EXPR = PARANTHESIZED_EXPR | MULTI_EXPR
PARANTHESIZED_EXPR = '(' EXPR ')'

3. In a testsuite one can have testsets which have an ID, a name and an expression. During the creation of a test session one can select a testset or explicitly give an expression. The test session then only contains all tests that match the expression. Additionally, the testsession will have an annotation which contains the testset name and the expression.

This issue is best split in three tasks along the individual features.

• Implement expression language incl. interpreter
• Add annotations to testcases and testsets to testsuites
• Implement selection during session creation

Steps to reproduce

• using ruruku_v0.3.1_darwin_386
• I generate the zupfnoter.yaml by myself. it looks like this:

---
case:
- description: "./test-source/030_Davids_Harfe_BW.abc"
  group: Zupfnoter
  id: "./test-source/030"
  minTesterCount: 0
  mustPass: false
  name: mal schaun ./test-source/030_Davids_Harfe_BW.abc
  steps: "\n\n## investigate ./test-source/030_Davids_Harfe_BW.abc\n\n<img src=\"http://localhost:3000/test-output/030_Davids_Harfe_BW_-test_a3.png\"
    alt=\"drawing\" width=\"100%\"/>\n<img src=\"http://localhost:3000/test-diff/030_Davids_Harfe_BW.diff.png\"
    alt=\"drawing\" width=\"100%\"/>\n\n  "
- description: "./test-source/083_Der_Mirjambrunnen_Bernhard.abc"
  group: Zupfnoter
  id: "./test-source/083"
  minTesterCount: 0
  mustPass: false
  name: mal schaun ./test-source/083_Der_Mirjambrunnen_Bernhard.abc
  steps: "\n\n## investigate ./test-source/083_Der_Mirjambrunnen_Bernhard.abc\n\n<img
    src=\"http://localhost:3000/test-output/083_Der_Mirjambrunnen_Bernhard_-test_a3.png\"
    alt=\"drawing\" width=\"100%\"/>\n<img src=\"http://localhost:3000/test-diff/083_Der_Mirjambrunnen_Bernhard.diff.png\"
    alt=\"drawing\" width=\"100%\"/>\n\n  "
name: zupfnoter
tags: {}
id: xplan
description: ""

• I start ruruku with

./ruruku_v0.3.1_darwin_386 start  zupfnoter.yaml}

• I login with admin / admin

I can see the testcases but I cannot claim. Ruruku says in the terminal

DEBU[0000] Set log level to debug                       
INFO[0000] Opened database                               filename=ruruku.db
INFO[0000] User created                                  password=admin username=admin
INFO[0000] Opened database                               filename=ruruku.db
INFO[0000] gRPC API server started                       port=1234
INFO[0000] UI web server started                         port=8080
INFO[0000] Using an auto-generated session name          name="yellow cardinal"
INFO[0000] Starting session                              id=yellow-cardinal-f90a9234 name="yellow cardinal"
INFO[0000] Session started                               id=yellow-cardinal-f90a9234
INFO[0000] Started server at https://localhost          
DEBU[0034] Successfully authenticated user               user=admin
DEBU[0039] Testcase claim recorded                       claim=true session=yellow-cardinal-f90a9234 testcase=./test-source/030 user=admin
DEBU[0041] Testcase claim recorded                       claim=true session=yellow-cardinal-f90a9234 testcase=./test-source/083 user=admin
DEBU[0337] Renewed user token                            user=admin

zupfnoter.yaml.zip

As a tester when I join a ruruku session I can't see what testplan we're testing or in which session I am. The name of testplan and session should be shown in the header of the ruruku page.

The test result dropdown in the "completion" form does not show the current selection. Users can select a value, but the widget remains "empty".

Now that testcases can be added, modified and removed from a running session, we want to extract a testplan from a running session. This way users can plan tests collaboratively or pull changes made during a testing session into their testplan.

We want a progress bar showing:

[PPPPPPPPUUUUUFFFFFFFXXXXXXXxxxx]
where
P is passed (green)
U is undecided (yellow)
F is failed (red)
X is claimed (grey)
x is unclaimed (lighter grey)

If a user opens a ruruku participation page without a token, the only error message is in the console upon connection. We should instead hide the username field and show an appropriate message.

After a user completes a test case run, that run needs to be matched up with their testcase claim, so that the status changes accordingly.

At the moment testcase runs and testcase claims also use a different format (group-qualified testcase ID vs casegroup and caseID). This should be harmonized.

When trying to restart a previous ruruku session (e.g. ruruku start --session-name foobar with a foobar.yaml present) the server won't start. There seems to be an issue in the YAML unmarshall process:

$ ruruku start --session-name ruruku-20181128-083215 testsuite.yaml
panic: reflect: reflect.Value.Set using unaddressable value [recovered]
        panic: reflect: reflect.Value.Set using unaddressable value

goroutine 1 [running]:
gopkg.in/yaml%2ev2.handleErr(0xc00014f938)
        /home/travis/gopath/src/gopkg.in/yaml.v2/yaml.go:249 +0x9a
panic(0x8db9c0, 0xc000221a40)
        /home/travis/.gimme/versions/go1.11.linux.amd64/src/runtime/panic.go:513 +0x1b9
reflect.flag.mustBeAssignable(0x16)
        /home/travis/.gimme/versions/go1.11.linux.amd64/src/reflect/value.go:234 +0x157
reflect.Value.Set(0x900740, 0x0, 0x16, 0x900740, 0xc0000b3f20, 0x16)
        /home/travis/.gimme/versions/go1.11.linux.amd64/src/reflect/value.go:1367 +0x2f
gopkg.in/yaml%2ev2.(*decoder).prepare(0xc0000b5f80, 0xc00021ed90, 0x900740, 0x0, 0x16, 0x0, 0x40e71d, 0xc0000b0170, 0xc000172d80)
        /home/travis/gopath/src/gopkg.in/yaml.v2/decode.go:302 +0x2bf
gopkg.in/yaml%2ev2.(*decoder).unmarshal(0xc0000b5f80, 0xc00021ed90, 0x900740, 0x0, 0x16, 0x1)
        /home/travis/gopath/src/gopkg.in/yaml.v2/decode.go:324 +0x74
gopkg.in/yaml%2ev2.(*decoder).document(0xc0000b5f80, 0xc00021ed20, 0x900740, 0x0, 0x16, 0xc00014f668)
        /home/travis/gopath/src/gopkg.in/yaml.v2/decode.go:344 +0x7a
gopkg.in/yaml%2ev2.(*decoder).unmarshal(0xc0000b5f80, 0xc00021ed20, 0x900740, 0x0, 0x16, 0x600)
        /home/travis/gopath/src/gopkg.in/yaml.v2/decode.go:320 +0x1d3
gopkg.in/yaml%2ev2.unmarshal(0xc000222000, 0x36c, 0x56c, 0x900740, 0x0, 0x0, 0x0, 0x0)
        /home/travis/gopath/src/gopkg.in/yaml.v2/yaml.go:148 +0x3a1
gopkg.in/yaml%2ev2.Unmarshal(0xc000222000, 0x36c, 0x56c, 0x900740, 0x0, 0x0, 0x0)
        /home/travis/gopath/src/gopkg.in/yaml.v2/yaml.go:81 +0x58
github.com/32leaves/ruruku/pkg/rurukulib/server.LoadFileStorage(0xc000217660, 0x1b, 0xc000172a80, 0xc0001f8ea0, 0x0, 0x0)
        /home/travis/gopath/src/github.com/32leaves/ruruku/pkg/rurukulib/server/storage.go:57 +0x145
github.com/32leaves/ruruku/pkg/rurukulib/server.LoadSessionOrNew(0x7fffe2905f93, 0x16, 0xc000172a80, 0x0, 0x0, 0x0)
        /home/travis/gopath/src/github.com/32leaves/ruruku/pkg/rurukulib/server/session.go:23 +0x240
github.com/32leaves/ruruku/pkg/rurukulib/server.Start(0xc00014fcb0, 0xc000172a80, 0x7fffe2905f93, 0x16, 0x0, 0xc00014fcc8)
        /home/travis/gopath/src/github.com/32leaves/ruruku/pkg/rurukulib/server/server.go:63 +0x79
github.com/32leaves/ruruku/cmd.glob..func3(0x1391480, 0xc0000e89b0, 0x1, 0x5)
        /home/travis/gopath/src/github.com/32leaves/ruruku/cmd/start.go:34 +0x17d
github.com/spf13/cobra.(*Command).execute(0x1391480, 0xc0000e8960, 0x5, 0x5, 0x1391480, 0xc0000e8960)
        /home/travis/gopath/src/github.com/spf13/cobra/command.go:766 +0x2cc
github.com/spf13/cobra.(*Command).ExecuteC(0x1391220, 0xc0000a3660, 0xf, 0xc0000af7b0)
        /home/travis/gopath/src/github.com/spf13/cobra/command.go:852 +0x2fd
github.com/spf13/cobra.(*Command).Execute(0x1391220, 0x407250, 0xc000098058)
        /home/travis/gopath/src/github.com/spf13/cobra/command.go:800 +0x2b
github.com/32leaves/ruruku/cmd.Execute()
        /home/travis/gopath/src/github.com/32leaves/ruruku/cmd/root.go:29 +0x2d
main.main()
        /home/travis/gopath/src/github.com/32leaves/ruruku/main.go:20 +0x20

tested in v0.1.3

At the moment users coming to this repo don't know how to use ruruku. Write a getting started section in the readme.md that describes the first steps.

Sometimes we decide that we do not want to execute a testcase in a session. Rather than remove the testcase, we want to make it explicit that the testcase should not be tested. In those cases a fourth test run state skipped would be handy. Skipped test cases would not influence the overall test result.

Using the following testplan

id: "1"
name: Das ist test 1
description: ""
case:
- id: "1"
  name: test 1
  group: one
  description: ""
  steps: ""
  mustpass: false
  mintestercount: 0
- id: "10"
  name: test 10
  group: one
  description: ""
  steps: ""
  mustpass: false
  mintestercount: 0
- id: "100"
  name: test 100
  group: one
  description: ""
  steps: ""
  mustpass: false
  mintestercount: 0

and confirming all three cases, I get

At the moment users have to tell each other on a side-channel which session to join. It would be nice if one could send a link to a session, e.g. https://demo.ruruku.sh/join/demo-session-1234

At the moment we have no tests for the ruruku init and ruruku init testcase CLI. These tests should automated.

Ruruku does not answer CORS preflight requests properly because the gRPC splitter in the server does not detect CORS preflight requests. This means that one cannot make gRPC web requests from another host - or more poignantly, future features which will rely on CORS will not work.

Add a command ruruku suite init to create a new test suite. This command would ask for details such as the name and description. In the same vein, we'd need a command ruruku suite add which adds a testcase to a suite by asking for the details in the same way.

github.com/manifoldco/promptui seems like a good candidate to implement this.

The release builds do not ship with the built React client at the moment which renders ruruku start useless. Release builds should package the necessary files.

Sometimes we do not want to use a token (e.g. during development in Gitpod the token is more annoying that useful). ruruku start should have a --no-token flag to disable the token all together.

At the moment ruruku is somewhat opportunistic about where it's hosted and makes a few assumptions about its hosting environment:

• each test session gets its own server
• we have access to the filesystem of the server to deposit the testplan and to retrieve the test protocol

A Gitpod workspace can maintain those requirements, but it is an odd way of working. What does one do with the test session protocol once the test is over? Place it next to the testsuite and commit it to Git?

It would be easier to manage things if we had a central ruruku server to which we can test plans, start test sessions, contribute test runs and get the status of a test execution. Some of those tasks are best done in a CLI, others in a UI, some in both. I will introduce such a central server which will sport a gRPC interface, maintain the data on its own (no external DBMS needed) and can be hosted from a single static binary or in a Docker container.

One key feature that makes ruruku charming is its low setup cost: you get a single binary and it does it all (host sessions, serve as CLI for creating plans, ...). This feature will be preserved: one will still be able use ruruku as it works today.

For claimed, items it would great to have a direct 'Passed', 'Failed' button.
Also, maybe you could put the form below the line to keep the context.
Claimed tests would automatically be opened.

Request

1. rururku takes much space on the screen. This could be improved by

   • add draggable dividers between test case and table
   • adjust the column width of the table by dragging
   • reduce the padding of the table cells

2. we need to many mouse clicks

   • do a one click accept which does claim and accept
   • replace the menu "passed, undecided, failed" by three buttons (under the comment) which close the testcase.

3. raise a warning if user leaves the page. In particular on touchpads (on Mac) it happens that one does the "back" gesture ...

4. ruruku should also snow the test ids which are often used as a reference in the Requirements tracing and to external test databases.

Use the GITPOD_WORKSPACE_URL env variable to show the correct server URL (incl. token) on the command line.

When I view a long list of testcases (long enough so that I have to scroll down), and open the sidebar (e.g. Contribute or testcase details), the sidebar opens at the top of the page and I have to scroll up to see it.

Starting a ruruku session on windows does use the configured port, but seemingly no port at all. The server starts properly, but is not reachable at the address it prints.

At the moment anyone can access a rukuku instance. Check for token when creating a websocket connection (welcome message needs token).

In preparation for the 0.1 release, we should have release builds using travis that upload their result to GitHub releases.