os-autoinst / openqa_review Goto Github PK

View Code? Open in Web Editor NEW

8.0 9.0 29.0 1.27 MB

A review helper script for openQA

Home Page: http://os-autoinst.github.io/openqa_review

License: MIT License

Python 95.81% Shell 4.19%

openqa openqa-review python webscraping hacktoberfest

openqa_review's Introduction

A review helper script for openQA.

For more details read the documentation within openqa_review/openqa_review.py.

Usage

Install requirements and package, using an isolated Python environment such as VirtualEnv.

pip3 install -r requirements.txt
pip3 install .

or if you are using openSUSE distribution, it is recommended to use zypper, e.g.:

zypper in python-openqa_review

Call openqa-review from PATH, e.g. with --help

openqa-review --help

If openqa-review has to access non public servers which make use of custom Certificate Authorities (CA's), it can happen that it fails because it can not verify the TLS certificate. By setting the environment variable REQUESTS_CA_BUNDLE, one can supply a custom ca-certificate store to openqa-review e.g.:

REQUESTS_CA_BUNDLE="/my/custom/ca-bundle.pem" openqa-review

If that doesn't work, you can try the following setting:

CURL_CA_BUNDLE=""

Communication

If you have questions, visit #opensuse-factory on libera.chat.

Contribute

This project lives in https://github.com/os-autoinst/openqa_review

Feel free to add issues in github or send pull requests.

TODOs and ideas are tracked in the file TODO as well as github issues.

How to run tests

Tests are implemented on top of pytest and tox. Tox can be invoked with no arguments to test all environments:

tox

It is also possible to run only a special environment, e.g. tox -e flake8 for the linter, tox -e py38 for running tests at the specified Python version (the interpreter needs to be installed for this version) or tox -e cov for the coverage report (see details below). These environments are configured in tox.ini.

To ensure files are passing the style check, invoke black . to apply correct formatting.

If you are running into issues installing dependencies (resulting in an error like "Could not build wheels…"), try to remove the temporary .tox directory (or the subfolder for the problematic environment within).

Run tests via `pytest`

You can also run the tests via pytest with one of the following commands:

pytest
# or
python3 -m pytest
# or, after running tox at least once:
./.tox/cov/bin/py.test

To test only certain functions, there are several possibilities:

pytest tests/test_openqa_review.py::test_reminder_comments_on_referenced_bugs_are_posted
pytest -k test_reminder_comments_on_referenced_bugs_are_posted

Generate coverage report

Run the following commands to generate a coverage report under coverage_html/:

tox -e cov
./.tox/cov/bin/coverage html

Rules for commits

Every commit is checked by a GitHub Workflow as soon as you create a pull request but you should run tox locally,
Make sure to keep the 100% test coverage, e.g. by adding test reference data for new scenarios. TDD is advised :-)
For git commit messages use the rules stated on How to Write a Git Commit Message as a reference

If this is too much hassle for you feel free to provide incomplete pull requests for consideration or create an issue with a code change proposal.

License

This project is licensed under the MIT license, see LICENSE file for details. Exceptions are the files under dashboard_files/ which are GPL-2 licensed coming from https://github.com/os-autoinst/openQA/.

openqa_review's People

Contributors

Stargazers

Watchers

openqa_review's Issues

fail to request data from bugzilla - status code: 401

Sample output

[cryptlvm_minimal_x](https://openqa.suse.de/tests/1012891 "Failed modules: hostname") -> [bsc#1028286](https://bugzilla.suse.com/show_bug.cgi?id=1028286) (Request to https://apibugzilla.suse.com/jsonrpc.cgi?method=Bug.get&params=%5B%7B%22ids%22%3A+%5B1028286%5D%7D%5D was not successful, status code: 401)

When all jobs in one architecture are incomplete, the output is "green"

When all jobs in one architecture are failed as incomplete (or cancelled,obsolete,skipped), the script output for the result of this arch is "green"

Test

Date: 2016-11-30 - 13:33
Build: 0179

Arch: aarch64
Status: Green

heartbeat_interval parameter makes tests/test_tumblesle_release.py to fail

test_notifications_are_sent is failing when tox is executed for openqa_review. In this particular test pika.BlockingConnection is used. Example of the failure can be found on https://travis-ci.org/os-autoinst/openqa_review/jobs/514592861. The problem seems to arise from the pika version for both python2 and python3.

error message from openqa-review

Output will be attached.

Add support for multi-version job groups

Motivation

E.g. http://openqa.suse.de/group_overview/110 for SLE15 as well as https://openqa.opensuse.org/group_overview/50 use the same job group for minor versions / service packs. For SLE15 the build number started from 0 again and openqa-review does not find the 15SP1 jobs as "last finished". This should be changed

Acceptance criteria

AC1: The latest two SLE15SP1 builds are found and compared automatically by openqa-review over SLE15(SP0)

Suggestions

We need some changes somewhere within the function "get_build_urls_to_compare"
Read out both the "version" as well as "build" from the job group page
Compare by using the max of version and build, e.g. by using a custom sort function for the call to "sorted" on finished_builds

Write more proper unit tests that could replace current integration tests

motivation

openqa_review has 100% statement and branch coverage so it's easy to start hacking and see how it behaves. The current tests are "integration tests" that rely on downloaded html source files and compare the generated result against reference reports in markdown. This is a complete testing approach but the two main limitations are:

tests can be slow more than 1 second for a complete test run can be considered too much for such a simple project
not so helpful test failures if tests fail the main problem shown is that the generated report does not look like the reference which is not helpful when trying to work on low-level internal functions.

suggestion

More proper unit tests (could also be doctests) should be added.

report on "cancelled/skipped" tests to report on regressions in sum of not executed tests (compare to tumblesle)

Only output "skipped test" if section is not empty

issue status should include priority of issue

bugs and issues have priority. Especially for bugzilla references within SUSE the priority is very important, e.g. to find failing tests with bugrefs that don't have a high priority as every failing test should probably be linked against a bug that has at least prio 3.

Request from PM mawerner: blocker flag in report

openqa_review fails if it can not access an restricted progress issue

observation

Traceback (most recent call last):
  File "/usr/bin/openqa-review", line 11, in <module>
    load_entry_point('openqa-review===.-0.0.0-', 'console_scripts', 'openqa-review')()
  File "/usr/lib/python2.7/site-packages/openqa_review/openqa_review.py", line 1003, in main
    report = generate_report(args)
  File "/usr/lib/python2.7/site-packages/openqa_review/openqa_review.py", line 969, in generate_report
    return Report(browser, args, root_url, job_groups)
  File "/usr/lib/python2.7/site-packages/openqa_review/openqa_review.py", line 919, in __init__
    self.report[k] = self._one_report(v)
  File "/usr/lib/python2.7/site-packages/openqa_review/openqa_review.py", line 930, in _one_report
    return ProductReport(self.browser, job_group_url, self.root_url, self.args)
  File "/usr/lib/python2.7/site-packages/openqa_review/openqa_review.py", line 787, in __init__
    self.reports[arch] = ArchReport(arch, results, args, root_url, progress_browser, bugzilla_browser, browser)
  File "/usr/lib/python2.7/site-packages/openqa_review/openqa_review.py", line 700, in __init__
    issue = Issue(bug['bugref'], bug['bugref_href'], self.args.query_issue_status, self.progress_browser, self.bugzilla_browser)
  File "/usr/lib/python2.7/site-packages/openqa_review/openqa_review.py", line 564, in __init__
    self.json = progress_browser.get_json(bugref_href + '.json')['issue']
  File "/usr/lib/python2.7/site-packages/openqa_review/browser.py", line 66, in get_json
    return self.get_page(url, as_json=True)
  File "/usr/lib/python2.7/site-packages/openqa_review/browser.py", line 80, in get_page
    raw = open(os.path.join(self.load_dir, filename)).read()
IOError: [Errno 2] No such file or directory: '/tmp/tmp.AufehQRMV7/https%3A::progress.opensuse.org:issues:11438.json'

steps to reproduce

link a private ticket in openqa
call openqa_review covering the referenced ticket from a failing job
see the crash

suggestion

the whole run of openqa_review should not fail if it can not read a single issue
next step: openqa_review should try to use authentication to read issues

requests.exceptions.ConnectionError: ('Connection aborted.', error("(104, 'ECONNRESET')",)) -> should be handled gracefully

Observation

from cron email:

Subject: Cron okurz@lord env openqa_host=https://openqa.opensuse.org recv=[email protected] html_target_file=openqa_opensuse_org_status.html TPL=$HOME/local/openqa_review/dashboard_files/dashboard.html.in $HOME/local/openqa_review/bin/openqa-review-daily-email; cp -a ~/{Export,public_html}/openqa_opensuse_org_status.html
Date: Sun, 5 Aug 2018 07:10:52 +0200 (CEST)

Traceback (most recent call last):
File "/usr/bin/openqa-review", line 9, in
load_entry_point('openqa-review==0.0.0', 'console_scripts', 'openqa-review')()
File "/usr/lib/python2.7/site-packages/openqa_review/openqa_review.py", line 1254, in main
reminder_comment_on_issues(report)
File "/usr/lib/python2.7/site-packages/openqa_review/openqa_review.py", line 1240, in reminder_comment_on_issues
reminder_comment_on_issue(ie, min_days_unchanged)
File "/usr/lib/python2.7/site-packages/openqa_review/openqa_review.py", line 1224, in reminder_comment_on_issue
issue.add_comment(comment)
File "/usr/lib/python2.7/site-packages/openqa_review/openqa_review.py", line 660, in add_comment
"is_private": True,
File "/usr/lib/python2.7/site-packages/openqa_review/browser.py", line 150, in json_rpc_post
r = requests.post(absolute_url, data=data, auth=self.auth, headers={'content-type': 'application/json'})
File "/usr/lib/python2.7/site-packages/requests/api.py", line 110, in post
return request('post', url, data=data, json=json, **kwargs)
File "/usr/lib/python2.7/site-packages/requests/api.py", line 56, in request
return session.request(method=method, url=url, **kwargs)
File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 475, in request
resp = self.send(prep, **send_kwargs)
File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 596, in send
r = adapter.send(request, **kwargs)
File "/usr/lib/python2.7/site-packages/requests/adapters.py", line 471, in send
raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', error("(104, 'ECONNRESET')",))

Reproducible

Seen sometimes already

Suggestions

104 should be handled gracefully.

Request from RM sbehlert: Include changes in numbers in report

As for example done in https://openqa.suse.de/group_overview/110#comment-112973 in the section "Changes since reference build" the request was to include numbers in changed scenarios in the generated report.

Review contains invalid HTML

The review accessible under https://w3.suse.de/~okurz/openqa_suse_de_status.html contains invalid HTML.

<p><strong>Arch:</strong> s390x <strong>Status: <font color="green">Green</font></strong> <hr> <strong>Arch:</strong> x86_64 <strong>Status: <font color="green">Green</font></strong></p>

There shouldn't be a closing </p> according to the validator. This is because the <hr> element is not inside the <p> element. It causes the <p> element to be implicitly closed before and hence the closing </p> is wrong.

Also the use of <font ... should be avoided.

This HTML output seems to be used directly and unaltered used in openQA comments causing invalid HTML in openQA.

tumblesle: distinguish the three results "passed", "softfailed", "failed"

So far "softfailed" are regarded same as "passed" which means a reduction of soft fails does not automatically mean an improvement for tumblesle-release

work on some issues in TODO file

The file https://github.com/okurz/openqa_review/blob/master/TODO has many ideas to work on. Pick any task and try your luck :-)

tumblesle-release does not send out notification if notification connection died previously

observation

[…]
DEBUG:openqa_review.tumblesle_release:Release DONE
ERROR:pika.adapters.base_connection:Socket Error: 104
WARNING:pika.adapters.base_connection:Socket closed when connection was open
WARNING:pika.connection:Disconnected from RabbitMQ at kazhua.suse.de:5672 (0): Not specified
Traceback (most recent call last):
  File "/home/tumblesle/venv/bin/tumblesle-release", line 9, in <module>
    load_entry_point('openqa-review===1.4.1-9-g73059a2', 'console_scripts', 'tumblesle-release')()
  File "/home/tumblesle/venv/lib/python2.7/site-packages/openqa_review/tumblesle_release.py", line 395, in main
    tr.run()
  File "/home/tumblesle/venv/lib/python2.7/site-packages/openqa_review/tumblesle_release.py", line 145, in run
    self.one_run()
  File "/home/tumblesle/venv/lib/python2.7/site-packages/openqa_review/tumblesle_release.py", line 156, in one_run
    self.release()
  File "/home/tumblesle/venv/lib/python2.7/site-packages/openqa_review/tumblesle_release.py", line 318, in release
    self.notify({'build': self.release_build}, topic='release')
  File "/home/tumblesle/venv/lib/python2.7/site-packages/openqa_review/tumblesle_release.py", line 136, in notify
    self.notify_channel.basic_publish(exchange='pubsub', routing_key='.'.join([self.notify_topic, topic]), body=body)
  File "/home/tumblesle/venv/lib/python2.7/site-packages/pika/adapters/blocking_connection.py", line 1978, in basic_publish
    mandatory, immediate)
  File "/home/tumblesle/venv/lib/python2.7/site-packages/pika/adapters/blocking_connection.py", line 2065, in publish
    self._flush_output()
  File "/home/tumblesle/venv/lib/python2.7/site-packages/pika/adapters/blocking_connection.py", line 1174, in _flush_output
    *waiters)
  File "/home/tumblesle/venv/lib/python2.7/site-packages/pika/adapters/blocking_connection.py", line 395, in _flush_output
    raise exceptions.ConnectionClosed()
pika.exceptions.ConnectionClosed
Exception pika.exceptions.ConnectionClosed: The AMQP connection was closed: () in <bound method TumblesleRelease.__del__ of <openqa_review.tumblesle_release.TumblesleRelease object at 0x7f1cae6fb110>> ignored

steps to reproduce

I assume we could do the following

start test server
let tumblesle-release connect to server initially
terminate notification server
let tumblesle-release run into the notification and see it fail or wait until the second time and it still fails although the server might be up again

suggestion

-        self.notify_channel.exchange_declare(exchange='pubsub', type='topic')
+        self.notify_channel.exchange_declare(exchange='pubsub', type='topic', passive=True, durable=True)

openqa-review in openSUSE Leap 42.2 script is not compatible with new openQA version

On openSUSE Leap 42.2, after installing openqa-review

zypper in python-openqa_review
zypper in python-configparser # because this dependency is not available yet (https://github.com/okurz/openqa_review/issues/56#issuecomment-300504567)

I get this error executing openqa_review

mansilla@copland:/var/log/apache2$ openqa-review                                                                                                                                                                                             
Traceback (most recent call last):
  File "/usr/bin/openqa-review", line 9, in <module>
    load_entry_point('openqa-review===.-0.0.0-', 'console_scripts', 'openqa-review')()
  File "/usr/lib/python2.7/site-packages/openqa_review/openqa_review.py", line 1115, in main
    report = generate_report(args)
  File "/usr/lib/python2.7/site-packages/openqa_review/openqa_review.py", line 1055, in generate_report
    return Report(browser, args, root_url, job_groups)
  File "/usr/lib/python2.7/site-packages/openqa_review/openqa_review.py", line 1008, in __init__
    self.report[k] = self._one_report(v)
  File "/usr/lib/python2.7/site-packages/openqa_review/openqa_review.py", line 1016, in _one_report
    return ProductReport(self.browser, job_group_url, self.root_url, self.args)
  File "/usr/lib/python2.7/site-packages/openqa_review/openqa_review.py", line 808, in __init__
    current_url, previous_url = get_build_urls_to_compare(browser, job_group_url, args.builds, args.against_reviewed, args.running_threshold)
  File "/usr/lib/python2.7/site-packages/openqa_review/openqa_review.py", line 427, in get_build_urls_to_compare
    finished_builds = find_builds(get_group_result(job_group), running_threshold)
  File "/usr/lib/python2.7/site-packages/openqa_review/openqa_review.py", line 378, in find_builds
    log.debug("Found the following finished non-empty builds: %s" % ', '.join(finished.keys()))
TypeError: sequence item 1: expected string or Unicode, NoneType found
mansilla@copland:/var/log/apache2$

Request from PM mawerner: skipped test cases in report

Numeric builds without a leading zero should sort numerically not alphabetically

Observation

It seems with #102 we have covered a part of the issue but not completely. SLE12SP4+SP5 is properly handled because builds have a leading 0 so an "alphabetic" sort will work but for SLE15 we have the problem that build 98 is considered higher than 102.

Acceptance criteria

AC1: A more recent build 102 is considered "later" or "higher" than 98
AC2: Builds including non-numerical characters don't crash the script

Problem

Basically there are two potentially conflicting build sorting approaches, by time of creation and by value of the build itself. Normally a more recent build is also higher by numerical value however not ensured to be. We could try to not sort on our own at all but rely on the order how it is presented by openQA or we get the information what is preferred from openQA and sort accordingly or we define our own rules with precedence.

Suggestion

Just trying to sort numerically with fallback to alphabetic sort if there are non-numeric characters besides interpunctation sounds good. Hopefully this is a "standard-problem" for which we find a good "standard-solution", at best without needing to define our own complicated algorithms.

if multiple tests show the same modules failing, the comment will be too long

Suggestion:
if more than e.g. 5 tests fail because of the same reason, write something like "several tests fail due to ..." in the review

comprehensive number of bugs with priority

Would be nice if there is a possibility to have a comprehensive number of bugs with priority for all the projects that do not focus on fixing bugs but "managing" them ;-)

Also with difference vs. comparison target

E.g.

P1: 42 (+3)
P2: 3 (-2)

bug references got "Ticket not found", but they exist

sles+extratests -> boo#1011931 (Ticket not found)

Existing Product bugs:

migration_offline_sle12 -> bsc#1012564 (Ticket not found)

tumblesle-release compares with reference that might be pruned from openQA already

observation

[Notice] -hermes to #qa-review- [07:05:30] tumbleSLE release: build: 0405
<c> [07:09:37] that is good?
<c> [07:11:28] that was a forced release? because 405 is not even tested
<okurz> tumblesle_release:0405: 1/0 vs. 0234: 1/1, clearly 1 passed+0 failed
is better than 1 passed and 1 failed ;-) but why does have the last tumblesle
release 1/1?

problem

https://oqXXX/group_overview/XX?limit_builds=200 shows the problem. 0234 is the reference but only 2 jobs still exist from that old "non-important" job so tumblesle-release should probably tag the last release

further details

I guess you can see it as a feature: if no build gets released for a very long time it just picks the most recent one

UnicodeEncodeError: 'ascii' codec can't encode character u'\u2026' in position 382: ordinal not in range(128)

Observation

Traceback (most recent call last):
  File "/usr/bin/openqa-review", line 9, in <module>
    load_entry_point('openqa-review==0.0.0', 'console_scripts', 'openqa-review')()
  File "/usr/lib/python2.7/site-packages/openqa_review/openqa_review.py", line 1245, in main
    print(report)
  File "/usr/lib/python2.7/site-packages/openqa_review/openqa_review.py", line 1141, in __str__
    report_str += '# %s\n\n%s\n---\n' % (k, v)
  File "/usr/lib/python2.7/site-packages/openqa_review/openqa_review.py", line 958, in __str__
    'arch_report': '\n---\n'.join(map(str, self.reports.values()))
  File "/usr/lib/python2.7/site-packages/openqa_review/openqa_review.py", line 892, in __str__
    'existing_product_issues': issue_listing('**Existing Product bugs:**', self.issues['existing']['product'], self.args.show_empty),
  File "/usr/lib/python2.7/site-packages/openqa_review/openqa_review.py", line 342, in issue_listing
    return '\n' + header + '\n\n' + ''.join(map(str, issues)) + '\n'
UnicodeEncodeError: 'ascii' codec can't encode character u'\u2026' in position 382: ordinal not in range(128)

found in email report from daily dashboard report generation. I assume a problem introduced by any person using a label with non-ascii characters.

Reproducible

Happens sometimes but not every day in the daily jobs

Further details

"daily dashboard report generation" is mainly https://github.com/okurz/openqa_review/blob/master/bin/openqa-review-daily-email triggered on a daily base from a cron job on lord.arch.suse.de for both o3 as well as osd.

we should also show "incomplete's" but as openQA issues

See e.g.
https://openqa.suse.de/tests/overview?distri=sle&version=12-SP2&build=2123&groupid=25&result=incomplete

a lot of incompletes are shown and all are properly labeled. This information would be good to have in the report. There is one product issue although that is a problem that the test should actually be failed instead of incomplete.

Request from SLE12SP4 RM Rado: List bugs appearing in multiple archs just once as common issues

filtered reports, e.g. "closed issues" should not include "skipped tests"

Automated notification test failure template

When there is a soft failure, the message posted into BZ neither openQA console dump may not contain information to uncover the failure cause. (Or it is well hidden for a person not so familiar with those reports.)

I suggest to modify record_soft_failure in such a way that the message would contain location of the actual code (similar to C assert). E.g link to the code

Soft fail at lib/sles4sap.pm:85, see [1]

[1] https://github.com/os-autoinst/os-autoinst-distri-opensuse/blob/7cb2bc0e2831c19b01777a0df019c5f50641125c/lib/sles4sap.pm#L85

Cc: @ricardobranco777 @okurz

bugzilla reminder comments should state version, e.g. on bug for openSUSE Leap 42.1 when it's still appearing in Leap 42.2

Network-settings

Hello,

i am trying to create a test with openQA and connect to network.
For example start machine and ping an adress (www.google.com) . But i cant find a way to give the openQA machine an ip-address. Do i have to use openvswitch ? Or os-autoinst-openvswitch ? Is there a guide for networking that could help me ?
Thanks a lot. Fabi

Misleading text for auto-generated message in progress

The same text message that we use in Bugzilla is used in Progress for the autogenerated message: "This bug is still referenced in a failing openQA test". IMHO it should be something more like "This failing openQA test is still referencing this resolved ticket" or something like that, which could potentially attract more our attention as QA Engineers:
https://progress.opensuse.org/issues/42842#note-5

retrieving bugs from bugzilla can fail and abort the script

observation

Cron <okurz@lord> env TPL=/suse/okurz/Export/dashboard_files/dashboard.html.in $HOME/local/openqa_review/bin/openqa-review-daily-email
Date: Today 08:00:55
From: "(Cron Daemon)" <[email protected]>
To: [email protected]
Traceback (most recent call last):
  File "/usr/bin/openqa-review", line 9, in <module>
    load_entry_point('openqa-review==0.0.0', 'console_scripts', 'openqa-review')()
  File "/usr/lib/python2.7/site-packages/openqa_review/openqa_review.py", line 1193, in main
    report = generate_report(args)
  File "/usr/lib/python2.7/site-packages/openqa_review/openqa_review.py", line 1129, in generate_report
    return Report(browser, args, root_url, job_groups)
  File "/usr/lib/python2.7/site-packages/openqa_review/openqa_review.py", line 1079, in __init__
    self.report[k] = self._one_report(v)
  File "/usr/lib/python2.7/site-packages/openqa_review/openqa_review.py", line 1090, in _one_report
    return ProductReport(self.browser, job_group_url, self.root_url, self.args)
  File "/usr/lib/python2.7/site-packages/openqa_review/openqa_review.py", line 910, in __init__
    self.reports[arch] = ArchReport(arch, results, args, root_url, progress_browser, bugzilla_browser, browser)
  File "/usr/lib/python2.7/site-packages/openqa_review/openqa_review.py", line 771, in __init__
    issue = Issue(bug['bugref'], bug['bugref_href'], self.args.query_issue_status, self.progress_browser, self.bugzilla_browser)
  File "/usr/lib/python2.7/site-packages/openqa_review/openqa_review.py", line 609, in __init__
    self.json = bugzilla_browser.json_rpc_get('/jsonrpc.cgi', 'Bug.get', {"ids": [self.bugid]})['result']['bugs'][0]
  File "/usr/lib/python2.7/site-packages/openqa_review/browser.py", line 129, in json_rpc_get
    return self.get_json(get_url.replace('http://dummy', ''), cache)
  File "/usr/lib/python2.7/site-packages/openqa_review/browser.py", line 75, in get_json
    return self.get_page(url, as_json=True, cache=cache)
  File "/usr/lib/python2.7/site-packages/openqa_review/browser.py", line 103, in get_page
    r = requests.get(absolute_url, auth=self.auth)
  File "/usr/lib/python2.7/site-packages/requests/api.py", line 70, in get
    return request('get', url, params=params, **kwargs)
  File "/usr/lib/python2.7/site-packages/requests/api.py", line 56, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 475, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 596, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python2.7/site-packages/requests/adapters.py", line 471, in send
    raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', error("(104, 'ECONNRESET')",))

reproducible

seems to happen near every day now when the automatic report is generated. Seems to be more reproducible now as we have more bugs to retrieve data for

problem

problems on retrieving single bugs should not abort the whole report generation

suggestion

handle one error gracefully, e.g. skip the retrieval of this one bug
retry multiple times to retrieve one bug

tumblesle_release stalls on network issues

observation

DEBUG:openqa_review.tumblesle_release:Checking last builds on http://openqa.suse.de ...
DEBUG:openqa_review.tumblesle_release:Finding most recent ISO matching regex '.*SP2.*Server.*x86\_64.*\Z(?ms)'
DEBUG:openqa_review.tumblesle_release:Found last ISO SLE-12-SP2-Server-MINI-ISO-x86_64-Build1651-Media.iso
DEBUG:openqa_review.tumblesle_release:Found last build 1651
DEBUG:openqa_review.tumblesle_release:Getting jobs in build 1651 ...


# pressed Ctrl-C here after it got stuck for very long, should have found build1652 already, probably stuck for at least 12h.

^CTraceback (most recent call last):
  File "/home/tumblesle/venv/bin/tumblesle-release", line 9, in <module>
    load_entry_point('openqa-review-script===0.4.0-27-gdfdc6f9', 'console_scripts', 'tumblesle-release')()
  File "/home/tumblesle/venv/lib/python2.7/site-packages/openqa_review/tumblesle_release.py", line 355, in main
    tr.run()
  File "/home/tumblesle/venv/lib/python2.7/site-packages/openqa_review/tumblesle_release.py", line 113, in run
    self.one_run()
  File "/home/tumblesle/venv/lib/python2.7/site-packages/openqa_review/tumblesle_release.py", line 121, in one_run
    self.check_last_builds()
  File "/home/tumblesle/venv/lib/python2.7/site-packages/openqa_review/tumblesle_release.py", line 170, in check_last_builds
    jobs_by_result['last'] = self.retrieve_jobs_by_result(build['last'])
  File "/home/tumblesle/venv/lib/python2.7/site-packages/openqa_review/tumblesle_release.py", line 141, in retrieve_jobs_by_result
    jobs_build = self.browser.get_json('/api/v1/jobs?state=done&latest=1&build=%s&group_id=%s' % (build, group_id))['jobs']
  File "/home/tumblesle/venv/lib/python2.7/site-packages/openqa_review/openqa_review.py", line 145, in get_json
    return self.get_page(url, as_json=True)
  File "/home/tumblesle/venv/lib/python2.7/site-packages/openqa_review/openqa_review.py", line 164, in get_page
    r = requests.get(absolute_url, verify=False)
  File "/home/tumblesle/venv/lib/python2.7/site-packages/requests/api.py", line 71, in get
    return request('get', url, params=params, **kwargs)
  File "/home/tumblesle/venv/lib/python2.7/site-packages/requests/api.py", line 57, in request
    return session.request(method=method, url=url, **kwargs)
  File "/home/tumblesle/venv/lib/python2.7/site-packages/requests/sessions.py", line 475, in request
    resp = self.send(prep, **send_kwargs)
  File "/home/tumblesle/venv/lib/python2.7/site-packages/requests/sessions.py", line 617, in send
    r.content
  File "/home/tumblesle/venv/lib/python2.7/site-packages/requests/models.py", line 741, in content
    self._content = bytes().join(self.iter_content(CONTENT_CHUNK_SIZE)) or bytes()
  File "/home/tumblesle/venv/lib/python2.7/site-packages/requests/models.py", line 664, in generate
    for chunk in self.raw.stream(chunk_size, decode_content=True):
  File "/home/tumblesle/venv/lib/python2.7/site-packages/requests/packages/urllib3/response.py", line 353, in stream
    data = self.read(amt=amt, decode_content=decode_content)
  File "/home/tumblesle/venv/lib/python2.7/site-packages/requests/packages/urllib3/response.py", line 310, in read
    data = self._fp.read(amt)
  File "/usr/lib64/python2.7/httplib.py", line 573, in read
    s = self.fp.read(amt)
  File "/usr/lib64/python2.7/socket.py", line 380, in read
    data = self._sock.recv(left)
KeyboardInterrupt

problem

H1. network issues

suggestion

Call requests.get with timeout. tumblesle_release actually continued just fine after pressing Ctrl-C once.

Retry on 504

Observation

Traceback (most recent call last):
  File "/usr/bin/openqa-review", line 9, in <module>
    load_entry_point('openqa-review==0.0.0', 'console_scripts', 'openqa-review')()
  File "/usr/lib/python2.7/site-packages/openqa_review/openqa_review.py", line 1233, in main
    print(report)
  File "/usr/lib/python2.7/site-packages/openqa_review/openqa_review.py", line 1129, in __str__
    report_str += '# %s\n\n%s\n---\n' % (k, v)
  File "/usr/lib/python2.7/site-packages/openqa_review/openqa_review.py", line 946, in __str__
    'arch_report': '\n---\n'.join(map(str, self.reports.values()))
  File "/usr/lib/python2.7/site-packages/openqa_review/openqa_review.py", line 881, in __str__
    'todo_issues': self._todo_issues_str(),
  File "/usr/lib/python2.7/site-packages/openqa_review/openqa_review.py", line 865, in _todo_issues_str
    'existing_issues': issue_listing('***existing issues***', self.issues['existing']['todo'], self.args.show_empty),
  File "/usr/lib/python2.7/site-packages/openqa_review/openqa_review.py", line 334, in issue_listing
    return '\n' + header + '\n\n' + ''.join(map(str, issues)) + '\n'
  File "/usr/lib/python2.7/site-packages/openqa_review/openqa_review.py", line 735, in __str__
    ', '.join(map(self._format_failure, self.failures)),
  File "/usr/lib/python2.7/site-packages/openqa_review/openqa_review.py", line 719, in _format_failure
    report_str = issue_report_link(self.root_url, f, self.test_browser) if (self.args.report_links and self.test_browser) else ''
  File "/usr/lib/python2.7/site-packages/openqa_review/openqa_review.py", line 488, in issue_report_link
    test_details_page = test_browser.get_soup(f['href'])
  File "/usr/lib/python2.7/site-packages/openqa_review/browser.py", line 71, in get_soup
    return BeautifulSoup(self.get_page(url), "html.parser")
  File "/usr/lib/python2.7/site-packages/openqa_review/browser.py", line 102, in get_page
    content = self._get(absolute_url, as_json=as_json)
  File "/usr/lib/python2.7/site-packages/openqa_review/browser.py", line 123, in _get
    raise DownloadError(msg)
openqa_review.browser.DownloadError: Request to https://openqa.opensuse.org/tests/654185 was not successful, status code: 504

Expected result

openqa-review should not fail on this but skip or retry gracefully

traceback when running openqa-review --help on Tumbleweed

openqa-review --help
Traceback (most recent call last):
File "/usr/bin/openqa-review", line 6, in
from pkg_resources import load_entry_point
File "/usr/lib/python2.7/site-packages/pkg_resources/init.py", line 2991, in
@_call_aside
File "/usr/lib/python2.7/site-packages/pkg_resources/init.py", line 2977, in _call_aside
f(*args, **kwargs)
File "/usr/lib/python2.7/site-packages/pkg_resources/init.py", line 3004, in _initialize_master_working_set
working_set = WorkingSet._build_master()
File "/usr/lib/python2.7/site-packages/pkg_resources/init.py", line 662, in _build_master
ws.require(requires)
File "/usr/lib/python2.7/site-packages/pkg_resources/init.py", line 970, in require
needed = self.resolve(parse_requirements(requirements))
File "/usr/lib/python2.7/site-packages/pkg_resources/init.py", line 856, in resolve
raise DistributionNotFound(req, requirers)
pkg_resources.DistributionNotFound: The 'configparser' distribution was not found and is required by openqa-review

"Missing architectures" table entry consumes next "hline"

report generates something like this


  Acceptance: SLE 12 SP2 1.Server

*Date:* 2016-09-19 - 08:00 *Build:* 2131

*Common issues:*

  * *Missing architectures*: x86_64, ppc64le
    ------------------------------------------------------------------------

*Arch:* aarch64 *Status: Red*

*TODO: review*

*/new issues/*

  * btrfs
  * minimal_x+uefi
  * sdk-ftp
  * textmode

------------------------------------------------------------------------
*Arch:* s390x *Status: Green*

------------------------------------------------------------------------

where the line after "missing architectures" should not be indented.

TypeError: 'NoneType' object has no attribute 'getitem' in get_test_details

Observation

Traceback (most recent call last):
  File "/usr/bin/openqa-review", line 9, in <module>
    load_entry_point('openqa-review==0.0.0', 'console_scripts', 'openqa-review')()
  File "/usr/lib/python2.7/site-packages/openqa_review/openqa_review.py", line 1220, in main
    report = generate_report(args)
  File "/usr/lib/python2.7/site-packages/openqa_review/openqa_review.py", line 1156, in generate_report
    return Report(browser, args, root_url, job_groups)
  File "/usr/lib/python2.7/site-packages/openqa_review/openqa_review.py", line 1106, in __init__
    self.report[k] = self._one_report(v)
  File "/usr/lib/python2.7/site-packages/openqa_review/openqa_review.py", line 1117, in _one_report
    return ProductReport(self.browser, job_group_url, self.root_url, self.args)
  File "/usr/lib/python2.7/site-packages/openqa_review/openqa_review.py", line 928, in __init__
    results = get_arch_state_results(arch, current_details, previous_details, args.output_state_results)
  File "/usr/lib/python2.7/site-packages/openqa_review/openqa_review.py", line 288, in get_arch_state_results
    states = SortedDict(get_state(v, test_results_previous_dict) for k, v in iteritems(test_results_dict))
  File "/usr/lib/python2.7/site-packages/sortedcontainers/sorteddict.py", line 140, in __init__
    self._update(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/sortedcontainers/sorteddict.py", line 338, in update
    self._dict_update(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/openqa_review/openqa_review.py", line 288, in <genexpr>
    states = SortedDict(get_state(v, test_results_previous_dict) for k, v in iteritems(test_results_dict))
  File "/usr/lib/python2.7/site-packages/openqa_review/openqa_review.py", line 276, in get_state
    state_dict.update(get_test_details(cur))
  File "/usr/lib/python2.7/site-packages/openqa_review/openqa_review.py", line 250, in get_test_details
    'failedmodules': [{'href': m.a['href'], 'name': m.text.strip(), 'needles': get_failed_needles(m)} for m in failedmodules]
TypeError: 'NoneType' object has no attribute '__getitem__'

Request from director: Rules why red/amber/green as legend in report

tumblesle: builds with jobs running can be seens as new release regardless of result

proposal

Count "running" same as "failed" to prevent a newer build not being seen as better as an old one that has been released while jobs have been running even though the corresponding jobs would fail eventually.

show severity rather than priority from QA perspective

Request from SLE RM sbehlert: Overview of changed failed/passed test modules (not just scenarios)

bugzilla bug filing assistant

user story

As a reviewer of failing tests in openQA I want to openqa-review to help me fill in bug reports with the common content to save some time when filing bugs

acceptance criteria

AC1: Text for bug entries is pre-generated, either as local text or within a browser window with the entries of a bug prefilled

ideas

provide a helper script parsing the information from a test
compare with previous jobs to fill "last good"
fill some statistics from previous jobs
add link to "latest in this scenario"

mark RESOLVED FIXED as red

Make use of new proper buglinks to soft failures within openqa-review reports

See https://progress.opensuse.org/issues/14310#change-30480

Missing dependency: python-configparser is required

The python-openqa_review package is installed by zypper command on my openSUSE Tumbleweed:

~> sudo zypper in python-openqa_review

~> openqa-review --help
Traceback (most recent call last):
  File "/usr/bin/openqa-review", line 6, in <module>
    from pkg_resources import load_entry_point
  File "/usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2991, in <module>
    @_call_aside
  File "/usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2977, in _call_aside
    f(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line 3004, in _initialize_master_working_set
    working_set = WorkingSet._build_master()
  File "/usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line 662, in _build_master
    ws.require(__requires__)
  File "/usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line 970, in require
    needed = self.resolve(parse_requirements(requirements))
  File "/usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line 856, in resolve
    raise DistributionNotFound(req, requirers)
pkg_resources.DistributionNotFound: The 'configparser' distribution was not found and is required by openqa-review

After installing python-configparser:
~> sudo zypper in python-configparser
~> openqa-review --help
Traceback (most recent call last):
  File "/usr/bin/openqa-review", line 11, in <module>
    load_entry_point('openqa-review===.-0.0.0-', 'console_scripts', 'openqa-review')()
  File "/usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line 567, in load_entry_point
    return get_distribution(dist).load_entry_point(group, name)
  File "/usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2604, in load_entry_point
    return ep.load()
  File "/usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2264, in load
    return self.resolve()
  File "/usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2270, in resolve
    module = __import__(self.module_name, fromlist=['__name__'], level=0)
  File "/usr/lib/python2.7/site-packages/openqa_review/openqa_review.py", line 105, in <module>
    from configparser import ConfigParser, NoSectionError, NoOptionError  # isort:skip can not make isort happy here
  File "/usr/lib/python2.7/site-packages/configparser.py", line 12, in <module>
    from backports.configparser import (
ImportError: No module named backports.configparser

The issue can be workaround by install configparser via pip
~> sudo pip install configparser
~> openqa-review --help

sort output for complete report based on custom ordering on index page, i.e. preserve ordering

Handle "boo#0" not as error in log but better

Acceptance criteria

AC1: boo#0 is not reported as error in the log but handled appropriately in the report

Suggestions

Adapt ttm in a way to do what openSUSE RMs need so that no one would use this bugref anymore.
If above not possible workaround in openqa_review by accepting the bugref as special

Further details

openSUSE RMs are using the invalid boo#0 bug reference to fool ttm which demands a ticket reference for each failure.

Log error:

ERROR:openqa_review.openqa_review:Error retrieving details for bugref boo#0 (https://bugzilla.opensuse.org/show_bug.cgi?id=0): 'NoneType' object has no attribute '__getitem__'

Request from SLE12SP4 RM Rado: Count bugs

add soft-fails without bugref to report instead of log

Motivation

In case openqa-review finds a soft-fail reference but without a valid bug reference this is reported in logs but rather should show up in the generated report as this is relevant information for the test reviewers.

Acceptance criteria

AC1: A soft-fail reference in test without a valid bugref is reported in the generated report
AC2: soft-fail references without valid bugref are not reported as error in log

Further details

Currently the log messages contain for example the following report:

ERROR:openqa_review.openqa_review:Could not find any soft failure reference within details of soft-failed job 'https://openqa.suse.de/tests/1580533'. Could be deleted workaround needle?.
ERROR:openqa_review.openqa_review:Failed to find valid bug tracker URL for {'state': 'NEW_SOFT_ISSUE', 'href': u'/tests/1562126', 'failedmodules': [], 'bugref': u'fate#321208: activating existing encrypted volume does *not* yield an encrypted system if not forcing'} with error u'ate'. Skipping current result

os-autoinst / openqa_review Goto Github PK

openqa_review's Introduction

Usage

Communication

Contribute

How to run tests

Run tests via pytest

Generate coverage report

Rules for commits

License

openqa_review's People

Contributors

Stargazers

Watchers

Forkers

openqa_review's Issues

Motivation

Acceptance criteria

Suggestions

motivation

suggestion

observation

steps to reproduce

suggestion

Observation

Reproducible

Suggestions

observation

steps to reproduce

suggestion

Observation

Acceptance criteria

Problem

Suggestion

observation

problem

further details

Observation

Reproducible

Further details

observation

reproducible

problem

suggestion

observation

problem

suggestion

Observation

Expected result

Observation

proposal

user story

acceptance criteria

ideas

Acceptance criteria

Suggestions

Further details

Motivation

Acceptance criteria

Further details

Recommend Projects

Recommend Topics

Recommend Org

Run tests via `pytest`