mozilla / coversheet Goto Github PK

Similar to mozilla/mozmill-ci#487 we need an upgrade for the pulse listener. The public access will be removed soon. CC'ing @markrcote.

Now that we can successfully run TPS tests via Jenkins, even some issues remain, we should make the switch and move the code from the Jenkins branch into master.

To backup the old code we should keep the current content of master, and store it in old-master.

The code is still under the MPL 1.1. @jonallengriffin I assume there is nothing which would hold us back from reducing the boilerplate and to make use of the MPL 2?

In order to run TPS testruns on different nodes simultaneously we need a different account for each testrun so they don't interfere.
We would make use of https://github.com/mozilla/fxa-python-client/ package for creating accounts and http://restmail.net/ service for emails.

My proposal is we create a python script for handling this, import it and call it in trigger.py script in tools job.

Lets re-organize the repository a bit. Given that each landing of the code should not be automatically on the production ci, we need a production branch for that. Also we can remove the formerly used Jenkins branch now.

I would keep the old-master branch for now. Just in case we missed something and need easy access to the code.

So we should do some minor updates for the status emails:

1. I don't see the needs to attach the full console log as attachment for successful (fixed) builds. Just strip that off.
2. For failing or aborted builds, lets try to compress the attachment. Reason is that the log is kinda huge (~1MB) and compressed it might be way lesser.

@cosmin-malutan can you please check that?

As seen today in the report email of a failing build, the failures are listed twice:

Test failures:
TEST-UNEXPECTED-FAIL | test_sync.js | [phase3] Exception caught: ASSERTION FAILED! places item not found; bookmark (uri: http://www.yahoo.com, title: testing Yahoo, location: menu/folderb) not found No traceback available
TEST-UNEXPECTED-FAIL | test_sync.js | [phase3] Exception caught: ASSERTION FAILED! places item not found; bookmark (uri: http://www.yahoo.com, title: testing Yahoo, location: menu/folderb) not found No traceback available

Instead of a single-line regex we might want to use a multi-line regex. Therefore we have to change BUILD_LOG_REGEX to BUILD_LOG_MULTILINE_REGEX. Then we could only match those entries which have an empty line before and after, and by that skip the summary listing.

To be able to do all the things we need in Jenkins a couple of plugins have to be added:

• buildtimeout - Automatically cancel builds after a given amount of time
• copyartifact - Copy artifacts from another job
• email-ext - Plugin to customize any sort of sending emails
• envinject - Handling of environment variables
• greenballs - Colored status display for job results
• mail-watcher-plugin - Monitor offline state of nodes
• monitoring - Monitor Jenkins status
• nodelabelparameter - Allow to run tests on specified nodes
• prioritysorter - Run jobs given a specified priority
• purge-build-queue - Remove all planned jobs from the build queue
• rebuild - Rebuild a job if it was failing
• show-build-parameters - Display the parameters for the jobs
• timestamper - Add timestamps to the console output
• ws-cleanup - Cleanup the workspace

This issue blocks #3.

As you can see in the log we fail uninstalling Firefox during cleanup:

00:01:29.737 No distributions at all found for requests (from fxa-python-client)
00:01:29.742 Storing debug log for failure in /home/mozauto/.pip/pip.log
00:01:30.055 Downloading: firefox-33.0a1.en-US.linux-x86_64.tar.bz2
00:01:30.055 Downloading: firefox-33.0a1.en-US.linux-x86_64.tests.zip
[..]
00:01:30.055 Failed to remove the Firefox build, please remove it manually
00:01:30.060 RunTPS instance has no attribute 'build_path'

A new version of fxa-python-client is out. It contains two fixes (better installation via deps, repeated checks for restmail.net if email has been arrived) we needed. So we might want to upgrade to it.

For the release see issue mozilla/fxa-python-client#21

Oh, and we should not forget to remove the workaround in our trigger script, which currently waits 2s. :)

I don't see why mozregression is necessary here. Instead mozinstall and mozdownload packages can be used to get all the necessary pieces done.

There was a split of test packages to individual zip files. Since then we no longer perform tps tests with coversheet. Reason is that we still rely on the old testsurl property in the normalized build notification. Instead we would have to use test_packages_url which points to a file like:

http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-linux64/1441706523/test_packages.json

With that info we can download the common test package.

@mhammond if it would be critical for you to get the tests running again in a short time, I might find some time to get this fixed. I assume those are the only tests we have for that Firefox component, or?

On bug 966434 I have updated tps to be able to use the Firefox Account authentication. We have to update the contained config file to be in sync.

Right now coversheet has to be installed globally. We should create a setup script which will setup an venv with all the appropriate packages and other pre-conditions to run tps tests. This is blocked on issue #10.

Right now a build triggered in Jenkins doesn't fail when some of the tests are failing:

08:56:17 TEST-UNEXPECTED-FAIL | test_tabs.js | [phase3] Exception caught: ASSERTION FAILED! error locating tab No traceback available
08:56:17 TEST-PASS | test_passwords.js |
08:56:18 Finished: SUCCESS

Here we have to fail. I assume the trigger.py doesn't care about the exit value of the tps testrunner script.

When a job has been finished a status email has to be send to the mailing list. This should be done for successful and failing jobs.

Therefore the email-ext plugin will be used in a similar way as we do for Mozmill-CI here:

https://github.com/mozilla/mozmill-ci/blob/master/jenkins-master/jobs/mozilla-aurora_addons/config.xml#L127

As we move on jenkins as CI infrastructure, when we receive pulse messages we have to trigger jenkins jobs instead of triggering testrun directly on the current machine. For that we would have to make use of jenkins python package similar we do on:
https://github.com/mozilla/mozmill-ci/blob/master/pulse.py

When CI has been finished running the test it fails with the following stacktrace when trying to send the report:

Traceback (most recent call last):
File "/home/ateamauto/coversheet/coversheet/results.py", line 85, in handleResults
self.sendEmail(body, sendTo)
File "/home/ateamauto/coversheet/coversheet/results.py", line 104, in sendEmail
buildUrl)
File "/home/ateamauto/coversheet/coversheet/emailtemplate.py", line 11, in GenerateEmailBody
'%Y%m%d%H%M%S')
File "/usr/lib/python2.7/_strptime.py", line 325, in _strptime
(data_string, format))
ValueError: time data 'unknown' does not match format '%Y%m%d%H%M%S'

The current version of pulsebuildmonitor is outdated and has to be updated to version 0.80.

Now running jobs on OSX is stable and we should trigger them automatically via pulse messages. For that we have to add the labels for platforms>mac:
https://github.com/mozilla/coversheet/blob/master/config/production/daily.json
https://github.com/mozilla/coversheet/blob/master/config/production/release.json

As it can be seen in logs like:
http://tps-ci-production.qa.scl3.mozilla.com:8080/job/mozilla-central_fx-account/206/console

03:19:09 GET https://api.accounts.firefox.com/__heartbeat__
03:19:09 restmail error
03:19:09 []

There is a failure in the verify call to fxa-client. Instead of the above the following has to be visible:

GET https://api.accounts.firefox.com/__heartbeat__
Verify URL: https://accounts.firefox.com/v1/verify_email?uid=da688c7359a24bfe99a922fa1c8a04fb&code=58437ef1b258f9c598c13cc5723ce9bb
Verified Acct

@jonallengriffin can you please move the repo under the Mozilla account? That would allow us to assign our contributors to issues, and propose mentored bugs in bugsahoy. Thanks.

Once we have jenkins landed we will have to create jobs for running TPS testruns.
At a moment we have only one type of a testrun and different release branches, that might change.

What I propose is we treat each branch separately, and create views for them as we have on mozmill-ci, and jobs for each view.
Right now we run a single type of testrun (plain TPS test and mozmill tests), but I think this might change in the future as mozmill tests are testing the UI and the other only the backend, if those would be splitted we will have them grouped in branch views.
I think we would need a Pre Build Job, or something similar in order to create the accounts, but that will be another issue.

What do you think?

Adding @whimboo for visibility.

As part of our goal for this quarter we want to add a CI system for TPS which will allow us to ran TPS test for each daily build for each platform. For achieving this in issue #3 we agreed on using Jenkins.
Using Jenkins for this require a little bit of work, so for splitting this in smaller commits we have to start with a plain Jenkins instance, on top of which we will continue with adding jobs, plugins, etc.
https://wiki.mozilla.org/QA/Automation/Projects/TPS#List_of_Tasks

If coversheet is run globally or in a venv, the download of the build and tests will happen into the same location where the coversheet files are located. We should download to a temporary folder.

   pathToBuild = os.path.join(os.path.dirname(os.path.abspath(__file__)),
                               buildName)

If we transition over to Jenkins this issue might be obsolete given that we can use its workspace folder.

zip_safe flag not set; analyzing archive contents...
httplib2.init: module references file
/mozilla/code/coversheet/downloadedtests/tps/INSTALL.sh: 75: /mozilla/code/coversheet/downloadedtests/tps/INSTALL.sh: Bad substitution

http://mxr.mozilla.org/mozilla-central/source/testing/tps/INSTALL.sh#75

NEWCONFIG=${CONFIG:0:${#CONFIG}-3}

Right now we are trying to start the binary even if not given. I think the code in https://github.com/jonallengriffin/coversheet/blob/master/coversheet/cli.py#L123 just misses a not, so that we don't start the listener. Would be good to also raise a proper exception and exit early.

Right now the pulse build listener isn't setup correctly. That means we also test other applications beside Firefox. In case of Xulrunner we download the builds, and fail when trying to download the tests.

We should ensure to only test Firefox builds.

As a follow up to bug 1009004 | https://bugzilla.mozilla.org/show_bug.cgi?id=1009004
we have to use create_venv.py script which will work on windows to.
We should fallback to INSTALL.sh if create_venv.py is not found.

This blocks issue #3

https://wiki.mozilla.org/QA/Automation/Projects/TPS#List_of_Tasks

Right now the tool has no queuing mechanism, that means in case of tests are running, we spawn a new process in parallel. Given that we are using a single account at the moment different tests would interfere with each other.

@jonallengriffin shall I find a way to dynamically create a random account via the tps extension which will be removed at the end of the test (if that is possible)?

Not sure why this is happening now - it was working yesterday night - but somehow we are not able to successfully find/download the requests package.

00:00:48.898 Downloading/unpacking requests (from fxa-python-client)
00:01:14.698 Real name of requirement requests is requests
00:01:29.721 Could not find any downloads that satisfy the requirement requests (from fxa-python-client)
00:01:29.721 Cleaning up...
00:01:29.737 No distributions at all found for requests (from fxa-python-client)
00:01:29.742 Storing debug log for failure in /home/mozauto/.pip/pip.log

As seen in our tps runs for beta builds, the download of the build and tests is broken because the URLs contain a whitespace. We might have to htmlencode the URL or adjust the trigger script to pass it in correctly within quotes.

Here the example:

URL_BUILDS: http://stage.mozilla.org/pub/mozilla.org/firefox/nightly/31.0b7-candidates/build1/mac/en-US/Firefox 31.0b7.dmg
URL_TESTS: http://stage.mozilla.org/pub/mozilla.org/firefox/nightly/31.0b7-candidates/build1/mac/en-US/Firefox 31.0b7.tests.zip

As far we need python and java installed on CI nodes, and with the landing of issue #34 we would need a couple of more dependencies, they should be documented here with all the steps for setting up.

Right now the console output for e.g. the info that we are downloading builds and tests is printed out very late in the process. This should happen immediately.

Example:

00:01:31.690 No distributions at all found for requests (from fxa-python-client)
00:01:31.690 Storing debug log for failure in /home/mozauto/.pip/pip.log
00:01:31.954 Downloading: firefox-32.0a2.en-US.linux-x86_64.tar.bz2
00:01:31.954 Downloading: firefox-32.0a2.en-US.linux-x86_64.tests.zip

The download is really the very first thing we do in this script. So not sure why it's getting printed when all files have been downloaded, unzipped, and the env already created.

Maybe using the logger instead of print statements would help us here.

This issue will care about the different nodes we use for testing on production.

There are cases when the trigger script in the tools job is trying to clean-up the firefox account at the end. This will fail if no account has been created during the run.

This blocks the landing of issue #3 on master.