Code Monkey home page Code Monkey logo

coversheet's Introduction

coversheet

Coversheet is a CI system for TPS, which allows to run tests for each daily build of Firefox across all platforms.

Setup

Before you can start the system the following commands have to be performed:

git clone git://github.com/mozilla/coversheet.git
cd coversheet
./setup.sh

You will need to have the Python header files installed:

  • Ubuntu: Install the package via: apt-get install python-dev
  • OSX, Windows: Install the latest Python 2.7

Startup

To start Jenkins simply run ./start.py from the coversheet directory. You can tell when Jenkins is running by looking out for "Jenkins is fully up and running" in the console output. You will also be able to view the web dashboard by pointing your browser at http://localhost:8080/

If this is the first time you've started Jenkins, or your workspaces have recently been deleted, you will need to run an admin job to finish the setup. Open http://localhost:8080/view/+admin/ and build the 'scripts' job.

Jenkins URL

If you intend to connect to this Jenkins instance from another machine (for example connecting additional nodes) you will need to update the Jenkins URL to the IP or DNS name. This can be found in http://localhost:8080/configure under the section headed "Jenkins Location".

Adding new Nodes

To add Jenkins slaves to your master you have to create new nodes. You can use one of the example nodes (Windows XP and Ubuntu) as a template. Once done the nodes have to be connected to the master. Therefore Java has to be installed on the node first.

Python needs to be installed and be available in PATH, also please make sure you have satisfied the requirements for fxa-python-client package.

Windows:

Go to www.java.com/download/ and install the latest version of Java JRE. Also make sure that the UAC is completely disabled, and the screensaver and any energy settings have been turned off.

Linux (Ubuntu):

Open the terminal or any other package manager and install the following packages:

sudo add-apt-repository ppa:webupd8team/java
sudo apt-get update
sudo apt-get install oracle-java7-installer

Also make sure that the screensaver and any energy settings have been turned off.

After Java has been installed open the appropriate node within Jenkins from the nodes web browser like:

http://IP:8080/computer/windows_xp_32_01/

Now click the Launch button and the node should automatically connect to the master. It will be used once a job for this type of platform has been requested by the Pulse consumer.

Using the Jenkins master as executor

If you want that the master node also executes jobs you will have to update its labels and add/modify the appropriate platforms, e.g. master mac 10.7 64bit for Mac OS X 10.7.

Testing changes

In order to check that patches will apply and no Jenkins configuration changes are missing from your changes you can run the run_tests.sh script. This uses Selenium and PhantomJS to save the configuration for each job and reports any unexpected changes. Note that you will need to download PhantomJS and put it in your path in order for these tests to run.

Merging branches

The main development on the coversheet code happens on the master branch. In not yet specified intervals we are merging changesets into the staging branch. It is used for testing all the new features before those go live on production. When running those merge tasks you will have to obey the following steps:

  1. Select the appropriate target branch
  2. Run 'git rebase master' for staging or 'git rebase staging' for production
  3. Run 'git pull' for the remote branch you want to push to
  4. Ensure the merged patches are on top of the branch
  5. Ensure that the Jenkins patch can be applied by running 'patch --dry-run -p1 <config/%BRANCH%/jenkins.patch'
  6. Run 'git push' for the remote branch

For emergency fixes we are using cherry-pick to port individual fixes to the staging and production branch:

  1. Select the appropriate target branch
  2. Run 'git cherry-pick %changeset%' to pick the specific changeset for the current branch
  3. Run 'git push' for the remote branch

Once the changes have been landed you will have to update the staging or production machines. Run the following steps:

  1. Run 'git reset --hard' to remove the locally applied patch
  2. Pull the latest changes with 'git pull'
  3. Apply the Jenkins patch with 'patch -p1 <config/%BRANCH%/jenkins.patch'
  4. Restart Jenkins

coversheet's People

Contributors

cosmin-malutan avatar sinemetu1 avatar whimboo avatar

Stargazers

 avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar

coversheet's Issues

Upgrade to fxa-python-client 1.2.0

A new version of fxa-python-client is out. It contains two fixes (better installation via deps, repeated checks for restmail.net if email has been arrived) we needed. So we might want to upgrade to it.

For the release see issue mozilla/fxa-python-client#21

Oh, and we should not forget to remove the workaround in our trigger script, which currently waits 2s. :)

Due to test package changes TPS tests are no longer executed

There was a split of test packages to individual zip files. Since then we no longer perform tps tests with coversheet. Reason is that we still rely on the old testsurl property in the normalized build notification. Instead we would have to use test_packages_url which points to a file like:

http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-linux64/1441706523/test_packages.json

With that info we can download the common test package.

@mhammond if it would be critical for you to get the tests running again in a short time, I might find some time to get this fixed. I assume those are the only tests we have for that Firefox component, or?

Sending the report fails because of "ValueError: time data 'unknown' does not match format '%Y%m%d%H%M%S'"

When CI has been finished running the test it fails with the following stacktrace when trying to send the report:

Traceback (most recent call last):
File "/home/ateamauto/coversheet/coversheet/results.py", line 85, in handleResults
self.sendEmail(body, sendTo)
File "/home/ateamauto/coversheet/coversheet/results.py", line 104, in sendEmail
buildUrl)
File "/home/ateamauto/coversheet/coversheet/emailtemplate.py", line 11, in GenerateEmailBody
'%Y%m%d%H%M%S')
File "/usr/lib/python2.7/_strptime.py", line 325, in _strptime
(data_string, format))
ValueError: time data 'unknown' does not match format '%Y%m%d%H%M%S'

Firefox Account is not getting verified

As it can be seen in logs like:
http://tps-ci-production.qa.scl3.mozilla.com:8080/job/mozilla-central_fx-account/206/console

03:19:09 GET https://api.accounts.firefox.com/__heartbeat__
03:19:09 restmail error
03:19:09 []

There is a failure in the verify call to fxa-client. Instead of the above the following has to be visible:

GET https://api.accounts.firefox.com/__heartbeat__
Verify URL: https://accounts.firefox.com/v1/verify_email?uid=da688c7359a24bfe99a922fa1c8a04fb&code=58437ef1b258f9c598c13cc5723ce9bb
Verified Acct

Update files to MPL2

The code is still under the MPL 1.1. @jonallengriffin I assume there is nothing which would hold us back from reducing the boilerplate and to make use of the MPL 2?

Build has to fail if tests are not successful

Right now a build triggered in Jenkins doesn't fail when some of the tests are failing:

08:56:17 TEST-UNEXPECTED-FAIL | test_tabs.js | [phase3] Exception caught: ASSERTION FAILED! error locating tab No traceback available
08:56:17 TEST-PASS | test_passwords.js |
08:56:18 Finished: SUCCESS

Here we have to fail. I assume the trigger.py doesn't care about the exit value of the tps testrunner script.

coversheet should not download builds to its own location

If coversheet is run globally or in a venv, the download of the build and tests will happen into the same location where the coversheet files are located. We should download to a temporary folder.

   pathToBuild = os.path.join(os.path.dirname(os.path.abspath(__file__)),
                               buildName)

If we transition over to Jenkins this issue might be obsolete given that we can use its workspace folder.

Add plain Jenkins support

As part of our goal for this quarter we want to add a CI system for TPS which will allow us to ran TPS test for each daily build for each platform. For achieving this in issue #3 we agreed on using Jenkins.
Using Jenkins for this require a little bit of work, so for splitting this in smaller commits we have to start with a plain Jenkins instance, on top of which we will continue with adding jobs, plugins, etc.
https://wiki.mozilla.org/QA/Automation/Projects/TPS#List_of_Tasks

Console output in trigger.py should be flushed, to not cause delays

Right now the console output for e.g. the info that we are downloading builds and tests is printed out very late in the process. This should happen immediately.

Example:

00:01:31.690 No distributions at all found for requests (from fxa-python-client)
00:01:31.690 Storing debug log for failure in /home/mozauto/.pip/pip.log
00:01:31.954 Downloading: firefox-32.0a2.en-US.linux-x86_64.tar.bz2
00:01:31.954 Downloading: firefox-32.0a2.en-US.linux-x86_64.tests.zip

The download is really the very first thing we do in this script. So not sure why it's getting printed when all files have been downloaded, unzipped, and the env already created.

Maybe using the logger instead of print statements would help us here.

Add new jobs for jenkins

Once we have jenkins landed we will have to create jobs for running TPS testruns.
At a moment we have only one type of a testrun and different release branches, that might change.

What I propose is we treat each branch separately, and create views for them as we have on mozmill-ci, and jobs for each view.
Right now we run a single type of testrun (plain TPS test and mozmill tests), but I think this might change in the future as mozmill tests are testing the UI and the other only the backend, if those would be splitted we will have them grouped in branch views.
I think we would need a Pre Build Job, or something similar in order to create the accounts, but that will be another issue.

What do you think?

Adding @whimboo for visibility.

Do not test xulrunner builds

Right now the pulse build listener isn't setup correctly. That means we also test other applications beside Firefox. In case of Xulrunner we download the builds, and fail when trying to download the tests.

We should ensure to only test Firefox builds.

Improve sending of status emails

So we should do some minor updates for the status emails:

  1. I don't see the needs to attach the full console log as attachment for successful (fixed) builds. Just strip that off.
  2. For failing or aborted builds, lets try to compress the attachment. Reason is that the log is kinda huge (~1MB) and compressed it might be way lesser.

@cosmin-malutan can you please check that?

trigger.py should only uninstall Firefox if it has been installed before

As you can see in the log we fail uninstalling Firefox during cleanup:

00:01:29.737 No distributions at all found for requests (from fxa-python-client)
00:01:29.742 Storing debug log for failure in /home/mozauto/.pip/pip.log
00:01:30.055 Downloading: firefox-33.0a1.en-US.linux-x86_64.tar.bz2
00:01:30.055 Downloading: firefox-33.0a1.en-US.linux-x86_64.tests.zip
[..]
00:01:30.055 Failed to remove the Firefox build, please remove it manually
00:01:30.060 RunTPS instance has no attribute 'build_path'

Failures are listed twice in the report emails

As seen today in the report email of a failing build, the failures are listed twice:

Test failures:
TEST-UNEXPECTED-FAIL | test_sync.js | [phase3] Exception caught: ASSERTION FAILED! places item not found; bookmark (uri: http://www.yahoo.com, title: testing Yahoo, location: menu/folderb) not found No traceback available
TEST-UNEXPECTED-FAIL | test_sync.js | [phase3] Exception caught: ASSERTION FAILED! places item not found; bookmark (uri: http://www.yahoo.com, title: testing Yahoo, location: menu/folderb) not found No traceback available

Instead of a single-line regex we might want to use a multi-line regex. Therefore we have to change BUILD_LOG_REGEX to BUILD_LOG_MULTILINE_REGEX. Then we could only match those entries which have an empty line before and after, and by that skip the summary listing.

Env creation fails because requests package cannot be found

Not sure why this is happening now - it was working yesterday night - but somehow we are not able to successfully find/download the requests package.

00:00:48.898 Downloading/unpacking requests (from fxa-python-client)
00:01:14.698 Real name of requirement requests is requests
00:01:29.721 Could not find any downloads that satisfy the requirement requests (from fxa-python-client)
00:01:29.721 Cleaning up...
00:01:29.737 No distributions at all found for requests (from fxa-python-client)
00:01:29.742 Storing debug log for failure in /home/mozauto/.pip/pip.log

Make Jenkins branch the active master branch

Now that we can successfully run TPS tests via Jenkins, even some issues remain, we should make the switch and move the code from the Jenkins branch into master.

To backup the old code we should keep the current content of master, and store it in old-master.

Make use of Jenkins for the TPS CI system

Right now the tool has no queuing mechanism, that means in case of tests are running, we spawn a new process in parallel. Given that we are using a single account at the moment different tests would interfere with each other.

@jonallengriffin shall I find a way to dynamically create a random account via the tps extension which will be removed at the end of the test (if that is possible)?

Coversheet should use its own virtualenv

Right now coversheet has to be installed globally. We should create a setup script which will setup an venv with all the appropriate packages and other pre-conditions to run tps tests. This is blocked on issue #10.

Add necessary Jenkins plugins

To be able to do all the things we need in Jenkins a couple of plugins have to be added:

  • buildtimeout - Automatically cancel builds after a given amount of time
  • copyartifact - Copy artifacts from another job
  • email-ext - Plugin to customize any sort of sending emails
  • envinject - Handling of environment variables
  • greenballs - Colored status display for job results
  • mail-watcher-plugin - Monitor offline state of nodes
  • monitoring - Monitor Jenkins status
  • nodelabelparameter - Allow to run tests on specified nodes
  • prioritysorter - Run jobs given a specified priority
  • purge-build-queue - Remove all planned jobs from the build queue
  • rebuild - Rebuild a job if it was failing
  • show-build-parameters - Display the parameters for the jobs
  • timestamper - Add timestamps to the console output
  • ws-cleanup - Cleanup the workspace

This issue blocks #3.

URLs for build and tests are not html encoded (space breaks download)

As seen in our tps runs for beta builds, the download of the build and tests is broken because the URLs contain a whitespace. We might have to htmlencode the URL or adjust the trigger script to pass it in correctly within quotes.

Here the example:

URL_BUILDS: http://stage.mozilla.org/pub/mozilla.org/firefox/nightly/31.0b7-candidates/build1/mac/en-US/Firefox 31.0b7.dmg
URL_TESTS: http://stage.mozilla.org/pub/mozilla.org/firefox/nightly/31.0b7-candidates/build1/mac/en-US/Firefox 31.0b7.tests.zip

Create production branch, and delete old un-used jenkins branch

Lets re-organize the repository a bit. Given that each landing of the code should not be automatically on the production ci, we need a production branch for that. Also we can remove the formerly used Jenkins branch now.

I would keep the old-master branch for now. Just in case we missed something and need easy access to the code.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.