groupon / dotci Goto Github PK

Private repository support was added recently (see #6), which is fantastic - thank you so much! The implementation as far as I can see depends on a single global deploy key being set with an expectation that this can be reused across all private repositories.

Unfortunately, as can be seen in the GitHub documentation here, deploy keys must be unique across an organisation.

As a consequence, adding a new DotCI job for a second private repository in the same organisation results in this error:

Is it a mis-configuration by me that is causing this, or is it a genuine issue?

I can't think of a good reason why "Delete workspace before build starts" should not be enabled.
It will prevent previous build artifacts/report on the master of prior builds
from being incorporated by the current artifacts/report running on the slave.

Not doing so can lead to false broken builds.
e.g its possible a prior build had generates a broken test report that exists on the master build.

It's rare but development may require previous report to generate differences against the current report. They can disable this field ... rather than forcing all most developments who create new builds to enable this parameter.

Putting this here just as a TODO resulting from #5

Private repos are not supported as of now.

I think it needs a little more work to fully support private repositories

Add a config preference in dotci configuration for enabling private repo setup.
Add deploy key to github repo when project is setup, and use that for git checkout.
Add configuration option to add deploy public key in dotci configuration.

In normal Jenkins, it is quite easy to search for a job; you start typing in the search field on the top left, it auto-completes, you click enter, and it loads it:

With DotCI, the typing and the autocomplete works, but then it is unable to find the job:

Also, searching without the autocomplete successfully finds the job, but still can't load it:

Searching for the Github group name does not work with autocomplete:

But if you type the correct name (case insensitive) and click enter, it loads the group's page:

Then searching within the group, the auto-complete works:

but hitting enter brings up the search results window:

and clicking on the link brings up the same page:

ad infinitum (well, I only clicked a few times, but I don't know why it would ever start working).

currently, a view listing all of our jobs shows the status of the last respective build, regardless of the branch. it would be nice if there was a way to see a similar view, but have the status for each respective job only be for master.

[ -x /usr/bin/socat ] && socat TCP-LISTEN:${port},fork TCP:${host}:${port}

At the moment, if you create a matrix of jobs in DotCI via a Groovy for loop, it forces the jobs to report their success/failure in the order they were started. In this way, if one job is hung but a lot of subsequent jobs are already done, you will never see the results of the subsequent jobs until the first job is completed. It would be better to simply report the completion of the jobs as they complete, instead of forcing the order they were started

https://wiki.jenkins-ci.org/display/JENKINS/Build+Monitor+Plugin

Jobs list doesn't have nested dotci jobs.

It would be great to define variables for a Jenkins installation which can be substituted in to a .ci.yml script; for example:

SELENIUM_GRID_URL (which could be different for different DotCI installations)

TEST_BROWSER_LIST

so that I could say in .ci.yml:

for (browser in ${TEST_BROWSER_LIST})
{
mvn test -DGRID_URL=${SELENIUM_GRID_URL} -Dwebdriver.browser=browser
}

Since we are running tests inside the container we will need a way to extract test artifacts for consumption by jenkins. This would allow users to review log files, screenshots, etc to determine what happened during the test run.

Proposed example of yaml design.

environment:
  language: node
  language_versions:
    - v0.10.26
  services:
    - fedora/memcached 
plugins:
  - artifacts: test/log/
docker:
  mount:
    - #{workspace}/test/log:someplace/where/test/log

This would run the corresponding docker volume mount commands.

% ... -v #{workspace}/test/log:someplace/where/test/log ...

The webhook is configured properly in the github settings, and Jenkins appears to be receiving the pushes as the following can be found in the log:

Jun 13, 2014 6:07:55 PM INFO com.groupon.jenkins.github.GithubWebhook processGitHubPayload
Received POST by r04r
Jun 13, 2014 6:07:56 PM INFO com.groupon.jenkins.github.GithubWebhook processGitHubPayload
Received kicking off build for https://github.com/r04r/TinyBBS2

But when looking at https://github.com/groupon/DotCi/blob/master/src/main/java/com/groupon/jenkins/github/GithubWebhook.java#L78 it seems that it fails to find the associated project to actually schedule the build? The project itself was configured through 'New DotCI Job' and the configured associated github repository appears to correctly be https://github.com/r04r/TinyBBS2.

The payload sent by github is:

{
  "ref": "refs/heads/master",
  "after": "1e04dfdac45e65b2b67bf0c452a17bb6546892f0",
  "before": "55609434587e7de558e3c8103cfea54d6225d35b",
  "created": false,
  "deleted": false,
  "forced": false,
  "compare": "https://github.com/r04r/TinyBBS2/compare/55609434587e...1e04dfdac45e",
  "commits": [
    {
      "id": "1e04dfdac45e65b2b67bf0c452a17bb6546892f0",
      "distinct": true,
      "message": "test jenkins",
      "timestamp": "2014-06-13T18:20:57+02:00",
      "url": "https://github.com/r04r/TinyBBS2/commit/1e04dfdac45e65b2b67bf0c452a17bb6546892f0",
      "author": {
        "name": "r04r",
        "email": "[email protected]",
        "username": "r04r"
      },
      "committer": {
        "name": "r04r",
        "email": "[email protected]",
        "username": "r04r"
      },
      "added": [

      ],
      "removed": [

      ],
      "modified": [
        ".ci.yml"
      ]
    }
  ],
  "head_commit": {
    "id": "1e04dfdac45e65b2b67bf0c452a17bb6546892f0",
    "distinct": true,
    "message": "test jenkins",
    "timestamp": "2014-06-13T18:20:57+02:00",
    "url": "https://github.com/r04r/TinyBBS2/commit/1e04dfdac45e65b2b67bf0c452a17bb6546892f0",
    "author": {
      "name": "r04r",
      "email": "[email protected]",
      "username": "r04r"
    },
    "committer": {
      "name": "r04r",
      "email": "[email protected]",
      "username": "r04r"
    },
    "added": [

    ],
    "removed": [

    ],
    "modified": [
      ".ci.yml"
    ]
  },
  "repository": {
    "id": 20500920,
    "name": "TinyBBS2",
    "url": "https://github.com/r04r/TinyBBS2",
    "description": "Incomplete.",
    "homepage": "",
    "watchers": 0,
    "stargazers": 0,
    "forks": 0,
    "fork": false,
    "size": 744,
    "owner": {
      "name": "r04r",
      "email": "[email protected]"
    },
    "private": false,
    "open_issues": 0,
    "has_issues": true,
    "has_downloads": true,
    "has_wiki": true,
    "language": "JavaScript",
    "created_at": 1401916265,
    "pushed_at": 1402676480,
    "master_branch": "master"
  },
  "pusher": {
    "name": "r04r",
    "email": "[email protected]"
  }
}

I would like to use your plugin but we are using LDAP for authentication and authorization of Jenkins, so I am unable to configure the Security Realm to GitHub Authentication Plugin (oauth). Would it be possible to make the usage optional? Would you accept a pull request?

My company want to use DotCi but we don't host our code in GitHub. It seems that DotCi use Github/GHE as it's source repository enforcedly. Do you guys have a plan to support other code hosting services?

When having a multi config run (e.g. .ci.yml with after: section), clicking the the 'Console output' of the Jenkins job shows a DotCI console output vs the build log as in single config run.

To get to the build log, you need to go to status then click the 'default' link, which is not easily discoverable .

Suggestion:

In the DotCI console output add direct links to every configs output for easier discoverability.

For example, a typical output looks like this

Started by GitHub push by xxx
[EnvInject] - Loading node environment variables.
Building remotely on xxx workspace xxxxxx
Triggering script=default
script=default appears cancelled: 1
Run script=default finished with : SUCCESS
Triggering script=post_build
Run script=post_build finished with : SUCCESS
setting commit status on Github for xxxxxxxxxxxx
Finished: SUCCESS

Clicking on 'script=default' or 'script=post_build' get you to the config's history rather than the output of the config for that run

The suggestion would be to add a link in the output to jump you directly the configs' output for that run

Since builds are kicked off by githook no builds in the queue should be viewed as duplicates.

Any ideas on how to prevent this? See details below
Null pointer exception:

java.lang.NullPointerException
    at com.groupon.jenkins.dynamic.buildconfiguration.template.DotCiTemplate.getMergedTemplate(DotCiTemplate.java:56)
    at com.groupon.jenkins.dynamic.buildconfiguration.EffectiveBuildConfigurationCalculator.calculateBuildConfiguration(EffectiveBuildConfigurationCalculator.java:47)
    at com.groupon.jenkins.dynamic.build.DynamicBuild$DynamicRunExecution.calculateBuildConfiguration(DynamicBuild.java:208)
    at com.groupon.jenkins.dynamic.build.DynamicBuild$DynamicRunExecution.doRun(DynamicBuild.java:180)
    at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:535)
    at hudson.model.Run.execute(Run.java:1732)
    at com.groupon.jenkins.dynamic.build.DynamicBuild.run(DynamicBuild.java:84)
    at hudson.model.ResourceController.execute(ResourceController.java:88)
    at hudson.model.Executor.run(Executor.java:234)

.ci.yml

environment:
  language: javascript
build:
  run:
    acceptance:  pybot --variable BROWSER:phantomjs robot/test/simple_test_suite.robot

The particular usecase is skipping builds when Readme.md changes

I receive the following exception when I click on the New Job Button:

javax.servlet.ServletException: java.lang.NoSuchMethodError: 
hudson.plugins.build_timeout.BuildTimeoutWrapper.<init>(IZZIILjava/lang/String;)V

I would appreciate any assistance.

Part of normal Jenkins functionality is the ability to schedule jobs to run on a timer, for example for daily job runs (helpful to make sure everything remains working even when code doesn't change). It would be very nice to embed this into .ci.yml files (hopefully is not too difficult)

Only param being set is BRANCH.

As discussed with you yesterday, it would be nice if there was a way we could have multiple DotCI scripts to do different things in the Github repo. At the moment we can do this by having multiple branches, but this requires that all changes are pushed into every branch to keep current, which is a pain.

1. Recently /job/a/job/file-transfer was created
2. Someone later reconfigured the name of the project to file-transfer_broken

The db count was one for both

• db.dotci_project.find({name:"file-transfer"}).count()
• db.dotci_project.find({name:"file-transfer_broken"}).count()

Note that neither project could be confirmDelete nor reconfigured back file-transfer because of this bad db state. A correction attempt was made by removing both entries from database but jenkins had to be restarted to pick up the alteration. It would be a nice if DotCi allowed flushing a particular project cache to avoid having to restart jenkins.

This ticket is about preventing project name reconfiguration if it's going to cause a bad db state.
Please present that field as plain text and not as a form entry field that can be altered during /configure

We have a build with the following .ci.yml configuration:

plugins:
- artifacts: '/screenshots//.'

to archive screenshots from failed browser tests.

However, if the browser tests pass, the screenshots directory is not present, and we would just like to do nothing, but DotCI treats this as a build failure, as shown below. We would like this to not be marked as a failure, minimally as an option in the .ci.yml (I don't know if there is any case where you would want it to fail if the files are not present, but as long as we can turn that off we're happy)

Triggering script=firefox-goodsbrowse-tests
Triggering script=chrome-goodsbrowse-tests
Triggering script=safari-goodsbrowse-tests
Triggering script=iexplorer8-goodsbrowse-tests
Triggering script=iexplorer9-goodsbrowse-tests
Triggering script=iexplorer10-goodsbrowse-tests
Triggering script=chrome-cart-tests
Run script=firefox-goodsbrowse-tests finished with : SUCCESS
Copying files :/screenshots//.
Run script=chrome-goodsbrowse-tests finished with : SUCCESS
Copying files :/screenshots//.
Run script=safari-goodsbrowse-tests finished with : SUCCESS
Copying files :/screenshots//.
Run script=iexplorer8-goodsbrowse-tests finished with : SUCCESS
Copying files :/screenshots//.
Run script=iexplorer9-goodsbrowse-tests finished with : SUCCESS
Copying files :/screenshots//.
Run script=iexplorer10-goodsbrowse-tests finished with : SUCCESS
Copying files :/screenshots//.
Run script=chrome-cart-tests finished with : SUCCESS
Copying files :/screenshots//.
Archiving artifacts
ERROR: No artifacts found that match the file pattern "/screenshots//.". Configuration error?
ERROR: ‘/screenshots//.’ doesn’t match anything: ‘’ exists but not ‘/screenshots//.’
Sending email notifications
setting commit status on Github for XXXX
Sending email notifications
Sending email to: [email protected]
Finished: FAILURE

javax.servlet.ServletException: org.apache.commons.jelly.JellyTagException: jar:file:/var/groupon/jenkins/war/WEB-INF/lib/jenkins-core-1.567.jar!/hudson/model/AbstractBuild/changes...: st:include No page found 'index.jelly' for class com.groupon.jenkins.dynamic.build.GithubChangeLogSet
at org.kohsuke.stapler.jelly.JellyFacet$1.dispatch(JellyFacet.java:103)
at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:728)
at org.kohsuke.stapler.Stapler.invoke(Stapler.java:858)

Right now they are stored indefinitely after the initial login.

I administrate an organization named Minichan containing one test project. When clicking on New DotCI Job I am only given the options to add projects that exist under my personal username (r04r).

Using a webhook needs that webhook url be accessible to github( and the whole world). Which is not ideal.

It marks build as fail as it can't find some artifacts.

This should NOT be a failure as some of these artifacts are only found when tests fails (screenshots artifacts for failed tests for example).

Could you enhance it so that it won't mark it as failure.? if there is already a way to do this, let me know.

ERROR: No artifacts found that match the file pattern "/screenshots//.,/target/surefire-reports/html//.,/target/surefire-reports/junitreports+//.,/target/surefire-reports/testRailReporter_results.xml". Configuration error?
ERROR: ‘/screenshots//.’ doesn’t match anything, but ‘/.’ does. Perhaps that’s what you mean?

Currently we support arbitrary naming of build execution

build:
  run:
     A:
       - do step A1..n
     B:
       - do step B1..n

We should support the same for packages ... to do away with the concept of language_versions. This will help support key differences between package dependency dependent on the language version rather than force to share the same commonality.
And this alteration does not interrupt the current inherited default use of language defined by https://github.com/groupon/DotCi/blob/master/src/main/resources/com/groupon/jenkins/buildconfiguration/base_yml/language

environment:
  language: ruby
  packages:
    A:
    - ruby-1.9.3_p484
    - some lib only required by ruby-1.9.x

    B:
     - ruby-2.x
     - some lib only required by ruby-2.xx

I've set up DotCI for a project that is hosted through a GitHub Enterprise (GHE) instance. After a certain period on the order of an hour or so, jobs will fail to be kicked off, manually kicked off jobs will hang, and the jenkins log shows 401 responses to API requests made to our GHE instance.

The only way I found to fix this is to delete the job and recreate it. It's a pretty easy solution, but it obviously breaks the whole CI aspect of DotCI :).

devA wants to merge 1 commit into relevance:master from devA:RELEVANCE-441

During

$  git clone  xxx.git .
Cloning into ....
$  git fetch origin +refs/pull/266/merge:
fatal: Couldn't find remote ref refs/pull/266/merge
fatal: The remote end hung up unexpectedly
Build step 'Execute shell' marked build as failure

The problem was resolved after the developer rebased from origin/master and resolved conflicts

Can DotCi capture errors during this step to provide the actual source of the problem and/or possible solutions to explore.

This would provide equivalent behavior to language_versions for testing across multiple versions of languages.

If I have a job http://jenkins/myJob, and it has a set of matrix jobs, m1, m2 and m3.

In the .ci.yml, the value of $BUILD_URL is http://jenkins/myJob/xx/, however when querying that variable from within the matrix job, the value becomes http://jenkins/myJob//scrip=m1/xx/.

So we need to have another variable to retrieve that URL of the master job (the http://jenkins/myJob/xx/) from within the matrix jobs

The reason that is needed, is that, within the test results reports we want to add hyperlink to artifacts (failed screenshots), and the artifacts are accessible only through the main job URL and not the matrix job's URL.

Currently the only way for DotCi to initiate a downstream build is inconvenient.
It involves remotely trigger via url by passing a bunch of build parameters ...

Use the following URL to trigger build remotely: 
JENKINS_URL/job/xxx/build?token=TOKEN_NAME or /buildWithParameters?token=TOKEN_NAME 
Optionally append &cause=Cause+Text to provide text that will be included in the recorded build cause.

It would be nice if there was a new DotCi syntax for initiating downstream build(s).
This would resolve cases where the code repository does not contain the deployment procedures, browser testing framework, etc.

build:
  [ before | run | after ]: 
    ...
    <% if( DOTCI_BRANCH =~ /deploy_request*/) { %>
    - downstream:
        deploy_foo:
           wait: true
           params:
             sha: $SHA
             branch: $DOTCI_BRANCH 
             repo: $GIT_URL
             environment: load_testing
        test_bar:
           wait: false
           params:
             environment: load_testing
             service: foo
    <% } %>
    ...  

I'm not sure how the syntax should be structured for sequential downstream builds versus parallel downstream builds.

Also since the above settings imply that not every branch would not invoke a downstream build.
Then its representation should only appear when viewing that build number job. It would be nice to if some downstream build status icon would also appear in the left side bar by that build number.

Everything works great using a Dockerfile and the following .ci.yml

build:
run: cd /app && npm test

When I add an after section for test

build:
run: cd /app && npm test
after: echo "hello"

The build get stuck w/ the following message in the console:

Started by GitHub push by Manchul
Building remotely on test (docker) in workspace /root/workspace/Manchul/travis_sample
Triggering script=default

** vs successful builds w/o after section has this line instead
[travis_sample] $ /bin/bash -le /tmp/hudson2719372970606347165.sh

I'm basically trying to see how the after section works and where... so that I can retag and push the successful docker images to our private registry. (If you have a suggestion on doing so would be great as well... or your deployment work flow from DotCi, kinda new to Jenkins)

Thanks

When using github authorizations webhooks are not properly kicked off unless the Grant READ permissions for Anonymous Users is checked.

The call is seen by dot-ci, as seen in the logs, but the job is not kicked off:

Jun 13, 2014 6:07:55 PM INFO com.groupon.jenkins.github.GithubWebhook processGitHubPayload
Received POST by r04r
Jun 13, 2014 6:07:56 PM INFO com.groupon.jenkins.github.GithubWebhook processGitHubPayload
Received kicking off build for https://github.com/r04r/TinyBBS2

See the below portions of code that were identified in #19:

• https://github.com/groupon/DotCi/blob/master/src/main/java/com/groupon/jenkins/github/GithubWebhook.java#L78
• https://github.com/jenkinsci/jenkins/blob/master/core/src/main/java/hudson/model/Items.java#L324

Anonymous read access should not be necessary in order for github webhooks to work.

I believe it currently always uses the latest stable version, which means any bugs in new releases (such as technomancy/leiningen#1555) come out of nowhere and can be difficult to workaround.

It would be nice if the "Build with Parameters" page queried Github for the available branches and displayed them in a drop down or list box to select, instead of having to know the exact name of the branch you want to run and having to type it in

use '--' for multi character flags and single '-' for single character flag.

Also make sure to delete previous webhook before adding a new one.

Demo from Setup-Cloud-CI-in-15-minutes-with-Jenkins,-DotCi,-Docker-and-DigitalOcean

I am getting error "install_packages: command not found" when I try a Java .ci.yml. Any idea on how to fix this?

.ci.yml file

environment:
  language: java
  language_versions: 1.6.0_24

build:
  info:
    - echo "Env:" ; env
    - java -version
  run:  gradlew build 

Error

Started by GitHub push by paulkling
Building remotely on slave (docker) in workspace /root/workspace/paulkling/gs-spring-boot
[gs-spring-boot] $ /bin/bash -le /tmp/hudson8788943577050406139.sh
stdin: is not a tty
$  install_packages java-1.6.0_24
/tmp/hudson8788943577050406139.sh: line 3: install_packages: command not found
Build step 'Execute shell' marked build as failure
Sending  email notifications

Need DotCi to enable webhook for tags to execute builds.
This will offer development the ability to configuring different behaviors for tags like nexus publish releases if a certain naming convention else ignore.

The first step would be to enable tags by undoing

• DotCi/src/test/java/com/groupon/jenkins/github/PayloadTest.java

  Lines 152 to 155 in 7f1b146

  	public void should_not_build_tags() throws IOException {
  	Payload payload = new Payload(readFile("push_tags.json"));
  	Assert.assertFalse(payload.needsBuild());
  	}

• https://github.com/groupon/DotCi/blob/7f1b14635fc596da66135234541264fd1fa9d803/src/main/java/com/groupon/jenkins/github/Payload.java#L104-#L106

The second step would be to supply a variable to be used that represents that tag.

I also propose an additional third step that involves defining an entire new build section for execution of code when a tag. This is a cleaner way to avoid git tag invocation of a build:run ... and vice-versa. Otherwise every developer will have to re-modify their existing .ci.yml to distinguish the new behavior support for git tag.

build:
  run:
     - ....
  tag:
    <% if (DOTCI_TAG =~ /some_naming_convention-SNAPSHOT/ %>
    - ... nexus publish snapshot
    <% else if (DOTCI_TAG =~ /some_naming_convention/ %>
    - ... nexus publish release
    <% } else { %>
    - skip # ignore some developer tag for their own sanity to mark their code
    <% } %>