Generate builds for PR's about ember-dev HOT 29 CLOSED

After reviewing this a bit more, it looks like Travis' secure env vars are not available to PR's (makes total sense).

Anyone see a way around this problem?

from ember-dev.

this is the same issue we are facing with saucelabs stuff.

@hjdivad may have some thoughts, as he spent some time on this.

from ember-dev.

@rjackson http://support.saucelabs.com/entries/25614798-How-can-we-set-up-an-open-source-account-that-runs-tests-on-people-s-pull-requests-

tl;dr sauce is working with travis on supporting secure environment variables for PRs for this exact use case.

To clarify, keys are available to PRs: what matters is that you have write access to the repo. So for a small repo with only a handful of contributors all with commit this system can sort of work[1].

[1] - There is a further caveat with sauce which is that any given account's key can only be used in one tunnel at a time. So you can't have two parallel builds using the same key to access sauce via sauce-connect. However, you can somewhat get around this by creating more keys with sub accounts.

from ember-dev.

@rjackson okay maybe I should respond in a way more relevant to what you're actually trying to do.

So the thing travis is working on to get saucelabs working may be useful here: the same token system would hopefully be usable beyond saucelabs.

What's the encrypted env you need for this feature? An s3 token?

from ember-dev.

@hjdivad - Thanks for your help on this!

In answer to your last question: I would like to create builds for each PR into an S3 bucket. This would mean I need access to the AWS Secret Token and AWS Access Key (I think that is what they call them). Both of which reside in the secure env vars for the host project (ember and ember-data).

Will read through the Sauce Labs link you posted tonight...

from ember-dev.

@rjackson makes a ton of sense. So this is basically going to be the same problem saucelabs has. Hopefully the token-based approach travis is working on will not be sauce-specific and can be used here.

Okay I have a terrible crazy idea for a workaround of the hackiest variety.

If we had some way of generating a publicly readable build artefact, then a separate ec2 instance could poll travis builds and upload to s3.

If travis doesn't have native support for public build artefacts we can simulate by outputting the built js to the log, which is public.

thoughts?

from ember-dev.

@hjdivad - Definitely could work! At that point though we might be better suited to just implement our own github service hooks for our needs. We would also likely be opening ourselves up to the same sort of security exposure that the Travis folks are worried about.

Another option in this scenario would be to make the S3 bucket publicly writeable from Travis's IP's via a custom S3 bucket policy.

from ember-dev.

@rjackson yes of course. Having a separate github hook makes way more sense than going too far down the hackety hack.

I'm not familiar enough with S3 to know how you avoid evil.com from running up your S3 bill arbitrarily with a publicly writable bucket.

from ember-dev.

@rjackson writing a custom github service hook is not too difficult. I think it would be worth it: this is a great feature and would really help people easily swap out ember implementations in their jsbins. You can do this now with an arbitrary jsbin by changing the URL to localhost, but that's obviously not shareable.

from ember-dev.

RE: S3 bucket policy

Well, I'm reasonably sure that we can limit the write access to specific IP's, but I'm not sure that really gives us any security to speak of (since thousands of projects are hosted on Travis and any one of them could write whatever they want...).

RE: custom GitHub hook

I am definitely down for this rabbit hole (assuming that the powers that be agree). The main issues that we need to understand and deal with are:

1. The potential for any PR submitting person to host their own javascript in our S3 bucket
2. Potentially have access to any/all of our code for the hook (including the ENV vars for pushing to S3).

I think that the first isn't a HUGE issue as long as we don't use the same bucket as assets that we inherently trust, but the second issue is a tougher nut to crack.

from ember-dev.

Just to illustrate what I mean in #2 above:

What if someone changed their Rakefile to have rake test call some other arbitrary command that they control? This command could send all of our ENV variables to a paste bin (exactly like what happened when Rubygems was comprimised a while back).

We might be able to work around this issue, but disallowing any submitted PR's code to be executed in our hook (the tests could still be done by Travis in a secure way). If we use a known good checkout of EmberDev to directly generate the assets we would probably be OK.

Thoughts?

from ember-dev.

@rjackson I was literally in the middle of writing something very similar to what you suggest

from ember-dev.

@rjackson so basically enable this only for PRs that only touch files in packages

from ember-dev.

As an aside, if we can figure this out we can probably circumvent the SauceLabs issue also...

from ember-dev.

@rjackson in fact, maybe the thing to do is not to check out the PR directly, but to only checkout its changes to packages.

from ember-dev.

@rjackson after all, it's not like it matters to build a pr-specific ember.js for PRs that only modify docs and, as you pointed out above, it's not really safe to build a pr that modifies Rakefile, Gemfile, &c.

from ember-dev.

@rjackson what do you think about

1. whitelisting paths for building a PR-specific ember.js
2. for that whitelist to initially just be packages

Do you see problems with this approach?

from ember-dev.

@hjdivad - I think that we can use a known good repo as a base, and then over write the packages directory from the PR repo. That way we know that no code will be executed by the PR (just rake-pipeline filters and such that do not execute the JS).

from ember-dev.

@hjdivad - You must be a mind reader :)

Yes, I think that should cover us fairly well. I can start on the general work, but I think that we need the powers that be to weigh in with their Yea/Nay...

from ember-dev.

Powers that be you say?

cc @wagenet

from ember-dev.

I'll begin playing around with the basic idea once I have finished a few more integration tests for the new multi-branch testing...

from ember-dev.

After more discussion in #ember-dev with @hjdivad I believe that we could also make the SauceLabs integration work for PR's using this same technique (we would have to queue the requests to only run a certain number of SauceLabs workers at once).

from ember-dev.

So it sounds like we'd only run cross-browser tests and generate builds for PRs that only touch packages. And, to be extra safe, we'd just copy over the pacakges directory into an existing known build?

This seems like it should be fine. If someone submits a tooling PR we likely don't need a new build or cross-browser tests. The standard Travis run should be sufficient to make sure that the test runner and general build process still work.

from ember-dev.

Ok, sounds good. I'll try to take a few minutes to layout the basic architecture and research how the Github integration stuff works.

Will update back here with more details in a few days...

from ember-dev.

👍

from ember-dev.

@rjackson see https://github.com/raggi/github_post_receive_server/ as a starting point. It has worked well for me in the past.

from ember-dev.

We also probably couldn't run phantomjs, since people might be able to figure out a way to use phantomjs to log out the environment.

from ember-dev.

@wycats - Good point.

I'm thinking to run the existing tests via Travis (without secure ENV vars) exactly as it is currently setup.

Then ALSO have this other github receiver to do the builds (possibly to a new bucket) and SauceLabs runs, then report the status of this runs (and the URL for the build) into the PR.

Make sense?

from ember-dev.

Seems fine 👍

from ember-dev.