Automated builds and testing about docker-stacks HOT 21 CLOSED

So, we have been doing this for some time. I think we can close this safely.

from docker-stacks.

Last I checked, it was hard to run your own docker on Travis, but that may not be true.

from docker-stacks.

I think that limitation still exists. I'll do some research into alternatives. Worst case, I figure I can roll something that listens for Docker Hub web hook post-build, pulls the image, runs it, and just makes sure the primary process comes up properly.

from docker-stacks.

As of Aug 19th, travis now supports custom docker containers.

http://blog.travis-ci.com/2015-08-19-using-docker-on-travis-ci/

from docker-stacks.

Circle CI supports Docker builds as well.

from docker-stacks.

I believe @jakirkham has had some success testing docker builds with CircleCI on the jupyter/notebook repo.

from docker-stacks.

Yes, personally, I would also vote for Circle CI. Though Travis CI does support docker it does so in a sudo required VM, which seems to take significantly longer to startup. A benefit of Circle CI is they allow one to ssh into the machine. I have used this feature and it is trivial to do. I can take a look at porting my existing stuff over to here with some modifications, but I won't be able to do so right away.

from docker-stacks.

I started looking at Circle CI a bit. The free account level allows a max of 1,500 build minutes (25 hrs) a month. I've seen a change to minimal-notebook result in rebuilds of all the substacks take over an hour. Pushing to the result images to Docker Hub takes even longer. I'm not sure if the stack definitions will change often or not going forward, but we'll need to keep the build limit in mind.

from docker-stacks.

I think one thing that will help keep build times reasonable would be to skip building components that aren't being changed.

An orthogonal issue is user wait time, which could probably be addressed by using a build matrix. However, this proves tricky as so many Dockerfiles here depend on the minimal-notebook. So, making changes to it would increase all build times.

from docker-stacks.

As this came back up again, I thought about it a little more. One thing that may help us here is preserving the docker build cache from session to session. It should help us keep the build times reasonable for the most part. Also, we can avoid trying to invent some more complex build system. Most of them will just reuse cache if they can.

We would want to make sure that base images like debian:jessie get repulled at startup by either deleting them or pulling them in ourselves. If they haven't change, it will make no difference. If they have, then it will obviously go much slower.

from docker-stacks.

Another problem to consider here is what to do when pushes to DH fail. Does CircleCI support retries when that happens? It's pretty typical that I get 500 errors when doing the manual pushes at present. Retry, retry, retry until everything is pushed has been the sad solution.

from docker-stacks.

Well, at a minimum, we can ssh into CircleCI when it fails and repush. So, that is definitely a plus compared to the other services where we simply have to rerun the whole build.

Though we could certainly have our own retry N times logic. We should probably have this anyways. Added in this issue ( #131 ).

from docker-stacks.

So, I gave a try at using Travis CI as it seemed we might be leaning in that direction. Surprisingly pulling in all the images did not put us over on memory usage. So, this is something worth keeping in mind. Also, it helped find some bugs so that was nice.

However, it looks like the build times are a bit long. We could probably compare max build times and resource limitations of other services, but I expect we will likely have the same issue. Instead I suggest that we think a bit about how to either cut down the build time or restructure the build process so as to fit within these limits.

One idea that might work would be to enable automated builds for the minimal base images. Namely, minimal-kernel and minimal-notebook. These are images that only depend on debian:jessie so don't require any other images in here. In the case of minimal-notebook, this is the base image for the rest of the stack. Alternatively, we could refactor out some apt-get steps and such from minimal-notebook and possibly minimal-kernel into a new internal image that is not intended for external consumption, but does help us reduce our build times. Though other ideas about how to solve this are certainly welcome.

from docker-stacks.

One idea that might work would be to enable automated builds for the minimal base images. Namely, minimal-kernel and minimal-notebook.

See downsides in the PR, namely, no way to tag per commit without manual setup on Docker Hub each time. The git commit = docker tag system has been working really well, IMHO. I'd hate to lose it.

It would be pretty trivial to stand up a small webhook receiver on the existing build VM and have it kick of the make release (in a loop of need be for the 500 errors). It would effectively accomplish what we've been doing manually so far which is tried and true.

from docker-stacks.

The VM doesn't sound like a bad idea. Though it would be really nice to have status updates for PRs. The problem right now is the builds are pretty long and so we are checking manually or discovering issues when we release.

It would be really nice to have the check occur before the merge and possibly let the contributor know about it. Do we have the bandwidth to do that with the VM?

from docker-stacks.

Though it would be really nice to have status updates for PRs.

Looks like GitHub can inform the webhook of a variety of events, PRs and changes to PRs included:

https://developer.github.com/webhooks/#events

It gets tricky on the VM if we start having it handle lots of PRs in parallel or in a queue. The main benefit of building merges only at the moment is that we leave all the old recent build on disk to get the caching speed. If we're building all sorts of PRs there too, we'll probably lose that benefit. I was thinking about two VMs, a staging one for PRs and another for merge builds, but even that separation will likely need some kind of image cleanup after the PR builds to avoid weird cross-build interactions which obviates the caching benefit.

It would be really nice to have the check occur before the merge

Thought exercise: what if we flipped the manual process to building PRs on the build VM instead of master? We'd go to the VM, pull the PR branch, run the equivalent of make release-all without the push step, and maybe even run a little smoke test suite (e.g., does a container per image start properly?). If everything works, then we'd go back to github, merge the PR to master, come back to the VM, pull master, and run make release-all, this time with the push to github. Nothing would rebuild because the PR just build cleanly.

If the PR did not build cleanly, then we might run make clean-last or some such to remove that build from the local Docker deamon in prep for the next PR.

I'm not suggesting we do all that manually. I'm just thinking through the necessary steps that get the benefits we want before asking: could we automate this flow?

from docker-stacks.

So we currently have a primitive version of CI with Travis. It doesn't do deployment. Also, it sometimes runs up against resource limitations with Travis. That all being said, we can get a better idea of how and when it works by trying it out.

from docker-stacks.

Yep. I say we let it be for a while and see what kinds of things work on travis and don't. Then we'll have more data about how to more forward with it.

from docker-stacks.

Actually, one thing we might want to consider during this "wait and see" period is to run the simplest possible test suite over the images build on travis:

for each container image:
  docker start that image with -d -P
  wait
  wget the exposed notebook port
  assert something from the notebook UI was retrieved

from docker-stacks.

Sounds like a good idea. Are you thinking about having this be a step in the Makefile?

from docker-stacks.

Yea. make test sounds reasonable so we can run it locally or on travis.

from docker-stacks.