Decide on main CI system about bitbox-base HOT 8 CLOSED

@Tomasvrba, please take a look at this issue so that we can discuss how to proceed

from bitbox-base.

For the purposes of this issue, I'll mention that I tried out setting up integration with the Shift Gitlab instance:

1. I added a bitbox-base project under my own user on the instance
2. configuration locally in the repo was trivial, see hkjn@56962cd
3. the only additional step required was configuring a specific test runner for the project, i.e an agent that can accept tasks and perform builds

With those steps, the main make rule for the master...hkjn:20190621-add-gitlab branch is off to the races. I'll comment here once it finishes. If we do decide to deploy CI to the Shift Gitlab instance, we have some open questions around:

1. how often to produce builds (every commit to every branch, or only master, or ..)
2. whether to use separate or shared test runners
3. how to best make build results accessible to external contributors

Edit: The first attempt to run the CI on Shift's Gitlab ran into this issue, where the job is terminated due to the log file becoming too large, after ~43 min of runtime. We've tried tweaking the config.toml for the job runner to allow for larger log output. The next issue likely would be that the timeout of 60min would be hit. I changed the Settings > CI / CD > General pipelines > Timeout setting to 6h for this test runner to work around that.

Edit 2: The next issue we hit was that some of the files generated in the build/ directory end up being owned by the superuser (the default inside Docker containers, which gets shared to the host system via the volume mount in our docker run commands). On subsequent builds, the Gitlab test runner can't remove these files, so the build immediately fails. Unfortunately, these steps happen before the Gitlab test runner even starts the commands in the .gitlab-ci.yml config, so there's no way to remove the files there. It seems that a workaround to this would be to run the Docker commands inside the container as the same unprivileged user as on the system, to preserve the file ownership.

from bitbox-base.

As a datapoint, we found that the experimental CircleCI integration could successfully build Armbian OS images (unlike Travis). It took 2h and 54min to finish the build.

from bitbox-base.

As another datapoint, on the Shift Gitlab instance we managed to produce an Armbian image successfully, after 19m and 39s:

b4ec202a2170eb3c23e1dfec2f103dc1a64dd3e0f03b469d69232cbb4990da8b  output/images/Armbian_5.89_Rockpro64_Debian_stretch_default_4.4.180.img

I'd suggest we address the issues mentioned earlier and move in the direction of using Shift's Gitlab exclusively, unless there's reasons to do something else.

from bitbox-base.

The experimental Gitlab build integration I mentioned in my last comment in master...hkjn:20190621-add-gitlab branch mostly works fine, and gives superior performance than circleci.com (not surprisingly, since that is a free service while Gitlab is a self-hosted CI system using a decently powerful VM).

The reason I say it "mostly" works fine is that there is an issue where the Dockerized build runs as the super-user (id -u is 0) inside the container, and the bind mount of the build/ directory ends up being owned by the super-user on the host as well. This makes subsequent runs fail, since the non-privileged Gitlab test runner attempts to clean up files that are not part of the repo. Since this step occurs before Gitlab even starts running the build scripts we provide, it's also hard to work around the issue ourselves.

The best path forward would be to run using the same uid and gid inside the build container as on the host. That both achieves the specific goal of allowing Gitlab to build our project without tripping over itself on subsequent builds, and also prevents surprising developers when their unprivileged user doesn't have access to delete files under build/. I've started on this work, and it doesn't look too difficult.

from bitbox-base.

it doesn't look to [sic] difficult.

Famous last words..

I've sent out #98, which does achieve the result that build/ is owned by the non-privileged user, however in order to achieve this for users who (wisely!) require sudo to run docker commands, the host user id needs to be explicitly set:

$ sudo make docker-build-go BUILDER_UID=$(id -u)

from bitbox-base.

There were further issues with 0e5efd5 (the commit added in #98) that we only noticed post-merge, which caused the sudo make command to fail if the BUILDER_UID parameter wasn't specified.

I've sent #99 to revert 0e5efd5 until we can figure out a way to achieve a method to run builds while maintaining the ownership of the build/ directory for the non-privileged user, as I described in an earlier comment:

The reason I say it "mostly" works fine is that there is an issue where the Dockerized build runs as the super-user (id -u is 0) inside the container, and the bind mount of the build/ directory ends up being owned by the super-user on the host as well. This makes subsequent runs fail, since the non-privileged Gitlab test runner attempts to clean up files that are not part of the repo. Since this step occurs before Gitlab even starts running the build scripts we provide, it's also hard to work around the issue ourselves.

from bitbox-base.

Closing due to inactivity. Not a pressing issue, can take this up next year.

from bitbox-base.