Comments (10)
Remote building is now supported in the unstable branch. If enabled, Colmena will copy system profile derivations to the target nodes and initiate the builds there. There is no need to configure designated builders beforehand.
This avoids copying back the build results like the native distributed build feature in Nix, and can hopefully make it easier to use Colmena on bandwidth-constrained machines as well as on macOS which is now a supported platform.
It can be enabled by:
- Setting
deployment.buildOnTarget = true;
in the node configuration, or - Setting
--build-on-target
on the command line. This overrides alldeployment.buildOnTarget
configurations for this run. You can also temporarily disable the feature for all nodes with--no-build-on-target
.
from colmena.
A possible improvement over the current implementation: make use of a local --eval-store
to avoid copying the derivations.
from colmena.
This approach is also interesting. I basically uses your CI to pull store paths: https://determinate.systems/posts/hydra-deployment-source-of-truth
This way one does not need to re-evaluate on the target. In case of hydra evaluation happens in parallel.
from colmena.
This is what I am using right now instead: https://github.com/Mic92/doctor-cluster-config/blob/master/fabfile.py
from colmena.
I know that some machines are not powerful enough to evaluate locally.
However in this case it is still faster to login to a faster machine and run nixos-rebuild
there:
$ ssh strong-machine nixos-rebuild switch --flake ${targetPath}/dotfiles#weakmachine --build-host localhost --target-host root@weekmachine
from colmena.
I was thinking about remote eval as well. Currently, most parts of the deployment process are already host-agnostic with implementations for "local" and "SSH" hosts abstracted out, and we can hopefully add evaluation to that list as well.
The problem with remote eval is that we need to define a "boundary" for the configurations that will be copied to the remote host. This is simple with Flake URIs, but then only remote flakes (github:
, https://
, but not path:
) will work.
from colmena.
I would do the following: Copy all flake inputs with nix copy
to the evaluation target and than do rsync
on the main flake to a fixed directory. Why rsync
? If you do many small changes than it is a lot faster to use rsync
than nix copy
.
It could sync the main flake to /var/lib/colmena/
. This also has the advantage that one can run nixos-rebuild --flake /var/lib/colmena
on the host could be used without having to rely on colmena
. This is quite useful to rescue a machine that no longer has network access or you even have to repair it with nixos-install
from colmena.
my remote build script looks like this:
#!/usr/bin/env bash
set -x
[ $# -ne 1 ] && echo build-remotely BUILDHOST && exit 1
target_configuration=$(hostname)
remote_builder=$1
export NIX_SSHOPTS="-oStrictHostKeyChecking=no"
drv=$(nix --pure-eval eval --raw ".#nixosConfigurations.${target_configuration}.config.system.build.toplevel.drvPath")
nix copy -s --derivation $drv --to ssh://$remote_builder
remote_result=$(ssh $remote_builder nix-store -r $drv)
nix copy --no-check-sigs --from ssh://$remote_builder $remote_result
in colmena's case, the nix copy destination would obviously be the target host.
I'm just now checking out colmena. If it suits my usecase nicely, I might try to implement remote building support.
from colmena.
I'm just now checking out colmena. If it suits my usecase nicely, I might try to implement remote building support.
Remote building is already supported by Colmena, which simply uses Nix's native distributed building functionality. You can have a file like:
ssh://builder@host aarch64-linux /path/to/your/ssh.key 16 2 kvm,big-parallel
and then specify it as the meta.machinesFile
in your config. Colmena will then pass the contents of the file in --builders
to Nix (#21). If you want, you can also set them globally in nix.buildMachines
. Nix will do the same thing your script would (copying the derivation as well as the input closure to a remote machine then copying the results back), but with support for multiple builders and basic scheduling.
Remote evaluation, however, is another thing which needs to be investigated.
from colmena.
@zhaofengli It would still be great to be able to rebuild on each specific host (instead of one designated builder). Pretty sure none of the other Nix deployment tools can do this either.
from colmena.
Related Issues (20)
- Feature Request : healthchecks HOT 6
- Fails when using a security key that requires presence verification
- User presence prompt not showing
- streaming evaluator with non-flake deployment and relative path broken
- Use of `vulnix` with colmena HOT 1
- Restart key service unit on change HOT 1
- Deploying a single user profile?
- `colmena apply-local --sudo` succeeds but with exit code 101 HOT 2
- Rebooting should not activate then reboot HOT 3
- --build-on-target fails on macos HOT 3
- Feature request: repl integration HOT 3
- Key services don't work correctly HOT 2
- Feature Request - storing gcroot under /nix/var/nix/ HOT 3
- allow sudo password entry HOT 1
- Streaming evaluator returns exit code 0 even if evaluation fails HOT 1
- Uploading keys is erroring and I don't know why HOT 2
- Allow deploying nix-darwin machines. HOT 1
- Add an option to upload the configuration but don't apply it. HOT 2
- Integration with nix-output-monitor when available HOT 1
- Integration with nvd when available HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from colmena.