Code Monkey home page Code Monkey logo

devour-flake's Introduction

devour-flake

Devour all outputs in a flake.

Why?

Running nix build .#a .#b ... .#z on a flake with that many outputs can be rather slow if the evaluation of those packages are non-trivial, as is often the case when using IFD. Nix evaluates separately for each of the argument (as if not using the eval cache).

To workaround this, we create a "consumer" flake that will depend on all outputs in the given input flake, and then run nix build on the consumer flake, which will then evaluate the input flake's packages only once.

devour-flake currently detects the following flake outputs:

Type Output Key
Standard flake outputs packages, apps, checks, devShells
NixOS nixosConfigurations.*
nix-darwin darwinConfigurations.*
home-manager legacyPackages.${system}.homeConfigurations.*

Usage

To build all of the nammayatri flake outputs for example:

nix build github:srid/devour-flake \
  -L --no-link --print-out-paths \
  --override-input flake github:nammayatri/nammayatri

Pipe this to | cachix push <name> to push all flake outputs to cachix!

Nix app

Add this repo as a non-flake input:

{
  inputs = {
    devour-flake.url = "github:srid/devour-flake";
    devour-flake.flake = false;
  };
}

Then, add an overlay entry to your nixpkgs:

{
  devour-flake = self.callPackage inputs.devour-flake { };
}

Use pkgs.devour-flake to get a convenient executable that will devour the given flake and spit out the out paths. You can then use this in CI to build all outputs of a flake.

nix-build-all

Note

See also: nixci, an improved version of nix-build-all.

For a CI-friendly command that builds all flake outputs, in addition to checking for flake.lock consistency, use:

{ pkgs, ... }:

pkgs.writeShellApplication {
  name = "nix-build-all";
  runtimeInputs = [
    pkgs.nix
    pkgs.devour-flake
  ];
  text = ''
    # Make sure that flake.lock is sync
    nix flake lock --no-update-lock-file

    # Do a full nix build (all outputs)
    # This uses https://github.com/srid/devour-flake
    devour-flake . "$@"
  '';
}

Who uses it

devour-flake's People

Contributors

ipetkov avatar nbraud avatar srid avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar

Forkers

yabosa nbraud

devour-flake's Issues

Allow supporting nix-build-uncached

The current devour-flake build script is hostile to performing incremental CI builds because building the top level derivation requires fully downloading all derivations from the flake (if there are many such derivations or their artifacts are large, the CI is going to waste cycles and bandwidth downloading artifacts from the cache).

A better approach would be for the script to still evaluate the all-encompassing-derivation but instead of building it directly, it should output only the input derivations. Then the caller can do whatever they want with those actual derivations (pass them to nix build or nix-build-uncached etc.).

For example:

FLAKE="$1"
shift 1 || true

nix derivation show ${./.}#default \
  "$@" \
  --override-input flake "$FLAKE" \ |
  jq 'to_entries[].value.inputDrvs | to_entries[].key'

Missing dependency on findutils

While attempting to use devour-flake in CI, I ran into the following bug:

/nix/store/yhmxdqspqchpd8cb15gwc8ybpiybh5bz-devour-flake/bin/devour-flake: line 15: xargs: command not found
warning: not writing modified lock file of flake 'path:/nix/store/cc4dp3h7nck0w6xz0jfsbchyklfmblry-p8vfkmfh43yy8lmz1r65dr9yq3ql7cd7-source':
• Added input 'flake':
    'path:/nix/store/lklg6iihnwkxfxcwj8ik1a145z4296cc-source?lastModified=0&narHash=sha256-M%2BBd3q9raBWjZsVxWzEKusGlKcQAqvp4zisplb%2BQwqM%3D' (1970-01-01)
• Added input 'flake/devour-flake':
    'github:srid/devour-flake/30a34036b29b0d12989ef6c8be77aa949d85aef5' (2023-09-07)
• Added input 'flake/flake-utils':
    'github:numtide/flake-utils/ff7b65b44d01cf9ba6a71320833626af21126384' (2023-09-12)
• Added input 'flake/flake-utils/systems':
    'github:nix-systems/default/da67096a3b9bf56a91d16901293e51ba5b49a27e' (2023-04-09)
• Added input 'flake/nixpkgs':
    'github:NixOS/nixpkgs/90e85bc7c1a6fc0760a94ace129d3a1c61c3d035' (2023-10-29)
error: writing to file: Broken pipe

Here's the full CI run

Error: cannot coerce a set to a string

I'm trying to run nix run nixpkgs#nixci on my flake, but I'm encountering this error from devour-flake.

🍏 .
🐚 nix --extra-experimental-features 'nix-command flakes' eval '.#nixci.default' --json️
🍎 default.<root>
🐚 nix --extra-experimental-features 'nix-command flakes' flake lock --no-update-lock-file .️
🐚 nix --extra-experimental-features 'nix-command flakes' build '/nix/store/p8vfkmfh43yy8lmz1r65dr9yq3ql7cd7-source#default' -L --no-link --print-out-paths --override-input flake . --refresh -j auto️
error:
       … while calling the 'derivationStrict' builtin

         at /derivation-internal.nix:9:12:

            8|
            9|   strict = derivationStrict drvAttrs;
             |            ^
           10|

       … while evaluating derivation 'devour-output'
         whose name attribute is located at /nix/store/qjqzr7905xxv3rv70z7mkpp06lalnzzb-source/pkgs/stdenv/generic/make-derivation.nix:300:7

       … while evaluating attribute 'text' of derivation 'devour-output'

         at /nix/store/qjqzr7905xxv3rv70z7mkpp06lalnzzb-source/pkgs/build-support/trivial-builders/default.nix:148:16:

          147|     runCommand name
          148|       { inherit text executable checkPhase allowSubstitutes preferLocalBuild;
             |                ^
          149|         passAsFile = [ "text" ];

       error: cannot coerce a set to a string
Error: devour-flake failed to run (exited: 1)

It seems it's happening because one of my outputs is actually of the form:

      packages.pythonPkgs = {
        package1 = pkgs.callPackage ./packages/mypackage1/. {};
        package2 = pkgs.callPackage ./packages/mypackage2/. {};
      };

Is there a way to make that structure work with nixci/devour-output? I'm would like to clearly separate python packages in a pythonPkgs key. Is my way of handling this non standard maybe? 🤔

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.