Doesn't watch paths through IFD + filterSource (unsure if proper diagnosis) about lorri HOT 10 OPEN

To get around such problems, I'd love to have a lorri reload for manual (or externally triggered) reloading.

from lorri.

I’ve seen this, too. I think we’ll need some additional setup for properly recognizing IFD paths. I had started a minimal repro somewhere, will push it once it’s ready.

from lorri.

Here’s an old issue I just migrated that talks about this: #31

from lorri.

I tried to get around this issue for developing a cabal-based project, but ran into issues. Documenting this here in the hope it saves someone some time.

Haskell package environment sucks in the source
I ran cabal2nix by hand but lorri watched my whole directory. This surprised me, since the environment derived from the shell.nix produced by cabal2nix --shell . > shell.nix should not contain the source: it should only contain the dependencies of the package under development. This turns out to be a known issue: NixOS/nixpkgs#51079.

Haskell package environment not directly overridable
Since cabal2nix produces src = ./., I thought my issue might be due to #58. To get around it, I overrode the source with the following default.nix:

with import <nixpkgs> {};

let
  drv = haskellPackages.callPackage (import ./package.nix) { };
  drv' = drv.overrideAttrs (attrs: rec {
      src = nix-gitignore.gitignoreSource [".gitignore" "*.nix" "*.cabal" ".envrc"] ./.;
  })
in
  drv'.env

created the simple shell.nix:

import ./default.nix

and generated package.nix with cabal2nix . > package.nix. But this did not solve my problem. lorri was still watching the whole directory. Digging in the code, I think the issue is with the use of shellFor to generate the haskell package derivations env attribute introduced in NixOS/nixpkgs@5067773e39a. To be able to use shellFor, the fix operator is used which prevents the derivation from being truly overridable: overriding the source takes effect in the derivation but not in its env. The following default.nix is not ideal but fixes the issue:

with import <nixpkgs> {};

let
  drv = haskellPackages.callPackage (import ./package.nix) { };
  drv' = drv.overrideAttrs (attrs: rec {
      src = nix-gitignore.gitignoreSource [".gitignore" "*.nix" "*.cabal" ".envrc"] ./.;
  });
in
  haskellPackages.shellFor { packages = _: [drv']; }

from lorri.

You don't even need IFD to reproduce this issue. Just running the imported derivation with a filtered source is sufficient:

let
  pkgs = import <nixpkgs> {};
  deps = pkgs.stdenv.mkDerivation {
    name = "deps";
    src = builtins.filterSource (path: type: baseNameOf path == "example") ./.;
    installPhase = ''
      echo "pkgs: with pkgs; [ $(cat example) ]" > $out
    '';
  };
in deps

Running the code that lorri uses internally to gather sources to watch shows that no reference to example is emmitted

$ nix-build -vv --no-out-link --expr "import ./logged-evaluation.nix { src = ./just-filter.nix; }" 2>&1 | grep example
<no output>

I haven't tried this with the latest version on master yet, so the results may have changed.

On a slightly unrelated note, callCabal2nix is supposed to have a filter already (see source code), but that seems to be kind of broken and the full source is included in the build for the cabal2nix-derivation anyways. :/

from lorri.

@anka-213, in your example, lorri is not expected to track the file example. It is a source file that does not have an influence on the shell environment that nix-shell should set up. @infinisil's example is different because he uses deps in the nativeBuildInputs field, which means, the content of the pkg file will impact the content of the shell environment.

from lorri.

@dudebout Aha, I see. However, the fact that nothing is logged in my example is probably the cause for not being able to track the sources through IFD. No info about what sources are being copied is being logged.

I am also a bit confused why manual filtering makes any difference when using callCabal2nix, since that already does filtering. For example:

pkgs.haskellPackages.callCabal2nix "example" ./. {}

will track the whole directory and nix-shell -vv will print

copied source '/path/to/example' -> '/nix/store/someHash-example'

while

pkgs.haskellPackages.callCabal2nix "example" (pkgs.lib.cleanSource ./.) {}

will not print such a copy statement and not track anything but the nix-file itself,
despite both commands copying exactly the same subset of the directory (the cabal-file) before generating the nix-derivation with cabal2nix.

Is this caused by NixOS/nixpkgs#51079? If so, adding an empty filter around src seems to be an extremely hacky workaround to that issue.

from lorri.

I think what your are after is point 2. in #6 (comment)

from lorri.

@dudebout Aha, I see. However, the fact that nothing is logged in my example is probably the cause for not being able to track the sources through IFD. No info about what sources are being copied is being logged.

I have also come across this. Looking at Nix' source code, I got as far as follows:

• "copied source" is only logged by copyPathToStore
• copyPathToStore is only called from coerceToString
• coerceToString is called from all over the place

But it looks like the following are true:

• src = ./. or similar lead to copyPathToStore being called, so we get "copied source"
• src = builtins.filterSource (...) ./. and src = builtins.path (...) ./. use different code paths and no "copied source" is logged by Nix.

As a result, lorri adds the former to the watch list, but not the latter.

We had discussions about changing Nix' output some time ago, I think it is time to revive those discussions.

from lorri.

(Sorry about the spam, I just need a place to record this somewhere ...)

Grepping through the Nix codebase, here's what I've found: the path and filterSource primops both instantiate a PathFilter which is then passed around.

• rg -F PathFilter shows that the argument of type PathFilter is always called filter.
• rg '\Wfilter\(' shows that the only time a function called filter is called in the entire codebase is in archive.cc::dump.

So in order to capture which files, post-filter, Nix actually uses, one would need to record those files here (or somewhere downstream, e.g. in the consumer of the sink into which the filtered files are dumped).

from lorri.