As reported in #159 (comment), a configuration fails to build if one of the source archives tracked in archive.lock is removed from the original location. Using archives is generally safer for packages on GNU ELPA, as some packages on the registry are built in specific ways.

Apparently, when a package on GNU ELPA is updated, the previous version is compressed with lzip, and the initial uncompressed archive is removed. This breaks reproducibility of this project. Also, .tar.lz is not natively supported by Nix flakes, so it wouldn't fit the current system of archive.lock anyway.

I don't come up with a solution right now. Perhaps some extra infrastructure will be needed? I wouldn't be happy with it, but the issue needs to be addressed to make this project usable.

I am a bit confused on native-comp support.
Could you provide documentation on the state of nativ-comp support in twist.nix?

It would be really nice to be able to compile all packages using native-comp.

At present, there are two ways in twist to read the main file of a package to parse the headers:

• Directly using builtins.readFile. This is not IFD, and it is effective when mainIsAscii or defaultMainIsAscii (#167) is set to true.
• Take the first 1500 characters of the file using head command and read the output using buitlins.readFile. This is IFD, which we want to avoid for package development.

The latter is required because buitlins.readFile can't process a file that contains non-ASCII characters. For example, org-ql.el contains control characters, so org-ql can't be added as a dependency without IFD.

This is inconvenient and makes rice unusable for many packages.

A workaround would be a generation approach: persist the header data to a file and read from it when available. We can probably save the metadata of all packages to a JSON file in the lock directory during lock. A downside of this approach is that the lock directory will contain another file, which will increase the repository size. Bonus is the user can see the diffs of dependencies and the build process may be faster.

CI failed due to a recent addition of a library to bbdb package:

    > In toplevel form:
   > bbdb-notmuch.el:27:2: Error: Cannot open load file: No such file or directory, notmuch-show

bbdb-notmuch.el is a recent addition to bbdb package, which is found in this diff.

This is likely a configuration problem. I will fix the test case later.

Now package-vc.el supports a new format elpa-packages.eld (see https://github.com/emacsmirror/emacs/blob/309e6aaa6867cd9a33e185d929afe18a660a8151/lisp/emacs-lisp/package-vc.el#L241) and package-build plans on supporting it (melpa/melpa#8435). It may be a good idea for twist to support it as well. It will allow to unify the logic for MELPA and elpa-packages.

The usage of comp-async-runnings:
https://github.com/emacs-twist/twist.nix/blob/master/pkgs/emacs/build/comp-native.el#L17

Which apparently is a private function will break after this commit:
emacs-mirror/emacs@fd61cf3

Not sure what to do, if just using the private function or if it needs a new solution.

This is a really strange one. I ran into the issue of this working locally, but not on CI.

It somehow finds the package in elpa (I guess, since it enters elpa.nix) on CI, but not locally. So the locked file is not correct. Locally the package is not found in the top inventories so it reaches the archive and hence when running nix run .#lock and nix run .#update it generates the archive.lock instead of adding it to the flake.lock.

If I put the archive above GNU ELPA it works on CI as well.

I have the current defined inventory:

[
  {
    name = "custom";
    type = "melpa";
    path = ../recipes;
  }
  {
    name = "gnu";
    type = "elpa";
    path = self.inputs.gnu-elpa.outPath + "/elpa-packages";
    core-src = emacsSrc;
    auto-sync-only = true;
  }
  {
    name = "melpa";
    type = "melpa";
    path = self.inputs.melpa.outPath + "/recipes";
  }
  {
    name = "nongnu";
    type = "elpa";
    path = self.inputs.nongnu-elpa.outPath + "/elpa-packages";
  }
  {
    name = "gnu-archive";
    type = "archive";
    url = "https://elpa.gnu.org/packages/";
  }
]

and archive.lock:

{
...
  "jinx": {
    "archive": {
      "narHash": "sha256-NA2ulVZuhetCYxwvUgOv8nDAdC9LYFDpj7O42pD/wyI=",
      "type": "tarball",
      "url": "https://elpa.gnu.org/packages/jinx-0.6.tar"
    },
    "inventory": {
      "type": "archive",
      "url": "https://elpa.gnu.org/packages/"
    },
    "packageRequires": {
      "compat": "29.1.4.0",
      "emacs": "27.1"
    },
    "version": "0.6"
  },
...
}

Here is the CI build:
https://github.com/terlar/emacs-config/actions/runs/4751840604/jobs/8441454659
and the source commit:
https://github.com/terlar/emacs-config/tree/0f1615ddb22b81d9f61a22166cbe95e8a8819fcf

Did you run into something like this?

The goal of this project is to provide a framework with which the user can lock and update individual packages in his/her config. The primary reason behind this goal is security, and the issue has been discussed several times:

• Why do we trust package.el?
• Are emacs package repositories a security risk?
• Borg vs Straight.el?

(There are more reddit threads on the topic, and you can easily reach them via DuckDuckGo.)

I haven't described it in the README yet. I find it difficult to concisely describe the reason, but I will have to do that.

cbowdon provides the following checklist, and Twist would help with the first criteria:

• reviewing every single update prior to installing (hardly feasible)
• code signing (not sure if package.el supports this though)
• waiting before updating, in hopes that an attack would be discovered before you install
• installing fewer packages to limit potential exposure
• generic measures like not running as root and having endpoint protection on Windows

To build some Emacs Lisp packages in Nix, Twist requires the user to override the derivations (example). At present, the API does not allow easily sharing overrides between multiple users. Ideally, there should be a community-maintained repository for an overlay, and the user should be able to define his/her own overrides in addition to that.

This is applied to both Emacs configurations and nomake. NoMake is a framework for linting and developing Emacs Lisp packages. If a package contains a dynamic module or depends on another package that requires one, it will require an override.

Some packages also have missing dependencies (which are upstream issues), which can be workaround like this.

• Provide workarounds as a separate repository, in a form of overlay.
• Refine the configuration API to allow composition of overrides, if necessary. not needed after #128

Regarding: https://github.com/emacs-twist/twist.nix/tree/master/pkgs/build-support/elisp
Suggestion: https://github.com/BrianHicks/tree-grepper, to be simple and easier.

Upon updating magit-section to my lockDir, I have captured its build failure error.

log:

sandbox setup: bind mounting '/nix/store/zzx3hx8awlz31spdj8ms14wd98v1yfpb-db-5.3.28' to '/nix/store/rlfar4f97kg5f5674rfk6cbl8zcjpdg3-emacs-git-20231211-0-magit-section-3.3.0.50-git.drv.chroot/nix/store/zzx3hx8awlz31spdj8ms14wd98v1yfpb-db-5.3.28'
sandbox setup: closing leaked FD 3
sandbox setup: closing leaked FD 4
sandbox setup: closing leaked FD 5
sandbox setup: closing leaked FD 6
sandbox setup: closing leaked FD 7
sandbox setup: closing leaked FD 8
sandbox setup: closing leaked FD 9
sandbox setup: closing leaked FD 10
sandbox setup: closing leaked FD 11
sandbox setup: closing leaked FD 12
sandbox setup: closing leaked FD 13
sandbox setup: closing leaked FD 14
sandbox setup: closing leaked FD 15
sandbox setup: closing leaked FD 16
sandbox setup: closing leaked FD 17
sandbox setup: closing leaked FD 20
building '/nix/store/rlfar4f97kg5f5674rfk6cbl8zcjpdg3-emacs-git-20231211-0-magit-section-3.3.0.50-git.drv'...
building of '/nix/store/rlfar4f97kg5f5674rfk6cbl8zcjpdg3-emacs-git-20231211-0-magit-section-3.3.0.50-git.drv^out' from .drv file: got EOF
building of '/nix/store/rlfar4f97kg5f5674rfk6cbl8zcjpdg3-emacs-git-20231211-0-magit-section-3.3.0.50-git.drv^out' from .drv file: woken up
building of '/nix/store/rlfar4f97kg5f5674rfk6cbl8zcjpdg3-emacs-git-20231211-0-magit-section-3.3.0.50-git.drv^out' from .drv file: build done
killing process 658448
builder process for '/nix/store/rlfar4f97kg5f5674rfk6cbl8zcjpdg3-emacs-git-20231211-0-magit-section-3.3.0.50-git.drv' finished
killing all processes running under uid '30001'
lock released on '/nix/store/531l4la167vn9zql2qsc4zimssw0ld17-emacs-git-20231211-0-magit-section-3.3.0.50-git.lock'
lock released on '/nix/store/l7kfqgmni29g9l1flrzrg9vzhdlxccdl-emacs-git-20231211-0-magit-section-3.3.0.50-git-info.lock'
building of '/nix/store/rlfar4f97kg5f5674rfk6cbl8zcjpdg3-emacs-git-20231211-0-magit-section-3.3.0.50-git.drv^out' from .drv file: done
error: builder for '/nix/store/rlfar4f97kg5f5674rfk6cbl8zcjpdg3-emacs-git-20231211-0-magit-section-3.3.0.50-git.drv' failed with exit code 1;
       last 8 log lines:
       > Running phase: unpackPhase
       > unpacking source archive /nix/store/bkkz28wc6ppz7dpwzxkzg9pwxxrg3xff-source
       > source root is /build/build
       > Running phase: patchPhase
       > Running phase: buildPhase
       > mv: missing destination file operand after '.'
       > Try 'mv --help' for more information.
       > /nix/store/d4jf1cbbk494zwgbqz31pxgigpsbh6w2-stdenv-linux/setup: line 131: pop_var_context: head of shell_variables not a function context
       For full logs, run 'nix log /nix/store/rlfar4f97kg5f5674rfk6cbl8zcjpdg3-emacs-git-20231211-0-magit-section-3.3.0.50-git.drv'.
building of '/nix/store/7g5m5xnvw6rw2mcw8h0qdz5i5vbzx9k4-emacs.drv^out' from .drv file: waitee 'building of '/nix/store/rlfar4f97kg5f5674rfk6cbl8zcjpdg3-emacs-git-20231211-0-magit-section-3.3.0.50-git.drv^out' from .drv file' done; 3 left
building of '/nix/store/i0g1sx3clfcrdak3bflys5dwzh2p8g2d-elisp-digest.json.drv^out' from .drv file: goal destroyed
building of '/nix/store/hhzmvsganhs5ph7piir8s3va1rfmj2nj-elisp-packages.drv^out' from .drv file: goal destroyed
building of '/nix/store/dayjk42pigrn6147cny98g222r8d6rr8-emacs-git-20231211-0-nix-mode-1.5.0.drv^out' from .drv file: goal destroyed
building of '/nix/store/7g5m5xnvw6rw2mcw8h0qdz5i5vbzx9k4-emacs.drv^out' from .drv file: woken up
building of '/nix/store/rlfar4f97kg5f5674rfk6cbl8zcjpdg3-emacs-git-20231211-0-magit-section-3.3.0.50-git.drv^out' from .drv file: goal destroyed
building of '/nix/store/7g5m5xnvw6rw2mcw8h0qdz5i5vbzx9k4-emacs.drv^out' from .drv file: all inputs realised
building of '/nix/store/7g5m5xnvw6rw2mcw8h0qdz5i5vbzx9k4-emacs.drv^out' from .drv file: done
building of '/nix/store/7g5m5xnvw6rw2mcw8h0qdz5i5vbzx9k4-emacs.drv^out' from .drv file: goal destroyed
error: 1 dependencies of derivation '/nix/store/7g5m5xnvw6rw2mcw8h0qdz5i5vbzx9k4-emacs.drv' failed to build

It seems that renamePhase failed during the execution of mv docs/*.* ..

I am at a loss regarding this, how should I resolve it?

As additional info， the flake of magit-section:

    "magit-section": {
      "flake": false,
      "locked": {
        "lastModified": 1704784205,
        "narHash": "sha256-qiQQ1nKm+f1Hx9cipHSmxP6ED2SntnWx/qm1erUS1t4=",
        "owner": "magit",
        "repo": "magit",
        "rev": "c3b7fd7dc43dd87468a86aef3d59576ad79fbc16",
        "type": "github"
      },
      "original": {
        "owner": "magit",
        "repo": "magit",
        "type": "github"
      }
    },

I am still not sure what the procedure is to update the lock.

We do have nix run .#lock --impure and nix run .#update --impure. But neither seems to update packages unless you remove the current lock information first. It just updates packages that are modified, but don't update to later versions for the ones that exist.

E.g. this works:

rm lock/*
nix run .#lock --impure

Is this the procedure or should these other helper commands work in some way?

As reported in #155, Nix regular expressions differ across platforms (see NixOS/nix#1537). Add cross platform tests to CI and fix issues if there are packages that fail to build on Darwin.

I ran into the dired-hacks packages using Package-requires: instead of Package-Requires: and because of that the dependency resolution didn't work.

I created a PR to fix this:
https://github.com/Fuco1/dired-hacks/pull/194/files

But perhaps the parser should be case-insensitive?

It would be nice if the build function was exposed so one could make custom derivations:
https://github.com/emacs-twist/twist.nix/blob/master/pkgs/emacs/build/default.nix

The use case is that I want to build my configuration and not install it as a package but still would like to byte compile it. Perhaps the correct thing is to make packages out of this not sure, but at least that is what I am doing currently.

I have achieved this by a bit of a hack using the trivialBuild instead:

final: prev: {
  emacsEnv =
    (final.emacsTwist {
      emacsPackage = final.emacsPgtkNativeComp.overrideAttrs (_: {version = "29.0.50";});

      initFiles = [(final.tangleOrgBabelFile "init.el" ./init.org {})];

      lockDir = ./lock;
      inventories = import ./nix/inventories.nix {
        inherit self;
        emacsSrc = final.emacsPgtkNativeComp.src.outPath;
      };

      inputOverrides = import ./nix/inputOverrides.nix;
    })
      .overrideScope' (tfinal: tprev: {
        elispPackages = tprev.elispPackages.overrideScope' (
          prev.callPackage ./nix/packageOverrides.nix {inherit (tprev) emacs;}
        );
      });

  emacsConfig = let
    emacs = let
      self =
        final.emacsEnv
        // {
          inherit (final.emacsEnv.emacs) meta;
          overrideAttrs = _: self;
        };
    in
      self;

    attrs = nixpkgs.lib.optionalAttrs (self ? lastModifiedDate) {
      version = nixpkgs.lib.substring 0 8 self.lastModifiedDate;
    };
  in
    (prev.emacsPackagesFor emacs).callPackage ./. attrs;
}

Then later it is used in custom packages:

trivialBuild {
  pname = "config-init";
  inherit version;

  src = lib.sourceByRegex ./. ["init.org" "lisp" "lisp/.*.el$"];

  buildPhase = ''
    emacs --batch --quick \
      --load org \
      *.org \
      --funcall org-babel-tangle

    mkdir -p .xdg-config
    ln -s $PWD .xdg-config/emacs
    export XDG_CONFIG_HOME="$PWD/.xdg-config"

    emacs -L . --batch --eval '(setq byte-compile-error-on-warn t)' -f batch-byte-compile *.el
  '';
}

See https://zimbatm.com/notes/1000-instances-of-nixpkgs.

The current implementation heavily use overlays, but actually it is better to avoid it unless there is a good reason for it. The primary API is currently provided as an overlay, it will probably better to provide a function that takes pkgs as an argument, and the function should be provided under lib in the flake.

To speed up Emacs I've been experimenting with having a derivation that generates package-quickstart via a derivation. I currently do this (given an Emacs package with all packages on path):

trivialBuild {
  pname = "config-package-quickstart";
  inherit version;

  dontUnpack = true;

  buildPhase = ''
    emacs --batch --quick \
      --load package \
      --eval '(setq package-quickstart-file "package-quickstart.el")' \
      --eval '(setq package-quickstart t)' \
      --funcall package-quickstart-refresh
  '';
}

I think package-quickstart integration into the nix Emacs Eco-system could be beneficial. Since it is a build-time thing that is dependent on it's inputs.

Recently, I am trying to incorporate mu4e into my configuration, but I don't know how build it by Twist.

The building of mu4e is dependent on the artifacts of mu.

For example, as build by nixpkgs elispPackages.

https://github.com/NixOS/nixpkgs/blob/b9c6b11eafbb693c3d9d820bea1636a51c87b2a5/pkgs/applications/editors/emacs/elisp-packages/manual-packages/mu4e/default.nix#L7-L33

I attempted to independently build its Emacs Lisp portion, but it appears that there are many missing components that I am unsure how to address. For example, mu-guile and a mu4e-config.el generated from the template.

https://github.com/djcb/mu/blob/62f0a9a902031d172e03e8caaaad013a1798749f/mu4e/meson.build#L19-L27

I am interested in understanding the methods available for extracting the mu output and integrating it with twist. Alternatively, are there other reproducible build strategies?

OT
So I'm finally getting around to explore this and doing my own take on using this and I'm avoiding emacsmirror.
/OT

When I try to run the emacsTwist function on my configuration and I guess it tries to look up a package from the elpa inventory it fails when trying to find cycle-quotes. I checked the elpa-packages file and noticed ("cycle-quotes" :url nil) and I'm guessing that is the case why it is not found.

Reading at the top of this file I see:

;; For KIND :url URL is the URL of the remote git repository that we want
;; to track, or `nil' if there is no upstream, i.e., if development of this
;; package takes place directly on the `externals/<name>' branch of the
;; `elpa.git' repository.  For packages of KIND :core URL must be a list of:
;;    STRING = A file-name to copy from Emacs repo.
;;    (STRING STRING) = A file-name to copy renamed from Emacs repo.

I'm guessing :url nil, e.g. looking inside externals/<name> is not currently supported or do you have any other idea why it would fail finding this specific package?

@nix { "action": "setPhase", "phase": "unpackPhase" }
unpacking sources
unpacking source archive /nix/store/9gg7ryydfkhvmh0vl2aalq5h3jl57igs-source
source root is /build/build
@nix { "action": "setPhase", "phase": "patchPhase" }
patching sources
@nix { "action": "setPhase", "phase": "buildPhase" }
building
Package autoload is deprecated
  SCRAPE   .
  INFO     Scraping files for persist-autoloads.el... 
  INFO     Scraping files for persist-autoloads.el...done
persist.texi:55: raising the section level of @section which is too low
/nix/store/gv2cl6qvvslz5h15vqd89f1rpvrdg5yc-stdenv-linux/setup: line 146: pop_var_context: head of shell_variables not a function context

when installing my setup

Given how emacs-overlay sets the versions and how the code relies on version here:

• https://github.com/emacs-twist/twist.nix/blob/master/pkgs/emacs/wrapper.nix#L91
• https://github.com/emacs-twist/twist.nix/blob/master/pkgs/emacs/wrapper.nix#L121

You will get invalid symlinks as the version 20221216.0 will be symlinked but the actual value should be 30.0.50 for example.

I know before you had the check that verified the version and now that is removed. Just curious if you are aware of this one or if we should document the workaround .overrideAttrs (_: {version = "30.0.50";}) for example. Best would be if the actual version was available somewhere.

Hello, I am sorry to raise this here as it is per se not an issue with twist.nix. However I figured you might have run into the same thing and found a solution. Or perhaps there is some weird thing making me run into this but not others.

The issue is with tracking the flake input git+https://git.savannah.gnu.org/git/emacs/elpa.git?ref=main. I have builds working at some revision and next time I try to re-run it, it fails with this kind of message:

       error: Cannot find Git revision 'd5ea227a3729357b6c4c09bd033db9445fa34048' in ref 'main' of repository 'https://git.savannah.gnu.org/git/emacs/elpa.git'! Please make sure that the rev exists on the ref you've specified or add allRefs = true; to fetchGit.

If I search the ELPA git repo, I can find the commit:
https://git.savannah.gnu.org/gitweb/?p=emacs/elpa.git;a=commit;h=d5ea227a3729357b6c4c09bd033db9445fa34048

It also seems to still be associated with the main branch (* elpa-packages (window-tool-bar): New core package):
https://git.savannah.gnu.org/gitweb/?p=emacs/elpa.git;a=shortlog;h=refs/heads/main

I only see this behavior with ELPA and it is causing failures for pipelines that worked in the past and only way I found to move forward was to update the input, but it keeps happening.

Feel free to close, in the meantime I will keep investigating...

The latest version of Emacs fails to start (emacs-twist/examples#29 (comment)):

Error using execdir /nix/store/3m4jv6pk4hgjmyrr9g5pchf8h2g5v23j-emacs/bin/:
emacs: /nix/store/3m4jv6pk4hgjmyrr9g5pchf8h2g5v23j-emacs/bin/../native-lisp/28.1.91-002d2166/preloaded/window-0d1b8b93-7ef4271a.eln: cannot open shared object file: No such file or directory

This is due to how twist organizes artifacts. I will work on it later.

Recently this was merged: NixOS/nixpkgs#235859

I think this affects the wrapper:

Probably a less breaking change from this change would be to still have this attribute propagated on the Emacs derivation as well not only in the input.

Hi! I was just wondering if it's on the roadmap to update twist such that it doesn't make so many API requests to github? I have an api key set up which helps some but I still get rate limited. Using a vpn is my current workaround but that doesn't seem ideal.