uthar / nix-cl Goto Github PK

Will require un-hardcoding from LD_LIBRARY_PATH. Maybe use DYLD_LIBRARY_PATH depending on the system

Might be cool for docker images, to limit their size and in general to limit closure size.

cl-ana is a "Free (GPL) Common Lisp data analysis library with emphasis on modularity and conceptual clarity. "

Build is broken, can be repaired by this patch:

--- a/pkgs/development/lisp-modules-new/ql.nix
+++ b/pkgs/development/lisp-modules-new/ql.nix
@@ -127,6 +127,15 @@ let
       nativeLibs = [ libfixposix ];
       systems = [ "iolib" "iolib/os" "iolib/pathnames" ];
     };
+    "cl-ana.hdf-cffi" = pkg: {
+      nativeBuildInputs = [ hdf5 pkg-config ];
+      nativeLibs = [ hdf5 ];
+      NIX_LDFLAGS = [ "-lhdf5" ];
+    };
+    gsll = pkg: {
+      nativeBuildInputs = [ gsl ];
+      nativeLibs = [ gsl ];
+    };
   };
 
   qlpkgs =

i.e. introduction of ${lisp}Attrs (program,pkg,flags,faslExt) instead of a lisp command and introduction of asdf parameter

Probably can wait, because this api is extremely experimental as well, will probably go with lisp.pkgs/lisp.withPackages/lisp.buildASDFSystem style.

I have a Lisp package that needs to add a directory to XDG_DATA_DIRS. What's the right way to do that?

(It's vk that needs to find vulkan-validation-layers for some data files loaded by the native libraries at runtime.)

refactor everything to use generic hashes instead of hard coded sha256

Hey I don't quite understand which nixpkgs version nix-cl is supposed to use for libraries?

In practice the chain of dependencies seems to be github:uthar/nix-cl -> github:uthar/dev -> nixpkgs i.e. that the nixpkgs version will be determined by the contents of https://github.com/Uthar/dev/blob/master/flake.lock.

That seems a bit random, though? Is it just that using flakes means pinning everything by default anyway?

I ask because I need to make a not-quite-trivial update to the Vulkan shaderc library in nixpkgs (lukego/nixpkgs@059fb38) in order to fix an obscure runtime incompatibility in the Lisp vk library that only shows up when running real application code (not during build.) I'm planning to upstream that change to nixpkgs but I don't really understand the flow for landing it in nix-cl where it is needed e.g. when I make a PR to package vk.

Tips welcome, I don't know how understood all this flakes stuff is really in general :)

How could we automate discovery of the libraries and programs from nixpkgs that each Lisp system depends on?

I suspect this will become more important over time if we want to support custom Lisp distributions (#25) because then we can't entirely rely on hand-crafted dependency declarations based on the latest Quicklisp e.g. on kons-9 they are forking a bunch of dependencies and requiring many new foreign libraries on the forks.

I don't know what the best approach is but here is a first idea that I'm toying around with. This doesn't involve screen-scraping the failing build logs (yet...)

Algorithm to discover dependencies of a Lisp package:

Builds with no libraries?
  yes ->
    Done: no dependencies.
  no ->
    Builds with "jumbo" set of all potential anticipated dependencies?
      yes ->
        Progress: we are able to build the package!
        Recommended next step:
          Iteratively search for minimal subset of packages that are truly needed.
      no ->
        Broken package: human needs to check for reason e.g.
          - completely broken
          - unsupported on the lisp/arch
          - depends on unanticipated package from nixpkgs (must add to "jumbo" set)
          - depends on something outside nixpkgs (must bundle somehow)

I'd like to test this idea by writing down a "jumbo" set of all the packages that Lisp systems might depend on, based on browsing build logs, and then see what effect this has e.g. does it make a lot more systems work.

If it does have a big effect then we could think about how to "shrink" the dependencies for each package so they don't all depend on everything. (Or if it doesn't improve the package situation much maybe we needn't bother.)

Here's the first tiny snippet of libraries that I've seen referenced in failing build logs:

ncurses, renderdoc, geoip, xorg.libX11, libcerf, libblas, libdrm, libfann, libfarmhash, glpk, gsl, libiio, tokyocabinet, leveldb, lmdb.

If the total ends up being less than ~50 that might be reasonable to maintain as our "jumbo" collection.

be more in line with other things in nixpkgs

To be able to specify something like this in build-asdf-system

https://github.com/nuddyco/lispnix/blob/f7ecdd252fa91268f7bb0d1b8b67d918c1cbe457/kons-9.nix#L16

McCLIM build is not working. I don't understand exactly why yet.

There seem to be a couple of issues:

• mcclim-bezier package is being flagged as broken. (I see now this was done manually, possibly due to problem below.)
• mcclim-bezier package fails because it tries to compile a file from mcclim-render into read-only nix store.

Reproduce on master branch:

$ lwp sbcl mcclim-bezier
error: builder for '/nix/store/1nhzagh6w25zsy2vzfmf128yvs654jfv-mcclim-bezier-20221106-git.drv' failed with exit code 1;
       last 10 log lines:
       > ; compiling file "/nix/store/8612lj65srfmy9a59l0l0f2w4igz0ir8-source/Extensions/fonts/ttf-medium-mixin.lisp" (written 01 JAN 1970 12:00:01 AM):
       >
       > ; wrote /build/source/Extensions/fonts/ttf-medium-mixin-tmpJAIDFZTC.fasl
       > ; compilation finished in 0:00:00.033
       > ;
       > ; compilation unit aborted
       > ;   caught 1 fatal ERROR condition
       > BUILD FAILED: Error opening #P"/nix/store/k7kcqlpc7a7p20bcarfxn7alyvryw98p-mcclim-render-20221106-git/Extensions/render/package-tmp8V3J6PE9.fasl":
       >
       >                 Permission denied
       For full logs, run 'nix log /nix/store/1nhzagh6w25zsy2vzfmf128yvs654jfv-mcclim-bezier-20221106-git.drv'.
error: 1 dependencies of derivation '/nix/store/p07vdqm699c6di1irnlp7skalwnnfhhm-sbcl-with-packages.drv' failed to build

Porting nix-cl to another project means making the flake more accessible, so if there is any work/comments on this already, it would be useful for a PR which will come in the following days. Much love and thanks in advance. Much gratitude for this project. Lisp will win, in the end.

Ressources already found:

• #29

PS: Land survey, gyroscopes and celestial navigation, each individually proves Earth is level.

Could we get an update to the March dist update of Quicklisp? If I am not mistaken, the last update in this repo is from December. Thanks.

http://blog.quicklisp.org/2022/04/march-2022-quicklisp-dist-update-now.html

I am looking for a way to customize the set of Lisp package derivations before they are built. Is this supported in the API? Or else how might we support it?

For example I would like to add a failureHook so that broken packages will be "built" into debug information.

I tried to do it like this:

let pkgs = import nixpkgs {};
    baseLispPackages = pkgs.lispPackages_new.sbclPackages;
    updateDerivation = d: d.overrideAttrs (o: {
      failureHook = ''
          ...
        '';
    });
    lispPackages = builtins.mapAttrs (n: v: updateDerivation v) baseLispPackages;
    pkg = builtins.getAttr lispPackage lispPackages;
    in

but I don't think this works because the modified derivations will include references to old unmodified definitions of their dependencies. Somehow I need to have the updated derivations all refer to each other.

I have a few ideas of how to approach this...

1. Add an API function to return Lisp package definitions as attribute sets so that they can be modified before calling build-asdf-system.
2. Make Lisp package definitions into functions that are called with callPackage to resolve their dependencies before creating their derivation.
3. Stealing some clever ideas from other languages' packagings...

But I'm not sure which approach makes sense so I'd be glad for some advice.

UI toolkit bindings https://github.com/bohonghuang/cl-gtk4

I attempt to start a slynk server from nix-configuration only. For this I need the slynk-mrepl packages and others. When attempting to build them, I get the following. Is there any way I can help with this? Thank you.

$ nix-build https://github.com/Uthar/nix-cl/archive/master.tar.gz -A sbclPackages.slynk_slash_mrepl

this derivation will be built:
  /nix/store/ypbnbsw51vp25wz160647wnjkf91sp52-slynk_mrepl-20220707-git.drv
building '/nix/store/ypbnbsw51vp25wz160647wnjkf91sp52-slynk_mrepl-20220707-git.drv'...
unpacking sources
unpacking source archive /nix/store/bn2hyl4659b8rj24i9xp1jscnv7mcn0f-source
source root is source
patching sources
configuring
no configure script, doing nothing
building
SLYNK's ASDF loader finished.
Unhandled SB-INT:SIMPLE-FILE-ERROR in thread #<SB-THREAD:THREAD "main thread" RUNNING
                                                {1004C181B3}>:
  Error opening #P"/nix/store/wxcdkva1kp8isnadn5rvgp6rvlaayg55-slynk-20220707-git/contrib/slynk-mrepl-tmpGHU3ALSV.fasl":

    Permission denied

Backtrace for: #<SB-THREAD:THREAD "main thread" RUNNING {1004C181B3}>
0: (SB-DEBUG::DEBUGGER-DISABLED-HOOK #<SB-INT:SIMPLE-FILE-ERROR "Error opening ~S" {10033A7E33}> #<unused argument> :QUIT T)
1: (SB-DEBUG::RUN-HOOK *INVOKE-DEBUGGER-HOOK* #<SB-INT:SIMPLE-FILE-ERROR "Error opening ~S" {10033A7E33}>)
2: (INVOKE-DEBUGGER #<SB-INT:SIMPLE-FILE-ERROR "Error opening ~S" {10033A7E33}>)
3: (ERROR SB-INT:SIMPLE-FILE-ERROR :PATHNAME #P"/nix/store/wxcdkva1kp8isnadn5rvgp6rvlaayg55-slynk-20220707-git/contrib/slynk-mrepl-tmpGHU3ALSV.fasl" :MESSAGE "Permission denied" :FORMAT-CONTROL "Error opening ~S" :FORMAT-ARGUMENTS (#P"/nix/store/wxcdkva1kp8isnadn5rvgp6rvlaayg55-slynk-20220707-git/contrib/slynk-mrepl-tmpGHU3ALSV.fasl"))
4: (SB-IMPL::FILE-PERROR #P"/nix/store/wxcdkva1kp8isnadn5rvgp6rvlaayg55-slynk-20220707-git/contrib/slynk-mrepl-tmpGHU3ALSV.fasl" 13 "Error opening ~S" #P"/nix/store/wxcdkva1kp8isnadn5rvgp6rvlaayg55-slynk-20220707-git/contrib/slynk-mrepl-tmpGHU3ALSV.fasl")
5: (SB-IMPL::%OPEN-ERROR #P"/nix/store/wxcdkva1kp8isnadn5rvgp6rvlaayg55-slynk-20220707-git/contrib/slynk-mrepl-tmpGHU3ALSV.fasl" 13 NIL :CREATE)
6: (OPEN #P"/nix/store/wxcdkva1kp8isnadn5rvgp6rvlaayg55-slynk-20220707-git/contrib/slynk-mrepl-tmpGHU3ALSV.fasl" :DIRECTION :IO :ELEMENT-TYPE :DEFAULT :IF-EXISTS NIL :IF-DOES-NOT-EXIST :CREATE :EXTERNAL-FORMAT :UTF-8 :CLASS SB-SYS:FD-STREAM)
7: (UIOP/STREAM:CALL-WITH-TEMPORARY-FILE #<FUNCTION (FLET "BEFORE234" :IN UIOP/STREAM::GET-TEMPORARY-FILE) {5346555B}> :WANT-STREAM-P NIL :WANT-PATHNAME-P T :DIRECTION :IO :KEEP T :AFTER NIL :DIRECTORY #P"/nix/store/wxcdkva1kp8isnadn5rvgp6rvlaayg55-slynk-20220707-git/contrib/" :TYPE "fasl" :PREFIX "slynk-mrepl-tmp" :SUFFIX NIL :ELEMENT-TYPE NIL :EXTERNAL-FORMAT NIL)
8: (UIOP/LISP-BUILD:COMPILE-FILE* #P"/nix/store/wxcdkva1kp8isnadn5rvgp6rvlaayg55-slynk-20220707-git/contrib/slynk-mrepl.lisp" :OUTPUT-FILE #P"/nix/store/wxcdkva1kp8isnadn5rvgp6rvlaayg55-slynk-20220707-git/contrib/slynk-mrepl.fasl" :EXTERNAL-FORMAT :UTF-8 :WARNINGS-FILE NIL)
9: (ASDF/LISP-ACTION:PERFORM-LISP-COMPILATION #<ASDF/LISP-ACTION:COMPILE-OP > #<ASDF/LISP-ACTION:CL-SOURCE-FILE "slynk/mrepl" "../contrib/slynk-mrepl">)
10: ((SB-PCL::EMF ASDF/ACTION:PERFORM) #<unused argument> #<unused argument> #<ASDF/LISP-ACTION:COMPILE-OP > #<ASDF/LISP-ACTION:CL-SOURCE-FILE "slynk/mrepl" "../contrib/slynk-mrepl">)
11: ((LAMBDA NIL :IN ASDF/ACTION:CALL-WHILE-VISITING-ACTION))
12: ((:METHOD ASDF/ACTION:PERFORM-WITH-RESTARTS :AROUND (T T)) #<ASDF/LISP-ACTION:COMPILE-OP > #<ASDF/LISP-ACTION:CL-SOURCE-FILE "slynk/mrepl" "../contrib/slynk-mrepl">) [fast-method]
13: ((:METHOD ASDF/PLAN:PERFORM-PLAN (T)) #<ASDF/PLAN:SEQUENTIAL-PLAN {1002516553}>) [fast-method]
14: ((FLET SB-C::WITH-IT :IN SB-C::%WITH-COMPILATION-UNIT))
15: ((:METHOD ASDF/PLAN:PERFORM-PLAN :AROUND (T)) #<ASDF/PLAN:SEQUENTIAL-PLAN {1002516553}>) [fast-method]
16: ((:METHOD ASDF/OPERATE:OPERATE (ASDF/OPERATION:OPERATION ASDF/COMPONENT:COMPONENT)) #<ASDF/LISP-ACTION:LOAD-OP > #<ASDF/SYSTEM:SYSTEM "slynk/mrepl"> :PLAN-CLASS NIL :PLAN-OPTIONS NIL) [fast-method]
17: ((SB-PCL::EMF ASDF/OPERATE:OPERATE) #<unused argument> #<unused argument> #<ASDF/LISP-ACTION:LOAD-OP > #<ASDF/SYSTEM:SYSTEM "slynk/mrepl">)
18: ((LAMBDA NIL :IN ASDF/OPERATE:OPERATE))
19: ((:METHOD ASDF/OPERATE:OPERATE :AROUND (T T)) #<ASDF/LISP-ACTION:LOAD-OP > #<ASDF/SYSTEM:SYSTEM "slynk/mrepl">) [fast-method]
20: ((SB-PCL::EMF ASDF/OPERATE:OPERATE) #<unused argument> #<unused argument> ASDF/LISP-ACTION:LOAD-OP SLYNK/MREPL)
21: ((LAMBDA NIL :IN ASDF/OPERATE:OPERATE))
22: ((:METHOD ASDF/OPERATE:OPERATE :AROUND (T T)) ASDF/LISP-ACTION:LOAD-OP SLYNK/MREPL) [fast-method]
23: (ASDF/SESSION:CALL-WITH-ASDF-SESSION #<FUNCTION (LAMBDA NIL :IN ASDF/OPERATE:OPERATE) {10017A464B}> :OVERRIDE T :KEY NIL :OVERRIDE-CACHE T :OVERRIDE-FORCING NIL)
24: ((LAMBDA NIL :IN ASDF/OPERATE:OPERATE))
25: (ASDF/SESSION:CALL-WITH-ASDF-SESSION #<FUNCTION (LAMBDA NIL :IN ASDF/OPERATE:OPERATE) {10030E0D6B}> :OVERRIDE NIL :KEY NIL :OVERRIDE-CACHE NIL :OVERRIDE-FORCING NIL)
26: ((:METHOD ASDF/OPERATE:OPERATE :AROUND (T T)) ASDF/LISP-ACTION:LOAD-OP SLYNK/MREPL) [fast-method]
27: (ASDF/OPERATE:LOAD-SYSTEM SLYNK/MREPL)
28: ((LAMBDA NIL :IN "/nix/store/15ywyl8x06j5xmi9c51pchhhnm160mlv-build-slynk_mrepl.lisp"))
29: (SB-INT:SIMPLE-EVAL-IN-LEXENV (DOLIST (S (QUOTE (SLYNK/MREPL))) (ASDF/OPERATE:LOAD-SYSTEM S)) #<NULL-LEXENV>)
30: (EVAL-TLF (DOLIST (S (QUOTE (SLYNK/MREPL))) (ASDF/OPERATE:LOAD-SYSTEM S)) 1 NIL)
31: ((LABELS SB-FASL::EVAL-FORM :IN SB-INT:LOAD-AS-SOURCE) (DOLIST (S (QUOTE (SLYNK/MREPL))) (ASDF/OPERATE:LOAD-SYSTEM S)) 1)
32: ((LAMBDA (SB-KERNEL:FORM &KEY :CURRENT-INDEX &ALLOW-OTHER-KEYS) :IN SB-INT:LOAD-AS-SOURCE) (DOLIST (S (QUOTE (SLYNK/MREPL))) (ASDF/OPERATE:LOAD-SYSTEM S)) :CURRENT-INDEX 1)
33: (SB-C::%DO-FORMS-FROM-INFO #<FUNCTION (LAMBDA (SB-KERNEL:FORM &KEY :CURRENT-INDEX &ALLOW-OTHER-KEYS) :IN SB-INT:LOAD-AS-SOURCE) {10016567DB}> #<SB-C::SOURCE-INFO {10016567A3}> SB-C::INPUT-ERROR-IN-LOAD)
34: (SB-INT:LOAD-AS-SOURCE #<SB-SYS:FD-STREAM for "file /nix/store/15ywyl8x06j5xmi9c51pchhhnm160mlv-build-slynk_mrepl.lisp" {1001650E73}> :VERBOSE NIL :PRINT NIL :CONTEXT "loading")
35: ((LABELS SB-FASL::LOAD-STREAM-1 :IN LOAD) #<SB-SYS:FD-STREAM for "file /nix/store/15ywyl8x06j5xmi9c51pchhhnm160mlv-build-slynk_mrepl.lisp" {1001650E73}> NIL)
36: (SB-FASL::CALL-WITH-LOAD-BINDINGS #<FUNCTION (LABELS SB-FASL::LOAD-STREAM-1 :IN LOAD) {7FFFF784F80B}> #<SB-SYS:FD-STREAM for "file /nix/store/15ywyl8x06j5xmi9c51pchhhnm160mlv-build-slynk_mrepl.lisp" {1001650E73}> NIL #<SB-SYS:FD-STREAM for "file /nix/store/15ywyl8x06j5xmi9c51pchhhnm160mlv-build-slynk_mrepl.lisp" {1001650E73}>)
37: (LOAD #<SB-SYS:FD-STREAM for "file /nix/store/15ywyl8x06j5xmi9c51pchhhnm160mlv-build-slynk_mrepl.lisp" {1001650E73}> :VERBOSE NIL :PRINT NIL :IF-DOES-NOT-EXIST :ERROR :EXTERNAL-FORMAT :DEFAULT)
38: ((FLET SB-IMPL::LOAD-SCRIPT :IN SB-IMPL::PROCESS-SCRIPT) #<SB-SYS:FD-STREAM for "file /nix/store/15ywyl8x06j5xmi9c51pchhhnm160mlv-build-slynk_mrepl.lisp" {1001650E73}>)
39: ((FLET SB-UNIX::BODY :IN SB-IMPL::PROCESS-SCRIPT))
40: ((FLET "WITHOUT-INTERRUPTS-BODY-11" :IN SB-IMPL::PROCESS-SCRIPT))
41: (SB-IMPL::PROCESS-SCRIPT "/nix/store/15ywyl8x06j5xmi9c51pchhhnm160mlv-build-slynk_mrepl.lisp")
42: (SB-IMPL::TOPLEVEL-INIT)
43: ((FLET SB-UNIX::BODY :IN SB-IMPL::START-LISP))
44: ((FLET "WITHOUT-INTERRUPTS-BODY-3" :IN SB-IMPL::START-LISP))
45: (SB-IMPL::%START-LISP)

unhandled condition in --disable-debugger mode, quitting
;
; compilation unit aborted
;   caught 1 fatal ERROR condition
error: builder for '/nix/store/ypbnbsw51vp25wz160647wnjkf91sp52-slynk_mrepl-20220707-git.drv' failed with exit code 1;
       last 10 log lines:
       > 41: (SB-IMPL::PROCESS-SCRIPT "/nix/store/15ywyl8x06j5xmi9c51pchhhnm160mlv-build-slynk_mrepl.lisp")
       > 42: (SB-IMPL::TOPLEVEL-INIT)
       > 43: ((FLET SB-UNIX::BODY :IN SB-IMPL::START-LISP))
       > 44: ((FLET "WITHOUT-INTERRUPTS-BODY-3" :IN SB-IMPL::START-LISP))
       > 45: (SB-IMPL::%START-LISP)
       >
       > unhandled condition in --disable-debugger mode, quitting
       > ;
       > ; compilation unit aborted
       > ;   caught 1 fatal ERROR condition
       For full logs, run 'nix log /nix/store/ypbnbsw51vp25wz160647wnjkf91sp52-slynk_mrepl-20220707-git.drv'.

Consider whether ASDF should be pinned and load'ed between each build run and during start of lisp wrappers.

Essentially overriding the implementation-dependent (require 'asdf) - Could lead to more consistent results, like in SBCL where ASDF doesn'y support package-inferred-system

When attempting to run the example from the readme, I get the following error:

$ HOME=$PWD nix-shell -p 'with import ./. {}; sbclWithPackages (ps: [ps.hunchentoot ps.sqlite])' --run "sbcl --noinform --eval '(require :asdf)' --eval '(asdf:load-system :hunchentoot)'"

debugger invoked on a SB-INT:SIMPLE-FILE-ERROR in thread
#<THREAD "main thread" RUNNING {1004B78323}>:
  Can't create directory /nix/store/wywjn6x9hcn5qd5z8nmiw49r0agniqak-babel-20200925-git/build

Type HELP for debugger help, or (SB-EXT:EXIT) to exit from SBCL.

restarts (invokable by number or by possibly-abbreviated name):
  0: [RETRY                        ] Retry directory creation.
  1: [RETRY                        ] Retry
                                     compiling #<CL-SOURCE-FILE "asdf" "build" "asdf">.
  2: [ACCEPT                       ] Continue, treating
                                     compiling #<CL-SOURCE-FILE "asdf" "build" "asdf">
                                     as having been successful.
  3:                                 Retry ASDF operation.
  4: [CLEAR-CONFIGURATION-AND-RETRY] Retry ASDF operation after resetting the
                                     configuration.
  5: [CONTINUE                     ] Ignore runtime option --eval "(asdf:load-system :hunchentoot)".
  6: [ABORT                        ] Skip rest of --eval and --load options.
  7:                                 Skip to toplevel READ/EVAL/PRINT loop.
  8: [EXIT                         ] Exit SBCL (calling #'EXIT, killing the process).

(SB-KERNEL::%FILE-ERROR #P"/nix/store/wywjn6x9hcn5qd5z8nmiw49r0agniqak-babel-20200925-git/build/asdf.fasl" "Can't create directory ~A~:[~;,~%a file with ~
                          the same name already exists.~]" "/nix/store/wywjn6x9hcn5qd5z8nmiw49r0agniqak-babel-20200925-git/build" NIL)
0]
;
; compilation unit aborted
;   caught 1 fatal ERROR condition
* 

Does this maybe have something todo with asdf's output path translations?

Just a small tracking issue, that the July update is available.

http://blog.quicklisp.org/2022/07/july-2022-quicklisp-dist-update-now.html

Hi,

eventually, we should add some metadata to the generated packages, like "description" and "license" ( anything else ? ). Currently, this is what I get with the old and the new package set:

nix-repl> lispPackages.cl-async.meta
{ available = true; broken = false; description = "Asynchronous operations for Common Lisp."; insecure = false; name = "lisp-cl-async-20211020-git"; outputsToInstall = [ "out" ]; position = ".../pkgs/development/lisp-modules/define-package.nix:115"; unfree = false; unsupported = false; version = "20211020-git"; }

nix-repl> lispPackages_new.sbclPackages.cl-async.meta
{ available = true; broken = false; insecure = false; name = "cl-async-20211020-git"; outputsToInstall = [ "out" ]; position = ".../pkgs/development/lisp-modules-new/imported.nix:6488"; unfree = false; unsupported = false; }

Metadata is also useful, if we want to be able to make the package set discoverable and eventually make them searchable on https://search.nixos.org.

For that though, we need to call recurseIntoAttrs onto the package set that is bound to the lisp implementation, like sbclPackages. As you can see in you PR NixOS/nixpkgs#172234, ofborg is not seeing anything to rebuild.

This is experimental work-in-progress but I am running a Hydra (Nix CI) instance with builds of the Lisp packages: https://hydra.nuddy.co/

I started last month with one 16-core Ryzen3 CPU running Linux/x86-64 builds.

This week I added an 80-core Ampere ARM64 build machine and just this moment kicked off an experimental larger build of {sbcl,ecl,abcl} on {x86-64, i686, arm64} at NixOS/nixpkgs#193754 (comment).

The next machine I add will be a Mac M1. Then we should have quite good cross-platform test coverage for Lisp packages.

The tests are very basic right now, and only served up in the raw Hydra webUI, but the intention is to also generate some human-readable reports and e.g. to directly monitor some key projects like SBCL to test changes before they are released.

If anyone wants to collaborate on this, e.g. to have Hydra test some branches of their own for Lisp packages, just leave a comment and I will try to help. The big idea is just to efficiently find and fix problems in Lisp libraries (or their Nix packagings.)

It's a 3d graphics development tool

see cfd4a3d

currently that's done manually in nix code (flattenedDeps)

normally, there are setup hooks in nixpkgs that e.g. add all jars from /share/java to the class path.

maybe a similar mechanism with share/lisp could be used to reduce the amount of hackery in nix code

http://clisp.org

See NixOS/nixpkgs#193754

These are broken as of 60dbe8f:

• commonqt
• qt-libs
• qtools

There's a patch to make them work:

From d7ea3d708bf0a430ac7e23b4f5cbff7160d4b8d0 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Kasper=20Ga=C5=82kowski?= <[email protected]>
Date: Sun, 9 Oct 2022 19:34:49 +0200
Subject: [PATCH 1/3] smokegen: init at v4.14.3

---
maintainers/maintainer-list.nix               |  6 ++++++
.../libraries/smokegen/default.nix            | 21 +++++++++++++++++++
pkgs/top-level/all-packages.nix               |  2 ++
3 files changed, 29 insertions(+)
create mode 100644 pkgs/development/libraries/smokegen/default.nix

diff --git a/maintainers/maintainer-list.nix b/maintainers/maintainer-list.nix
index 33b6269fcbd..4ff47316875 100644
--- a/maintainers/maintainer-list.nix
+++ b/maintainers/maintainer-list.nix
@@ -13980,6 +13980,12 @@
    githubId = 928084;
    name = "Utku Demir";
  };
+  uthar = {
+    email = "[email protected]";
+    github = "uthar";
+    githubId = 15697697;
+    name = "Kasper Gałkowski";
+  };
  uvnikita = {
    email = "[email protected]";
    github = "uvNikita";
diff --git a/pkgs/development/libraries/smokegen/default.nix b/pkgs/development/libraries/smokegen/default.nix
new file mode 100644
index 00000000000..643ae1064db
--- /dev/null
+++ b/pkgs/development/libraries/smokegen/default.nix
@@ -0,0 +1,21 @@
+{ pkgs, lib, ... }:
+
+pkgs.stdenv.mkDerivation rec {
+  pname = "smokegen";
+  version = "v4.14.3";
+  src = pkgs.fetchzip {
+    url = "https://invent.kde.org/unmaintained/${pname}/-/archive/${version}/${pname}-${version}.tar.gz";
+    hash = "sha256-finsoruPeJZLawIjNUJ25Pq54eaCByfALVraNQJPk7c=";
+  };
+  buildInputs = [ pkgs.cmake pkgs.qt4 ];
+  buildPhase = ''
+      cmake .
+    '';
+  meta = with lib; {
+    description = "A general purpose C++ parser with a plugin infrastructure";
+    homepage = "https://invent.kde.org/unmaintained/smokegen";
+    license = licenses.gpl2Only;
+    platforms = platforms.unix;
+    maintainers = with maintainers; [ uthar ];
+  };
+}
diff --git a/pkgs/top-level/all-packages.nix b/pkgs/top-level/all-packages.nix
index 0586a6575fd..3e40a0b4cec 100644
--- a/pkgs/top-level/all-packages.nix
+++ b/pkgs/top-level/all-packages.nix
@@ -4733,6 +4733,8 @@ with pkgs;

  simg2img = callPackage ../tools/filesystems/simg2img { };

+  smokegen = callPackage ../development/libraries/smokegen {};
+
  snazy = callPackage ../development/tools/snazy { };

  snippetpixie = callPackage ../tools/text/snippetpixie { };
-- 
2.34.1


From fd3646f5177ec74430218f1f24c1bc40be4990a2 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Kasper=20Ga=C5=82kowski?= <[email protected]>
Date: Sun, 9 Oct 2022 19:36:54 +0200
Subject: [PATCH 2/3] smokeqt: init at v4.14.3

---
.../development/libraries/smokeqt/default.nix | 21 +++++++++++++++++++
pkgs/top-level/all-packages.nix               |  2 ++
2 files changed, 23 insertions(+)
create mode 100644 pkgs/development/libraries/smokeqt/default.nix

diff --git a/pkgs/development/libraries/smokeqt/default.nix b/pkgs/development/libraries/smokeqt/default.nix
new file mode 100644
index 00000000000..145a9f243bb
--- /dev/null
+++ b/pkgs/development/libraries/smokeqt/default.nix
@@ -0,0 +1,21 @@
+{ pkgs, lib, ... }:
+
+pkgs.stdenv.mkDerivation rec {
+  pname = "smokeqt";
+  version = "v4.14.3";
+  src = pkgs.fetchzip {
+    url = "https://invent.kde.org/unmaintained/${pname}/-/archive/${version}/${pname}-${version}.tar.gz";
+    hash = "sha256-8FiEGF8gduVw5I/bi2wExGUWmjIjYEhWpjpXKJGBNMg=";
+  };
+  cmakeFlags = [
+    "-DCMAKE_CXX_STANDARD=98"
+  ];
+  buildInputs = [ pkgs.cmake pkgs.qt4 pkgs.smokegen ];
+  meta = with lib; {
+    description = "Bindings for the Qt libraries";
+    homepage = "https://invent.kde.org/unmaintained/smokeqt";
+    license = licenses.gpl2Only;
+    platforms = platforms.unix;
+    maintainers = with maintainers; [ uthar ];
+  };
+}
diff --git a/pkgs/top-level/all-packages.nix b/pkgs/top-level/all-packages.nix
index 3e40a0b4cec..6362b923c37 100644
--- a/pkgs/top-level/all-packages.nix
+++ b/pkgs/top-level/all-packages.nix
@@ -4735,6 +4735,8 @@ with pkgs;

  smokegen = callPackage ../development/libraries/smokegen {};

+  smokeqt = callPackage ../development/libraries/smokeqt {};
+
  snazy = callPackage ../development/tools/snazy { };

  snippetpixie = callPackage ../tools/text/snippetpixie { };
-- 
2.34.1


From 422f1c24bd9ebb0c413d74bf11f28eb82441e525 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Kasper=20Ga=C5=82kowski?= <[email protected]>
Date: Sun, 9 Oct 2022 19:38:32 +0200
Subject: [PATCH 3/3] sbclPackages: fix build of qt, qt-libs and qtools

---
.../development/lisp-modules-new/packages.nix | 91 +++++++++++++++++++
.../patches/qt-libs-dont-download.patch       | 36 ++++++++
.../patches/qtools-use-nix-libs.patch         | 19 ++++
3 files changed, 146 insertions(+)
create mode 100644 pkgs/development/lisp-modules-new/patches/qt-libs-dont-download.patch
create mode 100644 pkgs/development/lisp-modules-new/patches/qtools-use-nix-libs.patch

diff --git a/pkgs/development/lisp-modules-new/packages.nix b/pkgs/development/lisp-modules-new/packages.nix
index 97c1cf0fca5..1ad06f615b5 100644
--- a/pkgs/development/lisp-modules-new/packages.nix
+++ b/pkgs/development/lisp-modules-new/packages.nix
@@ -11,6 +11,9 @@ let
    optionals
    hasSuffix
    splitString
+    remove
+    optionalString
+    stringLength
  ;

  # Used by builds that would otherwise attempt to write into storeDir.
@@ -42,6 +45,11 @@ let
      # Patches are already applied in `build`
      patches = [];
      src = build;
+      # TODO(kasper): handle this with a setup hook
+      LD_LIBRARY_PATH =
+        build.LD_LIBRARY_PATH
+        + (optionalString (stringLength build.LD_LIBRARY_PATH != 0) ":")
+        + "${build}";
    });

  # A little hacky
@@ -332,6 +340,89 @@ let
    version = "f19162e76";
  });

+  qt = let
+    rev = "dffff3ee3dbd0686c85c323f579b8bbf4881e60e";
+  in build-with-compile-into-pwd rec {
+    pname = "commonqt";
+    version = builtins.substring 0 7 rev;
+
+    src = pkgs.fetchFromGitHub {
+      inherit rev;
+      owner = pname;
+      repo = pname;
+      hash = "sha256-GAgwT0D9mIkYPTHfCH/KxxIv7b6QGwcxwZE7ehH5xug=";
+    };
+
+    buildInputs = [ pkgs.qt4 ];
+    nativeBuildInputs = [ pkgs.smokegen pkgs.smokeqt ];
+    nativeLibs = [ pkgs.qt4 pkgs.smokegen pkgs.smokeqt ];
+
+    systems = [ "qt" ];
+
+    lispLibs = with ql; [
+      cffi named-readtables cl-ppcre alexandria
+      closer-mop iterate trivial-garbage bordeaux-threads
+    ];
+  };
+
+  qt-libs = build-with-compile-into-pwd {
+    inherit (ql.qt-libs) pname version src;
+    patches = [ ./patches/qt-libs-dont-download.patch ];
+    prePatch = ''
+      substituteInPlace systems/*.asd --replace ":qt+libs" ":qt"
+      echo "LD Path: $LD_LIBRARY_PATH"
+    '';
+    lispLibs = ql.qt-libs.lispLibs ++ [ qt ];
+    systems = [
+      "qt-libs"
+      "commonqt"
+      # "phonon"
+      # "qimageblitz"
+      # "qsci"
+      "qt3support"
+      "qtcore"
+      "qtdbus"
+      "qtdeclarative"
+      "qtgui"
+      "qthelp"
+      "qtnetwork"
+      "qtopengl"
+      "qtscript"
+      "qtsql"
+      "qtsvg"
+      "qttest"
+      "qtuitools"
+      # "qtwebkit"
+      "qtxml"
+      "qtxmlpatterns"
+      # "qwt"
+      "smokebase"
+    ];
+  };
+  commonqt = qt-libs;
+  qt3support = qt-libs;
+  qtcore = qt-libs;
+  qtdbus = qt-libs;
+  qtdeclarative = qt-libs;
+  qtgui = qt-libs;
+  qthelp = qt-libs;
+  qtnetwork = qt-libs;
+  qtopengl = qt-libs;
+  qtscript = qt-libs;
+  qtsql = qt-libs;
+  qtsvg = qt-libs;
+  qttest = qt-libs;
+  qtuitools = qt-libs;
+  qtxml = qt-libs;
+  qtxmlpatterns = qt-libs;
+  smokebase = qt-libs;
+
+  qtools = build-with-compile-into-pwd {
+    inherit (ql.qtools) pname version src nativeLibs;
+    lispLibs = [ qt ] ++ remove ql.qt_plus_libs ql.qtools.lispLibs ++ [ qt-libs ];
+    patches = [ ./patches/qtools-use-nix-libs.patch ];
+  };
+
  };

in packages
diff --git a/pkgs/development/lisp-modules-new/patches/qt-libs-dont-download.patch b/pkgs/development/lisp-modules-new/patches/qt-libs-dont-download.patch
new file mode 100644
index 00000000000..e21a44c2a17
--- /dev/null
+++ b/pkgs/development/lisp-modules-new/patches/qt-libs-dont-download.patch
@@ -0,0 +1,36 @@
+--- a/qt-libs.asd
++++ b/qt-libs.asd
+@@ -17,5 +17,4 @@
+   :components ((:file "qt-libs"))
+   :depends-on (:qt-lib-generator
+                :cl-ppcre
+-               :cffi)
+-  :perform (asdf:load-op :after (op c) (uiop:symbol-call :qt-libs :ensure-standalone-libs)))
++               :cffi))
+--- a/qt-libs.lisp
++++ b/qt-libs.lisp
+@@ -94,16 +94,14 @@
+   standalone-dir)
+
+ (defun %ensure-lib-loaded (file)
+-  (let ((file (etypecase file
+-                (pathname file)
+-                (string (installed-library-file file))))
+-        (name (intern (string-upcase (pathname-name file))))
+-        #+sbcl(sb-ext:*muffled-warnings* 'style-warning))
+-    (cffi::register-foreign-library
+-     name `((T ,file))
+-     :search-path (to-directory file))
+-    (unless (cffi:foreign-library-loaded-p name)
+-      (cffi:load-foreign-library name))))
++ (let ((name (make-pathname :name (format nil "lib~a" file)
++                             :type #+unix "so"
++                                   #+darwin "dylib")))
++    (or (find-if (lambda (lib)
++                   (equal (cffi:foreign-library-pathname lib)
++                          (namestring name)))
++                 (cffi:list-foreign-libraries))
++        (cffi:load-foreign-library name))))
+
+ (defun ensure-lib-loaded (file)
+   (cond ((pathnamep file)
\ No newline at end of file
diff --git a/pkgs/development/lisp-modules-new/patches/qtools-use-nix-libs.patch b/pkgs/development/lisp-modules-new/patches/qtools-use-nix-libs.patch
new file mode 100644
index 00000000000..b380894b04a
--- /dev/null
+++ b/pkgs/development/lisp-modules-new/patches/qtools-use-nix-libs.patch
@@ -0,0 +1,19 @@
+Dont use the qt+libs system for managing Qt dependencies, because Nix provides
+them already.
+Don't build the deploy.lisp helper file, because Nix also can handle deployment.
+--- a/qtools.asd
++++ b/qtools.asd
+@@ -33,10 +33,9 @@
+                (:file "generate")
+                (:file "dynamic")
+                (:file "precompile")
+-               (:file "deploy")
+                (:file "fast-call")
+                (:file "documentation"))
+-  :depends-on (:qt+libs
++  :depends-on (:qt
+                :deploy
+                :cl-ppcre
+                :closer-mop
+
+Diff finished.  Sun Oct  2 14:38:06 2022
\ No newline at end of file
-- 
2.34.1

I noticed that Nix crashes with a stack overflow when I try to build the full set of packages for sbcl like this:

$ nix-build . -A lispPackages_new.sbclPackages
error: stack overflow (possible infinite recursion)

Does anyone understand what the problem is? (Is it a true infinite recursion problems or just a deep recursion due to coding style that would need to be changed?)

My use case is that I would like to test building all packages via a Hydra CI instance. For this I need to evaluate a Nix expression that returns a complete (or at least ample) set of package derivations.

there should be something akin to buildPythonApplication from nixpkgs, but for ASDF lisp, perhaps using asdf:make combined with asdf:program-op.

Don't use execv flags to load ASDF, because for SBCL it makes it impossible to use C runtime flags.

sbcl --load /nix/store/.../asdf.fasl --dynamic-space-size 4096

fatal error before reaching READ-EVAL-PRINT loop: 
  C runtime option --dynamic-space-size in the middle of Lisp options.

Don't know how to do it, I tried with a pipe to stdin, but got really weird results... It worked but then the repl wouldn't catch EOF from Ctrl+D, and it would't catch SIGINT from Ctrl+C.

Glad my issue about reusing quicklisp-to-nix was noticed.

I have started writing an overlay for Common Lisp in December but there were questions I could not answer myself. Now I think that I overcomplicated stuff. Anyway, here are the things I could not decide on:

1. Usually we have sevaral .asd files with different systems. Should those be separate nix packages or not?
2. Should we run tests or they can be ignored?
3. Systems can have different dependencies on different implementations (noticed in tests, where sbcl has sb-rt and ccl requires rt)

Wonder what your thoughts on these are.

Also, I thought that CL in nix should not depend on quicklisp, having its own package set. But earlier today I had a thought that that could have been too much.

For constructing CL_SOURCE_REGISTRY I had a function similar to the one I found in your code. I have just written a setup-hook to replace that here.

With this mess I wanted to generate package definitions without quicklisp.

Overall my nix knowledge is not really great, and I would love to contribute instead of writing from scratch.

It would be nice to run tests (asdf:test-system) alongside each package to see if it works.

Now test systems are built separately, but with (asdf:load-system), and it doesn't fit nicely into mkDerivation checkPhase.

Maybe could default to a package/test or package/tests system name?

Is there an easy way to override the source of a Lisp package when it includes many sub-packages?

For example I'd like to replace the source to mcclim but we have many derivations (mcclim, mcclim-bezier, ...) and they each have a separate src attribute that references an external url/sha256. The best option I can think of is to override all of those derivations together.

It might be neat though if the sources themselves were a named derivation that could be overridden...? Or what's the least-bad way to do it today? :)

Hey is there a vision for how we can use nix-cl to define custom Lisp package distributions e.g. capturing the exact dependencies of a Lisp-based application?

I recently discovered a project called Qlot. This looks pretty convenient for writing a list of packages and having flexibility about where each one should be fetch from. Seems to also have functionality to generate a lock file to capture exact versions. Maybe it would make sense to support building package sets from qlock files in addition to Quicklisp distribution files?

I have been wondering about this in the context of kons-9 (kaveh808/kons-9#191) where the project started off using Quicklisp for everything but sometimes needs more specific versions of things. I wrote Nix derivations for the special dependencies by hand but that was a bit laborious e.g. writing down the list of dependent packages, writing down the list of systems, etc, which nix-cl automates so nicely when sourcing from quicklisp.