j3pic / lisp-binary Goto Github PK

View Code? Open in Web Editor NEW

90.0 90.0 15.0 1.22 MB

A library to easily read and write complex binary formats.

License: GNU General Public License v3.0

Common Lisp 99.83% Shell 0.17%

lisp-binary's People

Contributors

Stargazers

Watchers

Forkers

bharath1097 christophejunke pfdietz ralt adiayyakad cl98k alanruttenberg phmarek yssource bohonghuang ldbeth goenninger-b-t apr3vau joelreymont

lisp-binary's Issues

TravisCI is gone.

Tests are totally broken for Travis CI. It seems to be an unfixable issue with my account. The only alternative that will work is to manually set up a box and a GitHub integration for tests, or else pay for CircleCI or another alternative.

Support cut-off data structures?

I'm using lisp-binary to look at data contained in strace output; for example, sendto against UDP port 53 is very likely to be a DNS query or answer.

Now I can't always rely on having the full data block available - strace has a default limit, and even when its string dump length is set via -s (which has performance implications!) there's no guarantee that it'll always be high enough. (TLS handshakes might be several KB, depending on the SSL certificates!)

From the strace syscall output I can infer the data block length (it says how many bytes were read or written even if not all are dumped!);
but simply padding the binary data (eg. with NULs) and then use something like :junk-allowed doesn't always work -- some data structures would be invalid (like UTF 8) and would need specialized handling for each member.

So I'd like to have a parameter that says "parse as far as possible, and leave the rest unbound" (or at default values) -- more or less the opposite of parse-integers :junk-allowed.

Thanks!

Bit streams in CCL: BIT-STREAM's slot definitions are too strict for how they are used within the library.

(defclass bit-stream (fundamental-binary-stream)
  ((element-bits :type fixnum :initform 8 :initarg :element-bits)
   (real-stream :type stream :initarg :real-stream)
   #+unix (unix-fd :type fixnum :initarg :unix-fd)
   #+windows (file-handle :type integer :initarg :file-handle)
   (last-byte :type unsigned-byte :initform 0)
   (last-op :type unsigned-byte :initform nil)
   (bits-left :type integer :initform 0)
   (byte-order :type keyword :initarg :byte-order :initform :little-endian)))

You can have an initform of nil for a slot type of unsigned-byte under SBCL, but under CCL this is an error instead of a warning. This part of the library does not work.

SBCL runs out of heap space during compilation

Hi,
I experienced the issue mentioned in the title whenevever I tried to compile the library with SBCL 1.4.3.
I was told by the friendly folks on #sbcl to run SBCL with the --dynamic-space-size=8129 and it worked.

Since it took some hours of headscratching to find the solution, I wanded to post it here for others.

defbinary sensitive to value of print-case

(let ((*print-case* :downcase))
 (defbinary foo () (size 0 :type (unsigned-byte 32)) (type "    " :type (fixed-length-string 4)))
 (with-open-binary-file (f "myfile" :direction :input) 
   (read-binary 'foo f)))

-> error The function |MAKE-foo| is undefined.

Could fix with:

diff --git a/binary-2.lisp b/binary-2.lisp
index d1f532a..da03c34 100644
--- a/binary-2.lisp
+++ b/binary-2.lisp
@@ -41,7 +41,8 @@
     (or (if (consp con)
 	    (car con)
 	    con)
-	(intern (format nil "MAKE-~a" name) (symbol-package name)))))
+	(let ((*print-case* :upcase))
+          (intern (format nil "MAKE-~a" name) (symbol-package name))))))
 
 (defun expand-defbinary-field (name default-value &rest other-keys &key type (byte-order :little-endian)
 							      byte-count-name
diff --git a/utils.lisp b/utils.lisp
index 4822961..cb6863d 100644
--- a/utils.lisp
+++ b/utils.lisp
@@ -214,10 +214,11 @@ to a list of variables, and will expand to a nested MULTIPLE-VALUE-BIND expressi
        node)) form))
 
 (defun expand-struct-like-defclass-slot (class-name name default-value &key type)
-  `(,name :accessor ,(intern (format nil "~a-~a" class-name name))
-	  :initform ,default-value
-	  :initarg ,(intern (symbol-name name) :keyword)
-	  ,@(if type `(:type ,type))))
+  (let ((*print-case* :upcase))
+    `(,name :accessor ,(intern (format nil "~a-~a" class-name name))
+	    :initform ,default-value
+	    :initarg ,(intern (symbol-name name) :keyword)
+	    ,@(if type `(:type ,type)))))
 
 (defun add-default-value-if-needed (def)
   (if (keywordp (car def))

read-integer constantly returns zero on closed socket stream

Hello, nice library!
This could just be a matter of documenting?
I see that read-integer uses read-sequence via read-bytes but doesn't check that bytes-read is non-zero. however since bytes-read is returned as the second value, the user can check this condition for themselves.
But it is unexpected, (expect an error in my opinion) and it would be nice if it was noted in the docstring.

Slots of `:include`d structures not read

(lisp-binary:defbinary foo ()
  (slot-1 0 :type (unsigned-byte 8)))

(lisp-binary:defbinary bar (:include foo)
  (slot-2 0 :type (unsigned-byte 8)))

When reading a structure BAR, SLOT-1 is never read in - and so the whole data is read from offsets and wrong.
Eg. SLOT-2 is read from the position where SLOT-1 resides in the binary.

I guess that all :included structures' slots must be (recursively?) used in the constructor.

Can't call READ-BINARY on CCL under MacOS

(defpackage mpeg-ts-converter
  (:use :common-lisp :lisp-binary))

(in-package :mpeg-ts-converter)

(define-enum adaption-field 1/4 ()
  (payload-only #b01)
  (adaption-only #b10)
  (adaption-and-payload #b11))
  

(defbinary ts-header ()
  (sync 0 :type (magic :actual-type (unsigned-byte 8)
		       :value #x47))
  (transport-error 0 :type (unsigned-byte 1))
  (payload-unit-start-indicator 0 :type (unsigned-byte 1))
  (transport-priority-indicator 0 :type (unsigned-byte 1))
  (packet-identifier 0 :type (unsigned-byte 13))
  (scramble-control 0 :type (unsigned-byte 2))
  (adaption-indicator :payload-only :type adaption-field))

(with-open-binary-file (in "/tmp/test.ts")
  (read-binary 'ts-header in))

The error:

There is no applicable method for the generic function:
  #<STANDARD-GENERIC-FUNCTION READ-BINARY #x302001B0408F>
when called with arguments:
  (TS-HEADER
   #<BASIC-FILE-BINARY-INPUT-STREAM ("/tmp/grinch/1.mp4"/4 ISO-8859-1) #x302001EB4C3D>)
   [Condition of type CCL:NO-APPLICABLE-METHOD-EXISTS]

Restarts:
 0: [CONTINUE] Try calling it again
 1: [RETRY] Retry SLIME REPL evaluation request.
 2: [*ABORT] Return to SLIME's top level.
 3: [ABORT-BREAK] Reset this thread
 4: [ABORT] Kill this thread

Backtrace:
  0: (#<CCL::STANDARD-KERNEL-METHOD NO-APPLICABLE-METHOD (T)> #<STANDARD-GENERIC-FUNCTION READ-BINARY #x302001B0408F> TS-HEADER #<BASIC-FILE-BINARY-INPUT-STREAM ("/tmp/grinch/1.mp4"/4 ISO-8859-1) #x302001E..
  1: (NIL #<Unknown Arguments>)
  2: (NIL #<Unknown Arguments>)
  3: (#<Anonymous Function #x302001EB562F>)
  4: (CCL::CHEAP-EVAL (WITH-OPEN-BINARY-FILE (IN "/tmp/grinch/1.mp4") (READ-BINARY 'TS-HEADER IN)))

Using macros to build custom types

Let's say i have a custom reader and writer function that I want to convert into a macro to add to my structs.

Here's an example.

(defmacro test-custom (size)
`(custom
:reader (lambda (stream) (read-bytes ,size stream))
:writer (lambda (object stream) (write-bytes object stream ,size))))

(defbinary test
(a 0 :type (eval (test-custom 1))))

I'm not entirely sure if this would even work, because despite it passing compilation I don't think defbinary actually registered that this is a custom read and write function. So is there any conceivable way this could work?

Thank you.

Dependencies are fucked in the latest Quicklisp distro in SBCL 1.1.6.

They removed LISP-BINARY from Quicklisp, but also, Moptilities and CLOSER-MOP seem to be broken on SBCL 1.1.6, which is the latest version available for MacOS.

I don't know if the problem is the SBCL version or if the libraries are just broken.

There are no known workarounds.

My CCL installation is still working, but that may be because I'm using an older version of those libraries.

Heap exhaustion when reading structs

Oftentimes when I attempt to read from to a binary struct, I get a heap-exhausted-error.

What should one do to prevent this from happening, would closing some other applications suffice?

EDIT: I should state that I have written the file to immediately read an object on loading, so I don't think this is garbage collection.

PAD-FIXED-LENGTH-STRING is wrong for non-8-bit string encodings

lisp-binary::pad-fixed-length-string adds its padding before encoding. This works fine for encodings such as ASCII and Latin-1, but will fail for encodings such as UTF-8, in which the byte length may differ from the character length.

Padding the string after encoding might not work either, since the padding bytes would have to respect, (and therefore the writer generator must be aware of), the rules of each encoding.

One way to fix it could be to use a binary-search algorithm to find the right number of padding characters to add to the pre-encoded string to get the desired post-encoding length. In the worst case scenario, this could require re-encoding the same string dozens of times.

counted-array's write-binary does not respect byte-order

The argument is not passed there https://github.com/j3pic/lisp-binary/blob/master/types.lisp#L313

Doesn't build on Raspberry Pi

Error message:

failed AVER: (TYPEP OFFSET '(UNSIGNED-BYTE 12))
This is probably a bug in SBCL itself. (Alternatively, SBCL
might have been corrupted by bad user code, e.g. by an undefined
Lisp operation like (FMAKUNBOUND 'COMPILE), or by stray pointers
from alien code or from unsafe Lisp code; or there might be a
bug in the OS or hardware that SBCL is running on.) If it seems
to be a bug in SBCL itself, the maintainers would like to know
about it. Bug reports are welcome on the SBCL mailing lists,
which you can find at <http://sbcl.sourceforge.net/>.
   [Condition of type SB-INT:BUG]

Backtrace:
  0: (SB-INT:BUG "~@<failed AVER: ~2I~_~A~:>" (TYPEP SB-VM::OFFSET (QUOTE (UNSIGNED-BYTE 12))))
  1: (SB-IMPL::%FAILED-AVER (TYPEP SB-VM::OFFSET (QUOTE (UNSIGNED-BYTE 12))))
  2: (SB-VM::EMIT-LOAD/STORE-INSTRUCTION #<SB-ASSEM:SEGMENT :TYPE :REGULAR> :AL :STORE :WORD #<SB-C:TN t1[R0]> #S(SB-VM::MEMORY-OPERAND :BASE #<SB-C:TN t2[CFP]> :OFFSET 4896 :DIRECTION :UP :MODE :OFFSET))
  3: (SB-VM::STR-INST-EMITTER #<SB-ASSEM:SEGMENT :TYPE :REGULAR> #<SB-C::VOP :INFO SB-C:MOVE :ARGS #<SB-C:TN-REF :TN #<SB-C:TN t3[R0]> :WRITE-P NIL :VOP SB-C:MOVE> :RESULTS #<SB-C:TN-REF :TN #<SB-C:TN t4[C..
  4: (SB-VM::STORE-STACK #<SB-C::VOP :INFO SB-C:MOVE :ARGS #<SB-C:TN-REF :TN #<SB-C:TN t3[R0]> :WRITE-P NIL :VOP SB-C:MOVE> :RESULTS #<SB-C:TN-REF :TN #<SB-C:TN t4[CS1224]> :WRITE-P T :VOP SB-C:MOVE>> #<SB..
  5: (SB-C::GENERATE-CODE #<SB-C:COMPONENT :NAME (SB-C::ESCAPE-FUN #:EXIT-BLOCK-0) {51C4AC19}>)
  6: (SB-C::%COMPILE-COMPONENT #<SB-C:COMPONENT :NAME (SB-C::ESCAPE-FUN #:EXIT-BLOCK-0) {51C4AC19}>)
  7: (SB-C::COMPILE-COMPONENT #<SB-C:COMPONENT :NAME (SB-C::ESCAPE-FUN #:EXIT-BLOCK-0) {51C4AC19}>)
  8: (SB-C::%COMPILE (SB-INT:NAMED-LAMBDA LISP-BINARY::EXPAND-DEFBINARY-TYPE-FIELD (LISP-BINARY::STRUCT-NAME LISP-BINARY::TYPE-INFO) (DECLARE (TYPE LISP-BINARY::DEFBINARY-TYPE LISP-BINARY::TYPE-INFO) (OPTI..
  9: (SB-C::FOPCOMPILE-FUNCTION (SB-INT:NAMED-LAMBDA LISP-BINARY::EXPAND-DEFBINARY-TYPE-FIELD (LISP-BINARY::STRUCT-NAME LISP-BINARY::TYPE-INFO) (DECLARE (TYPE LISP-BINARY::DEFBINARY-TYPE LISP-BINARY::TYPE-..
 10: (SB-C::FOPCOMPILE (SB-IMPL::%DEFUN (QUOTE LISP-BINARY::EXPAND-DEFBINARY-TYPE-FIELD) (SB-INT:NAMED-LAMBDA LISP-BINARY::EXPAND-DEFBINARY-TYPE-FIELD (LISP-BINARY::STRUCT-NAME LISP-BINARY::TYPE-INFO) (DEC..
 11: (SB-C::CONVERT-AND-MAYBE-COMPILE (SB-IMPL::%DEFUN (QUOTE LISP-BINARY::EXPAND-DEFBINARY-TYPE-FIELD) (SB-INT:NAMED-LAMBDA LISP-BINARY::EXPAND-DEFBINARY-TYPE-FIELD (LISP-BINARY::STRUCT-NAME LISP-BINARY::..
 12: ((FLET SB-C::DEFAULT-PROCESSOR :IN SB-C::PROCESS-TOPLEVEL-FORM) (SB-IMPL::%DEFUN (QUOTE LISP-BINARY::EXPAND-DEFBINARY-TYPE-FIELD) (SB-INT:NAMED-LAMBDA LISP-BINARY::EXPAND-DEFBINARY-TYPE-FIELD (LISP-BI..
 13: (SB-C::PROCESS-TOPLEVEL-FORM (SB-IMPL::%DEFUN (QUOTE LISP-BINARY::EXPAND-DEFBINARY-TYPE-FIELD) (SB-INT:NAMED-LAMBDA LISP-BINARY::EXPAND-DEFBINARY-TYPE-FIELD (LISP-BINARY::STRUCT-NAME LISP-BINARY::TYPE..
 14: (SB-C::PROCESS-TOPLEVEL-PROGN ((EVAL-WHEN (:COMPILE-TOPLEVEL) (SB-C:%COMPILER-DEFUN # # T)) (SB-IMPL::%DEFUN (QUOTE LISP-BINARY::EXPAND-DEFBINARY-TYPE-FIELD) (SB-INT:NAMED-LAMBDA LISP-BINARY::EXPAND-D..
 15: (SB-C::PROCESS-TOPLEVEL-FORM (PROGN (EVAL-WHEN (:COMPILE-TOPLEVEL) (SB-C:%COMPILER-DEFUN # # T)) (SB-IMPL::%DEFUN (QUOTE LISP-BINARY::EXPAND-DEFBINARY-TYPE-FIELD) (SB-INT:NAMED-LAMBDA LISP-BINARY::EXP..
 16: ((FLET SB-C::DEFAULT-PROCESSOR :IN SB-C::PROCESS-TOPLEVEL-FORM) (DEFUN LISP-BINARY::EXPAND-DEFBINARY-TYPE-FIELD (LISP-BINARY::STRUCT-NAME LISP-BINARY::TYPE-INFO) "Expands the :TYPE field of a DEFBINAR..
 17: (SB-C::PROCESS-TOPLEVEL-FORM (DEFUN LISP-BINARY::EXPAND-DEFBINARY-TYPE-FIELD (LISP-BINARY::STRUCT-NAME LISP-BINARY::TYPE-INFO) "Expands the :TYPE field of a DEFBINARY form. Returns three values: ..)
 18: (SB-C::SUB-SUB-COMPILE-FILE #<SB-C::SOURCE-INFO >)
 19: ((FLET SB-THREAD::WITH-RECURSIVE-LOCK-THUNK :IN SB-C::SUB-COMPILE-FILE))
 20: ((LAMBDA NIL :IN SB-C::SUB-COMPILE-FILE))
 21: ((FLET SB-C::WITH-IT :IN SB-C::%WITH-COMPILATION-UNIT))
 22: (SB-C::SUB-COMPILE-FILE #<SB-C::SOURCE-INFO >)
 23: (COMPILE-FILE #P"/home/pi/quicklisp/local-projects/lisp-binary/binary-2.lisp" :OUTPUT-FILE #P"/home/pi/.cache/common-lisp/sbcl-1.2.4.debian-linux-arm/home/pi/quicklisp/local-projects/lisp-binary/binar..
 24: (UIOP/UTILITY:CALL-WITH-MUFFLED-CONDITIONS #<CLOSURE (LAMBDA NIL :IN UIOP/LISP-BUILD:COMPILE-FILE*) {51C4A53D}> NIL)
 25: (UIOP/PATHNAME:CALL-WITH-ENOUGH-PATHNAME #P"/home/pi/quicklisp/local-projects/lisp-binary/binary-2.lisp" NIL #<CLOSURE (LAMBDA (UIOP/LISP-BUILD::INPUT-FILE) :IN UIOP/LISP-BUILD:COMPILE-FILE*) {51C4A51..
 26: (UIOP/LISP-BUILD:COMPILE-FILE* #P"/home/pi/quicklisp/local-projects/lisp-binary/binary-2.lisp" :OUTPUT-FILE #P"/home/pi/.cache/common-lisp/sbcl-1.2.4.debian-linux-arm/home/pi/quicklisp/local-projects/..
 27: (ASDF/LISP-ACTION:PERFORM-LISP-COMPILATION #<ASDF/LISP-ACTION:COMPILE-OP > #<ASDF/LISP-ACTION:CL-SOURCE-FILE "lisp-binary" "binary-2">)
 28: ((SB-PCL::EMF ASDF/ACTION:PERFORM) #<unavailable argument> #<unavailable argument> #<ASDF/LISP-ACTION:COMPILE-OP > #<ASDF/LISP-ACTION:CL-SOURCE-FILE "lisp-binary" "binary-2">)
 29: ((:METHOD ASDF/ACTION:PERFORM-WITH-RESTARTS :AROUND (T T)) #<ASDF/LISP-ACTION:COMPILE-OP > #<ASDF/LISP-ACTION:CL-SOURCE-FILE "lisp-binary" "binary-2">) [fast-method]
 30: ((:METHOD ASDF/PLAN:PERFORM-PLAN (LIST)) ((#1=#<ASDF/LISP-ACTION:COMPILE-OP > . #2=#<ASDF/LISP-ACTION:CL-SOURCE-FILE #3="lisp-binary" "binary-2">) (#4=#<ASDF/LISP-ACTION:LOAD-OP :VERBOSE NIL> . #2#) (..
 31: ((FLET SB-C::WITH-IT :IN SB-C::%WITH-COMPILATION-UNIT))
 32: ((:METHOD ASDF/PLAN:PERFORM-PLAN :AROUND (T)) ((#1=#<ASDF/LISP-ACTION:COMPILE-OP > . #2=#<ASDF/LISP-ACTION:CL-SOURCE-FILE #3="lisp-binary" "binary-2">) (#4=#<ASDF/LISP-ACTION:LOAD-OP :VERBOSE NIL> . #..
 33: ((FLET SB-C::WITH-IT :IN SB-C::%WITH-COMPILATION-UNIT))
 34: ((:METHOD ASDF/PLAN:PERFORM-PLAN :AROUND (T)) #<ASDF/PLAN:SEQUENTIAL-PLAN {51C4A329}> :VERBOSE NIL) [fast-method]
 35: ((:METHOD ASDF/OPERATE:OPERATE (ASDF/OPERATION:OPERATION ASDF/COMPONENT:COMPONENT)) #<ASDF/LISP-ACTION:LOAD-OP :VERBOSE NIL> #<ASDF/SYSTEM:SYSTEM "lisp-binary"> :VERBOSE NIL) [fast-method]
 36: ((SB-PCL::EMF ASDF/OPERATE:OPERATE) #<unused argument> #<unused argument> #<ASDF/LISP-ACTION:LOAD-OP :VERBOSE NIL> #<ASDF/SYSTEM:SYSTEM "lisp-binary"> :VERBOSE NIL)
 37: ((LAMBDA NIL :IN ASDF/OPERATE:OPERATE))
 38: ((:METHOD ASDF/OPERATE:OPERATE :AROUND (T T)) #<ASDF/LISP-ACTION:LOAD-OP :VERBOSE NIL> #<ASDF/SYSTEM:SYSTEM "lisp-binary"> :VERBOSE NIL) [fast-method]
 39: ((SB-PCL::EMF ASDF/OPERATE:OPERATE) #<unused argument> #<unused argument> ASDF/LISP-ACTION:LOAD-OP "lisp-binary" :VERBOSE NIL)
 40: ((LAMBDA NIL :IN ASDF/OPERATE:OPERATE))
 41: (ASDF/CACHE:CALL-WITH-ASDF-CACHE #<CLOSURE (LAMBDA NIL :IN ASDF/OPERATE:OPERATE) {51C4A265}> :OVERRIDE NIL :KEY NIL)
 42: ((:METHOD ASDF/OPERATE:OPERATE :AROUND (T T)) ASDF/LISP-ACTION:LOAD-OP "lisp-binary" :VERBOSE NIL) [fast-method]
 43: ((:METHOD ASDF/OPERATE:OPERATE :AROUND (T T)) ASDF/LISP-ACTION:LOAD-OP "lisp-binary" :VERBOSE NIL) [fast-method]
 44: (ASDF/OPERATE:LOAD-SYSTEM "lisp-binary" :VERBOSE NIL)
 45: (QUICKLISP-CLIENT::CALL-WITH-MACROEXPAND-PROGRESS #<CLOSURE (LAMBDA NIL :IN QUICKLISP-CLIENT::APPLY-LOAD-STRATEGY) {51C4A1BD}>)
 46: (QUICKLISP-CLIENT::AUTOLOAD-SYSTEM-AND-DEPENDENCIES "lisp-binary" :PROMPT NIL)
 47: ((:METHOD QL-IMPL-UTIL::%CALL-WITH-QUIET-COMPILATION (T T)) #<unavailable argument> #<CLOSURE (FLET QUICKLISP-CLIENT::QL :IN QUICKLISP-CLIENT:QUICKLOAD) {51C4A125}>) [fast-method]
 48: ((:METHOD QL-IMPL-UTIL::%CALL-WITH-QUIET-COMPILATION :AROUND (QL-IMPL:SBCL T)) #<QL-IMPL:SBCL {51D3B7F9}> #<CLOSURE (FLET QUICKLISP-CLIENT::QL :IN QUICKLISP-CLIENT:QUICKLOAD) {51C4A125}>) [fast-method..
 49: ((:METHOD QUICKLISP-CLIENT:QUICKLOAD (T)) #<unavailable argument> :PROMPT NIL :SILENT NIL :VERBOSE NIL) [fast-method]
 50: (QL-DIST::CALL-WITH-CONSISTENT-DISTS #<CLOSURE (LAMBDA NIL :IN QUICKLISP-CLIENT:QUICKLOAD) {51C4A0FD}>)
 51: (SB-INT:SIMPLE-EVAL-IN-LEXENV (QUICKLISP-CLIENT:QUICKLOAD :LISP-BINARY) #<NULL-LEXENV>)
 52: (EVAL (QUICKLISP-CLIENT:QUICKLOAD :LISP-BINARY))
 53: (SWANK::EVAL-REGION "(ql:quickload :lisp-binary) ..)
 54: ((LAMBDA NIL :IN SWANK-REPL::REPL-EVAL))
 55: (SWANK-REPL::TRACK-PACKAGE #<CLOSURE (LAMBDA NIL :IN SWANK-REPL::REPL-EVAL) {51C4A05D}>)
 56: (SWANK::CALL-WITH-RETRY-RESTART "Retry SLIME REPL evaluation request." #<CLOSURE (LAMBDA NIL :IN SWANK-REPL::REPL-EVAL) {51C4A04D}>)
 57: (SWANK::CALL-WITH-BUFFER-SYNTAX NIL #<CLOSURE (LAMBDA NIL :IN SWANK-REPL::REPL-EVAL) {51C4A03D}>)
 58: (SWANK-REPL::REPL-EVAL "(ql:quickload :lisp-binary) ..)
 59: (SB-INT:SIMPLE-EVAL-IN-LEXENV (SWANK-REPL:LISTENER-EVAL "(ql:quickload :lisp-binary) ..)
 60: (EVAL (SWANK-REPL:LISTENER-EVAL "(ql:quickload :lisp-binary) ..)
      Locals:
        SB-DEBUG::ARG-0 = (SWANK-REPL:LISTENER-EVAL "(ql:quickload :lisp-binary)\n")
 61: (SWANK:EVAL-FOR-EMACS (SWANK-REPL:LISTENER-EVAL "(ql:quickload :lisp-binary) ..)
 62: (SWANK::PROCESS-REQUESTS T)
 63: ((LAMBDA NIL :IN SWANK::HANDLE-REQUESTS))
 64: ((LAMBDA NIL :IN SWANK::HANDLE-REQUESTS))
 65: (SWANK/SBCL::CALL-WITH-BREAK-HOOK #<FUNCTION SWANK:SWANK-DEBUGGER-HOOK> #<CLOSURE (LAMBDA NIL :IN SWANK::HANDLE-REQUESTS) {51A3CF95}>)
 66: ((FLET SWANK/BACKEND:CALL-WITH-DEBUGGER-HOOK :IN "/usr/share/common-lisp/source/slime/swank/sbcl.lisp") #<FUNCTION SWANK:SWANK-DEBUGGER-HOOK> #<CLOSURE (LAMBDA NIL :IN SWANK::HANDLE-REQUESTS) {51A3CF9..
 67: (SWANK::CALL-WITH-BINDINGS ((*STANDARD-OUTPUT* . #1=#<SWANK/GRAY::SLIME-OUTPUT-STREAM {51A9D7F9}>) (*STANDARD-INPUT* . #2=#<SWANK/GRAY::SLIME-INPUT-STREAM {51AA47E9}>) (*TRACE-OUTPUT* . #1#) (*ERROR-O..
 68: (SWANK::HANDLE-REQUESTS #<SWANK::SINGLETHREADED-CONNECTION {51A99A51}> T)
 69: ((LABELS SWANK/SBCL::RUN :IN SWANK/BACKEND:ADD-FD-HANDLER) 6)
 70: (SB-IMPL::SUB-SUB-SERVE-EVENT NIL NIL)
 71: (SB-IMPL::SUB-SERVE-EVENT NIL NIL NIL)
 72: (SB-SYS:WAIT-UNTIL-FD-USABLE 0 :INPUT NIL T)
 73: (SB-IMPL::REFILL-INPUT-BUFFER #<SB-SYS:FD-STREAM for "standard input" {51A99299}>)
 74: (SB-IMPL::INPUT-CHAR/UTF-8 #<SB-SYS:FD-STREAM for "standard input" {51A99299}> NIL 0)
 75: ((LAMBDA (&REST REST) :IN SB-IMPL::GET-EXTERNAL-FORMAT) #<SB-SYS:FD-STREAM for "standard input" {51A99299}> NIL 0)
 76: (READ-CHAR #<SB-SYS:FD-STREAM for "standard input" {51A99299}> NIL 0 #<unused argument>)
 77: (READ-CHAR #<SYNONYM-STREAM :SYMBOL SB-SYS:*STDIN* {4F0645F9}> NIL 0 #<unused argument>)
 78: (SB-IMPL::%READ-PRESERVING-WHITESPACE #<SYNONYM-STREAM :SYMBOL SB-SYS:*STDIN* {4F0645F9}> NIL (NIL) T)
 79: (SB-IMPL::%READ-PRESERVING-WHITESPACE #<SYNONYM-STREAM :SYMBOL SB-SYS:*STDIN* {4F0645F9}> NIL (NIL) NIL)
 80: (READ #<SYNONYM-STREAM :SYMBOL SB-SYS:*STDIN* {4F0645F9}> NIL (NIL) NIL)
 81: (SB-IMPL::REPL-READ-FORM-FUN #<SYNONYM-STREAM :SYMBOL SB-SYS:*STDIN* {4F0645F9}> #<unavailable argument>)
 82: (SB-IMPL::REPL-FUN NIL)
 83: ((LAMBDA NIL :IN SB-IMPL::TOPLEVEL-REPL))
 84: (SB-IMPL::%WITH-REBOUND-IO-SYNTAX #<CLOSURE (LAMBDA NIL :IN SB-IMPL::TOPLEVEL-REPL) {51A991ED}>)
 85: (SB-IMPL::TOPLEVEL-REPL NIL)
 86: (SB-IMPL::TOPLEVEL-INIT)
 87: ((FLET #:WITHOUT-INTERRUPTS-BODY-75 :IN SAVE-LISP-AND-DIE))
 88: ((LABELS SB-IMPL::RESTART-LISP :IN SAVE-LISP-AND-DIE))

SBCL version:

$ sbcl --version
SBCL 1.3.7.debian

Simple defbinary and also example from wiki fail to compile

Hi all:

The following code results in an error:

(defbinary packet-header (:export t :byte-order :big-endian) ((packet-type class-identifier-p indicators tsi tsf packet-count packet-size) 0 :type (bit-field :raw-type (unsigned-byte 32) :member-types ((unsigned-byte 4) (unsigned-byte 1) (unsigned-byte 3) (unsigned-byte 4) (unsigned-byte 16)))))

=>
In bitfield: Number of values 7 (PACKET-TYPE CLASS-IDENTIFIER-P INDICATORS TSI TSF PACKET-COUNT PACKET-SIZE) doesn't match number of types 0 NIL

I then tried to compile the example in the defbinary wiki page with a similar result. I also tried diiferent Lisp implementations with same results.

What am I doing wrong? Thanks for any hints!

Best,
Frank

Build failure with SBCL 2.2.10

The build log is included below. We are using a custom build script, but this should not be the cause of this failure, since the only thing that changed is the SBCL version from 2.2.9 to 2.2.10 to trigger this failure. The 2.2.10 changelog doesn't have anything suspicious, so it seems possible that this is an SBCL bug. Failure occurs while compiling open-binary which seems unproblematic. However compiling the function individually by pasting it into a REPL, doesn't trigger the problem.

; compiling file "/nix/store/0ls8aaw0yy77lk48hskp3liamr1vgfff-source/utils.lisp" (written 01 JAN 1970 12:00:01 AM):

; wrote //build/-nix-store-0ls8aaw0yy77lk48hskp3liamr1vgfff-source-utils.lisp.fasl
; compilation finished in 0:00:00.113
; compiling file "/nix/store/0ls8aaw0yy77lk48hskp3liamr1vgfff-source/integer.lisp" (written 01 JAN 1970 12:00:01 AM):

; wrote //build/-nix-store-0ls8aaw0yy77lk48hskp3liamr1vgfff-source-integer.lisp.fasl
; compilation finished in 0:00:00.064
; compiling file "/nix/store/0ls8aaw0yy77lk48hskp3liamr1vgfff-source/float.lisp" (written 01 JAN 1970 12:00:01 AM):

; wrote //build/-nix-store-0ls8aaw0yy77lk48hskp3liamr1vgfff-source-float.lisp.fasl
; compilation finished in 0:00:00.114
; compiling file "/nix/store/0ls8aaw0yy77lk48hskp3liamr1vgfff-source/simple-bit-stream.lisp" (written 01 JAN 1970 12:00:01 AM):

; wrote //build/-nix-store-0ls8aaw0yy77lk48hskp3liamr1vgfff-source-simple-bit-stream.lisp.fasl
; compilation finished in 0:00:00.071
; compiling file "/nix/store/0ls8aaw0yy77lk48hskp3liamr1vgfff-source/reverse-stream.lisp" (written 01 JAN 1970 12:00:01 AM):

; wrote //build/-nix-store-0ls8aaw0yy77lk48hskp3liamr1vgfff-source-reverse-stream.lisp.fasl
; compilation finished in 0:00:00.033
; compiling file "/nix/store/0ls8aaw0yy77lk48hskp3liamr1vgfff-source/binary-1.lisp" (written 01 JAN 1970 12:00:01 AM):
Unhandled TYPE-ERROR in thread #<SB-THREAD:THREAD "main thread" RUNNING
                                  {10010900A3}>:
  The value
    #<SB-C::OPTIONAL-DISPATCH
      :%SOURCE-NAME OPEN-BINARY
      :%DEBUG-NAME NIL
      :TYPE #<SB-KERNEL:FUN-TYPE (FUNCTION
                                  (T &KEY (:DIRECTION T) (:IF-EXISTS T)
                                   (:IF-DOES-NOT-EXIST T))
                                  (VALUES (OR STREAM NULL) &OPTIONAL))>
      :WHERE-FROM :DEFINED
      :ARGLIST (#<SB-C::LAMBDA-VAR :%SOURCE-NAME PATHNAME {100112D113}>
                #<SB-C::LAMBDA-VAR
                  :%SOURCE-NAME DIRECTION
                  :FLAGS (SB-C::DELETED)
                  :ARG-INFO #<SB-C::ARG-INFO
                              :KIND :KEYWORD
                              :DEFAULT :INPUT
                              :KEY :DIRECTION {1001340CE3}> {100123A673}>
                #<SB-C::LAMBDA-VAR
                  :%SOURCE-NAME IF-EXISTS
                  :FLAGS (SB-C::DELETED)
                  :ARG-INFO #<SB-C::ARG-INFO
                              :KIND :KEYWORD
                              :KEY :IF-EXISTS {1001340E53}> {100123A723}>
                #<SB-C::LAMBDA-VAR
                  :%SOURCE-NAME IF-DOES-NOT-EXIST
                  :FLAGS (SB-C::DELETED)
                  :ARG-INFO #<SB-C::ARG-INFO
                              :KIND :KEYWORD
                              :KEY :IF-DOES-NOT-EXIST {1001340FC3}> {100123A7D3}>)
      :ALLOWP NIL
      :KEYP T
      :MIN-ARGS 1
      :MAX-ARGS 1
      :ENTRY-POINTS (#<SB-C::CLAMBDA
                       :%SOURCE-NAME SB-C::.ANONYMOUS.
                       :%DEBUG-NAME (SB-C::&OPTIONAL-PROCESSOR OPEN-BINARY)
                       :KIND :DELETED
                       :TYPE #<SB-KERNEL:BUILT-IN-CLASSOID FUNCTION (read-only)>
                       :WHERE-FROM :DEFINED
                       :VARS (PATHNAME) {100123A883}>)
      :MORE-ENTRY #<SB-C::CLAMBDA
                    :%SOURCE-NAME SB-C::.ANONYMOUS.
                    :%DEBUG-NAME (SB-C::&MORE-PROCESSOR OPEN-BINARY)
                    :KIND :OPTIONAL
                    :TYPE #<SB-KERNEL:BUILT-IN-CLASSOID FUNCTION (read-only)>
                    :WHERE-FROM :DEFINED
                    :VARS (PATHNAME #:N-CONTEXT-18 #:N-COUNT-19) {10011A1F23}>
      :MAIN-ENTRY #<SB-C::CLAMBDA
                    :%SOURCE-NAME OPEN-BINARY
                    :%DEBUG-NAME NIL
                    :KIND :OPTIONAL
                    :TYPE #<SB-KERNEL:BUILT-IN-CLASSOID FUNCTION (read-only)>
                    :WHERE-FROM :DEFINED
                    :VARS (PATHNAME #:.DEFAULTING-TEMP. #:.DEFAULTING-TEMP.
                                    #:.DEFAULTING-TEMP.) {10049E2673}> {10010F83A3}>

  is not of type
    SB-C::CLAMBDA

Backtrace for: #<SB-THREAD:THREAD "main thread" RUNNING {10010900A3}>
0: (SB-C::SET-SLOT-OLD-P #<SB-C::COMBINATION :FUN #<SB-C::REF  :LEAF #<SB-C::GLOBAL-VAR :%SOURCE-NAME SB-KERNEL:%RPLACA :TYPE #1=#<SB-KERNEL:FUN-TYPE (FUNCTION (CONS T) (VALUES T &OPTIONAL))> :DEFINED-TYPE #1# :WHERE-FROM :DECLARED :KIND :GLOBAL-FUNCTION {10014E3C33}> {1001438B43}> :ARGS (#<SB-C::REF :%SOURCE-NAME #2=#:LIST5 :LEAF #<SB-C::LAMBDA-VAR :%SOURCE-NAME #2# :TYPE #<SB-KERNEL:CONS-TYPE CONS> {100123FE33}> {10013EF903}> #<SB-C::REF :%SOURCE-NAME #:INJECTEE1 :LEAF #<SB-C::LAMBDA-VAR :%SOURCE-NAME PATHNAME {100112D113}> {100123ABE3}>) {10014229A3}>)
1: ((SB-C::VOP SB-C:SET-SLOT) #<SB-C::VOP :INFO SB-C:SET-SLOT :ARGS #<SB-C:TN-REF :TN #<SB-C:TN t1[R11(d)] :NORMAL> :WRITE-P NIL :VOP SB-C:SET-SLOT> :RESULTS NIL :CODEGEN-INFO (SB-KERNEL:%RPLACA 0 7)>)
2: (SB-C::GENERATE-CODE #<SB-C:COMPONENT :NAME "DEFUN OPEN-BINARY" {1004A32F43}>)
3: (SB-C::%COMPILE-COMPONENT #<SB-C:COMPONENT :NAME "DEFUN OPEN-BINARY" {1004A32F43}>)
4: (SB-C::COMPILE-COMPONENT #<SB-C:COMPONENT :NAME "DEFUN OPEN-BINARY" {1004A32F43}>)
5: (SB-C::COMPILE-TOPLEVEL (#<SB-C::CLAMBDA :%SOURCE-NAME SB-C::.ANONYMOUS. :%DEBUG-NAME (SB-C::TOP-LEVEL-FORM (SB-IMPL::%DEFUN (QUOTE OPEN-BINARY) (SB-INT:NAMED-LAMBDA OPEN-BINARY # # "Like OPEN, but always opens a binary stream suitable for use with the DEFBINARY library." #))) :KIND :TOPLEVEL :TYPE #<SB-KERNEL:BUILT-IN-CLASSOID FUNCTION (read-only)> :WHERE-FROM :DEFINED :VARS NIL {10010F87A3}>) NIL)
6: (SB-C::CONVERT-AND-MAYBE-COMPILE (SB-IMPL::%DEFUN (QUOTE OPEN-BINARY) (SB-INT:NAMED-LAMBDA OPEN-BINARY (PATHNAME &KEY (DIRECTION :INPUT) IF-EXISTS IF-DOES-NOT-EXIST) (DECLARE (SB-C::TOP-LEVEL-FORM)) "Like OPEN, but always opens a binary stream suitable for use with the DEFBINARY library." (BLOCK OPEN-BINARY (APPLY (FUNCTION OPEN) (DIG #))))) (#1=(SB-IMPL::%DEFUN (QUOTE OPEN-BINARY) (SB-INT:NAMED-LAMBDA OPEN-BINARY (PATHNAME &KEY (DIRECTION :INPUT) IF-EXISTS IF-DOES-NOT-EXIST) (DECLARE (SB-C::TOP-LEVEL-FORM)) "Like OPEN, but always opens a binary stream suitable for use with the DEFBINARY library." (BLOCK OPEN-BINARY (APPLY # #)))) (PROGN (EVAL-WHEN (:COMPILE-TOPLEVEL) (SB-C:%COMPILER-DEFUN (QUOTE OPEN-BINARY) T NIL NIL)) #1#) SB-C::ORIGINAL-SOURCE-START 0 4))
7: (SB-C::PROCESS-TOPLEVEL-FORM (SB-IMPL::%DEFUN (QUOTE OPEN-BINARY) (SB-INT:NAMED-LAMBDA OPEN-BINARY (PATHNAME &KEY (DIRECTION :INPUT) IF-EXISTS IF-DOES-NOT-EXIST) (DECLARE (SB-C::TOP-LEVEL-FORM)) "Like OPEN, but always opens a binary stream suitable for use with the DEFBINARY library." (BLOCK OPEN-BINARY (APPLY (FUNCTION OPEN) (DIG #))))) ((PROGN (EVAL-WHEN (:COMPILE-TOPLEVEL) (SB-C:%COMPILER-DEFUN (QUOTE OPEN-BINARY) T NIL NIL)) (SB-IMPL::%DEFUN (QUOTE OPEN-BINARY) (SB-INT:NAMED-LAMBDA OPEN-BINARY (PATHNAME &KEY # IF-EXISTS IF-DOES-NOT-EXIST) (DECLARE #) "Like OPEN, but always opens a binary stream suitable for use with the DEFBINARY library." (BLOCK OPEN-BINARY #)))) SB-C::ORIGINAL-SOURCE-START 0 4) NIL)
8: (SB-C::PROCESS-TOPLEVEL-PROGN ((EVAL-WHEN (:COMPILE-TOPLEVEL) (SB-C:%COMPILER-DEFUN (QUOTE OPEN-BINARY) T NIL NIL)) (SB-IMPL::%DEFUN (QUOTE OPEN-BINARY) (SB-INT:NAMED-LAMBDA OPEN-BINARY (PATHNAME &KEY (DIRECTION :INPUT) IF-EXISTS IF-DOES-NOT-EXIST) (DECLARE (SB-C::TOP-LEVEL-FORM)) "Like OPEN, but always opens a binary stream suitable for use with the DEFBINARY library." (BLOCK OPEN-BINARY (APPLY # #))))) ((PROGN (EVAL-WHEN (:COMPILE-TOPLEVEL) (SB-C:%COMPILER-DEFUN (QUOTE OPEN-BINARY) T NIL NIL)) (SB-IMPL::%DEFUN (QUOTE OPEN-BINARY) (SB-INT:NAMED-LAMBDA OPEN-BINARY (PATHNAME &KEY # IF-EXISTS IF-DOES-NOT-EXIST) (DECLARE #) "Like OPEN, but always opens a binary stream suitable for use with the DEFBINARY library." (BLOCK OPEN-BINARY #)))) SB-C::ORIGINAL-SOURCE-START 0 4) NIL)
9: (SB-C::PROCESS-TOPLEVEL-FORM (PROGN (EVAL-WHEN (:COMPILE-TOPLEVEL) (SB-C:%COMPILER-DEFUN (QUOTE OPEN-BINARY) T NIL NIL)) (SB-IMPL::%DEFUN (QUOTE OPEN-BINARY) (SB-INT:NAMED-LAMBDA OPEN-BINARY (PATHNAME &KEY (DIRECTION :INPUT) IF-EXISTS IF-DOES-NOT-EXIST) (DECLARE (SB-C::TOP-LEVEL-FORM)) "Like OPEN, but always opens a binary stream suitable for use with the DEFBINARY library." (BLOCK OPEN-BINARY (APPLY # #))))) (SB-C::ORIGINAL-SOURCE-START 0 4) NIL)
10: (SB-C::PROCESS-TOPLEVEL-FORM (DEFUN OPEN-BINARY (PATHNAME &KEY (DIRECTION :INPUT) IF-EXISTS IF-DOES-NOT-EXIST) "Like OPEN, but always opens a binary stream suitable for use with the DEFBINARY library." (APPLY (FUNCTION OPEN) (DIG ((INJECT PATHNAME) :DIRECTION (INJECT DIRECTION) :ELEMENT-TYPE (UNSIGNED-BYTE 8) (SPLICE #) (SPLICE #))))) (SB-C::ORIGINAL-SOURCE-START 0 4) NIL)
11: ((LAMBDA (SB-KERNEL:FORM &KEY :CURRENT-INDEX &ALLOW-OTHER-KEYS) :IN SB-C::SUB-COMPILE-FILE) (DEFUN OPEN-BINARY (PATHNAME &KEY (DIRECTION :INPUT) IF-EXISTS IF-DOES-NOT-EXIST) "Like OPEN, but always opens a binary stream suitable for use with the DEFBINARY library." (APPLY (FUNCTION OPEN) (DIG ((INJECT PATHNAME) :DIRECTION (INJECT DIRECTION) :ELEMENT-TYPE (UNSIGNED-BYTE 8) (SPLICE #) (SPLICE #))))) :CURRENT-INDEX 4)
12: (SB-C::%DO-FORMS-FROM-INFO #<FUNCTION (LAMBDA (SB-KERNEL:FORM &KEY :CURRENT-INDEX &ALLOW-OTHER-KEYS) :IN SB-C::SUB-COMPILE-FILE) {52C208DB}> #<SB-C::SOURCE-INFO {10048CBB53}> SB-C::INPUT-ERROR-IN-COMPILE-FILE)
13: ((FLET "LAMBDA0" :IN "SYS:SRC;COMPILER;MAIN.LISP"))
14: ((FLET SB-C::WITH-IT :IN SB-C::%WITH-COMPILATION-UNIT))
15: (SB-C::SUB-COMPILE-FILE #<SB-C::SOURCE-INFO {10048CBB53}> NIL)
16: (COMPILE-FILE "/nix/store/0ls8aaw0yy77lk48hskp3liamr1vgfff-source/binary-1.lisp" :OUTPUT-FILE #P"//build/-nix-store-0ls8aaw0yy77lk48hskp3liamr1vgfff-source-binary-1.lisp.fasl" :VERBOSE NIL :PRINT NIL :EXTERNAL-FORMAT :DEFAULT :PROGRESS NIL :TRACE-FILE NIL :BLOCK-COMPILE NIL :ENTRY-POINTS NIL :EMIT-CFASL NIL)
17: (COMMON-LISP-USER::NIX-COMPILE-LISP "/nix/store/0ls8aaw0yy77lk48hskp3liamr1vgfff-source/binary-1.lisp")
18: ((LAMBDA NIL :IN "/nix/store/gqc6ldyg0n3frga5ij367irl9fn8q1yj-sbcl-compile.lisp"))
19: (SB-INT:SIMPLE-EVAL-IN-LEXENV (LET ((*COMPILE-VERBOSE* T) (COMMON-LISP-USER::CATTED-FASL (MAKE-PATHNAME :TYPE "fasl" :DIRECTORY (OR # #) :NAME "lisp-binary"))) (WITH-OPEN-FILE (COMMON-LISP-USER::FILE COMMON-LISP-USER::CATTED-FASL :DIRECTION :OUTPUT :IF-DOES-NOT-EXIST :CREATE) (SB-EXT:RUN-PROGRAM "cat" (MAPCAR (FUNCTION COMMON-LISP-USER::NIX-COMPILE-LISP) (QUOTE #)) :OUTPUT COMMON-LISP-USER::FILE :SEARCH T))) #<NULL-LEXENV>)
20: (SB-EXT:EVAL-TLF (LET ((*COMPILE-VERBOSE* T) (COMMON-LISP-USER::CATTED-FASL (MAKE-PATHNAME :TYPE "fasl" :DIRECTORY (OR # #) :NAME "lisp-binary"))) (WITH-OPEN-FILE (COMMON-LISP-USER::FILE COMMON-LISP-USER::CATTED-FASL :DIRECTION :OUTPUT :IF-DOES-NOT-EXIST :CREATE) (SB-EXT:RUN-PROGRAM "cat" (MAPCAR (FUNCTION COMMON-LISP-USER::NIX-COMPILE-LISP) (QUOTE #)) :OUTPUT COMMON-LISP-USER::FILE :SEARCH T))) 13 NIL)
21: ((LABELS SB-FASL::EVAL-FORM :IN SB-INT:LOAD-AS-SOURCE) (LET ((*COMPILE-VERBOSE* T) (COMMON-LISP-USER::CATTED-FASL (MAKE-PATHNAME :TYPE "fasl" :DIRECTORY (OR # #) :NAME "lisp-binary"))) (WITH-OPEN-FILE (COMMON-LISP-USER::FILE COMMON-LISP-USER::CATTED-FASL :DIRECTION :OUTPUT :IF-DOES-NOT-EXIST :CREATE) (SB-EXT:RUN-PROGRAM "cat" (MAPCAR (FUNCTION COMMON-LISP-USER::NIX-COMPILE-LISP) (QUOTE #)) :OUTPUT COMMON-LISP-USER::FILE :SEARCH T))) 13)
22: ((LAMBDA (SB-KERNEL:FORM &KEY :CURRENT-INDEX &ALLOW-OTHER-KEYS) :IN SB-INT:LOAD-AS-SOURCE) (LET ((*COMPILE-VERBOSE* T) (COMMON-LISP-USER::CATTED-FASL (MAKE-PATHNAME :TYPE "fasl" :DIRECTORY (OR # #) :NAME "lisp-binary"))) (WITH-OPEN-FILE (COMMON-LISP-USER::FILE COMMON-LISP-USER::CATTED-FASL :DIRECTION :OUTPUT :IF-DOES-NOT-EXIST :CREATE) (SB-EXT:RUN-PROGRAM "cat" (MAPCAR (FUNCTION COMMON-LISP-USER::NIX-COMPILE-LISP) (QUOTE #)) :OUTPUT COMMON-LISP-USER::FILE :SEARCH T))) :CURRENT-INDEX 13)
23: (SB-C::%DO-FORMS-FROM-INFO #<FUNCTION (LAMBDA (SB-KERNEL:FORM &KEY :CURRENT-INDEX &ALLOW-OTHER-KEYS) :IN SB-INT:LOAD-AS-SOURCE) {100106669B}> #<SB-C::SOURCE-INFO {1001066663}> SB-C::INPUT-ERROR-IN-LOAD)
24: (SB-INT:LOAD-AS-SOURCE #<SB-SYS:FD-STREAM for "file /nix/store/gqc6ldyg0n3frga5ij367irl9fn8q1yj-sbcl-compile.lisp" {1001060E33}> :VERBOSE NIL :PRINT NIL :CONTEXT "loading")
25: ((LABELS SB-FASL::LOAD-STREAM-1 :IN LOAD) #<SB-SYS:FD-STREAM for "file /nix/store/gqc6ldyg0n3frga5ij367irl9fn8q1yj-sbcl-compile.lisp" {1001060E33}> NIL)
26: (SB-FASL::CALL-WITH-LOAD-BINDINGS #<FUNCTION (LABELS SB-FASL::LOAD-STREAM-1 :IN LOAD) {7FFFF786F82B}> #<SB-SYS:FD-STREAM for "file /nix/store/gqc6ldyg0n3frga5ij367irl9fn8q1yj-sbcl-compile.lisp" {1001060E33}> NIL #<SB-SYS:FD-STREAM for "file /nix/store/gqc6ldyg0n3frga5ij367irl9fn8q1yj-sbcl-compile.lisp" {1001060E33}>)
27: (LOAD #<SB-SYS:FD-STREAM for "file /nix/store/gqc6ldyg0n3frga5ij367irl9fn8q1yj-sbcl-compile.lisp" {1001060E33}> :VERBOSE NIL :PRINT NIL :IF-DOES-NOT-EXIST :ERROR :EXTERNAL-FORMAT :DEFAULT)
28: ((FLET SB-IMPL::LOAD-SCRIPT :IN SB-IMPL::PROCESS-SCRIPT) #<SB-SYS:FD-STREAM for "file /nix/store/gqc6ldyg0n3frga5ij367irl9fn8q1yj-sbcl-compile.lisp" {1001060E33}>)
29: ((FLET SB-UNIX::BODY :IN SB-IMPL::PROCESS-SCRIPT))
30: ((FLET "WITHOUT-INTERRUPTS-BODY-11" :IN SB-IMPL::PROCESS-SCRIPT))
31: (SB-IMPL::PROCESS-SCRIPT "/nix/store/gqc6ldyg0n3frga5ij367irl9fn8q1yj-sbcl-compile.lisp")
32: (SB-IMPL::TOPLEVEL-INIT)
33: ((FLET SB-UNIX::BODY :IN SB-IMPL::START-LISP))
34: ((FLET "WITHOUT-INTERRUPTS-BODY-3" :IN SB-IMPL::START-LISP))
35: (SB-IMPL::%START-LISP)

unhandled condition in --disable-debugger mode, quitting
;
; compilation unit aborted
;   caught 1 fatal ERROR condition
; compilation aborted after 0:00:00.041

Heap exhaustion in SBCL when loading lisp-binary from Quicklisp

When attempting to load lisp-binary from Quicklisp in SBCL on x64 Ubuntu Linux, I'm getting heap exhaustion during garbage collection:

CL-USER> (ql:quickload :lisp-binary)
To load "lisp-binary":
Load 1 ASDF system:
lisp-binary
; Loading "lisp-binary"
..................................................
[package lisp-binary].............................
.........................
CL-USER> (format t "~:D" (sb-ext:dynamic-space-size))
1,073,741,824
NIL

Heap exhausted during garbage collection: 48 bytes available, 80 requested.
Gen StaPg UbSta LaSta LUbSt Boxed Unboxed LB LUB !move Alloc Waste Trig WP GCs Mem-age
0: 0 0 0 0 0 0 0 0 0 0 0 10737418 0 0 0,0000
1: 23421 23424 0 0 11056 4358 0 0 16 503874848 1211104 299799946 0 1 1,3608
2: 32766 32763 0 0 11001 4925 56 68 153 521300000 4626400 2000000 1404 0 0,2076
3: 0 0 0 0 0 0 0 0 0 0 0 2000000 0 0 0,0000
4: 0 0 0 0 0 0 0 0 0 0 0 2000000 0 0 0,0000
5: 0 0 0 0 0 0 0 0 0 0 0 2000000 0 0 0,0000
6: 0 0 0 0 1094 210 0 0 0 42729472 0 2000000 954 0 0,0000
Total bytes allocated = 1067904320
Dynamic-space-size bytes = 1073741824
GC control variables:
GC-INHIBIT = true
GC-PENDING = true
STOP-FOR-GC-PENDING = false
fatal error encountered in SBCL pid 26187(tid 140737231681280):
Heap exhausted, game over.

Is this a known issue? Is there a workaround for it?

Thanks for your assistance,
David

infinityp and nanp require CFFI.

I purportedly implemented a feature allowing the use of Lisp-Binary without CFFI. But I forgot about the fact that nanp and infinityp are just front-ends to their C counterparts. Also, the constants +inf, -inf, and quiet-nan are created by casting bit-patterns to float via CFFI.

In CCL, I avoid some of the problem by using CCL's extended constants. Newer versions of SBCL have constants for infinity, too, but I don't use them because SBCL didn't have them when I wrote the feature. I could begin using them now.

SBCL does not have a constant for NaN, but it does have sb-ext:float-nan-p. So I could avoid using C's isnan(), but I have no way to create one without CFFI.

missing () in define-enum documentation

at https://github.com/j3pic/lisp-binary/wiki/Enumerated-Types
the example:

(define-enum speed 2
...

needs to be:

(define-enum speed 2 ()
...

Annoying inline-expansion warning in Lispworks

Using read-binary-type and write-binary-type will produce annoying Warning: inline-expansion cannot be found in Lispworks while compiling, and I found that just putting the inline declaim before the declaration of the definition of functions will solve this problem - For example, moving line 856 of binary-1.lisp to line 839. However, besides of those annoying warnings, the functionality of the code have no problem, though I haven't profile the potential performance improvement.

Do you think it's matter and should make a change? I'm not very clear about whether there's potentially bad point for such a change.

And, Lisp-binary performs very well in Lispworks for me, even under heavy multiprocessing. Thank you very much :D!

Some systems failed to build for Quicklisp dist

Building with SBCL 2.1.0.29-a1ce59262 / ASDF 3.3.1 for quicklisp dist creation.

Trying to build commit id 02ce88f

lisp-binary-test fails to build because of a failure in lisp-binary.

lisp-binary fails to build with the following error:

; caught WARNING:
;   Derived type of SEQUENCE is (OR VECTOR SB-KERNEL:EXTENDED-SEQUENCE), conflicting with its asserted type LIST.
;   See also:
;     The SBCL Manual, Node "Handling of Types"
...
Unhandled UIOP/LISP-BUILD:COMPILE-FILE-ERROR in thread #<SB-THREAD:THREAD "main thread" RUNNING {1001A40103}>: COMPILE-FILE-ERROR while compiling #<CL-SOURCE-FILE "lisp-binary" "utils">

Full log here

Bit streams: STREAM-FINISH-OUTPUT is wrong

(defmethod stream-finish-output ((stream bit-stream))
  (unless (or (not (eq (slot-value stream 'last-op) :write))
          (= (slot-value stream 'bits-left) 0))
    (real-write-byte (slot-value stream 'last-byte)
             (slot-value stream 'real-stream))
    (finish-output (slot-value stream 'real-stream))))

If the stream is open in :big-endian mode, then the leftover bits should be bit-shifted so that the first bit is the high-order bit of the byte that gets written. Nothing needs to be done for :little-endian mode.

Support reading binary data from an (unsigned-byte 8) vector

The only way I could get data parsed from a (unsigned-byte 8) vector was this:

    (flexi-streams:with-input-from-sequence (s1 ub8-vector)
      (lisp-binary:with-wrapped-in-bit-stream (s2 s1)
        (lisp-binary:read-binary 'dns-data s2)))))

Perhaps such vectors could be allowed directly in READ-BINARY instead of a bit-stream?

Thanks!

Undefined variable LISP-BINARY::VALUE

A bug made it into Quicklisp that may result in this error being seen. It looks like I found it first. Fixed in #49.

PUSHNEW instead of PUSH into FEATURES

Should be PUSHNEW on both L31 and L32.

https://github.com/heegaiximephoomeeghahyaiseekh/lisp-binary/blob/cdc49baa1d30f4813c16076fda290f18d33c77c7/float.lisp#L31

Does not build in latest SBCL - type problem

This line seems to declare *standard-input* to be of type keyword, which signals a full warning in the latest sbcl.

lisp-binary/binary-1.lisp

Line 620 in ef019b0

(stream *standard-input* :type keyword)))

Full log here: http://report.quicklisp.org/2019-12-20/failure-report/lisp-binary.html#lisp-binary

defstruct options passed correctly?

The defstruct options are passed as a plist but for defstruct they are an alist, right?

e.g.

(defstruct (door (:conc-name dr-)) knob-color width material) =>  DOOR
    (setq my-door (make-door :knob-color 'red :width 5.0))

Patch below for what I think the correct behavior is, but maybe I've missed something.

index d1f532a..70d37da 100644
--- a/binary-2.lisp
+++ b/binary-2.lisp
@@ -985,7 +986,8 @@ FLOATING-POINT NUMBERS
 									 byte-order)))))))
 		(name-and-options (if defstruct-options
 				      (cons name
-					    (remove-plist-keys defstruct-options :byte-order))
+					(loop for (key value) on (remove-plist-keys defstruct-options :byte-order) by #'cddr
+                                                  collect (list key value)))
 				      name))
 		(documentation (if documentation
 				   (list documentation)

This library should not upgrade ASDF

According to the principle of least astonishment, I would never expect a library for reading binary data to have a side effect of upgrading my ASDF version.

What was the use case for that at all? If you had an implementation that had an ancient ASDF, why not patch it on the implementation level, so it bundles a new enough version of ASDF?

I think that 7afc423 should be reverted.

Need support for generic terminated arrays.

LISP-BINARY can handle terminated byte buffers and terminated strings, but it's also common in binary formats to have lists of complex structures terminated by some byte sequence. The byte sequence is often of a length that cleanly fits in all or a subset of the fields in the element structure.

For example: kaitai-io/kaitai_struct#445

Such data can currently be read/written using a custom type specifier.

Add CLASP to Travis.

Documentation: missing information

In an example,

              (defbinary stays-non-byte-aligned ()
                 (x 0 :type 3))

What does the 0 after the x mean? I could not find this in the documentation for DEFBINARY.

Need support for splitting an integer.

Consider the EDID format used by HDMI monitors: https://en.wikipedia.org/wiki/Extended_Display_Identification_Data#EDID_1.4_data_format

Part of this spec calls for an integer whose lowest 8 bits are followed by the lowest 8 bits of an unrelated integer, followed by the highest 4 bits of the first integer, then the highest 4 bits of the second integer.

Something like this:

(defbinary troll-format (:byte-order :big-endian)
    (number-of-frobs 0 :type (unsigned-byte 8))
    (unrelated-cruft 0 :type (unsigned-byte 8))
    (rest-of-number-of-frobs 0 :type (unsigned-byte 4)) ;; u mad bro?
    (rest-of-unrelated-cruft 0 :type (unsigned-byte 4)))

I've seen other formats that do things like this (but never an explanation of why someone would do
this), so perhaps there should be some way to specify that REST-OF-NUMBER-OF-FROBS is part of NUMBER-OF-FROBS (and which part it is in that case, since someone could specify a format in which the chunks are laid out in middle-endian order).