deep-symmetry / bytefield-svg Goto Github PK

Node module that generates byte field diagrams in SVG format

License: Eclipse Public License 2.0

Clojure 91.13% JavaScript 8.87%

clojure byte svg nodejs diagram-generator

bytefield-svg's Introduction

bytefield-svg

A Node module for generating byte field diagrams like this one. Inspired by the LaTeX bytefield package. Powered by a Clojure-based domain specific language (now built on top of SCI, the Small Clojure Interpreter).

Usage

This is published to npm, so you can install it for use in a Javascript project by running:

npm install bytefield-svg

Or you can install it globally for use anywhere by running:

npm install -g bytefield-svg

The language you use to create diagrams has its own documentation site.

Invoking from Javascript

Once installed, you can generate diagrams in your code like this:

const generate = require('bytefield-svg');

const source = `
;; Put your diagram DSL here, or read it from a file, or build it...
`;
const diagram = generate(source);
process.stdout.write(diagram);

By default, a full SVG file is generated, complete with XML version and namespaces. If you want to generate a simple <svg> tag which is suitable for embedding inside an HTML document, you can request that by calling generate like this:

// setup code omitted...
const diagram = generate(source, { "embedded": true });

Of course, you can do other things than writing the diagram to standard out. For a few more examples of usage, you can see the cli.js source in this project which implements the command-line interface, our next topic:

Invoking from the Command Line

This package also installs a command-line tool. If you have installed it globally, you can simply invoke it as bytefield-svg. If you have installed it locally, you can invoke it using npx bytefield-svg within your project.

With no arguments, the tool will read the diagram source from standard in, and write it to standard out. So you can generate the example diagram from this Read Me, as long as you have the test.edn file, by running:

bytefield-svg <test.edn >test.svg

You can also use the -s or --source command-line argument to specify that the tool should read from a named file rather than standard in, and -o or --output to write to a named file rather than standard out, which might be helpful in a scripting pipeline:

bytefield-svg --source test.edn --output test.svg

If you supply just a filename with no command-line flag, it is assumed to be the diagram source file.

Normally the output is a full SVG file, complete with XML version information and namespaces. If instead you want to generate a simple SVG tag which is suitable for embedding inside an HTML document, you can supply the -e or --embedded flag.

Invoking it with -h or --help displays this usage information.

-h, --help            Display this usage guide.
-s, --source string   File from which to read the diagram source, defaults to
                      standard in.
-o, --output string   File to which to write the SVG diagram, defaults to
                      standard out.
-e, --embedded        Emit a simple <svg> tag suitable for embedding in an
                      HTML document. (The default is to emit a full SVG file
                      with XML version and namespaces.)

Getting Help

Deep Symmetry’s projects are generously sponsored with hosting by Zulip, an open-source modern team chat app designed to keep both live and asynchronous conversations organized. Thanks to them, you can chat with our community, ask questions, get inspiration, and share your own ideas.

Background

The DSL has been nicely validated by porting all of the LaTeX documents I needed it for to an Antora documentation site.

As that site suggests, this package’s main purpose is to act as an Asciidoctor extension, and in fact asciidoctor-bytefield has now been published to enable that.

Building

To build a development build of bytefield-svg from source, clone the repository and make sure you have Node.js and the Clojure CLI tools installed, then from the top-level directory of your cloned repo run:

npm install
npm run build

This will create the file lib.js. At that point, you can generate the sample diagram by running:

node cli.js test.edn >test.svg

(The test.edn file is present in this project. It renders a diagram from the above-linked documentation site. With some well-designed helper functions in site’s own include file, the source for an even more attractive version of the diagram shrinks to this).

The DSL documentation is hosted on deepsymmetry.org, and built out of the doc folder, which includes build instructions. (They are slightly more complex than if you were using asciidoctor-bytefield, because they want to build against the latest version of bytefield-svg in case they are demonstrating unreleased features that haven’t yet made it to the that extension.)

To check for outdated dependencies, you can run:

clojure -M:outdated

Releasing

To cut a release, check for outdated dependencies as above, update the version in package.json, tag and push to GitHub, then run:

npm install
npm run release
npm publish

License

Distributed under the Eclipse Public License 2.0. By using this software in any fashion, you are agreeing to be bound by the terms of this license. You must not remove this notice, or any other, from this software.

bytefield-svg's People

Contributors

Stargazers

Watchers

Forkers

djencks metacritical ggrossetie i-fimafeng stjordanis jaredreisinger borkdude eval kumaraish 0xflotus wmat deining culyun volaris marcpa00 vitaly-z sthagen markopesevski

bytefield-svg's Issues

Omit XML processing structures for generating HTML embedded SVG

Is your feature request related to a problem? Please describe.
When generating SVG that is to be spliced into an HTML document rather than used as a standalone file, the <?xml version…> tag, as well as the <xmlns …> and <xmlsn:svg …> tags are not necessary or valid, so it would be nice to be able to omit them.

Describe the solution you'd like
There needs to be some way to configure the generator to know that it is operating in an embedded HTML mode. Perhaps a second optional argument that is an options map? If that proves too hard, then a new var could be introduced and a def statement could be used in the diagram definition to change it, but that would force users to take care of the issue on each diagram, or if the asciidoctor-bytefield extension prepended it, then the reported source lines would all be off by one.

Additional context
The HTML checker at https://validator.w3.org/nu/ can be used to verify the solution.

boxes-per-row can't do 32?

Is your feature request related to a problem? Please describe.
I am trying to describe a structure that has 32-bit words, but boxes-per-row will not accept a value above 16 with an error about vector length.

Describe the solution you'd like
(def boxes-per-row 32) shouldn't give an error of “No item 16 in vector of length 16”.

Describe alternatives you've considered
I modified the core.cljs build-initial-globals to make the default boxes-per-row equal to 64, but I get the same error.

Additional context
None

Set svg background

Is your feature request related to a problem? Please describe.

We are using bytefield-svg in wiki.js via kroki. That works great in light mode, but when in dark mode, it's impossible to see, as it is "black on black", due to the background of the svg (and the boxes) being transparent, and the foreground color being black.

Describe the solution you'd like

I would like to be able to set the background color of the whole svg to a color like white, so that it works both in light and dark mode.

Describe alternatives you've considered

Alternatively, I could set the background color of all boxes, but that does not seem to work for draw-gap and draw-bottom, and probably also not for the labels etc. In any way, it would be quite cumbersome and repetitive to have to do that for every element.

Support padding to a particular address

@Swiftb0y had another good idea, which is to be able to draw enough padding to get to a particular address in the packet.

Here are a pair of functions that achieve that result. Either start a library of useful functions people can copy for their own shared include files, or more likely build these in and document them—I can see them being useful for many situations.

(defn next-address
  "Calculates the address at which the next box will be drawn."
  []
  (let [{:keys [address column]} @diagram-state]
    (+ address column)))

(defn draw-padding
  "Draws enough related boxes to reach the specified address. If no
  `label` is supplied, draws a zero byte in each box. If `attr-spec`
  is supplied, it is passed along to `draw-related-boxes` along with
  each copy of the label."
  ([address]
   (draw-padding address 0 nil))
  ([address label]
   (draw-padding address label nil))
  ([address label attr-spec]
   (draw-related-boxes (repeat (- address (next-address)) label) attr-spec)))

Add more examples

I am donating the files attached examples which illustrates some network protocols. The ones that adjust the row length to 32 are a bit buggy in the row headers (since they increment by 16 every time).

The examples otherwise include headers for IPv4, IPv6, UDP, TCP, Ethernet frames, and CCSDS space packets.

protocols.zip

Add support for bit fields

More ideas coming out of the current reverse-engineering project @Swiftb0y is working on:

It would be nice to make it easier for people to draw bit fields. One foundational step is to enhance the automatic label creation logic so that boolean? values are rendered as one-digit hex-style 0 and 1 labels. Then we could add functions like this:

(defn- normalize-bit
  "Converts a value to either `true` or `false`. All non-zero numbers
  become `true`. Other values are tested for truthiness and translated
  to `true` or `false` accordingly."
  [value]
  (cond
    (number? value)
    (zero? value)

    :else
    (if value true false)))

(defn- number-as-bits
  "Takes a number and transforms it into a sequence of `boolean` bit
  values of the specified length."
  [number length]
  (map #(normalize-bit (bit-test number %)) (reverse (range length))))

Which would support diagrams like this:

(def box-width 20)
(def boxes-per-row 8)
(draw-column-headers {:labels (reverse (take 8 column-labels))})
(draw-related-boxes (number-as-bits 0xd3 8))

Declare the namespaces in the <svg>

Is your feature request related to a problem? Please describe.
If you embed the SVG produced by bytefield-svg in a HTML page and serve that page as text/html then xmlns attributes are not required.
But, if the page is served as image/svg+xml or any other MIME type that causes the user agent to use an XML parser then the xmlns attributes are required. Otherwise the browser won't recognize the content and display XML (instead of an image).
For instance: https://a.kroki.io/bytefield/svg/test

It's recommended to declared the namespaces you use in your XML files. If you don't, user agents such as Firefox won't recognize your content and will simply show the XML markup or inform the user that there's an error in the XML. It's a good idea to use a template that includes all the commonly used namespace declarations when creating new SVG files. If you don't already have one, make one up starting with the following code

https://developer.mozilla.org/en-US/docs/Web/SVG/Namespaces_Crash_Course#Conclusion

Describe the solution you'd like
Add the following namespaces:

<svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink">
</svg>

Describe alternatives you've considered
I guess not doing it? 🤔 😆

Variable row height

The row-height can be set globally, but resetting it before e.g. the next row means that we get overlapping rows (if we reduce the row height).

We can insert a row manually, but then the automatic labels with offsets are messed up, like this:

(def row-height 60) 
(draw-column-headers)
(draw-box "version" {:span 3})
(draw-box (text "type" {:writing-mode "vertical-rl"}) {:span 1})
(draw-box "dfh" {:span 1})
(draw-box "apid" {:span 11})

(next-row)
(def row-height 30) 
(draw-box "seq. flags" {:span 2})
(draw-box "sequence count" {:span 14})

(draw-box "packet length" {:span 16})

(draw-box "shf" {:span 1})
(draw-box "pusvers" {:span 3})
(draw-box "ack" {:span 4})
(draw-box "service type" {:span 8})
(draw-box "service subtype" {:span 8})

(draw-gap)

(draw-box "packet error control" {:span 16})

We can insert draw-row-header "00" before, the (next-row), but that doesn't increment the automatic row headers.

I suppose this is also a question of documentation in the end, how do we manually draw the row header address and increment it for the next line?

Allow boxes-per-row > 16

Is your feature request related to a problem? Please describe.
The variable "boxes-per-row" cannot be incremented larger than 16, only decreased. When larger than 16, an error is returned: "No item x in vector of length 16", So I assume each row is fixed at a maximum of 16 elements.

The diagrams I am creating would be much better visualised as a single row, with a number of bytes > 16, which cannot be achieved. (Multiple diagrams will be created one below the next, in order to visually show subtle differences in each bytefield diagram).

Describe the solution you'd like
Allow more than 16 boxes per row, I assume the heading would require two character: 00, 01 ... 0f, 11, 12, 13. I assume the row offsets values would be handled correctly by default.

Describe alternatives you've considered
Accept the limit of 16 and let my diagram to wrap onto a second row at the cost of loosing the subtitle visual differences between diagrams.

Additional context

draw-box should auto advance rows

When drawing a box would start at just past the end of the current row, an automatic next-row operation should occur. (If the previous box extended past the last column, an error should be thrown, the document author will have to decide how to represent that themselves.)

Right before auto-advancing, the row header should be auto-generated. (If we are currently on the first row, generate the first row header and then the next row header, because we don’t want the headers to be there if there is just one row.) We will provide a default header function, which renders the current byte address as a two-digit (or more) hex value, although if a gap has been generated before this row, this will be prefixed with “i+” to indicate that the address is relative to the end of the gap. This can be overridden by passing a :header-fn attribute.

Small/vertical draw-gap function

Is your feature request related to a problem? Please describe.
Currently, variable length data is being represented by using the draw-gap function which creates a horizontal gap, allocating at least an entire row of space.
When generating diagrams that only contain a single row of fields, using draw-gap to indicate variable length data takes up much more space than justified for the diagram.

Describe the solution you'd like
I propose to add a function to create a gap spanning only a single couple of cells on a single row.
This could either be accomplished by letting the draw-gap function draw this version of the diagram when the cap doesn't span past the row or by splitting that function into two different functions (eg draw-large-gap and draw-small-gap or draw-vertical-gap and draw-horizontal-gap.

Additional context
The produced SVG would rougly look like the following ascii art (spanning only a single row):

+----------/  /--------+
|         /  /         +
+--------/  /----------+

Add draw-column-headers Function in bytefield-svg

A new function/helper called draw-column-headers. This would make the header line format resemble the one used in this example: Packet Diagram Example.

Have draw-gap handle labels and finishing previous row

Experience with drawing more complex diagrams shows a very common pattern of having to finish off the preceding row, usually with text describing the content of the gap, before drawing the gap. This could all be automated in draw-gap for another significant increase in concision and convenience in the DSL.

Allow custom box drawing

Passing a function as the label argument to draw-box could call that function with the bounding box and supplied attributes, and leave it to the function to call append-svg with whatever it wants. This could support arbitrary graphical snazziness.

Support automatic dark mode

It would be nice if you could support automatic dark mode, like this:

<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 64 64">
   <style>
        path {
            fill: black;
        }
        @media (prefers-color-scheme: dark) {
            path { fill: white; }
        }
    </style>
    <path d="..."/>
</svg>

Unfortunately I couldn't find a way to insert arbitrary styles/CSS.

Box borders should be more flexible

In addition to taking a simple set of keywords, as it currently does, it should also accept a map, in which the border keywords can be assigned individual attr-specs. If the value associated with a border key is simply true or the same keyword as the border (which would be the result of looking it up in a set), a default attr-spec is used (:border-unrelated or :border-related as appropriate). If the value is another keyword, then it is looked up in the attr-spec map. If it is a map, it is used as the border attributes.

Pull in some useful functions from other namespaces

It would be nice to have some other utility functions available for drawing diagrams, things from clojure.set, possibly math functions, clojure.pprint, and so on. Also document their availability at the top of the Functions page too.

Add helper for related boxes

For generating a series of bytes without boundaries between them on a row, something like:

(draw-related-boxes labels attr-spec)

would iterate over the labels calling draw-box for each, but overriding the attr-spec for the first, middle, and last boxes to adjust the borders.

draw-line should take an attr-spec

This will give it more flexibility and the convenience of using named line styles, as well as being consistent with the new text approach.

Consider an option to set the viewport

@Mogztter suggests defining the viewBox attribute in order to allow diagrams to resize.

As currently generated:

With viewBox and max-width: 100%:

Since the current behavior is consistent with other image formats like PNG, I think it should probably remain the default, but it would be nice to offer people an option to adjust these settings.

Although, looking at those examples, it does resize very nicely, so that should perhaps be the default.

CLI Error - Could Not Resolve Symbol: Address

Description

Running bytefield-svg -s <file> -o <file> or echo "<bytefield syntax>" | bytefield-svg -s" results the the error described in the issue title:

CLI Error Output:

$HOME/.local/lib/node_modules/bytefield-svg/lib.js:1163
function m_(a,b,c){if(gb(uY)){var d=b.message;if(m(d)){if(-1!=d.indexOf("[at"))throw b;c=F(c);var e=ta(c);c=zd.Gc(e,U,U.La(a));a=zd.Gc(e,S,S.La(a));if(m(m(c)?a:c))throw d=p.La(d),e=u(b_),e=m(e)?[p.La(e),", "].join(""):null,d=[d," [at ",e,"line ",p.La(c),", column ",p.La(a),"]"].join(""),On.Gc(d,ll.He(he([new h(null,4,[sC,it,U,c,S,a,Cv,d],null),Pn(b)])),b);}}throw b;}
                                                                               ^
Nn [Error]: Could not resolve symbol: Address [at line 2, column 11]
    at new Nn ($HOME/.local/lib/node_modules/bytefield-svg/lib.js:675:26)
    at On.Gc ($HOME/.local/lib/node_modules/bytefield-svg/lib.js:677:74)
    at On.Na ($HOME/.local/lib/node_modules/bytefield-svg/lib.js:677:31)
    at l_ ($HOME/.local/lib/node_modules/bytefield-svg/lib.js:1162:204)
    at k_ ($HOME/.local/lib/node_modules/bytefield-svg/lib.js:1161:632)
    at $HOME/.local/lib/node_modules/bytefield-svg/lib.js:1199:324
    at H_ ($HOME/.local/lib/node_modules/bytefield-svg/lib.js:1199:378)
    at J_ ($HOME/.local/lib/node_modules/bytefield-svg/lib.js:1230:44)
    at $HOME/.local/lib/node_modules/bytefield-svg/lib.js:1199:451
    at $HOME/.local/lib/node_modules/bytefield-svg/lib.js:356:81 {
  data: h {
    v: null,
    A: 3,
    l: [
      y {
        kb: null,
        name: 'type',
        la: 'type',
        yc: 1174270348,
        o: 2153775105,
        F: 4096
      },
      y {
        kb: 'sci',
        name: 'error',
        la: 'sci/error',
        yc: -979082803,
        o: 2153775105,
        F: 4096
      },
      y {
        kb: null,
        name: 'line',
        la: 'line',
        yc: 212345235,
        o: 2153775105,
        F: 4096
      },
      2,
      y {
        kb: null,
        name: 'column',
        la: 'column',
        yc: 2078222095,
        o: 2153775105,
        F: 4096
      },
      11
    ],
    Ie: null,
    o: 16647951,
    F: 139268
  },
  cause: null,
  description: undefined,
  number: undefined,
  fileName: undefined,
  lineNumber: undefined,
  columnNumber: undefined
}

Node.js v18.16.0

Reproduce

For STDIN my exact syntax was:

$: echo "(draw-column-headers)
(draw-box "Address" {:span 4})
(draw-box "Size" {:span 2})
(draw-box 0 {:span 2})
(draw-gap "Payload")
(draw-bottom)" | bytefield-svg -s

From FILE:

$: echo "(draw-column-headers)
(draw-box "Address" {:span 4})
(draw-box "Size" {:span 2})
(draw-box 0 {:span 2})
(draw-gap "Payload")
(draw-bottom)" >> byte.test

$: bytefield-svg -s ./byte.test

Expected Behavior

I expected the normal behavior for whatever these commands reproduce.

Desktop

OS: Pop OS 22.04 LTS
Java Version:

openjdk 11.0.19 2023-04-18
OpenJDK Runtime Environment (build 11.0.19+7-post-Ubuntu-0ubuntu122.04.1)
OpenJDK 64-Bit Server VM (build 11.0.19+7-post-Ubuntu-0ubuntu122.04.1, mixed mode, sharing)

Project Version: Whatever the lastest version in the NPM repository is, as of June 30 2023.

Long boxes

When working with autogenerated diagrams of network packets (bit level), we sometimes end up with fields crossing multiple lines and being in general unhelpfully unaligned.

Due to the unalignedness, it is hard to define a row length that will fit all values, in some cases this causes boxes to spill over into the next row, which generates an exception.

For example, a protocol has a double, aligned at a byte boundary, we then use draw-box with a span of 8.

Can we get a function like: draw-long-box or something like that that handles overflow into next line automatically? This of-course raises the issue as to what to draw in the lines (e.g. repeating the label, continuation symbol of some sort, label at the longest box sequence (i.e. the second row if the sequence is 1, 7)).

defattrs should call eval-attribute-spec

In writing up an example for #15 in the user guide, I realized that when you use defattrs to build a new attribute expression, you are often going to want to start with existing attribute specs, and so will almost always want to call eval-attribute-spec. It should just do that with its argument always.

Once this is done, of course the defattrs documentation will need to be updated, but so should the vertical text example in draw-box.

Support rotation of box labels

When printing labels for shorter boxes, the text often do not fit.
It would be nice if we could rotate the labels in the. boxes in these cases (combined with increasing the row height).