babashka / bbin Goto Github PK

Docs

bbin upgrade [script]

Upgrade a script

# Install from Git tag
$ bbin install io.github.babashka/neil --git/tag v0.1.44
$ neil --version
neil 0.1.44

# Upgrade to latest Git tag
$ bbin upgrade neil
$ neil --version
neil 0.1.45

# Install from HTTP script
$ bbin install https://raw.githubusercontent.com/borkdude/tools/main/antq.clj

# Re-fetch HTTP script
$ bbin upgrade antq

Implementation Details

A non-exhaustive list of examples to consider for upgrading:

{:lib io.github.babashka/neil,
 :coords
 {:git/url "https://github.com/babashka/neil",
  :git/tag "v0.1.45",
  :git/sha "0474d4cb5cfb0207265a4508a0e82ae7a293ab61"}}

{:lib org.babashka/http-server, :coords {:mvn/version "0.1.11"}}

{:coords
  {:bbin/url
   "https://raw.githubusercontent.com/borkdude/tools/main/antq.clj"}}

Like #46 scripts installed from a git dependency don't have to be called via process/exec. Instead, they can be called directly in combination with babashka.deps/add-deps.

The time savings depend on your local hardware, but should be something like:

time bb -e 'nil'
bb -e 'nil'  0.02s user 0.01s system 83% cpu 0.036 total

(Some background in a recent discussion around Carve)

Source: https://clojure.org/reference/deps_and_cli#_coord_attributes
Related: https://clojure.atlassian.net/browse/TDEPS-228

This is currently failing, but I think we can make it work.

$ echo '(println "Hello world")' > foo.clj

$ bbin install foo.clj
{:coords
 {:bbin/url
  "file:///Users/rads/src/bbin/foo.clj"}}

$ foo
/Users/rads/.local/bin/foo: line 1: println: command not found
/Users/rads/.local/bin/foo: line 4: syntax error near unexpected token `;'
/Users/rads/.local/bin/foo: line 4: `; :bbin/start'

Issue #35 is not finished yet, but we still want to support XDG_DATA_HOME in the meantime (as described in the README).

Originally reported in Slack: https://clojurians.slack.com/archives/C0400TZENE7/p1679138773023159

Maybe something like this if we don't want to provide SHAs all the time:

# This fails with an untrusted URL error
bbin install io.github.foo/bar

# We can work around it by adding a "trusted user" before installing
bbin trust --github/user foo
bbin install io.github.foo/bar

Awesome project, thanks for creating it! I tried it out and everything is working so far.

I have a repo containing multiple babashka scripts, each with its own main.

For now, I need to specify the main opts and the script name on the bbin install line, like this:

bbin install --as empath io.github.justone/bb-scripts --main-opts '["-m" "empath"]'

Since the :bbin/bin value in the repo's bb.edn file is a map, how about supporting a way of picking which script is installed? Something like this:

bbin install io.github.justone/bb-scripts --select empath

It could even be extended to support an :all keyword to install all scripts found.

Thanks again.

Right now this is a manual step in the README.

I tried running bbin install io.github.nextjournal/garden-cli (which I had previously installed) and got the following error:

mk@mkair ~ [1]> bbin version
bbin 0.1.9

mk@mkair ~ [1]> bbin install io.github.nextjournal/garden-cli
{:coords
 #:git{:url "https://github.com/nextjournal/garden-cli.git",
       :tag "",
       :sha ""},
 :lib io.github.nextjournal/garden-cli}
Error building classpath. Library io.github.nextjournal/garden-cli has invalid tag: 
----- Error --------------------------------------------------------------------
Type:     java.io.FileNotFoundException
Message:  /Users/mk/.clojure/.cpcache/A1A0BDF8D59CA7999543C1499C00A8D9.cp (No such file or directory)
Location: /opt/homebrew/bin/bbin:589:13

----- Context ------------------------------------------------------------------
585:                   {:coords {:bbin/url (str "file://" (get-in header [:coords :local/root]))}}
586:                   header)
587:         _ (pprint header' cli-opts)
588:         _ (when-not (#{::no-lib} lib)
589:             (deps/add-deps {:deps script-deps}))
                 ^--- /Users/mk/.clojure/.cpcache/A1A0BDF8D59CA7999543C1499C00A8D9.cp (No such file or directory)
590:         script-root (fs/canonicalize (or (get-in header [:coords :local/root])
591:                                          (local-lib-path script-deps))
592:                                      {:nofollow-links true})
593:         bin-config (load-bin-config script-root)
594:         script-name (or (:as cli-opts)

----- Stack trace --------------------------------------------------------------
clojure.java.io/fn--11617                       - <built-in>
clojure.java.io/fn--11569/G--11523--11576       - <built-in>
clojure.java.io/fn--11591                       - <built-in>
clojure.java.io/fn--11530/G--11519--11537       - <built-in>
clojure.java.io/reader                          - <built-in>
... (run with --debug to see elided elements)
babashka.bbin.cli/bbin                          - /opt/homebrew/bin/bbin:825:1
babashka.bbin.cli                               - /opt/homebrew/bin/bbin:831:3
clojure.core/apply                              - <built-in>
babashka.bbin.cli                               - /opt/homebrew/bin/bbin:835:3
babashka.bbin.cli                               - /opt/homebrew/bin/bbin:833:1

It works when passing the --latest-sha param.

For when a library does not have a tag, it might be worth considering installing the latest sha instead. Alternatively the error message could be improved.

We already store pods in ~/.babashka/pods. I think it would make sense to use the ~/.babashka/bbin directory. This saves creating another bb-related directory on the top level of a user directory.

Check the pods project how this directory is detected. It respects XDG_ environment variables.

cc @rads @bobisageek

From the README:

# TODO: Not implemented yet, but possibly supported in the future
$ bbin install https://gist.github.com/1d7670142f8117fa78d7db40a9d6ee80.git
$ bbin install [email protected]:1d7670142f8117fa78d7db40a9d6ee80.git
$ bbin install https://github.com/babashka/http-server/releases/download/v0.1.11/http-server.jar
$ bbin install foo.clj
$ bbin install ~/src/watch
$ bbin install http-server.jar

Without validation, we get errors like this when the :bbin/bin value is malformed:

----- Error --------------------------------------------------------------------
Type:     java.lang.NullPointerException
Location: /Users/rads/.gitlibs/libs/io.github.rads/deps-info/a32493dd7f27be7e6f108719756516ed7e6c56c7/infer/src/rads/deps_info/git.clj:82:11

bbin install <uberjar> will create a wrapper script that relies on babashka.process/exec to execute the jar. This means that babashka is invoked twice, adding extra startup time for the script (also see #45).

After some personal testing and a conversation with @borkdude in Slack I believe this is unnecessary.

Proposed alternative

• The wrapper script would need to be adapted to something like:

(require '[babashka.classpath :refer [add-classpath]])
(add-classpath "the-uber-jar.jar")
(require '[the-main-ns])
(apply the-main-ns/-main *command-line-args*)

• The install process has to find the main class, and fail if it cannot find the main-ns (and maybe also when it cannot find a -main in that namespace).

I've played around with the compiled code from Babashka itself to find the main ns and I found that this runs in interpreted Babashka as well. (here and here)

I'll create a PR for this issue somewhere the coming days unless there are reasons for babashka.process/exec that I missed.

Splitting the metadata has the following benefits:

• You can re-install scripts, even if the bin dir is wiped
• You only have to back up those .edn files and can put them into source control
• It feels less magic than having inline comments in scripts
• Makes more sense with XDG conventions of keeping data and config separate

I haven't done any profiling yet but it's likely slow because we're reading and parsing all the scripts in bbin bin every time to generate the list. This could be improved with #34 but there may be easier fixes outside of that too.

Pull requests are welcome. 🙂

In my opinion the :coords are more readable with this set to false.

Right now bbin only looks in deps.edn for dependencies since we're using babashka.deps/add-deps. In some cases, though, we need to add a special spec.alpha dependency for Babashka on top of the deps.edn:

 :deps {org.babashka/spec.alpha {:git/url "https://github.com/babashka/spec.alpha"
                                 :git/sha "8df0712896f596680da7a32ae44bb000b7e45e68"}
        local/deps {:local/root "."}}

The problem is if we include the org.babashka/spec.alpha in deps.edn, we can no longer use the library as a dependency in Clojure JVM.

If we look in bb.edn for dependencies, this will only work with Gitlibs. Is there a better way to solve this?

I encountered several issues running several installed project on Windows 10 Home Single Language

1. Firstly, I tried installing a local bb project (initially tried with my existing project, but then I reproduced it to a folder that consist only bb.edn with :bbin/bin and src/hello.clj that only print hello). I install them with bbin install D:/keychera/projects/hello and the installation was successful but when I ran the project, I got this error

Error during loading bb.edn:
Exception in thread "main" java.lang.RuntimeException: Unsupported escape character: \k
        at clojure.lang.Util.runtimeException(Util.java:221)
        at clojure.lang.EdnReader$StringReader.invoke(EdnReader.java:457)
        at clojure.lang.EdnReader.readDelimitedList(EdnReader.java:757)
       ...omitted

2. initially, I thought there was something wrong with my project, but then I tried installing one of the example bbin install io.github.babashka/neil, I got an error as well when running them

----- Error --------------------------------------------------------------------
Type:     java.lang.IllegalArgumentException
Message:  'other' has different root
Location: C:\Users\Kyoga\.gitlibs\libs\io.github.babashka\neil\350c7ce94060bb0a6e57b32d35c37d9b17d88970\neil:8:1

----- Context ------------------------------------------------------------------
 4: ;; DO NOT EDIT DIRECTLY. GENERATED FROM scr/babashka/neil.clj
 5:
 6: (require '[babashka.deps :as deps])
 7:
 8: (deps/add-deps '{:deps {borkdude/rewrite-edn {:mvn/version "0.4.6"}
    ^--- 'other' has different root
...omitted

3. however, weirdly, I tried installing the second example bbin install https://gist.githubusercontent.com/rads/da8ecbce63fe305f3520637810ff9506/raw/25e47ce2fb5f9a7f9d12a20423e801b64c20e787/portal.clj and it ran just fine

so I am not so sure what is going on here

also regarding the first error, I did some investigation, Unsupported escape character: \k apparently is from the first
letter of the folder name in the path name provided, and I did move around the hello project to several places and the error is the same but the character will show the first letter of the path folder. I also tried installing the project with different slash and using relative folder as well but the error stays the same. I also tried this in both cmd and Powershell and there are no differences

Right now the install steps are specific to ZSH. For compatibility and ease of installation, we should use a manual install script. See Babashka for an example: https://github.com/babashka/babashka#installer-script

This is from Windows Terminal + Powershell:

~ $ bbin install io.github.rads/watch
----- Error --------------------------------------------------------------------
Type:     java.nio.file.AccessDeniedException
Message:  C:\Users\.bbin
Location: C:\Users\alesn\scoop\apps\bbin\current\bbin:61:3

----- Context ------------------------------------------------------------------
57:          (when-let [v (:local/root cli-opts)]
58:            {:local/root (str (fs/canonicalize v {:nofollow-links true}))})))
59:
60: (defn ensure-bbin-dirs [cli-opts]
61:   (fs/create-dirs (bin-dir cli-opts)))
      ^--- C:\Users\.bbin
62:
63: (ns babashka.bbin.scripts
64:   (:require [babashka.fs :as fs]
65:             [babashka.deps :as deps]
66:             [rads.deps-info.infer :as deps-info-infer]

----- Stack trace --------------------------------------------------------------
babashka.fs/create-dirs             - <built-in>
babashka.bbin.util/ensure-bbin-dirs - C:\Users\alesn\scoop\apps\bbin\current\bbin:61:3
babashka.bbin.util/ensure-bbin-dirs - C:\Users\alesn\scoop\apps\bbin\current\bbin:60:1
babashka.bbin.scripts               - C:\Users\alesn\scoop\apps\bbin\current\bbin:337:7
babashka.bbin.cli                   - C:\Users\alesn\scoop\apps\bbin\current\bbin:373:10
babashka.bbin.cli/bbin              - C:\Users\alesn\scoop\apps\bbin\current\bbin:403:5
babashka.bbin.cli/bbin              - C:\Users\alesn\scoop\apps\bbin\current\bbin:400:1
babashka.bbin.cli                   - C:\Users\alesn\scoop\apps\bbin\current\bbin:406:3
clojure.core/apply                  - <built-in>
babashka.bbin.cli                   - C:\Users\alesn\scoop\apps\bbin\current\bbin:409:3

Error reported on
https://github.com/babashka/bbin/blob/main/bbin#L61

But in fact the issue seems to be here
https://github.com/babashka/bbin/blob/main/bbin#L47

Looks like (fs/expand-home) is not working correctly and it forgets last component from the path. I'll do a bug report to babashka/fs too.

My recommendation is to get user directory from environment variables which are mutually exclusive on each platform, something along these lines:

(def ^:dynamic *bbin-root*
  (str (or (.get (System/getenv) "HOME")
           (.get (System/getenv) "USERPROFILE"))
  File/separator
  ".bbin"))

Probably just wait a moment to fs/expand-home to be fixed.

bbin install should fail when a binary with the same name exists at the path.
It should IMO allow user to force installation with a flag.

This could be a security issue, especially with package owners allowed to set the binary name.
Some people might chose to override things like ls or other stuff.

Current status:

• bbin installs binary commands as they are
• If a binary is present it will be overwritten

~/.local/bin$ ls -lah neil
-rwxr-xr-x 1 ieugen ieugen 1,2K sep 27 11:05 neil

BABASHKA_BBIN_DIR=~/.local bbin install io.github.babashka/bbin --latest-sha --as neil
{:coords
 {:git/url "https://github.com/babashka/bbin",
  :git/sha "6fde1b1dbfaef3063eb1eba4899a730bf703c792"},
 :lib io.github.babashka/bbin}

~/.local/bin$ ls -lah neil
-rwxr-xr-x 1 ieugen ieugen 1,1K sep 27 11:26 neil

cat neil 
#!/usr/bin/env bb

; :bbin/start
;
; {:coords
;  {:git/url "https://github.com/babashka/bbin",
;   :git/sha "6fde1b1dbfaef3063eb1eba4899a730bf703c792"},
;  :lib io.github.babashka/bbin}
;
; :bbin/end

(require '[babashka.process :as process]
         '[babashka.fs :as fs]
         '[clojure.string :as str])

(def script-root "/home/ieugen/.gitlibs/libs/io.github.babashka/bbin/6fde1b1dbfaef3063eb1eba4899a730bf703c792")
(def script-lib 'io.github.babashka/bbin)

Might be related to

• #34
• #18
• #35

I ran into some errors when using local/deps and :local/root during initial development. That said I need to do some more investigation on what was actually going on. It's possible this may not be an issue anymore.

pre-emptive apology
I don't mean to be the "but what about Windows?" guy, but...

problem statement
Installing a local clj on Windows throws:

>bbin version
bbin 0.1.3

>bbin install hellocal.clj
{:coords {:bbin/url "file://C:\\testing_stuff\\hellocal.clj"}}
----- Error --------------------------------------------------------------------
Type:     java.lang.IllegalArgumentException
Message:  C:\testing-stuff\hellocal is not a relative path
Location: C:\path\to\bbin:398:38
...
398:         script-file (fs/canonicalize (fs/file (util/bin-dir cli-opts) script-name)
                                          ^--- C:\testing-stuff\hellocal is not a relative path
... (longer details elided for brevity)

but works if an alias is supplied:

>bbin install hellocal.clj --as hellocal
{:coords {:bbin/url "file://C:\\testing_stuff\\hellocal.clj"}}

analysis

• the error and the fact that it works with an alias suggests that script-name is problematic
• script-name comes from file-path->script-name, which is splitting the path on '/', which doesn't produce the desired result on a Windows canonicalized path
• I haven't tested on a local jar yet, but I assume the same symptom and root cause would pop up there

I haven't started on a PR yet, but a little thinking about it raised another question in my head:
The current file-path->script-name takes the first split of the file name on .. Would it be desirable to drop the extension instead? By which I mean 'foo.bar.clj' would be installed as 'foo.bar'. I realize this is very unlikely to make a difference because of namespace names and what not, but I was considering using fs/strip-ext in the PR, which would technically be different in the edge case.

Like bbin install, but it doesn't save a script to to the bbin bin path. It just runs the bb command with the correct arguments.

$ bbin exec io.github.rads/watch --version
watch 0.0.4

My private repo cannot be installed because it has a dot in the name. When I change the repo name and replace the dot for a dash (-) it works again.

bbin install io.github.jeroenvandijk/aws.console --as aws.console-github
----- Error --------------------------------------------------------------------
Type:     clojure.lang.ExceptionInfo
Message:  Invalid script coordinates.
If you're trying to install from the filesystem, make sure the path actually exists.
Data:     {:script/lib "io.github.jeroenvandijk/aws.console", :procurer :unknown-procurer, :artifact :unknown-artifact}
Location: /usr/local/bin/bbin:750:11

----- Context ------------------------------------------------------------------
746:           [:local :dir] (install-deps-git-or-local cli-opts' summary)
747:           [:local :file] (install-local-script cli-opts')
748:           [:local :jar] (install-local-jar cli-opts')
749:           [:maven :jar] (install-deps-maven cli-opts')
750:           (throw (ex-info "Invalid script coordinates.\nIf you're trying to install from the filesystem, make sure the path actually exists."
               ^--- Invalid script coordinates.
If you're trying to install from the filesystem, make sure the path actually exists.
751:                           {:script/lib (:script/lib cli-opts')
752:            

The bbin install command should print human-friendly text like "Installed cljfmt in ~/.babashka/bbin/binor so instead of printing EDN, perhaps with--format edn. Similar for bbin ls`.

Minor issue: the EDN should be printed with *print-namespace-maps* false.

This would allow people to install scripts from a URL which requires auth.

Thank you for creating this project. I did some tests and things work in a very clean way.

I'm trying to minimize the startup time of my cli tool. When I install it with bbin it adds around 40ms extra due to how the wrapper script works (invoking bb one more time). By writing my own wrapper script I could keep the total around 40ms, but this would limit the distribution options (only exact url or local file). Maybe it is possible to have bbin provide a hook that would be called during installation in order to create a custom wrapper script?

You might reason 40ms is not that much, but I'm working on a command line tool that I intend to use regularly and as a part of a chain of other binaries. Sometimes the binary is even executed multiple times in one invocation. This means that every addition in latency quickly adds up.

Maybe related to #18 and #40

I'll do some experimentation with my own wrapper script and report back later.

From the Design Docs:

• We can install dependencies from untrusted sources, but only with an explicit SHA.
  • For a Git repo, it's a commit SHA.
  • For a file, it's a SHA256 hash of the file contents.

Related: #7

Since :bbin/bin is a map, I assumed installing multiple binaries was okay:

{:deps {io.github.teodorlu/shed {:local/root "."}}
 :bbin/bin {update-repos {:main-opts ["-m" "teodorlu.shed.update-repos/-main"]}
            browsetxt {:main-opts ["-m" "teodorlu.shed.browsetxt/-main"]}}}

(source)

But this seems to only install update-repos. Originally asked on Slack.

Should bbin be able to install multiple binaries, and not just one?

bbin install currently does not seem to work with non-raw gist links. It might be a nice convenience feature to add such capability.

I tried to run bbin with a non-raw gist link: bbin install https://gist.github.com/chase-lambert/c5533d8e8fbb71268a25e83ecf8e3cc6 but got the following error:

----- Error --------------------------------------------------------------------
Type:     clojure.lang.ExceptionInfo
Message:  Invalid script coordinates.
If you're trying to install from the filesystem, make sure the path actually exists.
Data:     {:script/lib "https://gist.github.com/chase-lambert/c5533d8e8fbb71268a25e83ecf8e3cc6", :procurer :http, :artifact :unknown-artifact}
Location: /home/chase/.babashka/bbin/bin/bbin:629:11

----- Context ------------------------------------------------------------------
625:           [:local :dir] (install-deps-git-or-local cli-opts' summary)
626:           [:local :file] (install-local-script cli-opts')
627:           [:local :jar] (install-local-jar cli-opts')
628:           [:maven :jar] (install-deps-maven cli-opts')
629:           (throw (ex-info "Invalid script coordinates.\nIf you're trying to install from the filesystem, make sure the path actually exists."
               ^--- Invalid script coordinates.
If you're trying to install from the filesystem, make sure the path actually exists.
630:                           {:script/lib (:script/lib cli-opts')
631:                            :procurer procurer
632:                            :artifact artifact})))))))
633:
634: (defn uninstall [cli-opts]

----- Stack trace --------------------------------------------------------------
babashka.bbin.scripts  - /home/chase/.babashka/bbin/bin/bbin:629:11
babashka.bbin.cli/run  - /home/chase/.babashka/bbin/bin/bbin:657:5
babashka.bbin.cli/run  - /home/chase/.babashka/bbin/bin/bbin:654:1
babashka.bbin.cli      - /home/chase/.babashka/bbin/bin/bbin:671:10
babashka.bbin.cli/bbin - /home/chase/.babashka/bbin/bin/bbin:706:5
babashka.bbin.cli/bbin - /home/chase/.babashka/bbin/bin/bbin:703:1
babashka.bbin.cli      - /home/chase/.babashka/bbin/bin/bbin:709:3
clojure.core/apply     - <built-in>
babashka.bbin.cli      - /home/chase/.babashka/bbin/bin/bbin:713:3
babashka.bbin.cli      - /home/chase/.babashka/bbin/bin/bbin:711:1

Everything worked as expected when I used the raw link: https://gist.githubusercontent.com/chase-lambert/c5533d8e8fbb71268a25e83ecf8e3cc6/raw/9cc7b4208f9bed2609f4736c4f469e1828273af5/chatgpt-cli.clj

Proposed Changes

Reference: https://specifications.freedesktop.org/basedir-spec/basedir-spec-latest.html

• Scripts will be installed to $BABASHKA_BBIN_BIN_DIR (there is no explicit XDG variable for this)
  • Default: $HOME/.local/bin
• Cached JARs will be installed to $BABASHKA_BBIN_JARS_DIR
  • Default: $XDG_CACHE_HOME/babashka/bbin/jars -> $HOME/.cache/babashka/bbin/jars
• In the future, script metadata can be installed to $BABASHKA_BBIN_SCRIPTS_DIR
  • Default: $XDG_CONFIG_HOME/babashka/bbin/scripts -> $HOME/.config/babashka/bbin/scripts
  • We should also support $XDG_CONFIG_DIRS

Follow up from #25.

I'm not aware of any blockers at the moment. After the macOS/Linux implementation solidifies a bit more, I should be able to port the scripts to Windows pretty easily.

Right now adding ~/.bbin/bin to PATH is a manual step in the README.