• create repo for installer requests: https://github.com/webinstall/webi-installer-requests
• copy existing .github/ISSUE_TEMPLATE
• update .github/ISSUE_TEMPLATE
• update link on webinstall.dev

Add rnd

This is two parts:

1. A program called rnd that spits out a random string (for passwords, tokens, etc)
2. Make installable via webi (could be done by a different person, such as myself)

Currently I have some aliases for creating random strings:

webi aliasman

aliasman alias rnd='xxd -l24 -ps /dev/urandom | xxd -r -ps | base64 | tr -d = | tr + - | tr / _'
aliasman alias xrnd='xxd -l24 -ps /dev/urandom'

It would be trivial to create a cross-platform Go program that spits out a random string.

rnd
2> (196-bit, base64)
> Zys_yZqEySPS5il0l1Vew_y46fQwAqvL

rnd --hex --bits 128
2> (128-bit, hexidecimal)
> 506cc039ef529b631d3d6a6baff6717e

Honestly, I'd prefer it in Rust, but I have yet to see a simple solution for cross compiling to multiple platforms from a Linux host (yes, Windows too, no the user to separately install the visual studio runtime distribution is not acceptable).

Getting this on webi would as simple as copying _example, updating the github releases
info, and doing a find and replace on a few file system path names.

There are several Go programs on here which are released with goreleaser which could serve as examples.

Part 2. How to create a webi installer

Skills required

• Basic Command Line knowledge (mkdir, mv, ls, tar, unzip, variables)

Steps

1. Clone and setup the webi packages repo

   git clone [email protected]:webinstall/packages.git
   pushd packages/
   npm install

2. Copy the example template and update with info from Official Releases:
   https://github.com/___CHANGE/ME___/releases

   rsync -av _example/ CHANGE-ME/

   • update CHANGE-ME/release.js to use the official repo
   • Learn how CHANGE-ME unpacks (i.e. as a single file? as a .tar.gz? as
     a .tar.gz with a folder named CHANGE-ME?)
   • find and replace to change the name
     • update CHANGE-ME/install.sh (see bat and jq as examples)
     • update CHANGE-ME/install.ps1 (see bat and jq as examples)
3. Needs an updated tagline and cheat sheet
   • update CHANGE-ME/README.md
     • official URL
     • tagline
     • Switch versions
     • description / summary
     • General pointers on usage (and perhaps "gotchas")

It's also okay to have multiple people work on part of this (i.e. the Cheat
Sheet can be done independently from the install.sh)

We need machine-friendly ways of editing certain aspects of the config - which will likely require splitting it out into more files and whatnot.

See webinstall/webi-installers#641 (comment).

For example:

vimtool get ale.fixers
vimtool add ale.fixers.markdown prettier

I hesitate to say "we should just do it as JSON and generate the config each time" because that's well outside the convention (whereas having a few extra files is odd, but not crazy), but... 🤷‍♂️

• purchase the domains
• setup standard html-only redirects
• webi.sh should always show the posix script
• webi.ms should always show the powershell script
• no change to any scripts that rely on webinstall.dev

winget is a built-in tool available on Windows 10 and 11.

See https://docs.microsoft.com/en-us/windows/package-manager/package/repository.

The updated webi install command could be:

winget webi && webi node

Webi has become far too complex.

I wanted something with near-0 magic - which is why I opted for shell scripts.

But as I've added one case, and then another, and then another (and pivoted from the helper script to the webi command), it's bloated way too far.

There is a lot of necessary complexity, but there's also a lot of leftovers from the learning process.

My thoughts on fixing that:

• webi needs to be its own command, not embedded in the bootstrap
  (less scary, easier to lint, more fun to add --whatever too)
• webi should be offline-first
  (ex: switch versions, reinstall most recent version)
• and yet... everything that can be decided ahead of time (online), should be
  • webi_download should be templated based on curl and wget
  • webi_extract should be templated based on tar, xz, git etc
  • add new APIs to support newer webi command
• rename bootstrap.sh to curl-pipe-bootstrap.sh for clarity
• rename template.sh to install.tpl.sh for clarity
• link to generate script from website (since the one in the repo may be more abstract)

This shouldn't really be Webi v2.0.0 - because it should be a smooth refactor with additional API on the server-side, not breaking changes, nor a from-scratch rewrite, nor a philosophical shift from the original goal.

Thoughts

• Git UA? https://stackoverflow.com/questions/29117355/git-clone-how-to-fake-user-agent

Deserves an overview video, for sure.
https://webinstall.dev/vim-essential/

Sustain

• users + paid commercial use
• let users save custom bundle scripts https://webi.ms/@dude/my-vpses (or https://dude.webi.ms/my-vpses?)

Simplify

• eliminate pathman
  • for Windows webinstall/webi-installers#505
  • for Unix https://github.com/webinstall/webi-installers/pulls/612
• eliminate the need for a bootstrap script
  • (include pathman and zip/tar detect function in the installer itself?)

Streamline

• create installers based on heuristics (per version?)
• don't exclude BSD
• add all domains to the HSTS preload list
  • webinstall.dev: https://hstspreload.org/?domain=webinstall.dev
  • webi.sh: https://hstspreload.org/?domain=webi.sh
  • webi.ms: https://hstspreload.org/?domain=webi.ms

Ok. This really isn't a super amazing feature request, but I was just polishing up my readme for a bit and went down a rabbit hole. I decided to add some of those neat little "badges" or shields and dug into how to use the shields.io api. And then I got the bright idea to create a Webi badge using:

https://img.shields.io/static/v1?label=webi&message=gprox&color=6c71c4&labelColor=fdf6e3&logoWidth=10&logo=data:image/svg+xml;base64,PHN2ZyBhcmlhLWhpZGRlbj0idHJ1ZSIgZGF0YS1wcmVmaXg9ImZhcyIgZGF0YS1pY29uPSJib2x0IiB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciIHZpZXdCb3g9IjAgMCAzMjAgNTEyIiB3aWR0aD0iMzQiIGhlaWdodD0iNTYiPjxwYXRoIGZpbGw9IiM2YzcxYzQiIGQ9Ik0yOTYgMTYwSDE4MC42bDQyLjYtMTI5LjhDMjI3LjIgMTUgMjE1LjcgMCAyMDAgMEg1NkM0NCAwIDMzLjggOC45IDMyLjIgMjAuOGwtMzIgMjQwQy0xLjcgMjc1LjIgOS41IDI4OCAyNCAyODhoMTE4LjdMOTYuNiA0ODIuNWMtMy42IDE1LjIgOCAyOS41IDIzLjMgMjkuNSA4LjQgMCAxNi40LTQuNCAyMC44LTEybDE3Ni0zMDRjOS4zLTE1LjktMi4yLTM2LTIwLjctMzZ6Ii8+PC9zdmc+Cg==

Which pretty much just follows the pattern:

https://img.shields.io/static/v1

query params:
label: webi
message: gprox
color: 6c71c4
labelColor: fdf6e3
logoWidth: 10
logo: data:image/svg+xml;base64,<BASE_64 OF THE WEBI BOLT SVG>

It'd be kind of cool if someone wanted to just add a badge on their repo to direct people to a recognizable link to the web installer website. Not sure if you can package the badge-url into the api somehow and let people put that on their website/github or something.

But also, I dont feel super strong about this, haha. Just thought it was kind of fun to have.

What do you think about using Mastodon instead of Twitter?

Update: Confirmed

Not sure what I was looking at before, but the difference between these two looks right:

note: also useful: https://github.blog/2020-12-21-get-up-to-speed-with-partial-clone-and-shallow-clone/

git ls-remote --tags https://github.com/jshint/jshint

• -c 'versionsort.suffix=-' means that 1.0.0-rc4 sorts before 1.0.0
• --sort=version:refname is numeric sort order (1.2.0 is before 1.10.0)
• --sort=-version:refname changes the sort order to number version, descending
• when ^{} is in the output, it's the true commit (i.e. the tag is dereferenced from the commit)

Before:

• 1.1.0
• 1.10.0
• 1.2.0

Ascending Sort

--sort=version:refname

git -c 'versionsort.suffix=-' \
    ls-remote --tags --sort='version:refname' \
    https://github.com/jshint/jshint

After:

• 1.1.0
• 1.2.0
• 1.10.0

Descending Sort

--sort=-version:refname

git -c 'versionsort.suffix=-' \
    ls-remote --tags --sort='-version:refname' \
    https://github.com/jshint/jshint

After:

• 1.10.0
• 1.2.0
• 1.1.0
• 1.0.0
• 1.0.0-rc4

Release Type Comparison

• Binary Release

  {
    "name": "sclient_1.5.0_darwin_amd64v2.tar.xz",
    "version": "1.5.0",
    "lts": false,
    "channel": "stable",
    "date": "2023-10-28",
    "os": "macos",
    "arch": "amd64",
    "ext": "tar.xz",
    "download": "https://github.com/therootcompany/sclient/releases/download/v1.5.0/sclient_1.5.0_darwin_amd64v2.tar.xz"
  }

• Source Release

  {
    "name": "BeyondCodeBootcamp-aliasman-v1.1.2-0-g0e5e1c1.tar.gz",
    "version": "1.1.2",
    "lts": false,
    "channel": "stable",
    "date": "2023-02-23",
    "os": "linux",
    "arch": "amd64",
    "ext": "tar.gz",
    "download": "https://codeload.github.com/BeyondCodeBootcamp/aliasman/legacy.tar.gz/refs/tags/v1.1.2"
  }

• Git Release

  {
    "name": "aliasman",
    "version": "1.1.2",
    "lts": false,
    "channel": "stable",
    "date": "2023-02-23",
    "os": "linux",
    "arch": "amd64",
    "ext": "git",
    "download": "https://github.com/BeyondCodeBootcamp/aliasman.git"
  }

Original

Most (all?) of the vim plugins are direct via Git (not GitHub, just git).

For anything that has a tagged git source we could switch to showing versions using git ls-remote:

git -c 'versionsort.suffix=-' \
    ls-remote --tags --sort='version:refname' \
    https://github.com/BeyondCodeBootcamp/aliasman

6a0ee1723b2106e117fc0f3862ca340eab8c26c5	refs/tags/v1.0.0
4171e2fe58122c61a100941c05ef963d80b413f4	refs/tags/v1.0.1

(doesn't seem sorted by version at all...)

This may include replacing the newly-added GitHub Source adapter, in most cases.

This would be helpful for automated checks to determine which packages have changed which OSes, Arches, etc they support.

It would also be useful for checking which packages are similar in terms of being msvc or musl or xz, etc.

When I first wrote this I was aware that Windows had symlinks, but that they required administrator privileges.

However, Windows also has "junctions" which, for all intents and purposes, appear to be symlinks... but technically are not... somehow. Anyway, they do NOT require admin privs.

It would speed up Windows installs significantly to use junctions.

Example:

https://github.com/webinstall/webi-installers/blob/84722bd1409d7774f5032e04d4f1aca00ebe2fa3/atomicparsley/install.ps1#L60C1-L60C60

Instead of

Copy-Item -Path "$pkg_src" -Destination "$pkg_dst" -Recurse

we could do a junction (if it's a directory and not just a file).

Future Updates

This may mean that we'd want to install all packages into
.local/opt/<package-name>-<version>/bin/<bin-name>
so that we could junction to
.local/opt/<package-name>/bin/<bin-name>

GitHub Actions would be very useful to test if an installer has broken due to some code change or a new package release. A daily runner would be able to find these issues before our users do.

Something like:

• webinstall/webi-installers#346 (comment)
• webinstall/webi-installers#714 (comment)

Short, 5 minute.

Not so long as https://www.youtube.com/watch?v=hj814rdTDm0&list=PLxki0D-ilnqbXhctTFCDGH3fSN7l7ATkL

We need a Tips 'n' Tricks blog

• How to host Node on Digital Ocean
  • node
  • caddy
  • serviceman
• How to self-host imgur clone
• How to self-host cors proxy
• etc

We want to update the husky pre-commit file fmt (i.e. prettier, shfmt) and lint (i.e. jshint, shellcheck) so that:

• only files that are staged for commit are checked
• use the "autofix" capabilities of prettier and shellcheck where possible

Here's an example that can be used to "autofix" with prettier (and there's a link to a similar script for shellcheck - both may need tweaking):

#!/usr/bin/env sh
. "$(dirname -- "$0")/_/husky.sh"

# from https://prettier.io/docs/en/precommit.html#option-6-shell-script

# Should be functionally equivalent to `npm run fmt`
# We're not using basetag
#npx basetag rebase
FILES=$(
    git diff --cached --name-only --diff-filter=ACMR | sed 's| |\\ |g'
)
if [ -z "$FILES" ]; then
    exit 0
fi

echo "$FILES" | xargs npx prettier@2 --ignore-unknown --write
echo "$FILES" | xargs git add

# Should be functionally equivalent to `npm run lint`
JS_FILES=$(
    echo "$FILES" | grep '\.js$' || true
)
if [ -z "$JS_FILES" ]; then
    exit 0
fi
echo "$JS_FILES" | xargs npx jshint@2 -c .jshintrc.lax

See also:

• https://prettier.io/docs/en/precommit.html#option-6-shell-script
• koalaman/shellcheck#1220 (comment)

The pyenv installer needs to test for the existence of zsh and fish and then update the appropriate config files.

Fish does NOT have a config directory or file by default. Here are some places to check to see if fish is installed.

• ~/.local/share/fish/fish_history
• ~/.config/fish
• ${WEBI_OPT_PATH}/fish
• "$(command -v fish)"

A very high percentage of developer tools that provide packaged releases (~99%) can be categorized into a finite set of formats that can be installed with a finite set of rules - probably around 6 for unpacking, and 6 for type of contents.

We should be able to create almost all (80%+) installers from a set of rules.

Rules

Typically we're dealing with one of these package styles:

# bare (like ollama)
thing-1.0-linux

# bare, compressed
unxz thing-1.0-linux.xz

# compressed container
tar xvf thing-1.0-linux.tar.xz
unzip thing-1.0-linux.zip
diskutil ... thing-1.0-arm.dmg

Once unpacked, the directory structure almost always falls into one of these hierarchies:

./thing
./bin/thing
./thing-1.0/thing
./thing-1.0/bin/thing

# macOS apps (iTerm2, Brave)
Thing.app/

# macOS faux apps (fish, cmake)
Thing.app/Contents/thing
Thing.app/Contents/bin/thing

# Windows files have .exe
thing.exe

We should always know the exact name of the command that will be installed.

We should be able to determine if there are other files or just the one.

We should be able to tell if the manpage file exists or not.

Exceptions

• License
• Readme
• Shell Completions
• Top-level Man pages

If we detect extra loose files outside the opt directory structure, we can "best guess it" flag it for manual review.

• Go (has two bin dirs)
• Node (no bin on Windows)

Notes

we probably shouldn't attempt this in shell, but...

# count how deep "thing" is once unpacked 
echo "thing/bin/thing" | tr -dc '/' | wc -c

Currently webi is only updated when a curl https://webinstall.dev/xxxx is run. It should definitely update when webi webi is run, and perhaps any time webi xxxx is run.

• Switch webi script heredoc to heredoc literal

  - cat << EOF > ...
  + cat <<'EOF' > ...

• Move webi script to ./webi/webi.sh
• Update serveInstaller and servePackage/serveScript to string-replace ./webi/webi.sh

• webi info <installer> should show (and possibly open) the URL with the cheat sheet.
• webi versions <installer> should show the relevant https://webinstall.dev/api/releases/<installer>.tab ?os=macos&arch=aarch64&pretty=true
• webi discord-invite to join the discord
• webi twitter-follow to follow on twitter

I don't quite understand the purpose of yet another alternative when we already have tools like Scoop, Winget, and Choco. Don't these tools already cover all use cases?

To me this seems to be almost exactly the same as Scoop.
Can someone explain diffs to me?

Hi, I'm presently developing something which as a high degree of overlap with webi.
http://github.com/elasticdotventures/_b00t_

It's definitely not same-same -- I'm planning to add webi in places where you have web installers. If webi can run bash then webi could use b00t installers, sort of incestuous.

My primary focus is menus around tool path selection and deployment (basically templated menus and interface libraries), a lot of devops/gitops type behaviors.

I like how you've configured everything to install in .local quite a bit. Anyway mostly wanted to say hi and cool project. Initiate a conversation about how we might work collaborate. Cheers!

Obviously it's a dumb idea to embed a script in a heredoc of another script.

We need a different solution for _webi/bootstrap.sh, and it may require a shudder build step... 😬

Thoughts

How about we just make the webi command its own command and installer, just like ssh-adduser?

Or maybe even host it on GitHub releases so it gets its own version?

Maybe just check if the downloaded version is more than 5 minutes old? (bad idea)

Run if WEBI_INSTALL=true is not set. 👍

Implementation thoughts

webi webi and webi update should manually update webi, regardless of version.

if command -v curl > /dev/null 2> /dev/null; then
    if ! curl -fsSL "$my_installer_url" -H "User-Agent: curl $WEBI_UA" \
        -o "$WEBI_BOOT/$my_package-bootstrap.sh"; then
        echo >&2 "error fetching '$my_installer_url'"
        exit 1
    fi
else
    if ! wget -q "$my_installer_url" --user-agent="wget $WEBI_UA" \
        -O "$WEBI_BOOT/$my_package-bootstrap.sh"; then
        echo >&2 "error fetching '$my_installer_url'"
        exit 1
    fi
fi

    webi_download \
        "$WEBI_HOST/packages/${my_cmd}/${my_cmd}.sh" \
        "$HOME/.local/bin/${my_cmd}"