Installing the ESP32 SDK through xs-dev setup --device=esp32 requires an extra step.

Environment is a MacBook Air M1 2020 running on macOS 13.1 (22C65).

⠧ Installing esp-idf tooling
All done! You can now run:

  . ./export.sh

✔ Installing esp-idf tooling
ℹ Sourcing esp-idf environment
✔ 
  Successfully set up esp32 platform support for Moddable!
  Test out the setup by starting a new terminal session, plugging in your device, and running: xs-dev run --example helloworld --device=esp32
  If there is trouble finding the correct port, pass the "--port" flag to the above command with the path to the /dev.cu.* that matches your device.

Problem is, the correct folder for running . ./export.sh is not mentioned .

Correct procedure is

cd /Users/USER/.local/share/esp32/esp-idf
. ./export.sh

Could xs-dev manage this extra step?

Thank you!

If the user does a setup and ignores the instructions to "start a new terminal session" and jumps directly to "run" the error message is misleading:

$ xs-dev setup
✔ Moddable SDK successfully set up! Start a new terminal session and run the "helloworld example": xs-dev run --example helloworld
$ xs-dev run --example helloworld
Example project does not exist.
Lookup the available examples: xs-dev run --list-examples
$ xs-dev run --list-examples
/opt/homebrew/lib/node_modules/xs-dev/node_modules/gluegun/build/index.js:15
    throw up;
    ^

Error: At least one choice must be selectable

xs-dev might want to have some preflight checks that it uses consistently to determine if the install is good (for the Moddable SDK itself, and for each device target). This will never be perfect, but some are pretty straightforward - like a missing $MODDABLE environment variable.

Here's an example from the another perspective. The user does a teardown and then tries to use the install:

$ xs-dev teardown
$ xs-dev run --example piu/balls
Example project does not exist.
Lookup the available examples: xs-dev run --list-examples

After install Pico support on macOS Monterey (12.3.1) building helloworld...

xs-dev run --example helloworld --device pico

...fails because pioasm can't be found.

$ xs-dev run --example helloworld --device pico
⠧ Building and deploying project /Users/peterhoddie/.local/share/moddable/examples/helloworld on pico/usr/local/lib/node_modules/xs-dev/node_modules/gluegun/build/index.js:15
    throw up;
    ^

Error: Command failed with exit code 2: mcconfig -m -p pico -t all -o /Users/peterhoddie/.local/share/moddable/build -d
make: /Users/peterhoddie/pico/pico-sdk/build/pioasm/pioasm: No such file or directory
make: *** [/Users/peterhoddie/.local/share/moddable/build/tmp/pico/debug/helloworld/ws2812.pio.h] Error 1
make: *** Waiting for unfinished jobs....
/Users/peterhoddie/.local/share/moddable/xs/platforms/pico/xsPlatform.c: In function 'fxCreateMachinePlatform':
/Users/peterhoddie/.local/share/moddable/xs/platforms/pico/xsPlatform.c:79:2: warning: implicit declaration of function 'modMachineTaskInit' [-Wimplicit-function-declaration]
   79 |  modMachineTaskInit(the);
      |  ^~~~~~~~~~~~~~~~~~
/Users/peterhoddie/.local/share/moddable/xs/platforms/mc/xsHosts.c: In function 'modRunMachineSetup':
/Users/peterhoddie/.local/share/moddable/xs/platforms/mc/xsHosts.c:559:2: warning: implicit declaration of function 'modInstrumentationSetup'; did you mean 'modInstrumentationSet_'? [-Wimplicit-function-declaration]
  559 |  modInstrumentationSetup(the);
      |  ^~~~~~~~~~~~~~~~~~~~~~~
      |  modInstrumentationSet_
/Users/peterhoddie/.local/share/moddable/xs/platforms/pico/xsPlatform.c: In function 'fxDeleteMachinePlatform':
/Users/peterhoddie/.local/share/moddable/xs/platforms/pico/xsPlatform.c:103:2: warning: implicit declaration of function 'modMachineTaskUninit' [-Wimplicit-function-declaration]
  103 |  modMachineTaskUninit(the);
      |  ^~~~~~~~~~~~~~~~~~~~
/Users/peterhoddie/.local/share/moddable/xs/platforms/pico/xsPlatform.c: In function 'fxReceiveLoop':
/Users/peterhoddie/.local/share/moddable/xs/platforms/pico/xsPlatform.c:462:6: warning: implicit declaration of function 'modMessagePostToMachine' [-Wimplicit-function-declaration]
  462 |      modMessagePostToMachine(current, NULL, 0xffff, doDebugCommand, current);
      |      ^~~~~~~~~~~~~~~~~~~~~~~
/Users/peterhoddie/.local/share/moddable/xs/platforms/pico/xsPlatform.c: In function 'fxInNetworkDebugLoop':
/Users/peterhoddie/.local/share/moddable/xs/platforms/pico/xsPlatform.c:723:66: warning: comparison between pointer and integer
  723 |  return the->DEBUG_LOOP && the->connection && (kSerialConnection != the->connection);
      |                                                                  ^~
/var/folders/nd/2xq97b3j4fgd9zqrz1hlykvm0000gn/T//ccxgp5iL.s: Assembler messages:
/var/folders/nd/2xq97b3j4fgd9zqrz1hlykvm0000gn/T//ccxgp5iL.s:457: Warning: setting incorrect section attributes for .data.xsram
/var/folders/nd/2xq97b3j4fgd9zqrz1hlykvm0000gn/T//ccC4rD4c.s: Assembler messages:
/var/folders/nd/2xq97b3j4fgd9zqrz1hlykvm0000gn/T//ccC4rD4c.s:2846: Warning: setting incorrect section attributes for .rodata
# compile pio:  ws2812.pio
# library xs: xsHosts.c
# library xs: xsHost.c
# library xs: xsPlatform.c
# library xs: xsAll.c
    at makeError (/usr/local/lib/node_modules/xs-dev/node_modules/execa/lib/error.js:60:11)
    at handlePromise (/usr/local/lib/node_modules/xs-dev/node_modules/execa/index.js:118:26)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5) {
  shortMessage: 'Command failed with exit code 2: mcconfig -m -p pico -t all -o /Users/peterhoddie/.local/share/moddable/build -d',
  command: 'mcconfig -m -p pico -t all -o /Users/peterhoddie/.local/share/moddable/build -d',
  escapedCommand: 'mcconfig -m -p pico -t all -o "/Users/peterhoddie/.local/share/moddable/build" -d',
  exitCode: 2,
  signal: undefined,
  signalDescription: undefined,
  stdout: '# compile pio:  ws2812.pio\n' +
    '# library xs: xsHosts.c\n' +
    '# library xs: xsHost.c\n' +
    '# library xs: xsPlatform.c\n' +
    '# library xs: xsAll.c',
  stderr: 'make: /Users/peterhoddie/pico/pico-sdk/build/pioasm/pioasm: No such file or directory\n' +
    'make: *** [/Users/peterhoddie/.local/share/moddable/build/tmp/pico/debug/helloworld/ws2812.pio.h] Error 1\n' +
    'make: *** Waiting for unfinished jobs....\n' +
    "/Users/peterhoddie/.local/share/moddable/xs/platforms/pico/xsPlatform.c: In function 'fxCreateMachinePlatform':\n" +
    "/Users/peterhoddie/.local/share/moddable/xs/platforms/pico/xsPlatform.c:79:2: warning: implicit declaration of function 'modMachineTaskInit' [-Wimplicit-function-declaration]\n" +
    '   79 |  modMachineTaskInit(the);\n' +
    '      |  ^~~~~~~~~~~~~~~~~~\n' +
    "/Users/peterhoddie/.local/share/moddable/xs/platforms/mc/xsHosts.c: In function 'modRunMachineSetup':\n" +
    "/Users/peterhoddie/.local/share/moddable/xs/platforms/mc/xsHosts.c:559:2: warning: implicit declaration of function 'modInstrumentationSetup'; did you mean 'modInstrumentationSet_'? [-Wimplicit-function-declaration]\n" +
    '  559 |  modInstrumentationSetup(the);\n' +
    '      |  ^~~~~~~~~~~~~~~~~~~~~~~\n' +
    '      |  modInstrumentationSet_\n' +
    "/Users/peterhoddie/.local/share/moddable/xs/platforms/pico/xsPlatform.c: In function 'fxDeleteMachinePlatform':\n" +
    "/Users/peterhoddie/.local/share/moddable/xs/platforms/pico/xsPlatform.c:103:2: warning: implicit declaration of function 'modMachineTaskUninit' [-Wimplicit-function-declaration]\n" +
    '  103 |  modMachineTaskUninit(the);\n' +
    '      |  ^~~~~~~~~~~~~~~~~~~~\n' +
    "/Users/peterhoddie/.local/share/moddable/xs/platforms/pico/xsPlatform.c: In function 'fxReceiveLoop':\n" +
    "/Users/peterhoddie/.local/share/moddable/xs/platforms/pico/xsPlatform.c:462:6: warning: implicit declaration of function 'modMessagePostToMachine' [-Wimplicit-function-declaration]\n" +
    '  462 |      modMessagePostToMachine(current, NULL, 0xffff, doDebugCommand, current);\n' +
    '      |      ^~~~~~~~~~~~~~~~~~~~~~~\n' +
    "/Users/peterhoddie/.local/share/moddable/xs/platforms/pico/xsPlatform.c: In function 'fxInNetworkDebugLoop':\n" +
    '/Users/peterhoddie/.local/share/moddable/xs/platforms/pico/xsPlatform.c:723:66: warning: comparison between pointer and integer\n' +
    '  723 |  return the->DEBUG_LOOP && the->connection && (kSerialConnection != the->connection);\n' +
    '      |                                                                  ^~\n' +
    '/var/folders/nd/2xq97b3j4fgd9zqrz1hlykvm0000gn/T//ccxgp5iL.s: Assembler messages:\n' +
    '/var/folders/nd/2xq97b3j4fgd9zqrz1hlykvm0000gn/T//ccxgp5iL.s:457: Warning: setting incorrect section attributes for .data.xsram\n' +
    '/var/folders/nd/2xq97b3j4fgd9zqrz1hlykvm0000gn/T//ccC4rD4c.s: Assembler messages:\n' +
    '/var/folders/nd/2xq97b3j4fgd9zqrz1hlykvm0000gn/T//ccC4rD4c.s:2846: Warning: setting incorrect section attributes for .rodata',
  failed: true,
  timedOut: false,
  isCanceled: false,
  killed: false
}

Node.js v18.12.1

It may be that the Pico setup code in xs-dev needs to be updated. The dependency on pioasm is relatively new, arriving when Pico W platform support was implemented. @mkellner maybe able to provide some information to help with this.

I am setting up a dedicated OSX laptop for Moddable/xs-dev development and running into an issue following the documentation.

After preparing the laptop with the requirements when I run the xs-dev setup command I get a permissions error running the script.
permission denied, mkdir '/usr/local/lib/node_modules/xs-dev'

I have a freshly formatted MacBook Pro with Ventura 13.2.1 installed. Xcode and its CLI tools installed. I have only one account on this laptop and I am the Administrator.
I installed Node.js (followed the link in xs-dev/introduction and installed from the node.js OSX installer) I can confirm in terminal that node v18.15.0 and rpm 9.5.0 are installed. I can see that /usr/local/bin is in my $PATH.
I have installed Homebrew.

When running the xs-dev setup command I get:
npm ERR! code EACCES
npm ERR! syscall mkdir
npm ERR! path /usr/local/lib/node_modules/xs-dev
npm ERR! errno -13
npm ERR! Error: EACCES: permission denied, mkdir '/usr/local/lib/node_modules/xs-dev'
npm ERR! [Error: EACCES: permission denied, mkdir '/usr/local/lib/node_modules/xs-dev'] {
npm ERR! errno: -13,
npm ERR! code: 'EACCES',
npm ERR! syscall: 'mkdir',
npm ERR! path: '/usr/local/lib/node_modules/xs-dev'
npm ERR! }
npm ERR!
npm ERR! The operation was rejected by your operating system.
npm ERR! It is likely you do not have the permissions to access this file as the current user
npm ERR!
npm ERR! If you believe this might be a permissions issue, please double-check the
npm ERR! permissions of the file and its containing directories, or try running
npm ERR! the command again as root/Administrator.

I am assuming that because I installed node.js as an administrator permissions should not be an issue.
Advice would be appreciated, thanks

When trying to run an example that does not exist, the error message indicates that the manifest is missing.

> xs-dev run --example piu/DOES_NOT_EXIST
Example project must contain a manifest.json.
Lookup the available examples: xs-dev run --list-examples

It would be more helpful to indicate that the example wasn't found.

In the current architecture of the CLI, the error handling is often repeated and singular to each task. Updating any error messaging or handling can result in tedious work. The stack traces printed by unhandled errors can be intimidating and unhelpful to those that encounter them.

Using something like modern-errors, there could be custom errors for raising known issues and consolidated handlers higher up in the chain. Unknown error handling could also result in more helpful output, including a prompt to automatically create an issue in the xs-dev repo with the stack trace and any other environment info to help with debugging.

The scan feature works very nicely for ESP8266 and ESP32. But, when neither of those is connected and instead something else is (e.g. Pico), scan exits with an exception:

$ xs-dev scan
⠧ Scanning for devices.../usr/local/lib/node_modules/xs-dev/node_modules/gluegun/build/index.js:15
    throw up;
    ^

Error: Command failed with exit code 1: esptool.py --port /dev/tty.usbmodem14101 read_mac
Traceback (most recent call last):
  File "/Users/hoddie/.espressif/python_env/idf4.4_py3.8_env/lib/python3.8/site-packages/serial/serialposix.py", line 398, in _reconfigure_port
    orig_attr = termios.tcgetattr(self.fd)
termios.error: (6, 'Device not configured')

During handling of the above exception, another exception occurred:
...

@phoddie mentioned this in #1, related to the hard-coded arch -arm64 command prefix before running any brew install processes, which failed on Intel-based Mac computers. I ran into this issue again when testing the #6 pico support branch, where Node was running under Rosetta but homebrew was running natively on arm64.

Example error output:

Error: Cannot install under Rosetta 2 in ARM default prefix (/opt/homebrew)!

To resolve this, the logic for determining the correct arch prefix cannot use the process.arch or os.arch() references in NodeJS, rather examining the output of brew --prefix. This should be abstracted to a helper function (brewInstall(packageName)) to keep that logic in one place and replace all manual evocations of system.exec('brew install').

I've started experimenting with pre-building the Moddable tooling for macos and linux (both x86_64) using GitHub Actions: https://github.com/HipsterBrown/moddable-prebuild

I manually kick off this action whenever I noticed new features within the Moddable repo, which leads to a gzip, tarball release for each platform under the latest tag that matches the commit from Moddable. This process could eventually be triggered by a regular cron job that watches for new commits.

Using these prebuilt releases could speed up the initial setup and future updates for each platform, since the build step would be skipped, especially on resource-lite machines.

Caveats:
I'm currently manually building the macos arm64 releases locally and adding them to the tagged release, so the cron job optimization would not be reliable until GitHub Actions supports arm64 environments or the build process updated to use a custom runner with support for more platforms (including Windows).

I'm trying out the Pico support on macOS. The setup succeeds without an error: xs-dev setup --device=pico. Running helloworld compiles the C binaries, builds the XS byte code, and links. All good.

Then it fails here:

make: /Users/hoddie/pico/pico-sdk/build/elf2uf2/elf2uf2: No such file or directory
make: *** [/Users/hoddie/.local/share/moddable/build/bin/pico/debug/helloworld/xs_pico.uf2] Error 1
    at makeError (/usr/local/lib/node_modules/xs-dev/node_modules/execa/lib/error.js:60:11)
    at handlePromise (/usr/local/lib/node_modules/xs-dev/node_modules/execa/index.js:118:26)
    at processTicksAndRejections (node:internal/process/task_queues:96:5) {
  shortMessage: 'Command failed with exit code 2: mcconfig -d -m -p pico',

There is an older copy of the pico-sdk on my computer at that path, but it is not linked to by any environment variable. The copy of pico-sdk installed by xs-dev is at /Users/hoddie/.local/share/pico/pico-sdk and does appear to contain elf2uf2 at the expected path.

While the underlying tooling for ESP32 and ESP8266 generally works to discover dev board ports, it would be helpful to provide a command or flag to scan from xs-dev and display a nice output.

This could start by shelling to esptool.py and parsing the output. It would be even better to find a way of scanning from NodeJS; perhaps through serialport, node-usb, or something else like the esptool-js utility (https://github.com/espressif/esptool-js).

Once pico support lands in the main branch, this command could expand to look for those boards as well.

A nice to have after scanning would be to save a discovered port to some project configuration file, something like the .env file described in another issue.

Starting in Moddable SDK 3.6.0 mcconfig implements the -l command line option to redirect debugging output to the console. This is necessary for headless development machines and convenient for developers who prefer to run without a debugger (I don't understand why, but I suppose it is familiar because of Node.js). This feature is implemented as scripts that run under Node.js. Before using the log feature, developers must first do an npm install to get the dependencies as documented under mcconfig:

cd $MODDABLE/tools/xsbug-log
npm install

Since xs-dev is already taking care of setting-up just about everything else and since it is running in Node.js we know npm is available, what do you think about also setting-up the log feature?

As mentioned in #1, @phoddie mentioned:

Is there a way to unsetup? Since the .profile is modified automatically, it could be nice to store it with the tool somehow.

Since this CLI tool appends several exports and modifies the global PATH, cleaning up those changes could be a hassle to automate. It is also a "leaky abstraction" to have all of those changes visible to the end user, who might be familiar with maintaining their shell profile.

If those changes were collected under a single mod-export.sh (similar to the export.sh file used by the esp-idf tooling), then appending a single source $MOD_PATH/mod-export.sh (MOD_PATH being somewhere under the ~/.local/share directory with the other installed dependencies) to the shell profile will be much easier to maintain long term and remove through a potential xs-dev teardown command.

The scan feature of xs-dev is incredibly useful for checking that devices are correct connected. Recent changes to the Moddable SDK are causing it to fail:

> xs-dev scan
esptool.py required to scan for Espressif devices. Setup environment for ESP8266 or ESP32:
 xs-dev setup --device esp32
 xs-dev setup --device esp8266.
⚠ No available devices found.

Before the changes, the Moddable SDK required the user to source $IDF_PATH/exports.sh before running any ESP32-related commands. Now that is taken care of automatically by the build. As a result, esptool.py isn't available by default when xs-dev runs. The text above is misleading, because it suggests that the ESP32 tools aren't installed, though they are.

At a minimum, it seems like the text above could be updated so that when $IDF_PATH is defined, a more accurate message is disabled. Perhaps it would make sense to perform the source automatically in that case, so the user doesn't have to. I'm unsure which approach makes more sense here.

Web developers will have Node.js and npm installed. Embedded developers may not. I started from a (nearly) fresh macOS set-up and failed at npm install -g xs-dev. A link to https://nodejs.org/en/download/ would help.

Alas, that installed but then failed on npm install -g xs-dev with a permission error.

npm notice 
npm notice New major version of npm available! 8.19.2 -> 9.1.2
npm notice Changelog: https://github.com/npm/cli/releases/tag/v9.1.2
npm notice Run npm install -g [email protected] to update!
npm notice 
npm ERR! code EACCES
npm ERR! syscall mkdir
npm ERR! path /usr/local/lib/node_modules/xs-dev
npm ERR! errno -13
npm ERR! Error: EACCES: permission denied, mkdir '/usr/local/lib/node_modules/xs-dev'
npm ERR!  [Error: EACCES: permission denied, mkdir '/usr/local/lib/node_modules/xs-dev'] {
npm ERR!   errno: -13,
npm ERR!   code: 'EACCES',
npm ERR!   syscall: 'mkdir',
npm ERR!   path: '/usr/local/lib/node_modules/xs-dev'
npm ERR! }
npm ERR! 
npm ERR! The operation was rejected by your operating system.
npm ERR! It is likely you do not have the permissions to access this file as the current user
npm ERR! 
npm ERR! If you believe this might be a permissions issue, please double-check the
npm ERR! permissions of the file and its containing directories, or try running
npm ERR! the command again as root/Administrator.

npm ERR! A complete log of this run can be found in:
npm ERR!     /Users/peterhoddie/.npm/_logs/2022-11-22T16_41_21_507Z-debug-0.log

The installation succeeds if prefixed with sudo .

On macOS, xs-dev puts all its downloaded directories and generated files (xs-dev-export.sh) in ~/.local/share/. After installing support for several devices, that results in many directories owned by xs-dev (moddable, esp32, esp, pico, etc). Since ~/.local/share/ is of a shared location, maybe instead there should be a single xs-dev directory that contains all of the downloaded directories and generated files?

It might not matter for functionality but it would be much more clean when browsing the file system which pieces are managed by xs-dev.

xs-setup installs the ESP32 tools to ~/.local/share/ but the ESP8266 tools are at ~/esp. Using the ESP_BASE environment variable it looks possible to move the ESP tools to ~/.local/share/ as well.

The ESP_BASE name is a little too generic as it was added when there was only ESP8266. It isn't much used. What do you think about renaming it to ESP8266_BASE or even MODDABLE_ESP8266_BASE?

Take advantage of the brew bundle command to define and install all external package dependencies. This makes it easier to maintain and hands off the responsibility of conditionally installing those packages to Homebrew. We could also make Homebrew the default installer for Mac, Linux, and Window WSL, but that's a different discussion.

For apt-get installation, we could use a similar apt-requirements.txt file.

For Windows folks not using WSL, there is a potential solution to explore with winget scripting.

After a successfully xs-dev setup, executing...

xs-dev teardown

...leaves the xsbug symlink the Applications folder. When reinstalling, that results in a warnings:

ℹ xsbug.app symlink already exists

The new Pico scan feature is great:

✔ Found the following available devices!
  Port                         Device                      Features                                                                   
  /dev/cu.usbserial-01711461   ESP32-D0WDQ6 (revision 1)   WiFi, BT, Dual Core, 240MHz, VRef calibration in efuse, Coding Scheme None 
  /dev/cu.usbserial-0001       ESP8266EX                   WiFi                                                                       
  /dev/cu.usbmodem14101        pico                        USB stdin / stdout                                                         

However, the scan fails if the Pico is in BOOTSEL mode. That's because there is no serial port available in BOOTSEL mode. so picotool is never run. picotool does find the device in both BOOTSEL and application modes.

xs-dev scan only runs picotool if the serial port is present, which means the device is in application mode. It looks like we can force the device into application mode:

picotool reboot -f -a

This would need to be done before getting the list of serial ports. Perhaps there would need to be a short delay after reboot to allow the serial port to appear. After that, it looks like picotool info can be used with a --address option to scan one particular device (assuming you can get the USB device address from the Serial scan).

Following through on the remote dependency "docs" and existing manifest includes management for Moddable modules, this is a proposal for managing third-party modules using npm and associated package.json as the standard shared by the JS ecosystem.

There are various levels at which this could work.

1. The npm CLI would be used directly to install dependencies for a project and xs-dev would handle configuring the manifest.json includes to point to the modules' manifest:

npm install dtex/j5e // adds j5e to package.json, pulled from GitHub
xs-dev include j5e/lib/led // adds the path to the j5e led dependency under node_modules

This takes advantage of the npm feature for installing modules from git, forgoing the need for module authors to publish their code to the central npm registry.

2. xs-dev provides a command to install dependencies using npm under the hood (see the gluegun packageManager module), and sets up an alias for the path to that module (see docs for the build field in the manifest):

xs-dev install dtex/j5e // adds j5e to package.json, creates J5E env variable for the path under node_modules under the "build" field in the manifest
xs-dev include j5e/lib/led // adds the path to the j5e led dependency from the J5E env variable

It is expected for folks using xs-dev to have npm (or some Node package manager) installed since that is the easiest way to get xs-dev at the moment. It is nice to collect the package management behavior under the single tool, just as xs-dev provides a command over CLIs like mcconfig. This is simply automating what could be done by hand, if needed. The gluegun feature will also detect yarn usage if that is preferred by the environment as well.

When working on a project with third-party modules, the xs-dev install command without any arguments will act the same as npm install and fetch any missing dependencies.

After a set-up of the Moddable SDK on macOS, but without having set-up the ESP32 tools, the failure message isn't helpful in guiding the user to the solution.

$ xs-dev run --example helloworld --device=esp32
⠸ Building and deploying project ~/.local/share/moddable/examples/helloworld on esp32

/opt/homebrew/lib/node_modules/xs-dev/node_modules/gluegun/build/index.js:15
    throw up;
    ^

Error: Command failed with exit code 2: mcconfig -m -p esp32 -t all -o ~/.local/share/moddable/build -d
fatal: not a git repository (or any of the parent directories): .git
~/.local/share/moddable/build/tmp/esp32/debug/helloworld/makefile:152: Could not detect ESP-IDF version.

Something I've been doing recently with the company I cofounded, is creating a common core application that runs our primary code. It handles initial server setup, ble wifi credential setting (improv-wifi spec), barebones/basic MQTT connection to home assistant, ota updates, etc. I wonder if it doesn't potentially make sense to create a stripped-down version to offer users to build from in xs-dev, that gives them some core features pre-programmed, an interface to interact with, and some compartments/mod options to extend the core app?

I imagine there are some common use cases (wifi onboarding was one for us) that pop up, some devs might like a framework with prebuilt features to start from.

The intriguing "add a remote dependency" feature has been pending in xs-dev for a while. We've recently done work in the Moddable SDK that may be helpful here. The feature is deployed in the current top of tree of the Moddable SDK.

Moddable SDK manifests can now include Git repositories directly. Both mcconfig and mcrun will automatically clone the repositories. The manifest document has the details.

Here's a contrived example:

{
	"include": [
		{
			"git": "https://github.com/phoddie/soundstream",
			"include": "./resource-stream/manifest.json"
		}
	]
}

Fair warning: this feature is new so it might not be perfect and the design might need refinement. Your feedback is welcome.

More testing on a (nearly) fresh install of macOS (Monterey, 12.3.1). If brew isn't installed, the ESP32 install fails when it tries to brew install python. At a minimum, it should probably display a more proactive error message so the user knows what to do.

$ xs-dev setup --device=esp32  
ℹ Ensuring esp32 install directory
✔ Cloning esp-idf repo
⠋ Installing build dependencies/usr/local/lib/node_modules/xs-dev/node_modules/gluegun/build/index.js:15
    throw up;
    ^

Error: Command failed with ENOENT: brew install python
spawn brew ENOENT
    at ChildProcess._handle.onexit (node:internal/child_process:283:19)
    at onErrorNT (node:internal/child_process:476:16)
    at process.processTicksAndRejections (node:internal/process/task_queues:82:21) {
  errno: -2,
  code: 'ENOENT',
  syscall: 'spawn brew',
  path: 'brew',
  spawnargs: [ 'install', 'python' ],
  originalMessage: 'spawn brew ENOENT',
  shortMessage: 'Command failed with ENOENT: brew install python\nspawn brew ENOENT',
  command: 'brew install python',
  escapedCommand: 'brew install python',
  exitCode: undefined,
  signal: undefined,
  signalDescription: undefined,
  stdout: '',
  stderr: '',
  failed: true,
  timedOut: false,
  isCanceled: false,
  killed: false
}

Node.js v18.12.1

After installing Homebrew, re-running xs-dev setup --device=esp32 succeeds.

See #32 (comment) and #32 (comment) for more details.

Example usage:

xs-dev init my-mqtt --example mqttbasic
Generating Moddable project "my-mqtt" from mqttbasic example

Allow for selecting example project base:

xs-dev init my-project --example

Display subset of matching examples if direct match is not found:

xs-dev init my-project --example http

While it is not very likely that users will run commands that rely on xs-dev-exports.sh existing before upserting config into it, since it is created during the default setup command, it would be safer to ensure the file exists before any potential write.

This could be done similar to the moddableExists check, at the top of any setup module. Or have it abstracted away through an updateExports helper that wraps the upsert helper and ensures the file exists, which would probably be more maintainable for new folks contributing to the codebase.

I usually run the eslint and prettier checks locally before publishing new package versions, but now that there are new contributors, it would be helpful to add an Action to run those checks on pull requests before allowing merges into the main branch.

It looks like Pico needs to be added to the teardown code.

While working on some separate improvements to this project, I was motivated to do some research into automating the xcode command line tools installation within xs-dev. This led me to the xcode-install gem, which provides a pretty clear process for downloading, extracting, and enabling xcode tooling on behalf of the user.

This is not an immediate need since Apple provides decent documentation and processes themselves for getting xcode; however, it would be an interesting improvement for the first-time user experience or fresh install environment.

The Moddable SDK uses the fontbm tool to allow projects to incorporate TrueType / OpenType fonts at arbitrary sizes. The fontbm tool is installed separately from the Moddable SDK and made available by the $FONTBM environment variable.

As an experiment to learn a bit more about xs-dev. I implemented a naive fontbm command to install fontbm on macOS. Despite limited experience with Node and this being my first encounter with Gluegun, it was reasonably straightforward to implement.

A few questions:

• Should this be its own command or an option for setup? It isn't a development platform so setup seemed a bit awkward
• If it is a separate command, is it reasonable to put the Mac, Windows, and Linux support all in a single source file?
• Is there a better way to check that FreeType is installed other than the hardcoded path used here? (It feels fragile)
• What other obvious things did I get wrong? ;)

When running network examples on a device, it is necessary to set the Wi-Fi credentials. This is most commonly done by passing the credentials on the command line arguments:

cd $MODDABLE/examples/network/http/httpget
mcconfig -d -m -p esp32 ssid=MyWiFi password="a secret"

This adds ssid and password to the properties of the mc/config module, which is automatically used at start-up to connect to Wi-Fi.

xs-dev does not currently use such command line arguments, nor does it generate an error when they are present:

xs-dev run --example network/http/httpsget ssid=MyWiFi password="a secret"

This feature is used in other places in the Moddable SDK, for example to configure the touch driver. It is also used by projects to change their behavior. It would be great to see it supported in xs-dev as well, using the same syntax if practical.

When installing / updating xs-dev there are some warnings that appear to be related to npm / Node. These aren't causing failures, but looking at this from the perspective of someone approaching this for the first time the warnings could be misinterpreted as problems that need to be correctly before proceeding, particularly the critical vulnerabilities.

npm update -g xs-dev
...
7 vulnerabilities (4 low, 1 moderate, 2 critical)
...

⠋ (node:20700) ExperimentalWarning: The Fetch API is an experimental feature. This feature could change at any time
(Use `node --trace-warnings ...` to show where the warning was created)

From testing on a (nearly) fresh macOS install (Monterey, 12.3.1), the esp8266 device setup fails when downloading the tools from moddable.com. FWIW – moddable.com uses LetsEncrypt for TLS certificates which can lead to unusual certificate configurations. The tool download URL does work in Safari on the same computer, so it seems valid.

xs-dev setup --device=esp8266
ℹ Ensuring esp directory
⠧ Downloading xtensa toolchain/usr/local/lib/node_modules/xs-dev/node_modules/gluegun/build/index.js:15
    throw up;
    ^

<ref *1> Error: write EPROTO 8045F00901000000:error:0A000172:SSL routines:tls12_check_peer_sigalg:wrong signature type:../deps/openssl/openssl/ssl/t1_lib.c:1572:

    at WriteWrap.onWriteComplete [as oncomplete] (node:internal/stream_base_commons:94:16) {
  errno: -100,
  code: 'EPROTO',
  syscall: 'write',
  config: {
    transitional: {
      silentJSONParsing: true,
      forcedJSONParsing: true,
      clarifyTimeoutError: false
    },
    adapter: [Function: httpAdapter],
    transformRequest: [ [Function: transformRequest] ],
    transformResponse: [ [Function: transformResponse] ],
    timeout: 0,
    xsrfCookieName: 'XSRF-TOKEN',
    xsrfHeaderName: 'X-XSRF-TOKEN',
    maxContentLength: -1,
    maxBodyLength: -1,
    validateStatus: [Function: validateStatus],
    headers: {
      Accept: 'application/json, text/plain, */*',
      'User-Agent': 'axios/0.24.0'
    },
    responseType: 'stream',
    method: 'get',
    url: 'https://www.moddable.com/private/esp8266.toolchain.darwin.tgz',
    data: undefined
  },
  request: <ref *4> Writable {
    _writableState: WritableState {
      objectMode: false,
      highWaterMark: 16384,
      finalCalled: false,
      needDrain: false,
      ending: false,
      ended: false,
      finished: false,
      destroyed: false,
      decodeStrings: true,
      defaultEncoding: 'utf8',
      length: 0,
      writing: false,
      corked: 0,
      sync: true,
      bufferProcessing: false,
      onwrite: [Function: bound onwrite],
      writecb: null,
      writelen: 0,
      afterWriteTickInfo: null,
      buffered: [],
      bufferedIndex: 0,
      allBuffers: true,
      allNoop: true,
      pendingcb: 0,
      constructed: true,
      prefinished: false,
      errorEmitted: false,
      emitClose: true,
      autoDestroy: true,
      errored: null,
      closed: false,
      closeEmitted: false,
      [Symbol(kOnFinished)]: []
    },
    _events: [Object: null prototype] {
      response: [Function: handleResponse],
      error: [Function: handleRequestError]
    },
    _eventsCount: 2,
    _maxListeners: undefined,
    _options: {
      maxRedirects: 21,
      maxBodyLength: 10485760,
      protocol: 'https:',
      path: '/private/esp8266.toolchain.darwin.tgz',
      method: 'GET',
      headers: {
        Accept: 'application/json, text/plain, */*',
        'User-Agent': 'axios/0.24.0'
      },
      agent: undefined,
      agents: { http: undefined, https: undefined },
      auth: undefined,
      hostname: 'www.moddable.com',
      port: null,
      nativeProtocols: {
        'http:': {
          _connectionListener: [Function: connectionListener],
          METHODS: [
            'ACL',         'BIND',       'CHECKOUT',
            'CONNECT',     'COPY',       'DELETE',
            'GET',         'HEAD',       'LINK',
            'LOCK',        'M-SEARCH',   'MERGE',
            'MKACTIVITY',  'MKCALENDAR', 'MKCOL',
            'MOVE',        'NOTIFY',     'OPTIONS',
            'PATCH',       'POST',       'PROPFIND',
            'PROPPATCH',   'PURGE',      'PUT',
            'REBIND',      'REPORT',     'SEARCH',
            'SOURCE',      'SUBSCRIBE',  'TRACE',
            'UNBIND',      'UNLINK',     'UNLOCK',
            'UNSUBSCRIBE'
          ],
          STATUS_CODES: {
            '100': 'Continue',
            '101': 'Switching Protocols',
            '102': 'Processing',
            '103': 'Early Hints',
            '200': 'OK',
            '201': 'Created',
            '202': 'Accepted',
            '203': 'Non-Authoritative Information',
            '204': 'No Content',
            '205': 'Reset Content',
            '206': 'Partial Content',
            '207': 'Multi-Status',
            '208': 'Already Reported',
            '226': 'IM Used',
            '300': 'Multiple Choices',
            '301': 'Moved Permanently',
            '302': 'Found',
            '303': 'See Other',
            '304': 'Not Modified',
            '305': 'Use Proxy',
            '307': 'Temporary Redirect',
            '308': 'Permanent Redirect',
            '400': 'Bad Request',
            '401': 'Unauthorized',
            '402': 'Payment Required',
            '403': 'Forbidden',
            '404': 'Not Found',
            '405': 'Method Not Allowed',
            '406': 'Not Acceptable',
            '407': 'Proxy Authentication Required',
            '408': 'Request Timeout',
            '409': 'Conflict',
            '410': 'Gone',
            '411': 'Length Required',
            '412': 'Precondition Failed',
            '413': 'Payload Too Large',
            '414': 'URI Too Long',
            '415': 'Unsupported Media Type',
            '416': 'Range Not Satisfiable',
            '417': 'Expectation Failed',
            '418': "I'm a Teapot",
            '421': 'Misdirected Request',
            '422': 'Unprocessable Entity',
            '423': 'Locked',
            '424': 'Failed Dependency',
            '425': 'Too Early',
            '426': 'Upgrade Required',
            '428': 'Precondition Required',
            '429': 'Too Many Requests',
            '431': 'Request Header Fields Too Large',
            '451': 'Unavailable For Legal Reasons',
            '500': 'Internal Server Error',
            '501': 'Not Implemented',
            '502': 'Bad Gateway',
            '503': 'Service Unavailable',
            '504': 'Gateway Timeout',
            '505': 'HTTP Version Not Supported',
            '506': 'Variant Also Negotiates',
            '507': 'Insufficient Storage',
            '508': 'Loop Detected',
            '509': 'Bandwidth Limit Exceeded',
            '510': 'Not Extended',
            '511': 'Network Authentication Required'
          },
          Agent: [Function: Agent] { defaultMaxSockets: Infinity },
          ClientRequest: [Function: ClientRequest],
          IncomingMessage: [Function: IncomingMessage],
          OutgoingMessage: [Function: OutgoingMessage],
          Server: [Function: Server],
          ServerResponse: [Function: ServerResponse],
          createServer: [Function: createServer],
          validateHeaderName: [Function: __node_internal_],
          validateHeaderValue: [Function: __node_internal_],
          get: [Function: get],
          request: [Function: request],
          setMaxIdleHTTPParsers: [Function: setMaxIdleHTTPParsers],
          maxHeaderSize: [Getter],
          globalAgent: [Getter/Setter]
        },
        'https:': {
          Agent: [Function: Agent],
          globalAgent: Agent {
            _events: [Object: null prototype],
            _eventsCount: 2,
            _maxListeners: undefined,
            defaultPort: 443,
            protocol: 'https:',
            options: [Object: null prototype],
            requests: [Object: null prototype] {},
            sockets: [Object: null prototype],
            freeSockets: [Object: null prototype] {},
            keepAliveMsecs: 1000,
            keepAlive: false,
            maxSockets: Infinity,
            maxFreeSockets: 256,
            scheduling: 'lifo',
            maxTotalSockets: Infinity,
            totalSocketCount: 1,
            maxCachedSessions: 100,
            _sessionCache: [Object],
            [Symbol(kCapture)]: false
          },
          Server: [Function: Server],
          createServer: [Function: createServer],
          get: [Function: get],
          request: [Function: request]
        }
      },
      pathname: '/private/esp8266.toolchain.darwin.tgz'
    },
    _ended: true,
    _ending: true,
    _redirectCount: 0,
    _redirects: [],
    _requestBodyLength: 0,
    _requestBodyBuffers: [],
    _onNativeResponse: [Function (anonymous)],
    _currentRequest: <ref *2> ClientRequest {
      _events: [Object: null prototype] {
        response: [Function: bound onceWrapper] {
          listener: [Function (anonymous)]
        },
        abort: [Function (anonymous)],
        aborted: [Function (anonymous)],
        connect: [Function (anonymous)],
        error: [Function (anonymous)],
        socket: [Function (anonymous)],
        timeout: [Function (anonymous)]
      },
      _eventsCount: 7,
      _maxListeners: undefined,
      outputData: [],
      outputSize: 0,
      writable: true,
      destroyed: false,
      _last: true,
      chunkedEncoding: false,
      shouldKeepAlive: false,
      maxRequestsOnConnectionReached: false,
      _defaultKeepAlive: true,
      useChunkedEncodingByDefault: false,
      sendDate: false,
      _removedConnection: false,
      _removedContLen: false,
      _removedTE: false,
      strictContentLength: false,
      _contentLength: 0,
      _hasBody: true,
      _trailer: '',
      finished: true,
      _headerSent: true,
      _closed: false,
      socket: <ref *3> TLSSocket {
        _tlsOptions: {
          allowHalfOpen: undefined,
          pipe: false,
          secureContext: SecureContext { context: SecureContext {} },
          isServer: false,
          requestCert: true,
          rejectUnauthorized: true,
          session: undefined,
          ALPNProtocols: undefined,
          requestOCSP: undefined,
          enableTrace: undefined,
          pskCallback: undefined,
          highWaterMark: undefined,
          onread: undefined,
          signal: undefined
        },
        _secureEstablished: false,
        _securePending: false,
        _newSessionPending: false,
        _controlReleased: true,
        secureConnecting: true,
        _SNICallback: null,
        servername: null,
        alpnProtocol: null,
        authorized: false,
        authorizationError: null,
        encrypted: true,
        _events: [Object: null prototype] {
          close: [
            [Function: onSocketCloseDestroySSL],
            [Function],
            [Function: onClose],
            [Function: socketCloseListener]
          ],
          end: [ [Function: onConnectEnd], [Function: onReadableStreamEnd] ],
          newListener: [Function: keylogNewListener],
          secure: [Function: onConnectSecure],
          session: [Function (anonymous)],
          free: [Function: onFree],
          timeout: [Function: onTimeout],
          agentRemove: [Function: onRemove],
          error: [Function: socketErrorListener],
          drain: [Function: ondrain]
        },
        _eventsCount: 10,
        connecting: false,
        _hadError: true,
        _parent: null,
        _host: 'www.moddable.com',
        _closeAfterHandlingError: false,
        _readableState: ReadableState {
          objectMode: false,
          highWaterMark: 16384,
          buffer: BufferList { head: null, tail: null, length: 0 },
          length: 0,
          pipes: [],
          flowing: true,
          ended: false,
          endEmitted: false,
          reading: true,
          constructed: true,
          sync: false,
          needReadable: true,
          emittedReadable: false,
          readableListening: false,
          resumeScheduled: false,
          errorEmitted: true,
          emitClose: false,
          autoDestroy: true,
          destroyed: true,
          errored: [Circular *1],
          closed: true,
          closeEmitted: true,
          defaultEncoding: 'utf8',
          awaitDrainWriters: null,
          multiAwaitDrain: false,
          readingMore: false,
          dataEmitted: false,
          decoder: null,
          encoding: null,
          [Symbol(kPaused)]: false
        },
        _maxListeners: undefined,
        _writableState: WritableState {
          objectMode: false,
          highWaterMark: 16384,
          finalCalled: false,
          needDrain: false,
          ending: false,
          ended: false,
          finished: false,
          destroyed: true,
          decodeStrings: false,
          defaultEncoding: 'utf8',
          length: 0,
          writing: false,
          corked: 0,
          sync: false,
          bufferProcessing: false,
          onwrite: [Function: bound onwrite],
          writecb: null,
          writelen: 0,
          afterWriteTickInfo: null,
          buffered: [],
          bufferedIndex: 0,
          allBuffers: true,
          allNoop: true,
          pendingcb: 0,
          constructed: true,
          prefinished: false,
          errorEmitted: true,
          emitClose: false,
          autoDestroy: true,
          errored: [Circular *1],
          closed: true,
          closeEmitted: true,
          [Symbol(kOnFinished)]: []
        },
        allowHalfOpen: false,
        _sockname: null,
        _pendingData: null,
        _pendingEncoding: '',
        server: undefined,
        _server: null,
        ssl: null,
        _requestCert: true,
        _rejectUnauthorized: true,
        parser: null,
        _httpMessage: [Circular *2],
        [Symbol(res)]: TLSWrap {
          _parent: TCP {
            reading: [Getter/Setter],
            onconnection: null,
            [Symbol(owner_symbol)]: [Circular *3],
            [Symbol(handle_onclose)]: [Function: done]
          },
          _parentWrap: undefined,
          _secureContext: SecureContext { context: SecureContext {} },
          reading: true,
          onkeylog: [Function: onkeylog],
          onhandshakestart: {},
          onhandshakedone: [Function (anonymous)],
          onocspresponse: [Function: onocspresponse],
          onnewsession: [Function: onnewsessionclient],
          onerror: [Function: onerror],
          [Symbol(owner_symbol)]: [Circular *3]
        },
        [Symbol(verified)]: false,
        [Symbol(pendingSession)]: null,
        [Symbol(async_id_symbol)]: 26,
        [Symbol(kHandle)]: null,
        [Symbol(lastWriteQueueSize)]: 166,
        [Symbol(timeout)]: null,
        [Symbol(kBuffer)]: null,
        [Symbol(kBufferCb)]: null,
        [Symbol(kBufferGen)]: null,
        [Symbol(kCapture)]: false,
        [Symbol(kSetNoDelay)]: false,
        [Symbol(kSetKeepAlive)]: false,
        [Symbol(kSetKeepAliveInitialDelay)]: 0,
        [Symbol(kBytesRead)]: 0,
        [Symbol(kBytesWritten)]: 166,
        [Symbol(connect-options)]: {
          rejectUnauthorized: true,
          ciphers: 'TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256:TLS_AES_128_GCM_SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA256:ECDHE-RSA-AES256-SHA384:DHE-RSA-AES256-SHA384:ECDHE-RSA-AES256-SHA256:DHE-RSA-AES256-SHA256:HIGH:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!SRP:!CAMELLIA',
          checkServerIdentity: [Function: checkServerIdentity],
          minDHSize: 1024,
          maxRedirects: 21,
          maxBodyLength: 10485760,
          protocol: 'https:',
          path: null,
          method: 'GET',
          headers: {
            Accept: 'application/json, text/plain, */*',
            'User-Agent': 'axios/0.24.0'
          },
          agent: undefined,
          agents: { http: undefined, https: undefined },
          auth: undefined,
          hostname: 'www.moddable.com',
          port: 443,
          nativeProtocols: { 'http:': [Object], 'https:': [Object] },
          pathname: '/private/esp8266.toolchain.darwin.tgz',
          _defaultAgent: Agent {
            _events: [Object: null prototype],
            _eventsCount: 2,
            _maxListeners: undefined,
            defaultPort: 443,
            protocol: 'https:',
            options: [Object: null prototype],
            requests: [Object: null prototype] {},
            sockets: [Object: null prototype],
            freeSockets: [Object: null prototype] {},
            keepAliveMsecs: 1000,
            keepAlive: false,
            maxSockets: Infinity,
            maxFreeSockets: 256,
            scheduling: 'lifo',
            maxTotalSockets: Infinity,
            totalSocketCount: 1,
            maxCachedSessions: 100,
            _sessionCache: [Object],
            [Symbol(kCapture)]: false
          },
          host: 'www.moddable.com',
          noDelay: true,
          servername: 'www.moddable.com',
          _agentKey: 'www.moddable.com:443:::::::::::::::::::::',
          encoding: null,
          singleUse: true
        }
      },
      _header: 'GET /private/esp8266.toolchain.darwin.tgz HTTP/1.1\r\n' +
        'Accept: application/json, text/plain, */*\r\n' +
        'User-Agent: axios/0.24.0\r\n' +
        'Host: www.moddable.com\r\n' +
        'Connection: close\r\n' +
        '\r\n',
      _keepAliveTimeout: 0,
      _onPendingData: [Function: nop],
      agent: Agent {
        _events: [Object: null prototype] {
          free: [Function (anonymous)],
          newListener: [Function: maybeEnableKeylog]
        },
        _eventsCount: 2,
        _maxListeners: undefined,
        defaultPort: 443,
        protocol: 'https:',
        options: [Object: null prototype] { noDelay: true, path: null },
        requests: [Object: null prototype] {},
        sockets: [Object: null prototype] {
          'www.moddable.com:443:::::::::::::::::::::': [ [TLSSocket] ]
        },
        freeSockets: [Object: null prototype] {},
        keepAliveMsecs: 1000,
        keepAlive: false,
        maxSockets: Infinity,
        maxFreeSockets: 256,
        scheduling: 'lifo',
        maxTotalSockets: Infinity,
        totalSocketCount: 1,
        maxCachedSessions: 100,
        _sessionCache: { map: {}, list: [] },
        [Symbol(kCapture)]: false
      },
      socketPath: undefined,
      method: 'GET',
      maxHeaderSize: undefined,
      insecureHTTPParser: undefined,
      path: '/private/esp8266.toolchain.darwin.tgz',
      _ended: false,
      res: null,
      aborted: false,
      timeoutCb: null,
      upgradeOrConnect: false,
      parser: null,
      maxHeadersCount: null,
      reusedSocket: false,
      host: 'www.moddable.com',
      protocol: 'https:',
      _redirectable: [Circular *4],
      [Symbol(kCapture)]: false,
      [Symbol(kBytesWritten)]: 0,
      [Symbol(kEndCalled)]: true,
      [Symbol(kNeedDrain)]: false,
      [Symbol(corked)]: 0,
      [Symbol(kOutHeaders)]: [Object: null prototype] {
        accept: [ 'Accept', 'application/json, text/plain, */*' ],
        'user-agent': [ 'User-Agent', 'axios/0.24.0' ],
        host: [ 'Host', 'www.moddable.com' ]
      },
      [Symbol(kUniqueHeaders)]: null
    },
    _currentUrl: 'https://www.moddable.com/private/esp8266.toolchain.darwin.tgz',
    [Symbol(kCapture)]: false
  },
  response: undefined,
  isAxiosError: true,
  toJSON: [Function: toJSON]
}

Node.js v18.12.1

Since one of the complexities of managing the xs-dev environment is making sure certain environment variables are sourced in the correct shell profile, maybe per-project configuration can use a conventional .env file to load into the process before certain commands like xs-dev run.

Along with existing configuration variables like UPLOAD_PORT, there could be a DEPLOY_TARGET or DEPLOY_DEVICE setting for the default --device (or platform target) for running projects.

So, hear me out. With a common host (mcconfig -m -d ), you can already load additional modules via mccrun as .xsa (xs archive format). I assume it bundles all the js files in the mod into the xsa.

A low brow hot reloader might:

1. In a separate process from xs-dev run, watch for file changes
2. Run the mccrun command (or leverage its same mechanism to push to the device) to add/override existing .xsa code
3. Reload the device

If it didn't kill the original process, that'd be great. If it supports over the wire pushing instead of serial, that'd also be great.

A high brow hot reloader might just break apart code into meaningful xsa for singular files, so we only need to push bytecode changes and not entire projects/mods

I know there's a xsdebug port that gets opened and is used to communicate back and forth for xsdebug as well as mccrun (might need to double check they're the same port).

The xs-dev repository readme.md is great: it quickly and clearly explains everything you need to know.

A few suggestions:

• Now that Pico support is merged, explain how to use that similar to esp8266, esp32, and wasm targets
• In addition to explaining how to install xs-dev, explain how to update to the latest version
• "Run an example (coming soon)" - running an example by name works, so "(coming soon)" can be removed

Based on recent user feedback, it's tough to know where an issue lies when using xs-dev to run a project after setting up for a platform. Even the run command allows choosing a device/platform that may not have been set up yet. That command can be "smarter" or more defensive ensuring known environment requirements before executing mcconfig.

Alongside this update to run, a "doctor" command (similar to brew doctor) could display helpful output about currently supported platforms, including dependency locations.

The xs-dev tools to manage projects and their manifests helpfully automate what are currently manual operations. In exploring it, I have a few observations to share:

• The list of available modules displayed with xs-dev include is really great.
• Entering xs-dev include http brings up the module picker, which makes sense because the correct name is network/http. It could be helpful if the picker filter was pre-populated with "http" to accelerate the search
• Using xs-dev include network/http twice results in the module being included twice. This is generally safe but definitely results in more confusing manifest.
• xs-dev include fails if the include property in the manifest is a string (as in $MODDABLE/examples/helloworld/manifest.json). mcconfig treats that special case as a single element array.
• xs-dev include fails there is no include property. That is rare, but is allowed.
• Sometimes you want to add an include to a specific platform rather than all targets. That is done using the platforms section of the manifest. It would be cool if xs-dev include network/http --device=pico could do that too.
• The message output by xs-dev remove network/http is not quite right: "Removing "http" to manifest includes". It should be "from" instead of "to".
• If the name given doesn't exist, xs-dev remove NO_SUCH_MODULE no error is reported, so typing errors will go unnoticed
• xs-dev remove http removes "network/http". That's either a bug or a feature. It might have unintended side-effects, especially if the module named is short like ("h"). Maybe it should fail and output the possible matches, or output the include it did remove, so it can be quickly noticed and fixed.
• xs-dev init foo outputs Run the project using: xs-dev run. That doesn't work without first doing a cd to the new directory.
• Running xs-dev init foo where foo already exists overwrites the files that are there. That could be destructive of changes the developer has made.
• The xs-dev init foo --typescript option is great, and the possibility to have other templates is intriguing. Maybe they should be clones of examples? The default JavaScript project already is helloworld. We could have a hello TypeScript project too, and others as needed. I often want to start from a network example (mqttbasic) and modify from there, so almost any example might be a starting point for init.

When building with device target esp8266 the output message uses esp. The inconsistency is potentially confusing.

$ xs-dev run --example helloworld --device=esp8266
⠋ Building and deploying project ~/.local/share/moddable/examples/helloworld on esp

This may not be something to solve in XS-Dev, but I wanted to put it out there.

By default, Top Level Await no longer works in XS entry files. It's only available in imported modules (This decision was made for performance reasons).

J5e examples assume TLA is available, and I'd like to keep the examples as simple as possible. No otherwise unnecessary async wrapper functions or Async IIFE's. It is possible to enable TLA everywhere by setting main_async: 1 under 'defines' in manifest.json.

Here's an example J5e manifest.json with main_async: 1:

{
	"include": [
		"$(MODDABLE)/modules/io/manifest.json",
                "$(j5e)/lib/button/manifest.json",
                "$(j5e)/lib/led/manifest.json"
	],
	"modules": {
		"*": "./main"
	},
	"defines": {
		"main_async": 1
	}
}

So, is it possible for xs-dev to help with this setting?

Maybe as a flag for init xs-dev init my-project io main-async?

I'm happy to work on a PR, but wanted to run it by your first. Is this a welcome addition, and if so do you have anything preferable to an init flag?

Making an explicit issue for this so people viewing the project know its on the roadmap. I don't have a development environment for running Windows, either natively or in a VM, so community help is required.

Open questions:

1. is there a popular package manager upon which xs-dev can depend, i.e. brew for Macos or apt for Debian linux? Some available options from basic research: chocolatey, scoop, winget

While looking into the Moddable install instructions for Windows, it appears more of the dependencies can be installed from Chocolatey than any of the other options.

3. how to manage setting up environment variables automatically, given there is no shell profile like unix systems?

4. is it possible to automate the espressif tooling without using the esp-idf windows installer? Or does xs-dev just provide some help downloading the installer and checking for those dependencies before continuing?

5. how does xs-dev treat WSL?

Hi!
I issued a xs-dev setup on my Raspberry Pi. This ended in with the following error message:

ℹ Moddable repo already installed
⠋ Installing simulator/usr/lib/node_modules/xs-dev/node_modules/gluegun/build/index.js:15
    throw up;
    ^

Error: Command failed with exit code 126: mcconfig -m -p x-lin /home/pi/.local/share/moddable/tools/xsbug/manifest.json
/home/pi/.local/share/moddable/build/bin/lin/release/mcconfig: line 3: /home/pi/.local/share/moddable/build/bin/lin/release/tools: cannot execute binary file: Exec format error
    at makeError (/usr/lib/node_modules/xs-dev/node_modules/execa/lib/error.js:60:11)
    at handlePromise (/usr/lib/node_modules/xs-dev/node_modules/execa/index.js:118:26)
    at processTicksAndRejections (node:internal/process/task_queues:96:5) {
  shortMessage: 'Command failed with exit code 126: mcconfig -m -p x-lin /home/pi/.local/share/moddable/tools/xsbug/manifest.json',
  command: 'mcconfig -m -p x-lin /home/pi/.local/share/moddable/tools/xsbug/manifest.json',
  escapedCommand: 'mcconfig -m -p x-lin "/home/pi/.local/share/moddable/tools/xsbug/manifest.json"',
  exitCode: 126,
  signal: undefined,
  signalDescription: undefined,
  stdout: '',
  stderr: '/home/pi/.local/share/moddable/build/bin/lin/release/mcconfig: line 3: /home/pi/.local/share/moddable/build/bin/lin/release/tools: cannot execute binary file: Exec format error',
  failed: true,
  timedOut: false,
  isCanceled: false,
  killed: false
}

Walking manually through the described setup procedure does not create an error.

Just like the --list-examples flag for the run command, any command that allows passing a --device flag should allow a --list-devices picker:

xs-dev run --list-devices

This 👆 should output possible targets, i.e not mac, lin, or win since those are the default targets based on the user's environment.

Question: what happens when the --list-examples and --list-devices flag are both passed?

Question: how many target devices should be shown? All the "root" platforms (esp8266, ESP32, etc) and the sub-platforms (moddable_one)?

Installing can be interrupted, so xs-dev should recover gracefully to avoid leaving the installation in a bad state. Installs are even somewhat likely to be interrupted because some of them take a while to execute (lots to download, for example).

I experimented with this on macOS by hitting Control-C to interrupt the Moddable SDK and ESP8266 setup. If the Moddable SDK setup is interrupted, it seems to always leave behind a /moddable directory. which is enough to cause later attempt to xs-dev setup to fail to complete a real setup. The ESP8266 seems to recover better, but it does leave a partial installation.

For setup, maybe it is best to remove all the bits of the setup fails for some reason? That won't work for update though. I'm not sure what best practices are here.

The good news is that xs-dev teardown does get back to a safe state so setup can proceed again. But, that's only an option once the developer understands that they have a partial install that needs to be cleaned up before proceeding.

Thank you for adding the instructions to the docs on how to update xs-dev. I've been using "npm uninstall..." and then reinstalling.

Using npm update fails on my Mac with the following:

$ npm update -g xs-dev@latest
npm ERR! code EACCES
npm ERR! syscall rename
npm ERR! path /usr/local/lib/node_modules/npm
npm ERR! dest /usr/local/lib/node_modules/.npm-i9nnxROI
npm ERR! errno -13
npm ERR! Error: EACCES: permission denied, rename '/usr/local/lib/node_modules/npm' -> '/usr/local/lib/node_modules/.npm-i9nnxROI'
npm ERR!  [Error: EACCES: permission denied, rename '/usr/local/lib/node_modules/npm' -> '/usr/local/lib/node_modules/.npm-i9nnxROI'] {
npm ERR!   errno: -13,
npm ERR!   code: 'EACCES',
npm ERR!   syscall: 'rename',
npm ERR!   path: '/usr/local/lib/node_modules/npm',
npm ERR!   dest: '/usr/local/lib/node_modules/.npm-i9nnxROI'
npm ERR! }
npm ERR! 
npm ERR! The operation was rejected by your operating system.
npm ERR! It is likely you do not have the permissions to access this file as the current user
npm ERR! 
npm ERR! If you believe this might be a permissions issue, please double-check the
npm ERR! permissions of the file and its containing directories, or try running
npm ERR! the command again as root/Administrator.

Removing @latest allows the operation to complete without an error.

The new documentation site looks great!

At the top of the introduction page, there is a list of "Features:" with links. The links all appear to be broken. For example, "Moddable SDK Setup" links to https://hipsterbrown.github.io/xs-dev/en/introduction/features/setup but the correct link, from the left side-bar. is https://hipsterbrown.github.io/xs-dev/en/introduction/features/update.

I spent a little time trying this out. It is very promising. I don't know much about Node and even less about zx, but the script is still pretty straightforward to follow. That's great.

The following notes are based on a naive first impression:

• If $MODDABLE is already defined, nothing much works. I'm not sure how to solve that, but perhaps a warning at least?
• Is there a way to unsetup? Since the .profile is modified automatically, it could be nice to store it with the tool somehow.
• The clone command in the readme.md does a cd into xs-dev instead of xs-setup (git clone https://github.com/HipsterBrown/xs-setup && cd xs-dev)
• On an x86 Mac, the ESP8266 and ESP32 installs fail with:

  $ arch -arm64 brew install python; arch -arm64 brew upgrade python
  arch: Unknown architecture: arm64
  Error: arch: Unknown architecture: arm64
      at file:///Users/hoddie/Projects/xs-setup/xs-dev.mjs:174:15
      exit code: 1
  

• ./xs-dev.mjs update does a full rebuild even if nothing changed
• ./xs-dev.mjs test works but ./xs-dev.mjs run-example helloworld exits after printing:

   ./xs-dev.mjs run-example helloworld
  

I'm assuming some of this is operator error, so I put it all in one issue. If you prefer, I can break it out into separate issues.

Idea:

1. It would be nice to be able to use mcrun hot reloading for the basis of development after flashing a common host.
   I have done some custom file-watcher/nodemon like dev in the past, i am going to take a crack at it.
2. I have also talked with a colleague/maintainer of node-env/node-build about adding support for xs as a target, this could help alleviate some symptoms associated with testing in node vs xs.

On macOS, start a build with

> xs-dev run --example piu/balls

Press Control C. The terminal prompt appears, as expected, but the build continues in the background, eventually launching xsbug and the simulator.