zkat / npx Goto Github PK

There's a particular corner case with npx usage that hasn't been quite specced out to satisfaction: local directory specifiers.

note: directory in npm-package-arg lingo includes regular directories as well as any files that don't end in .tar, .tgz, or .tar.gz. I'll be using that terminology here, with specific disambiguation as necessary.

The biggest problem is that there's several different situations where these specifiers can pop up, and we need to know what to actually do with them. I'll put a table here with the current + proposed behavior for the various situations, and hopefully we can come up with something useful here.

I definitely consider the current behavior to be buggy because I think literally no one wants what npx does for this right now.

Some npm packages don't export their primary binary under the same name as the package itself. In these cases, we should assume that if a package only defines a single binary, and you do npm <pkg> without a -p option, <command> will be translated to the actual command.

For example: npx grunt-cli instead of npx -p grunt-cli grunt.

For packages with multiple binaries, where none of the binaries have the same name as the package, this should not be done at all, and a -p option should be required.

The best way to do this would be to read npm's own output and parse out the binaries linked from there. This is kinda tricky right now 'cause the output just looks like /Users/usr/.npm/_npx/bin/standard -> /Users/usr/.npm/_npx/lib/node_modules/foo/bin/cmd.js and can be mixed in with arbitrary other output.

Instead, we'll need to patch npm so its --json output for npm i includes the linked binaries somewhere. We'll also have to output our own summary report in this case, in order to simulate the usual human output.

Platform: windows 7
npx: v9.0.7
npm: v5.2.0
node: v6.9.5
shell: CMD.EXE

I have installed the npx globally, then want to try some features, but got the error: 'TypeError: rimraf.sync is not a function'.

reproduction:

in a blank directory without a package.json

> ls -a
./  ../
> npx -p cowsay -c "cowsay hi"
Command failed: /usr/local/lib/node_modules/npm/bin/npm-cli.js run env --parseable
npm ERR! path /Users/jsn/foo/package.json
npm ERR! code ENOENT
npm ERR! errno -2
npm ERR! syscall open
npm ERR! enoent ENOENT: no such file or directory, open '/Users/jsn/foo/package.json'
npm ERR! enoent This is related to npm not being able to find a file.
npm ERR! enoent

npm ERR! A complete log of this run can be found in:
npm ERR!     /Users/jsn/.npm/_logs/2017-07-11T23_20_09_939Z-debug.log

> echo {} > package.json
> ls -a
./    ../   package.json
> npx -p cowsay -c "cowsay hi"
npx: installed 1 in 2.809s
 ____
< hi >
 ----
        \   ^__^
         \  (oo)\_______
            (__)\       )\/\
                ||----w |
                ||     ||
>

note, if there is a package.json in one of the parent directories (regardless of contents as long as it's valid json), it also lets npx -c... proceed normally

expected

npx -c would shell exec a string even without a package.json present

in usecases where you know you want the latest of a specific package, or you know you don't want the local version, it would be nice to be able to do something like npx --no-global foo to force npx to use its cached (or freshly downloaded) version of a package.

npx could use a manpage, just like npm.

Thanks for maintaining all this stuff <3

At some point, after running a few npx commands, I saw this prompt:

   ╭─────────────────────────────────────╮
   │                                     │
   │   Update available 9.0.3 → 9.2.2    │
   │    Run npm i -g libnpx to update    │
   │                                     │
   ╰─────────────────────────────────────╯

(I'm not sure how to reproduce seeing that prompt - looks to be on a timer of some sort?)

I ran the suggested commands:

$ npx --version
9.0.3
$ npm i -g libnpx
+ [email protected]
added 155 packages in 3.704s
$ npx --version
9.0.3

Note the version of npx has not changed.

I am using the npx which comes with npm:

$ which npx
/Users/jess.telford/.nvm/versions/node/v7.10.0/bin/npx
$ node --version
v7.10.0
$ npm --version
5.2.0

Extra Investigation

I am able to update the version of npx with npm i -g npx, however that appears to overwrite the global npm version of npx with the new version:

$ npm i -g npx
/Users/jess.telford/.nvm/versions/node/v7.10.0/bin/npx -> /Users/jess.telford/.nvm/versions/node/v7.10.0/lib/node_modules/npx/index.js
+ [email protected]
added 507 packages in 9.423s
$ npx --version
9.3.2

I'm not sure if this has any consequences when updating npm in the future?

Hi Kat,

Thanks for releasing npx! Sorry to file a bug so quickly. =(

Windows 7 64-bit
node: 8.1.3
npm: 5.2.0
npx: 9.0.3

This probably relates to #58 but is on a different setup with a seemingly different error.

I see the same issue with two different command shells, so to keep this on a common denominator across, just use cmd.exe to run this in:

Unrelated perhaps, but this gets interesting, too, when cowsay, etc. is already installed globally.

So far, we've been framing -p as "this is the package the binary comes from".

As of d9175e8, though, this understanding became something more like "this is a package that should be put in the current environment so that <command> or its sub-processes might find it".

What that means is we can totally have multiple -p options now for folks who want to set up a command that sets up a few tools at a time. One example of this might be:

$ npx -p node-bin@6 -p npm@4 npm --versions

This change has also created another problem that needs to be solved: the "hidden" _npx directory where non-installed binaries end up can now cause hidden, unintended conflicts. That means that we'll have to remove the directory between npx runs altogether. That, or we'll have to put each binary in its own prefix, so we can update them (#11) more efficiently, instead of doing a full install every time.

stuff like npx -p pkg -c 'echo foo | pkg' will break.

Will --loglevel=silent or -s be passed to npm?

There're others: https://docs.npmjs.com/misc/config#shorthands-and-other-cli-niceties

Ref: #62 (comment)

I'm confused 'cause this should totally be the right way to escape things for child_process.exec. Note that C:\"Program Files"\... is there, which should be the way to escape paths on Windows.

I'm thinking it's possible that this sort of escaping only applies when you're dealing with the binary path itself, and arguments don't get processed the same way and thus need to be escaped as args?

Would it be possible for npx to take an npm command like install [email protected] and be able to parse it into an AST into which people could abstract the desired behavior of that command?

So install [email protected] would turn into;

{
    "command": "install",
    "module": "tap-html",
    "version": "0.0.4"
}

needed for local-npm/local-npm#154 🤕

Arguably the code could be as simple as

var string = "install [email protected]";
parse(s); 
// |
// V
// {
//     "command": "install",
//     "module": "tap-html",
//     "version": "0.0.4"
// }

var parse = function parse(s) {
    var command = s.split(' ');
    var modV = command[1].split('@');
    
    return {
        "command": command[0],
        "module": modV[0],
        "version": modV[1]
    }
}

but just thought it would be something interesting to add to npx!

right now, npx just looks for "something called npm". It would be nice to be able to specify which one to use for installations.

npx's heuristic for detecting which bin file to use fails to detect a valid shebang, and results in not finding any bin for some packages.

npx is peeking at the first few bytes of a potential bin file and comparing it exactly to '#!/usr/bin/env node\n' (

however, a valid shebang can have more whitespace, such as `'#! /usr/bin/env node \n'

One potential approach would be to read more bytes from the candidate file, and then use a looser match. This is what node does to strip shebangs when loading a file: https://github.com/nodejs/node/blob/7e3a3c962f09233c53cee7ebe381341d7c8b7162/lib/internal/module.js#L48

reproduction:
gist with space in shebang (breaks): https://gist.github.com/js-n/a192c339b652dc29c90d8be397ee3cf7
gist without space (works): https://gist.github.com/js-n/eaab62ef315754e478791ebf63c1bf67

To truly be a replacement for npm run, npx needs to work on as many node versions as possible.

Currently, use of untranspiled const and destructuring, and probably other things, means it can't work prior to node 4.

Would it be possible to babel-transpile npx, so that your language level choice isn't a barrier to using it on older node versions (including travis-ci runs on those versions)?

Long story short, if I want to execute a script in a package that has an @ in the filename (like maybe its bin scripts follow a similar naming convention to systemd unit files or something), I should be able to do npx -p like-units [email protected] and not worry that port.service might be interpreted as a tag name for the like-units package (and, of course, if I want to specify a specific version of the package, I'd do it as npx -p [email protected] the-script and not npx -p the-package [email protected]).

This also draws a guaranteed wall between the command to be run and the arguments to npx so long as the package name is explicitly defined (one that can be enforced with -- a la other command-running programs' arguments, such as ssh); ideally, I should be able to drop npx --package $mypackage@$whatever -- "$@" as an alias into one of my scripts knowing I don't need to worry about some esoteric mypackage's -1 script being misinterpreted due to the npx layer.

npx blows up with ENOENT on Win32 because it uses spawn directly with npm and on Win32, spawn doesn't run down PATH. I wrote code to fix this in the spawn-rx package but I wasn't sure if you wanted to take a dependency on it or just want to copy-paste the code:

https://github.com/paulcbetts/spawn-rx/blob/master/src/index.ts#L91

Let me know and I'll send in a PR

(Been using npx since 1.0.2. Love it.)

I often want to run node --inspect and node --inspect-brk on apps that I also like to run via npx. For example: I want to breakpoint debug a mocha test, or do a performance profile of eslint .

AFAIK, there isnt' a decent workaround, so I always go back to node --inspect-brk ./node_modules/eslint/bin/eslint.js . which is fine but obviously it'd be nice to avoid this.

Could we add --inspect-brk and --inspect as supported npx flags?

The surface area is pretty minor for things that need to be translated: options, descriptions, and a couple of error messages. It should be pretty straightforward using https://npm.im/y18n 👍

Environment: Windows 10
npx Version: 9.4.1
See also: #84

Reason:
My user home directory contain whitespace, like foo bar

Right now, if you don't have a certain package in $PATH, we always do a npm install --global into $(npm config get cache)/_npx. This means that we'll run through an actual install cycle every time, instead of just making sure we're at the latest version.

So, we should use npm update instead, which should short-circuit this.

Unfortunately, npm is kinda wonky with npm update and globals right now, so we need to fix up npm update for this use case before we can take advantage of it.

npm run-script adds a bunch of environment variables when running scripts. Ideally, running npx will enrich the environment before execution, so both child processes and -c scripts run as if they were called by run-script.

This one's probably a bit of a doozy, since I think it might require yanking out npm's config monstrosity. But hey, if it's a good motivation to do that thing we're already intending to do, hooray? :)

Since there's a much greater chance of bad actors in environments you don't directly control (i.e., not npm), I think there should be some sort of warning message or confirmation step when using a URL-based command. Just a quick sanity check to make sure people don't install something nasty.

Thoughts on this? If there's interest, I can work up a PR.

(I've migrated this issue from its original home on the nodenv issues, where I closed it, thinking this probably needs to be handled on the npx side)

Using the nodenv node version manager, I want to use $ npx create-react-app my-cool-new-app where it installs a temporary create-react-app and calls it. So I installed node 8. I did nodenv global 8.0.0. I upgraded npm to make sure I had 5.2. I then ran:

npx create-react-app real-world-discussions-demo

Output:

nodenv: create-react-app: command not found

The `create-react-app' command exists in these Node versions:
  4.2.3

Oops! I had already installed create-react-app globally (with nodenv) using 4.2.3. But wait, I'm not trying to run a global command, right? That's the point of npx. After I deleted create-react-app out of 4.2.3, I ran the command again and it worked.

So my question is, why didn't this work as expected? It should have installed a temporary create-react-app in node 8 and called it, but instead it barfed, but that's exactly what npx is trying to prevent. The original npx article mentions that npx should work well with nvm, nave, or n, but doesn't mention nodenv.

I already asked Kat about this here, and she responded saying:

npx has no explicit support for any version managers. It does some installation and $PATH-manipulation work, and that’s it

I suspect npx would need to be aware of nodenv. My understanding is that nodenv uses shims in $PATH such that the system will use it first before global stuff, whereas the other node version managers modify things on a more global level. If I understand correctly, the other ones only allow you to have one node version active at a time. I could be wrong, but that sounds like it would account for the difference that this problem would not occur with the other version managers but would for nodenv.

So my feature request is: make npx even more awesome and support nodenv! I realize nodenv is less popular (why??), so I'm a diva here asking for this special case. No rush here for me; I've been using the workarounds mentioned in the article for some time :-) But, I just wanted to say thanks for making npx, it's a welcome tool!

• npx uglifyjs input.js reports not found: uglify-js
• npx uglify-js input.js reports not found: uglify-js
• npm install uglify-js && npx uglifyjs input.js works

In other words, it seems that when the package name and the "bin" name are different, it breaks?

It'd be great if when there was only one "bin", it would quietly use it, and when there was more than one "bin", it prompted me asking which one i wanted to use :-D

One of the nice convenience features of yarn is the ability to run scripts directly via yarn script-name.

This was the first thing I tried to do with npx. I kinda assumed it was there, but it seems it's not. Would it be possible to add?

If added, npm run build or npx build would do the same thing, assuming there's a script named build.

I think that for the feature to be of use, it would have to be the default and not hidden behind a flag, otherwise one is better off doing npm run script-name anyway.

Let me know what you think. Cheers!

jfc please just give us completion support, at least for existing binaries in the $PATH or in ~/.npm/_npx/bin. There's gotta be something nice and easy the cool kids use these days to set up easy completion stuff.

Do you wanna pick this up? I'd love the help.

After running npx https://gist.github.com/zkat/4bc19503fe9e9309e2bfaa2c58074d32 I get an the command "4bc19503fe9e9309e2bfaa2c58074d32" is wrong or could not be found error.

• Windows: 10
• Node v6.10.2
• npx 9.2.2

If I want to be able to use npx foo in CI, and in automated scripts, I have to know that the output will only be from foo.

However, I occasionally get output like this:

   ╭─────────────────────────────────────╮
   │                                     │
   │   Update available 5.2.0 → 9.3.2    │
   │     Run npm i -g npx to update      │
   │                                     │
   ╰─────────────────────────────────────╯

This is also problematic with npm, but that CLI is typically verbose, so nobody tends to rely on it.

Do I have to always include --silent (I don't get the update output predictably, so I can't really check if this even works), or can npx instead be changed to not add this disruptive (and unnecessary; if i want to update, i would have done so) output?

As also a brazilian portuguese speaker, I come with a slightly different translation propositions I found to be more accurate given the context I understood from the JSON messages.

% npx @coderbyheart/xkcd-password  
npx: installed 19 in 2.583s
command not found: xkcd-password
npx @coderbyheart/xkcd-password  4.93s user 0.35s system 189% cpu 2.794 total

The sourcecode for the package is here: https://github.com/coderbyheart/xkcd-password

The package is published as a scoped package at https://www.npmjs.com/package/@coderbyheart/xkcd-password

It seems npx ignores the scope when executing it.

Running npx from within the package source folder works:

~/git/xkcd-password (git)-[master] % npx @coderbyheart/xkcd-password                                                                                       :(
kerosine brunts shroffs aborally

"It'll be fine" they said

"you get the progress bar for free!" they said

and then... I tried to npx covfefe. And hoooooly cow.

Right now, every time npx runs, it removes all binaries from a single,
shared binary directory. This works fine in isolation, but if there's a long-running pipeline, it can cause odd errors. For example, in npx foo | npx bar | npx baz, each successive npx call would blow away the bin directory before executing the next command. If everything a command from that bin dir is already loaded into memory on execution, that's fine -- but some required binaries for later stuff might end up getting removed.

npx should have a better model for this, and, ideally, it'll also facilitate faster re-runs.

my step:

1. create dir 'export' on desktop
2. start cmd & cd to 'export'
3. run npx http-server ./

exception throwed

env info

C:\Users\gaopi\Desktop\export>node -v
v8.2.1

C:\Users\gaopi\Desktop\export>npm -v
5.3.0

C:\Users\gaopi\Desktop\export>npx -v
Command failed: C:\"Program Files"\nodejs\node.exe C:\Program Files\nodejs\node_modules\npm\bin\npm-cli.js config get cache --parseable
module.js:487
    throw err;
    ^

Error: Cannot find module 'C:\Program'
    at Function.Module._resolveFilename (module.js:485:15)
    at Function.Module._load (module.js:437:25)
    at Function.Module.runMain (module.js:605:10)
    at startup (bootstrap_node.js:158:16)
    at bootstrap_node.js:575:3

9.2.1

C:\Users\gaopi\Desktop\export>

Hey y'all! [email protected] added several new strings! Most of them are single-word translations so the usage examples actually have their names shown in the right language.

I already did all the languages I felt semi-confident in being able to at least butcher. I'd really appreciate it if y'all helped with the others! 779d950 is the commit that added the strings, and you can just copy-paste that and fill the rest in. LMK if you have any questions!

Needs updating

• Catalan - ca.json
• German - de.json (/cc @janl)
• English - en.json
• Spanish - es.json
• French - fr.json (/cc @gorhgorh)
• Japanese - ja.json (note: I butchered this badly. Please feel free to correct it!)
• Pirate - pirate.json (/cc @bnb @codetheory) (note: currently in #36)
• Portuguese (BR) - pt_BR.json (/cc @cyberglot)
• Romanian - ro.json (/cc @hisk)
• Russian - ru.json (/cc @sobolevn)
• Chinese (Simplified) - zh_CN.json (/cc @int64ago)

It would be helpful for documentation to clearly explain that if you run npx foo, then npx will always download foo@latest, rather than using the existing cached version of foo. (Assuming this is true)

Even without the global fallback (which nicely requires the @Version specifier), it's still pretty easy to goof up a name and download and execute something completely unexpected (for example "create-react-appp" with three ps is still available on the registry at this time).

A prompt of some kind could give the user time to review what they've just requested to be downloaded from the internet and executed with all the permissions of their user on their computer.

The prompt could force a complete 'yes' response in the same way you first agree to a fingerprint when using ssh.

npx could even display author information and popularity stats like download or star counts so the user can be very aware they're downloading what they think they are.

And of course none of this could be necessary if there's no need to download anything in the case the user's already agreed to this or the binary is available in node_modules/.bin.

Thanks for considering!

It would be really nice to have shorthands for all npm package types. I think it's something folks will "expect" but also it's just really convenient for a tool like npx to be able to do it. As with regular registry dependencies, these are of course just fudged based on the assumption that the binary is named the same as the "name" of the dep requested. Users can always -p <pkg> as an escape hatch!

Here are all the supported package types, along with the suggested "guesswork" binary:

It's fine for these to be kinda fudgy: they're meant to be there for convenience. These expansions should also be skipped entirely if a user does -p <pkg>.

Hello dear npx maintainers.

I've just set up auto shell fallback by adding

command_not_found_handle() {
  # Do not run within a pipe
  if test ! -t 1; then
    >&2 echo "command not found: $1"
    return 127
  fi
  echo "$1 not found. Trying with npx..." >&2
  if ! [[ $1 =~ @ ]]; then
    npx --no-install "$@"
  else
    npx "$@"
  fi
  return $?
}

into my .zsh_rc file.

I then do

npx awesome-npx

and the following happens!

❯ npx awesome-npx
npm ERR! path /Users/charlotte.spencer/ 🎵 I Have the Honour to Be Your Obedient Servant,🎵 ~ npm 📜🖋
npm ERR! code ENOENT
npm ERR! errno -2
npm ERR! syscall lstat
npm ERR! enoent ENOENT: no such file or directory, lstat '/Users/charlotte.spencer/ 🎵 I Have the Honour to Be Your Obedient Servant,🎵 ~ npm 📜🖋'
npm ERR! enoent This is related to npm not being able to find a file.
npm ERR! enoent

npm ERR! A complete log of this run can be found in:
npm ERR!     /Users/charlotte.spencer/.npm/_logs/2017-06-27T19_24_03_425Z-debug.log
Install for awesome-npx@latest failed with code 254

The end of any npm command I do prints out

🎵 I Have the Honour to Be Your Obedient Servant,🎵 ~ npm 📜🖋

which is an npm easter egg.

It seems that npx is taking this, and thinking it is part of the directory I am working in.

Pretty funny, but surely worth looking into.

~ your obedient servant, class

As part of replacing the current node process with the subcommand, npx does a lot of hand-wavy magic involving various bits of process and module state. One missing piece here is process.argv0, which needs to be configured to something that makes sense -- preferably to opts.command.

The end result is that you'll have stuff like this happen:

$ npx cmd -h
Usage: npx --cmd-opt <cmd-arg>

Its not entirely clear what the purpose of the --save functionality in this repo is. Is it a value add over npm? If not, it seems to just complicate the surface area of the project.

When trying to run an executable of a package that is not currently installed, npx fails to run underlying npm commands.

♥ npx ncp --help
Command failed: C:\Program Files\nodejs\npm.CMD config get cache
'C:\Program' is not recognized as an internal or external command,
operable program or batch file.

Any package that is currently not installed globally or in local scope ends with the same error.

• node -v: v6.9.4
• npm -v: 5.0.1
• npx -v: 2.1.0

this may or may be not related to #58 (i encountered the same issue, but only when running a module that has not been installed already).

when i try to reproduce the example from your blog post I get the error message from the title.
in more detail:

D:\programming\florp>cat package.json
{
  "name": "florp",
  "version": "1.0.0",
  "description": "",
  "main": "index.js",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "keywords": [],
  "author": "",
  "license": "ISC",
  "dependencies": {
    "cowsay": "^1.1.9"
  }
}

D:\programming\florp>npm -v
5.2.0

D:\programming\florp>npx -v
9.0.3

D:\programming\florp>ls .\node_modules\.bin
cowsay  cowsay.cmd  cowthink  cowthink.cmd

D:\programming\florp>npx cowsay hello!
npx: command not found: cowsay

Update:
I updated to 9.0.5, the issue is still the same.

This thing was a single-day weekend hack for me. I haven't written any tests, and all the testing I've done has been manual. I'm sure there's lots of bugs right now.

Gotta get those tests written, probably starting off with a utility that makes it easy to mock up different environments, so we can run the tests in isolation of the rest of the system. It's probably a good idea to mock up child_process, because npm is a monstrosity that we should NOT be shelling out to in the test suite lol.

I'm a little unclear on the proposed behavior around cached packages: if it already exists, is npx just going to link its existing binaries?

One of the things I'm thinking is that, considering the main use case I can see this being used for ("I found this npm package's command on Stack Overflow, let me try it"), npx should re-evaluate the latest version of the package available (against the semver) every time it's run, and update if a newer (non-prerelease) version is available (giving the same "against the latest version available" behavior it'd have if the package hadn't been retrieved yet).

Obviously that's a bit of network overhead that could introduce a lot of lag on every invocation, so maybe there should be global / user config and/or command-line options to elide the update check under certain circumstances. Some (but not all) of the dimensions I can see this being useful to slice on (not necessarily 1:1 with a desirable configuration surface):

• Available bandwidth / connectivity (for instance, I may want to only disable this check if I'm not connected to Wi-Fi, when I know it's not going to resolve)
• Which package is being checked against (I may have a script that runs little-linter individually against a hundred files, but big-huge-framework-generator only once per project, and I never want the technical debt that comes from being a single Big Huge Framework patch release behind)
• How recently the package was updated (and note that this can intersect the previous: I might want a package that gets multiple patches a day to check at every invocation, even invoked several times a minute, but a package that only gets updated annually I never want to check against)

As of [email protected], if the command you're trying to execute turns out to be a Node.js script, npx will do some trickery to "convert" that script into the main module for its own Node process, without spawning a child. This is a pretty significant optimization for what turns out to be a pretty common case for npx users.

Unfortunately, there's no Windows cmd shim detection right now, so this trick works exclusively on *nix platforms.

So, it'd be nice to safely detect the Windows cmd shims that npm installs into $PATH, and grab the Node.js script it's meant to execute -- and call that directly. This should make Windows execution much faster.

I want to be able to add a version specifier to request a particular version of a command. This could be useful either for checking out beta tagged versions that aren't yet "latest", or for going back in time and checking that something works with a specific prior version. It also makes it feel more npm-like.

right now, I get:

> npx standard@5 --version
6.0.4

and instead it would be awesome to get:

> npx standard@5 --version
5.4.1

(also, since npm info standard --json | js .version is 10.0.2, I'm not sure why I'm getting 6.0.4 other than assuming I've got it cached somewhere on my machine - but that's a different issue)

See yeoman/update-notifier#112

In this example, npx is using the locally-installed version of bower (node_modules/.bin/bower). update-notifier is correctly noticing that we're not using the latest version, but it suggests that npm i -g bower will fix that, which it will not, since it'll still use the locally-installed version, which is old.

For now, npx could probably fix this problem by setting isGlobal to the appropriate value in update-notifier?

https://github.com/yeoman/update-notifier#isglobal