zkat / cipm Goto Github PK

Passing --ignore-scripts seems to have no effect, or at least doesn't do what I would expect it to do: ignore preinstall/install/postinstall scripts of dependencies during installation.

I noticed there is at least code for accepting the parameter, but haven't looked further yet to see where it might be getting dropped.

I also haven't looked at #6 yet to see if the refactor inadvertently fixes this.

Hi! Thanks for awesome project, hope to see it along with npx as part of node install!

I'm trying to suppress all logs (except final message added X packages in Ys) by calling cipm --loglevel error, but it seems like does nothing - anyway I see info about each package and fetch info if cache is clear.

NodeJS 8.9.4, cipm 0.9.1

I was asked to open a corresponding issue for this problem in this repository. I won't copy the full body of that issue to this one, though. Here's the minimum repro though:

$ mkdir test-npm-ci-only-dev
$ cd test-npm-ci-only-dev
$ npm install -D typescript
npm notice created a lockfile as package-lock.json. You should commit this file.
npm WARN [email protected] No description
npm WARN [email protected] No repository field.

+ [email protected]
added 1 package from 1 contributor in 1.246s
$ rm -r node_modules/
$ npm ci --only=dev
added 0 packages in 0.019s

npm 6.x ci doesn't seem to respect the --no-optional flag or optional=false setting in .npmrc.

I'm trying to switch our CI build to using npm ci to speed up time, but it ends up taking more time because one of the optional dependency reaches out to the internet, causing the build to time out (our CI server is in an air-gapped environment).

Had a user with a messed-up system try it out and it turns out that if node-gyp isn't available globally, cipm can't run gyp. npm works around this by bundling its own node-gyp so you know it'll always be available.

Why does cipm need to be a separate tool?
Couldn't it be a command line flag for the existing npm tool?

install and postinstall should not be run until the package's dependencies have finished installing and running their own install and postinstall.

Getting this working will involve parsing the requires:[] fields on each dependency and building an ordered tree. Preferably, this tree will not just be flattened to a single list, so we have the ability to parallelize scripts at some point.

I think I'd like to block initial release on having at least some baseline i18n support. My preference would be to make it similar to the way npx does it, by using y18n.

Currently, npm ci runs the postinstall script if it is present, which works well enough, but I would like a little more control after installing during npm ci by running a postci script instead.

My use case is with CI services in a lerna maintained Git repo, where I would like to run lerna bootstrap in a postinstall script, while running lerna bootstrap --ci in a postci script.

(I found a similar request on npm.community.)

Hi, for latest libcipm 3.0.3 a vulnerable version of dependency npm-lifecycle was reported in npm audit. Could you please npm audit fix your package and publish to npm?

npm audit output follows:
High Arbitrary File Overwrite

Package tar

Patched in >=4.4.2

Dependency of npm

Path npm > libcipm > npm-lifecycle > node-gyp > tar

More info https://nodesecurity.io/advisories/803

It just crashes right now.

Keep in mind that this isn't as simple as ignoring failures: Any dependencies of the optionalDep which are not used by anything else need to be removed.

Looks like we're not gonna be able to get away with just invoking npm itself after all. We're gonna have to write config stuff ourselves or patch npm to allow outputting these vars

It looks like when #58 was resolved by merging #59 and libcipm was published to npm as v3.0.3, it was actually tagged and published as v3.0.3 without the merged fixes from #59.

The latest branch does indeed include the fixes from #59, however you can also see that the package.json file was not updated properly to v3.0.3.

Would it be possible to re-tag latest as v3.0.4 and republish? At least that way this fix could be included whenever the next NPM release is made available.

Thanks!

In order to support cache configuration, offline mode, private packages, alternative registries, etc, etc, we need to pass a blob of various pacote-related configs through. See lib/config/pacote.js in npm for an idea of what's needed.

These should also have similar or identical semantics to how they work in npm, and the options should be visible with cipm -h.

It appears that currently the --no-bin-links option - intended to avoid creating symlinks when doing npm install - is not supported in this API. Is this an intentional design decision, or something that just hasn't been implemented yet?

I haven't looked yet to see if this bit of functionality has already been extracted from npm core or whether it is planned or not, but it sure would be nice!

It looks like there's a difference in behavior between npm and cipm here, where bins aren't linked correctly by cipm in some cases. There's a pretty good repro in a gist by @mollerse, using the following package.json:

{
    "name": "npm-ci-repro",
    "version": "1.0.0",
    "description": "",
    "main": "index.js",
    "scripts": {
        "test": "rm -rf node_modules package-lock.json && npm i && ls node_modules/.bin && npm ci && ls node_modules/.bin"
    },
    "keywords": [],
    "license": "ISC",
    "devDependencies": {
        "json2csv": "3.11.5",
        "npm-run-all": "4.1.2"
    },
    "dependencies": {
        "d3": "4.11.0"
    }
}

cipm doesn't link binaries for deps yet, so lifecycle scripts that use them... can't.

It would be useful to allow installing without a package.json file present, using only package-lock.json. This would be particularly useful in Docker CI environments, where bumping the version field in package.json otherwise causes the Docker cache to be invalidated even if no new dependencies were added.

Running cipm (or through npm ci) in a directory with only package-lock.json results in an ENOENT error for package.json:

ENOENT: no such file or directory, open '/Users/victor/c/p/test-ci/package.json'
Error: ENOENT: no such file or directory, open '/Users/victor/c/p/test-ci/package.json'

Is package.json actually used for anything? The npm docs for package-lock.json do read:

version
The version of the package this is a package-lock for. This must match what's in package.json.

So maybe doing this would be a bad idea for some reason I don't understand?

git deps in npm have their deps installed and prepare scripts executed. We'll need to do the same for any git dependencies that go through cipm.

The best way to do this is probably just to do npm pack <git url> and let npm take care of it.

This is a continuation of #45, a side-effect of the provided fix for it in #46 and was introduced in [email protected].

For the case where a default install script is used and a binding.gyp file is present in the package root, npm ci will fail with packages that target a different platform that the one currently running.

Reproduction

To reproduce on Windows, use the following package.json, run npm install to create the package-lock.json, then npm ci.

Since [email protected] is the current latest, this is happening right now in [email protected]

{
  "dependencies": {
    "grunt-ts": "^5.0.0-beta.4"
  }
}

$ npm ci
npm WARN prepare removing existing node_modules/ before installation

> [email protected] install d:\repos\z\node_modules\fsevents
> node-gyp rebuild

d:\repos\z\node_modules\fsevents>if not defined npm_config_node_gyp (node "C:\Users\rrey\AppData\Roaming\npm\node_modules\npm\node_modules\npm-lifecycle\node-gyp-bin\\..\..\node_modules\node-gyp\bin\node-gyp.js" rebuild )  else (node "C:\Users\rrey\AppData\Roaming\npm\node_modules\npm\node_modules\node-gyp\bin\node-gyp.js" rebuild )
Traceback (most recent call last):
  File "C:\Users\rrey\AppData\Roaming\npm\node_modules\npm\node_modules\node-gyp\gyp\gyp_main.py", line 16, in <module>
    sys.exit(gyp.script_main())
  File "C:\Users\rrey\AppData\Roaming\npm\node_modules\npm\node_modules\node-gyp\gyp\pylib\gyp\__init__.py", line 545, in script_main
    return main(sys.argv[1:])
  File "C:\Users\rrey\AppData\Roaming\npm\node_modules\npm\node_modules\node-gyp\gyp\pylib\gyp\__init__.py", line 538, in main
    return gyp_main(args)
  File "C:\Users\rrey\AppData\Roaming\npm\node_modules\npm\node_modules\node-gyp\gyp\pylib\gyp\__init__.py", line 523, in gyp_main
    generator.GenerateOutput(flat_list, targets, data, params)
  File "C:\Users\rrey\AppData\Roaming\npm\node_modules\npm\node_modules\node-gyp\gyp\pylib\gyp\generator\msvs.py", line 2019, in GenerateOutput
    sln_projects, project_objects, flat=msvs_version.FlatSolution())
  File "C:\Users\rrey\AppData\Roaming\npm\node_modules\npm\node_modules\node-gyp\gyp\pylib\gyp\generator\msvs.py", line 1778, in _GatherSolutionFolders
    return _DictsToFolders('', root, flat)
  File "C:\Users\rrey\AppData\Roaming\npm\node_modules\npm\node_modules\node-gyp\gyp\pylib\gyp\generator\msvs.py", line 1731, in _DictsToFolders
    for folder, contents in bucket.iteritems():
AttributeError: 'MSVSProject' object has no attribute 'iteritems'
gyp ERR! configure error
gyp ERR! stack Error: `gyp` failed with exit code: 1
gyp ERR! stack     at ChildProcess.onCpExit (C:\Users\rrey\AppData\Roaming\npm\node_modules\npm\node_modules\node-gyp\lib\configure.js:336:16)
gyp ERR! stack     at ChildProcess.emit (events.js:127:13)
gyp ERR! stack     at Process.ChildProcess._handle.onexit (internal/child_process.js:209:12)
gyp ERR! System Windows_NT 10.0.17134
gyp ERR! command "C:\\Program Files\\nodejs\\node.exe" "C:\\Users\\rrey\\AppData\\Roaming\\npm\\node_modules\\npm\\node_modules\\node-gyp\\bin\\node-gyp.js" "rebuild"
gyp ERR! cwd d:\repos\z\node_modules\fsevents
gyp ERR! node -v v9.7.1
gyp ERR! node-gyp -v v3.6.2
gyp ERR! not ok
added 77 packages in 5.201sactTree: info lifecycle [email protected]~install: [email protected]

From #34

I was reading about the whole package-lock.json fiasco where the lock file wasn't really locking dependencies which is how I found a reference to cipm.

I was also looking for how to use the package-lock.json file instead of package.json to install the packages. NPM docs about package-lock.json nor package-locks tell you how you use it "only" what they are for:

Describe a single representation of a dependency tree such that teammates, deployments, and continuous integration are guaranteed to install exactly the same dependencies.

So I searched and found cipm. This worked fine at my local machine, but at Gitlab CI it didn't work. After some debugging I found out that the command didn't log errors but installed only the half of the dependencies right e.g. node-sass was not build as well as dependencies pulled in via git instead of npm registry. I searched again and found in an archived NPM issue that npm ci is out. This works as expected.

And now I'm confused 😕
Does npm ci use libcipm from this project or only cipm? In my opinion it would make sense to send people to npm ci instead of cipm in the Readme.

This was first opened upstream to npm at npm/npm#20275

It appears that the root cause of the upstream issue is that cipm is not considering the default value of the install script:

"install": "node-gyp rebuild":

If there is a binding.gyp file in the root of your package and you haven't defined your own install or preinstall scripts, npm will default the install command to compile using node-gyp.

Detection for the binding.gyp file should be run and pkg.scripts.install should be added to the pkg object before the install hook is called here:

Perhaps enable this as part of --low-mem. So it will also reduce memoization in pacote, etc.

So I got here because I was searching for a reliable way to install javascript dependencies based on a package-lock.json file. Several stack-overflow and git-hub issues linked to this repo.

Would my typical use case be to install libcpm globally or in my project as a dev-dependency?
How do I use the API for libcipm? Is it identical to the API for cipm?
What is the difference between libcipm and cipm and do I need both?

I would really appreciate an example of how to use this library to install javascript dependencies based on the versions that are "locked" in an already generated package-lock.json file (if that is indeed what this library intends to do).

npm itself now uses pacote@8, which has this fix (zkat/pacote#151), but I still sometimes encounter zkat/pacote#142 using [email protected] with Node 6.13.1 on macOS. The reason appears to be that cipm still has the old pacote dependency.

There are several places in cipm when an object is being passed to .then() in bluebird promise chain.
It works but it causes warning message as described here: https://github.com/petkaantonov/bluebird/blob/master/docs/docs/warning-explanations.md

So using npm ci creates a long warning stack with messages like

(node:21662) Warning: .then() only accepts functions but was passed: [object Object]

Everything still works but that's very frightening and I think that it is important to fix this warning and ask npm to bump cipm version. I will add pull request here in several minutes.

On virtual machines no limited(limited by os.cpus) workerFarm use many thread and fail with coredump.
Maybe we can add some environment variable for redefine default maxConcurrentWorkers in:

More details in: https://npm.community/t/npm-ci-failed-child-send-is-not-a-function/726/8