swarajban / npm-cache Goto Github PK
View Code? Open in Web Editor NEWCache (npm | bower | composer | jspm) installs
License: MIT License
Cache (npm | bower | composer | jspm) installs
License: MIT License
Hello. I want to execute this
npm install -g typings webpack-dev-server rimraf webpack
What is the proper syntax to tell npm-cache to execute npm install on specific pacakges only, and globally?
I don't see the option to specify which npm package I want npm-cache to install, nor the option to pass global. Does it support this?
On my build server I run "npm install --production". I tried "npm-cache install --production" and I get a usage error.
With the 0.3.5 release last night my npm-cache installs are all failing (exit code 1) with this error:
[npm-cache] [INFO] [bower] Dependency config file {my code directory}/bower.json does not exist. Skipping install
module.js:338
throw err;
^
Error: Cannot find module '{my code directory}/composer.json'
at Function.Module._resolveFilename (module.js:336:15)
at Function.Module._load (module.js:278:25)
at Module.require (module.js:365:17)
at require (module.js:384:17)
at Object.<anonymous> ({my code directory}/node_modules/npm-cache/cacheDependencyManagers/composerConfig.js:27:16)
at Module._compile (module.js:460:26)
at Object.Module._extensions..js (module.js:478:10)
at Module.load (module.js:355:32)
at Function.Module._load (module.js:310:12)
at Module.require (module.js:365:17)
0.3.4 does not have this issue, I do not need composer.json and bower.json to be present to run npm-cache install.
When I run npm install -g npm-cache
I get the following error message:
npm ERR! not a package C:\Users\Me\appdata\roaming\npm\npm-cache
npm ERR! addLocal Could not install C:\Users\Me\appdata\roaming\npm\npm-cache
npm ERR! Windows_NT 6.1.7601
npm ERR! argv "C:\\Program Files\\nodejs\\node.exe" "C:\\Users\\Me\\AppData\\Roaming\\npm\\node_modules\\npm\\bin\\npm-cli.js" "install" "-g" "npm-cache"
npm ERR! node v5.10.1
npm ERR! npm v3.10.2
npm ERR! path C:\Users\Me~1\AppData\Local\Temp\3\npm-15596-c052a037\unpack-6d94b7f9\package.json
npm ERR! code ENOENT
npm ERR! errno -4058
npm ERR! syscall open
npm ERR! enoent ENOENT: no such file or directory, open 'C:\Users\Me~1\AppData\Local\Temp\3\npm-15596-c052a037\unpack-6d94b7f9\package.json'
npm ERR! enoent ENOENT: no such file or directory, open 'C:\Users\Me~1\AppData\Local\Temp\3\npm-15596-c052a037\unpack-6d94b7f9\package.json'
npm ERR! enoent This is most likely not a problem with npm itself
npm ERR! enoent and is related to npm not being able to find a file.
npm ERR! enoent
npm ERR! Please include the following file with any support request:
npm ERR! C:\Users\buildmeister\appdata\roaming\npm\npm-debug.log
"Me" is a replacement for a longer username. Otherwise the log file contents are exactly as stated.
Here is the full log file:
0 info it worked if it ends with ok
1 verbose cli [ 'C:\\Program Files\\nodejs\\node.exe',
1 verbose cli 'C:\\Users\\Me\\AppData\\Roaming\\npm\\node_modules\\npm\\bin\\npm-cli.js',
1 verbose cli 'install',
1 verbose cli '-g',
1 verbose cli 'npm-cache' ]
2 info using [email protected]
3 info using [email protected]
4 silly loadCurrentTree Starting
5 silly install loadCurrentTree
6 silly install readGlobalPackageData
7 silly fetchPackageMetaData npm-cache
8 silly fetchOtherPackageData npm-cache
9 silly cache add args [ 'npm-cache', null ]
10 verbose cache add spec npm-cache
11 silly cache add parsed spec Result {
11 silly cache add raw: 'npm-cache',
11 silly cache add scope: null,
11 silly cache add escapedName: 'npm-cache',
11 silly cache add name: null,
11 silly cache add rawSpec: 'npm-cache',
11 silly cache add spec: 'C:\\Users\\Me\\appdata\\roaming\\npm\\npm-cache',
11 silly cache add type: 'local' }
12 silly addLocalTarball shasum (computed) f1f9665493e7089d78850103dd0e4d283416bd05
13 verbose addTmpTarball C:\Users\Me\appdata\roaming\npm\npm-cache not in flight; adding
14 verbose correctMkdir C:\Users\Me\AppData\Roaming\npm-cache correctMkdir not in flight; initializing
15 verbose addTmpTarball validating metadata from C:\Users\Me\appdata\roaming\npm\npm-cache
16 verbose tar unpack C:\Users\Me\appdata\roaming\npm\npm-cache
17 verbose tar unpacking to C:\Users\Me~1\AppData\Local\Temp\3\npm-15596-c052a037\unpack-6d94b7f9
18 silly gentlyRm C:\Users\Me~1\AppData\Local\Temp\3\npm-15596-c052a037\unpack-6d94b7f9 is being purged
19 verbose gentlyRm don't care about contents; nuking C:\Users\Me~1\AppData\Local\Temp\3\npm-15596-c052a037\unpack-6d94b7f9
20 silly gunzTarPerm modes [ '777', '666' ]
21 error not a package C:\Users\Me\appdata\roaming\npm\npm-cache
22 error addLocal Could not install C:\Users\Me\appdata\roaming\npm\npm-cache
23 silly fetchPackageMetaData Error: ENOENT: no such file or directory, open 'C:\Users\Me~1\AppData\Local\Temp\3\npm-15596-c052a037\unpack-6d94b7f9\package.json'
23 silly fetchPackageMetaData at Error (native)
23 silly fetchPackageMetaData error for npm-cache { [Error: ENOENT: no such file or directory, open 'C:\Users\Me~1\AppData\Local\Temp\3\npm-15596-c052a037\unpack-6d94b7f9\package.json']
23 silly fetchPackageMetaData errno: -4058,
23 silly fetchPackageMetaData code: 'ENOENT',
23 silly fetchPackageMetaData syscall: 'open',
23 silly fetchPackageMetaData path: 'C:\\Users\\Me~1\\AppData\\Local\\Temp\\3\\npm-15596-c052a037\\unpack-6d94b7f9\\package.json' }
24 silly rollbackFailedOptional Starting
25 silly rollbackFailedOptional Finishing
26 silly runTopLevelLifecycles Starting
27 silly runTopLevelLifecycles Finishing
28 silly install printInstalled
29 verbose stack Error: ENOENT: no such file or directory, open 'C:\Users\Me~1\AppData\Local\Temp\3\npm-15596-c052a037\unpack-6d94b7f9\package.json'
29 verbose stack at Error (native)
30 verbose cwd C:\Users\Me\appdata\roaming\npm
31 error Windows_NT 6.1.7601
32 error argv "C:\\Program Files\\nodejs\\node.exe" "C:\\Users\\Me\\AppData\\Roaming\\npm\\node_modules\\npm\\bin\\npm-cli.js" "install" "-g" "npm-cache"
33 error node v5.10.1
34 error npm v3.10.2
35 error path C:\Users\Me~1\AppData\Local\Temp\3\npm-15596-c052a037\unpack-6d94b7f9\package.json
36 error code ENOENT
37 error errno -4058
38 error syscall open
39 error enoent ENOENT: no such file or directory, open 'C:\Users\Me~1\AppData\Local\Temp\3\npm-15596-c052a037\unpack-6d94b7f9\package.json'
40 error enoent ENOENT: no such file or directory, open 'C:\Users\Me~1\AppData\Local\Temp\3\npm-15596-c052a037\unpack-6d94b7f9\package.json'
40 error enoent This is most likely not a problem with npm itself
40 error enoent and is related to npm not being able to find a file.
41 verbose exit [ -4058, true ]
Any help troubleshooting this problem would be appreciated.
I just tried to rebase on the PR you just merged and in my test it fails when trying to extract an existing cache file. Maybe it should invalidate the cache or add something to the cache hash filename.
[npm-cache] [INFO] [jspm] extracting dependencies from /Users/avweiss/.package_cache/jspm/0.16.27/0733d590c33222a8f95fe2090c28ccbd.tar.gz
[npm-cache] [ERROR] [jspm] Error extracting /Users/avweiss/.package_cache/jspm/0.16.27/0733d590c33222a8f95fe2090c28ccbd.tar.gz: Error: invalid tar file
[npm-cache] [ERROR] [jspm] Error extracting /Users/avweiss/.package_cache/jspm/0.16.27/0733d590c33222a8f95fe2090c28ccbd.tar.gz: Error: invalid tar file
[npm-cache] [ERROR] [jspm] Error extracting /Users/avweiss/.package_cache/jspm/0.16.27/0733d590c33222a8f95fe2090c28ccbd.tar.gz: Error: invalid tar file
.......
First of all - thank you. npm-cache install saves my nerves.
Since i work in a lot of small projects i realized how crazy it gets with modules. Yesterday in order to backup my full drive and update Ubuntu i deleted 10 GB of node modules. NPM is probably killing every developers SSD drive. Most of the projects use very similar modules, mostly even in the same version.
What i'd love would be a global npm storage that makes sure every module is only present once per version. In the projects node_modules folder there should be a symlink to the versions folder.
Why?
[npm-cache] [ERROR] [bower] error running bower install
[npm-cache] [ERROR] error installing dependencies
When running
npm-cache install npm --production
if you later run without --production on same package.json-file you get the same result
Ex
have a package.json file with devDependencies
run: npm-cache install npm --production
delete node_modules folder
run: npm-cache install npm
then node_modules doesn't container devDependencies
How about support for npm-shrinkwrap.json as the checksum source? You could have different npm-shrinkwrap.json from the same package.json, but npm-cache would not pick that up.
Hey,
I've just upgraded our CI runner to use 0.4.10 of npm-cache
, but I'm running into an issue. The first fresh npm-cache install
runs fine, but subsequent calls that use the cache cause ember-cli
to fail with:
ember -v
module.js:340
throw err;
^
Error: Cannot find module 'rsvp'
at Function.Module._resolveFilename (module.js:338:15)
at Function.Module._load (module.js:280:25)
at Module.require (module.js:364:17)
at require (module.js:380:17)
at Object. (/Users/jkay/node_modules/ember-cli/lib/ext/promise.js:3:15)
at Module._compile (module.js:456:26)
at Object.Module._extensions..js (module.js:474:10)
at Module.load (module.js:356:32)
at Function.Module._load (module.js:312:12)
at Module.require (module.js:364:17)
0.3.9 appears to work fine. Here is a list of the node_modules folder after a full install and after a cached npm-cache install. A tonne of packages are missing when installed from the cache using 0.4.x, but when using 0.3.x the installation works fine and all modules are un-archived correctly.
If I link a local directory as a dependency to my project (i.e. npm link my-local-package
), that dependency is then represented as a symlink in the node_modules
directory. If I now run npm-cache install
, and I had previously used a different version of that local package, it will unpack the cached version into the symlinked directory, overwriting whatever is currently there.
To avoid this, I would suggest that npm-cache install
removes any linked packages/symlinked directories before unpacking the cache.
Hi!
I'm using npm-cache as part of my deployment system, and found that if NPM_CACHE_DIR isn't set on the environment when called from child_process.exec
, it throws the following exception:
path.js:8
throw new TypeError('Path must be a string. Received ' +
^
TypeError: Path must be a string. Received undefined
at assertPath (path.js:8:11)
at Object.posix.resolve (path.js:426:5)
at main (/usr/lib/node_modules/npm-cache/index.js:36:48)
at Object.<anonymous> (/usr/lib/node_modules/npm-cache/index.js:159:1)
at Module._compile (module.js:410:26)
at Object.Module._extensions..js (module.js:417:10)
at Module.load (module.js:344:32)
at Function.Module._load (module.js:301:12)
at Function.Module.runMain (module.js:442:10)
at startup (node.js:136:18)
at node.js:966:3
I did the following to prevent this error: export NPM_CACHE_DIR=/path/to/cache && npm-cache install
Note: this happened even when specifying the cache directory through the cacheDirectory
or c
parameters.
After updating from version 0.5.1 to 0.5.3 today, npm-cache immediately throws the following error:
Error: Cannot find module 'fstream' at Function.Module._resolveFilename (module.js:336:15) at Function.Module._load (module.js:278:25) at Module.require (module.js:365:17) at require (module.js:384:17) at Object.<anonymous> (/Users/patrickdavis/cvent.git/developer-portal/developer-portal-ui/node_modules/npm-cache/cacheDependencyManagers/cacheDependencyManager.js:10:15) at Module._compile (module.js:460:26) at Object.Module._extensions..js (module.js:478:10) at Module.load (module.js:355:32) at Function.Module._load (module.js:310:12) at Module.require (module.js:365:17)
Not sure how fstream is supposed to be installed into npm-cache, but it's completely missing. Installing fstream and adding it to npm-cache's node_modules folder resolves this issue.
{
// ...
"devDependencies": {
"beepbeep": "^1.2.0",
"browser-sync": "^2.10.0",
"gulp-autoprefixer": "^3.1.0",
"gulp-concat": "^2.6.0",
"gulp-html-minifier": "^0.1.6",
"gulp-imagemin": "^2.4.0",
"gulp-less": "^3.0.5",
"gulp-rename": "^1.2.2",
"gulp-sourcemaps": "^1.6.0",
"gulp-tslint": "^4.1.0",
"gulp-typescript": "^2.9.2",
"gulp-uglify": "^1.5.1",
"gulp-uncss": "^1.0.4",
"gulp-util": "^3.0.7",
"imagemin-pngquant": "^4.2.0",
"less-plugin-clean-css": "^1.5.1",
"merge2": "^0.3.6",
"tslint": "^3.1.1"
}
}
gulp is globally installed
requiring gulp-less
ends in the following error. default npm install works fine
module.js:339
throw err;
^
Error: Cannot find module 'lodash'
at Function.Module._resolveFilename (module.js:337:15)
at Function.Module._load (module.js:287:25)
at Module.require (module.js:366:17)
at require (module.js:385:17)
at Object.<anonymous> (/var/www/xxx/builds/835110e/src/node_modules/gulp-less/node_modules/accord/lib/index.js:11:7)
at Object.<anonymous> (/var/www/xxx/builds/835110e/src/node_modules/gulp-less/node_modules/accord/lib/index.js:100:4)
at Module._compile (module.js:435:26)
at Object.Module._extensions..js (module.js:442:10)
at Module.load (module.js:356:32)
at Function.Module._load (module.js:311:12)
Frequently, when trying to execute the npm-cache install npm
script, I get this failed error message. If I run it again it usually succeeds. What is causing rmdir to fail? It only happens when node_modules is not empty.
npm ERR! Windows_NT 10.0.10586
npm ERR! argv "C:\\dev\\nodejs\\node.exe" "C:\\dev\\nodejs\\node_modules\\npm\\bin\\npm-cli.js" "cache" "clean"
npm ERR! node v4.4.4
npm ERR! npm v2.15.1
npm ERR! path C:\Users\nparson\AppData\Roaming\npm-cache
npm ERR! code EPERM
npm ERR! errno -4048
npm ERR! syscall rmdir
npm ERR! Error: EPERM: operation not permitted, rmdir 'C:\Users\nparson\AppData\Roaming\npm-cache'
npm ERR! at Error (native)
npm ERR! { [Error: EPERM: operation not permitted, rmdir 'C:\Users\nparson\AppData\Roaming\npm-cache']
npm ERR! errno: -4048,
npm ERR! code: 'EPERM',
npm ERR! syscall: 'rmdir',
npm ERR! path: 'C:\\Users\\nparson\\AppData\\Roaming\\npm-cache' }
npm ERR!
npm ERR! Please try running this command again as root/Administrator.
npm ERR! Please include the following file with any support request:
npm ERR! C:\dev\#Projects\ASP.Net.Core1.Angular2.Webpack.Starter\src\starterproject\npm-debug.log
npm ERR! Windows_NT 10.0.10586
npm ERR! argv "C:\\dev\\nodejs\\node.exe" "C:\\dev\\nodejs\\node_modules\\npm\\bin\\npm-cli.js" "run" "clean"
npm ERR! node v4.4.4
npm ERR! npm v2.15.1
npm ERR! code ELIFECYCLE
npm ERR! [email protected] preclean: `npm run installglobals && npm cache clean`
npm ERR! Exit status 4294963248
It seems related to #58 .
Hi,
I stumbled upon npm-cache yesterday and I totally loved the idea. Then I made a few tests with showed quite promising results (npm i
took 5,7 mins npm-cache install npm
3,43). After further investigation I found out that the slowest part of npm-cache install npm
is extracting the archive (obviously) and I wondered if we could add a switch to store the dependencies without compressing them. Because in our case the compressed tar-file has 219MB and the uncompressed node_modules directory has 246MB (and normally disk space is not the problem in a CI-environment) and just resolving the hash and copying a directory should be blazingly fast.
We could probably do something like npm-cache install --deflate
in order to just copy the node_modules to the cacheDirectory without compressing them. And for resolving the dependencies we could scan the cacheDirectory for either #{HASH}.tar.gz files or directories with the specified hash.
What do you think about that idea? If you like it as well I'd be happy to contribute a PR.
Word on the street is that the NPM progress bar is slow. As far as I can tell it's not explicitly disabled by this library, and seeing as the progress is hidden perhaps the --progress=false
flag should be passed by default?
Hi, I just noticed this package is using the md5 of composer.json to see if there's any changes. This seems really wrong to me. Every project should commit his composer.lock and that file should be used, not the json.
npm has the --registry
flag to allow it to use a different registry, such as a company-internal registry. Does npm-cache allow some sort of passthrough of config options to the eventual package manager?
Hi there,
If one has a path containing a space, e.g. /Users/John/Projects/My project
, npm-cache cannot tar packages, like the exemple below.
[npm-cache] [INFO] [bower] archiving dependencies from /Users/John/Projects/My project
tar: could not chdir to '/Users/John/Projects/My'
[npm-cache] [ERROR] [bower] error tar-ing /Users/John/Projects/My project
[npm-cache] [ERROR] error installing dependencies
I don't get any extracted modules. Everything looks fine in the command prompt.
It clears the old folder but never creates a new one.
C:\TeamCity\buildAgent\work\6e8ae23de4bcc9d9\***SE_src\***.SKT\***.SKT.Api\***.SKT.Api\Application>npm-cache install npm
[npm-cache] [INFO] using C:\Users\****\.package_cache as cache directory
[npm-cache] [INFO] [npm] config file exists
[npm-cache] [INFO] [npm] cli exists
[npm-cache] [INFO] [npm] hash of C:\TeamCity\buildAgent\work\6e8ae23de4bcc9d9\***SE_src\***.SKT\***.SKT.Api\***.SKT.Api\Application\package.json: 6cd7ceb02dc80477b9c0084e50507a72
[npm-cache] [INFO] [npm] cache exists
[npm-cache] [INFO] [npm] clearing installed dependencies at C:\TeamCity\buildAgent\work\6e8ae23de4bcc9d9\***SE_src\***.SKT\***.SKT.Api\***.SKT.Api\Application\node_modules
[npm-cache] [INFO] [npm] ...cleared
[npm-cache] [INFO] [npm] extracting dependencies from C:\Users\****\.package_cache\npm\3.6.0\6cd7ceb02dc80477b9c0084e50507a72.tar.gz
[npm-cache] [INFO] [npm] done extracting
[npm-cache] [INFO] successfully installed all dependencies
I'm using Windows Server. Any ideas?
Edit: I can add that bower_components works fine.
I might also add I'm using Node 5.7.0 NPM 3.6
Added an issue in node-tar: isaacs/node-tar#67
Sometimes it is helpful to know if the install will occur from cache or not, for conditional processing.
Something like: npm-cache iscached npm
that returns an error status code upon cache-miss, or the hash string if the cache exists.
Thanks for sharing your work. It has proven to be very useful for speeding up my CI pipeline.
Some background - My development CI process currently works like this:
1. npm-cache install npm # Install (hopefully from cache)
2. npm update # Update any new dependencies from last cache
3. npm-cache install --forceRefresh npm # Re-archive the dependencies
In my package.json some of my internal project dependency versions are set to "latest" (it's continuous delivery-related decision...). Since the cache is keyed to a hash of the package.json file itself and not the latest versions of each dependencies, I need to run an update after the initial install to pick up any changes from the first cache.
It would be great if I could do this all in one shot. Something like an: npm-cache update npm
, where upon a cache-hit, it extracts, runs update, then re-archives (otherwise on cache-miss, just install as normal).
Or instead/additionally, a new command like npm-cache rearchive npm
would be helpful in speeding up my step 3 (I don't need to actually run an install again, only need to cache it).
I might be missing an undocumented feature, but wanted to ask about this regardless. Thanks again!
npm run <script>
where any <script>
that refers to the node_modules/.bin
generally has problems. e.g. mocha and grunt which maybe installed in the the relative node_modules
folder
its pretty common to have dependencies
and devDependencies
in package.json
and use NODE_ENV=production
or npm install --production
to omit devDependencies
. Would be great if npm-cache
would consider NODE_ENV
and/or have a --production
flag and work the same way.
Hi,
I have a package.json with a postinstall script to run bower-installer.
When I use npm-cache the postinstall script is not run and bower dependencies are missing.
I need bower-installer to move/rename some files from bower packages. So using bower is not an option.
package.json
{
"name": "my-package",
"private": true,
"version": "0.1.0",
"repository": {
"type": "git",
"url": "http://github.com/some/repo.git"
},
"dependencies": {
"bower-installer": "git+ssh://[email protected]:rebuy-de/bower-installer.git#master"
},
"scripts": {
"postinstall": "node_modules/bower-installer/bower-installer.js"
}
}
The project is in two different folders due to some concurrency issues with our automatic and manual deployment.
So when npm-cache is run from folder 1 everything is fine. When run from folder 2 it uses the cached files, but there the bower dependencies from bower-installer are missing.
Is there a possibility to force run postinstall scripts from the package.json?
Am I doing it wrong?
Some node modules have a binary compilation piece to the (phantomjs for example) which results in an "npm install" that is only compatible with the verion of node that compiled it. So if I try to build the same project with two different versions of node, the second one gets an error because it pulls in the node_modules of the first one.
I could manually script different --cacheDirectory locations for each node version, but it would be nice if the tool did that. Maybe under the --cacheDirectory should be a directory for the node version used to create it, then the md5sum. For example:
~/.package_cache/0.10.32/8a944a089acdeeafbf831f58d2813c85.tar.gz
~/.package_cache/0.12.4/8a944a089acdeeafbf831f58d2813c85.tar.gz
This seemed like a great idea. It uses cache instead of a million http requests. Thats great. But why does it clear all my existing node_modules and bower_components every time I run "npm-cache install"?
I thought the point is to reduce the amount of time npm install takes. Now it just deletes the dependencies and reinstalls the exact same ones.
Can you make it check if installed dependencies matches the hashtag for cached dependencies, and if so, stop execution? That would make it really speedy.
I'm trying to run npm-cache install npm from windows shell and get the following:
[npm-cache] [INFO] [npm] installed npm dependencies, now archiving
[npm-cache] [INFO] [npm] archiving dependencies from node_modules
tar (child): Cannot execute remote shell: No such file or directory
tar (child): C:\Users\username.package_cache\18c7544ea9d5afdef076583351ae2
7a2.tar.gz: Cannot open: I/O error
tar (child): Error is not recoverable: exiting now
stdout: Invalid argument
gzip: [npm-cache] [ERROR] [npm] error tar-ing node_modules
rm: no such file or directory: C:\Users\username.package_cache\18c7544ea9d5afde
f076583351ae27a2.tar.gz
[npm-cache] [ERROR] error installing dependencies
The same works fine on linux.
npm-cache -v 0.3.0
Please add 'typings' package manager support
jspm is yet another package manager coming to frontend repositories. It would be great to also have support for caching those files.
The folder to be cached can be extrapolated from require('package.json').jspm.directories.packages
with a fallback to require('package.json').jspm.directories.baseURL
with a fallback to ./jspm_packages
It is ideal for my use case to use a directory in node_modules
for the cache. I can understand why that doesn't work. I tried it anyway, and the process seemed to hang after installing, and it turns out that it was writing to node_modules
directory during that time. It filled up 10GB. I believe it would have gone on to fill up the entire disk if not stopped. I propose one of the following:
cacheDirectory
is or is nested in node_modules
, throw an error that cacheDirectory
cannot be under node_modules
cacheDirectory
to function under node_modules
by ignoring it when interacting with node_modules
.I would like to submit a PR for this at some point. I think the first thing that needs to happen is for there to be some tests so that things can be changed and refactored safely. For now, I just wanted this issue to be known in case it helps someone avoid filling up their disk!
Is there a way to do this? I'm getting bower errors, and its pretty time consuming to set them up for my cluster slave nodes. Just need to install npm stuff, so is there a way to skip bower?
Hi! we're using npm-cache
to improve the time our CI takes to lint/test our project. We have a npm-shrinkwrap.json
used to freeze dependencies versions.
What I found is that if I update only a devDependency, the hash generated is the same (because it uses npm-shrinkwrap.json to generate the hash and it didn't change).
I know that I can use --dev
to add devDependencies
to npm-shrinkwrap, but I like how it works by default.
What I think will be a good solution is to add package.json
's devDependencies
here (https://github.com/swarajban/npm-cache/blob/master/cacheDependencyManagers/npmConfig.js#L28) if npm-shrinkwrap.json
didn't have them.
Do you think this is a good idea? If so, I can try to do a PR adding this feature.
Example modules structure:
{
"dependencies": {
"b": "file:../b"
}
}
Trying to run npm-cache install
in /vagrant/a leads to infinite repeat of npm install:
[npm-cache] [INFO] using /home/vagrant/.package_cache as cache directory
[npm-cache] [INFO] [composer] Dependency config file /vagrant/a/composer.json does not exist. Skipping install
[npm-cache] [INFO] [npm] config file exists
[npm-cache] [INFO] [npm] cli exists
[npm-cache] [INFO] [npm] hash of /vagrant/a/package.json: df06dcab837ecf93c78fdb2b768c83dc
[npm-cache] [INFO] [npm] running [npm install]...
> [email protected] install /vagrant/a
> npm-cache install npm
[npm-cache] [INFO] using /home/vagrant/.package_cache as cache directory
...
Ubuntu 14.04
Node: v5.9.0
npm: 3.7.3
I'd be happy to provide a pull request if anybody could give me a hint about what can cause this. I couldn't find such behavior in sources.
laptop:myproject adambaker$ npm-cache install npm
[npm-cache] [INFO] using /Users/adambaker/.package_cache as cache directory
[npm-cache] [INFO] [npm] config file exists
[npm-cache] [INFO] [npm] cli exists
[npm-cache] [INFO] [npm] hash of /Users/adambaker/Development/...projectnamehere.../ember/package.json:88508382ccc5f6dba26f358779b3f3e2
[npm-cache] [INFO] [npm] cache exists
[npm-cache] [INFO] [npm] clearing installed dependencies at /Users/adambaker/Development/...projectnamehere.../ember/node_modules
[npm-cache] [INFO] [npm] ...cleared
[npm-cache] [INFO] [npm] extracting dependencies from /Users/adambaker/.package_cache/npm/2.12.1/88508382ccc5f6dba26f358779b3f3e2.tar.gz
[npm-cache] [ERROR] [npm] Error extracting /Users/adambaker/.package_cache/npm/2.12.1/88508382ccc5f6dba26f358779b3f3e2.tar.gz: Error: unexpected eof
[npm-cache] [ERROR] error installing dependencies
I have played around with this for a while. I think this might be failing when the cache is just too large?
Hi,
Thanks for your work on a great tool, it's taken part of our build process down from 75 seconds to 3, which is obviously wonderful.
The only downside that we have with it right now is the lack of cache invalidation, currently cache will grow until you manually clear it. It would be great if there could be an optional, built-in invalidation. The most sensible for me would be to keep the X most recently used, where X could be passed in. I think you would need to touch each zip before you use it, to ensure it remains freshest and doesn't get deleted each time. What do you think?
I'm sorry I'm not nodeJS dev, otherwise I would have a go and make a PR!
Thanks!
I read another issue about performance. Ran across this test:
https://gist.github.com/mafintosh/8102201
Is it possible to use tar-fs instead of node-tar if this test is really correct, and I'm not misunderstanding anything.
we're seeing this error every few days in our ci
[npm-cache] [INFO] [npm] extracting dependencies from /path/to/caches/npm/2.3.0/fe56bf256353b99775417bb4904da265.tar.gz
gzip: stdin: decompression OK, trailing garbage ignored
tar: Child returned status 2
tar: Error is not recoverable: exiting now
[npm-cache] [ERROR] [npm] error untar-ing /path/to/caches/npm/2.3.0/fe56bf256353b99775417bb4904da265.tar.gz
[npm-cache] [ERROR] error installing dependencies
Fatal error: local() encountered an error (return code 1) while executing 'npm-cache install --cacheDirectory /path/to/caches/'
it seems like file is corrupted during download?
npm-cache should checksum downloaded dependencies?
$ cat package.json
{
"name": "machtfit",
"version": "1.0.0",
"devDependencies": {
"gulp": "^3.8.9",
"gulp-coffee": "^2.2.0",
"gulp-coffeelint": "^0.4.0",
"gulp-plumber": "^0.6.6",
"gulp-sourcemaps": "^1.2.4",
"gulp-util": "^3.0.1"
},
"private": true,
"repository": {
"type": "git",
"url": "https://github.com/machtfit/machtfit-oscar.git"
}
}
$ node --version
v0.12.2
$ npm --version
2.7.4
$ npm-cache --version
0.3.0
$ ls node_modules
ls: cannot access node_modules: No such file or directory
$ time npm-cache --cacheDirectory /home/django/.package_cache-0.12 install npm
real 0m0.118s
user 0m0.104s
sys 0m0.016s
$ ls node_modules
ls: cannot access node_modules: No such file or directory
$ time npm-cache -c /home/django/.package_cache-0.12 install npm
real 0m0.111s
user 0m0.100s
sys 0m0.016s
$ ls node_modules
ls: cannot access node_modules: No such file or directory
npm install
works fine:
$ time npm install --quiet
[email protected] node_modules/gulp-plumber
└── [email protected] ([email protected], [email protected])
[email protected] node_modules/gulp-sourcemaps
├── [email protected]
├── [email protected]
├── [email protected] ([email protected], [email protected])
├── [email protected] ([email protected], [email protected])
└── [email protected] ([email protected], [email protected])
[email protected] node_modules/gulp-coffee
├── [email protected]
├── [email protected] ([email protected], [email protected])
├── [email protected]
└── [email protected] ([email protected])
[email protected] node_modules/gulp-util
├── [email protected]
├── [email protected]
├── [email protected]
├── [email protected]
├── [email protected]
├── [email protected]
├── [email protected]
├── [email protected]
├── [email protected]
├── [email protected] ([email protected], [email protected])
├── [email protected] ([email protected], [email protected], [email protected], [email protected], [email protected])
├── [email protected] ([email protected], [email protected], [email protected], [email protected], [email protected], [email protected], [email protected], [email protected], [email protected])
├── [email protected] ([email protected], [email protected])
├── [email protected] ([email protected], [email protected])
└── [email protected] ([email protected])
[email protected] node_modules/gulp-coffeelint
├── [email protected] ([email protected], [email protected])
├── [email protected] ([email protected], [email protected])
├── [email protected]
└── [email protected] ([email protected], [email protected], [email protected], [email protected], [email protected], [email protected])
[email protected] node_modules/gulp
├── [email protected]
├── [email protected]
├── [email protected]
├── [email protected]
├── [email protected] ([email protected])
├── [email protected] ([email protected])
├── [email protected]
├── [email protected] ([email protected], [email protected], [email protected], [email protected], [email protected])
├── [email protected]
├── [email protected] ([email protected], [email protected], [email protected])
├── [email protected] ([email protected], [email protected], [email protected], [email protected])
└── [email protected] ([email protected], [email protected], [email protected], [email protected], [email protected], [email protected], [email protected], [email protected])
real 0m4.761s
user 0m4.140s
sys 0m0.528s
$ ls node_modules
gulp gulp-coffee gulp-coffeelint gulp-plumber gulp-sourcemaps gulp-util
I checked the archive and it's ok, and v0.4.10 do not have this bug. So the problem is due to using Decompress package. I suggest to switch back to targz or just call tar cli.
Windows doesn't have a tar exe so an error is thrown. This module is very useful and it would be nice to see it work in Windows.
I'm wondering is it possible to install single package? If not, will you plan add such feature? It would be very helpful.
Which branch should I create pull requests against? It looks like develop
is work in progress (portable tar
etc.) - which one should I use for a PR fixing some inconsistencies?
Hey, the way my project is setup i have node_modules
at cwd
, and jspm_packages
in my static folder which isn't. Kind of like a maven setup.
Does this lib have any options for that type of setup?
Thank you
Trying to run the npm-cache
command back to back results in an error coming up from RimRaf saying that a directory is not empty.
>npm-cache install --cacheDirectory "C:\Users\Andrew Stoker\AppData\Roaming\npm-cache" npm
[npm-cache] [INFO] using C:\Users\Andrew Stoker\AppData\Roaming\npm-cache as cache directory
[npm-cache] [INFO] [npm] config file exists
[npm-cache] [INFO] [npm] cli exists
[npm-cache] [INFO] [npm] hash of C:\Workflow\Branches\CentralPrototype\Projects\Source\WebApplication\package.json: bf9c78e40ae0c10c7636ed4d576043fa
[npm-cache] [INFO] [npm] cache exists
[npm-cache] [INFO] [npm] clearing installed dependencies at C:\Workflow\Branches\CentralPrototype\Projects\Source\WebApplication\node_modules
C:\Users\Andrew Stoker\AppData\Roaming\npm\node_modules\npm-cache\node_modules\rimraf\rimraf.js:304
throw er
^
Error: ENOTEMPTY: directory not empty, rmdir 'C:\Workflow\Branches\CentralPrototype\Projects\Source\WebApplication\node_modules\bower-installer\node_modules\glob'
at Error (native)
at Object.fs.rmdirSync (fs.js:806:18)
at rmkidsSync (C:\Users\Andrew Stoker\AppData\Roaming\npm\node_modules\npm-cache\node_modules\rimraf\rimraf.js:334:11)
at rmdirSync (C:\Users\Andrew Stoker\AppData\Roaming\npm\node_modules\npm-cache\node_modules\rimraf\rimraf.js:324:7)
at rimrafSync (C:\Users\Andrew Stoker\AppData\Roaming\npm\node_modules\npm-cache\node_modules\rimraf\rimraf.js:295:9)
at C:\Users\Andrew Stoker\AppData\Roaming\npm\node_modules\npm-cache\node_modules\rimraf\rimraf.js:332:5
at Array.forEach (native)
at rmkidsSync (C:\Users\Andrew Stoker\AppData\Roaming\npm\node_modules\npm-cache\node_modules\rimraf\rimraf.js:331:26)
at rmdirSync (C:\Users\Andrew Stoker\AppData\Roaming\npm\node_modules\npm-cache\node_modules\rimraf\rimraf.js:324:7)
at rimrafSync (C:\Users\Andrew Stoker\AppData\Roaming\npm\node_modules\npm-cache\node_modules\rimraf\rimraf.js:295:9)
Third attempt:
>npm-cache install --cacheDirectory "C:\Users\Andrew Stoker\AppData\Roaming\npm-cache" npm
[npm-cache] [INFO] using C:\Users\Andrew Stoker\AppData\Roaming\npm-cache as cache directory
[npm-cache] [INFO] [npm] config file exists
[npm-cache] [INFO] [npm] cli exists
[npm-cache] [INFO] [npm] hash of C:\Workflow\Branches\CentralPrototype\Projects\Source\WebApplication\package.json: bf9c78e40ae0c10c7636ed4d576043fa
[npm-cache] [INFO] [npm] cache exists
[npm-cache] [INFO] [npm] clearing installed dependencies at C:\Workflow\Branches\CentralPrototype\Projects\Source\WebApplication\node_modules
C:\Users\Andrew Stoker\AppData\Roaming\npm\node_modules\npm-cache\node_modules\rimraf\rimraf.js:304
throw er
^
Error: ENOTEMPTY: directory not empty, rmdir 'C:\Workflow\Branches\CentralPrototype\Projects\Source\WebApplication\node_modules\es6-module-loader\test'
at Error (native)
at Object.fs.rmdirSync (fs.js:806:18)
at rmkidsSync (C:\Users\Andrew Stoker\AppData\Roaming\npm\node_modules\npm-cache\node_modules\rimraf\rimraf.js:334:11)
at rmdirSync (C:\Users\Andrew Stoker\AppData\Roaming\npm\node_modules\npm-cache\node_modules\rimraf\rimraf.js:324:7)
at rimrafSync (C:\Users\Andrew Stoker\AppData\Roaming\npm\node_modules\npm-cache\node_modules\rimraf\rimraf.js:295:9)
at C:\Users\Andrew Stoker\AppData\Roaming\npm\node_modules\npm-cache\node_modules\rimraf\rimraf.js:332:5
at Array.forEach (native)
at rmkidsSync (C:\Users\Andrew Stoker\AppData\Roaming\npm\node_modules\npm-cache\node_modules\rimraf\rimraf.js:331:26)
at rmdirSync (C:\Users\Andrew Stoker\AppData\Roaming\npm\node_modules\npm-cache\node_modules\rimraf\rimraf.js:324:7)
at rimrafSync (C:\Users\Andrew Stoker\AppData\Roaming\npm\node_modules\npm-cache\node_modules\rimraf\rimraf.js:295:9)
The directory it's failing on is not consistent, which leads me to believe that it's doing a little bit more deleting every time.
Running v. 0.5.3 of npm-cache.
It would be really great if npm-cache could cache packages not only locally but at some server. Then we could use it as corporate package proxy server.
We use this package on our build server which runs many simultaneous builds at once. Today, we upgraded npm (which makes npm-cache use a new cache directory), and experienced that the new archive created by npm-cache was invalid. This caused subsequent runs of npm-cache install
to fail, since it tried to use an invalid archive.
This error seems to be caused by npm-cache archiving directly to the cache directory, which is the same for all instances of npm-cache with the same package.json file. I think the error occurred because two instances of npm-cache tried to write to the same file simultaneously, causing it to become invalid.
To fix this, npm-cache should create the archive in some unique location first, and then move it to the cache directory after it has completed creating the archive.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.