ymedlop / npm-cache-resource Goto Github PK
View Code? Open in Web Editor NEWConcourse resource for caching dependencies
Home Page: https://ymedlop.github.io/npm-cache-resource
License: MIT License
Concourse resource for caching dependencies
Home Page: https://ymedlop.github.io/npm-cache-resource
License: MIT License
We will be updating the app version in the package.json file for each release but don't want to rebuild the node modules if there are no changes to the dependencies.
My idea is to check whether the dependency objects differ and only rebuild the modules if they do. This way other changes to the package.json file won't cause the cache to be invalidated.
As like for the other releases of the resource, please provide a release on Docker hub that packages with node-12.
Hi there, we were using your docker image at the latest tag, since the commits that bumped the node version to 9 all our builds broke.
The build log is:
npm-repo-cache
Identity added: /tmp/git-resource-private-key (/tmp/git-resource-private-key)
Cloning into '/tmp/git-resource-destination.NCFmLo'...
git lfs fetch
git lfs checkout
95d574b [MOD] installingSupplier improvements
git submodule foreach git lfs fetch && git lfs checkout
Identity added: /tmp/git-resource-private-key (/tmp/git-resource-private-key)
/opt/resource/in: line 76: syntax error: unexpected end of file
Any idea why this is suddenly happening?
Thanks
I'm making a proof of concept of concourse and I have migrated the original app code to a new git repo (from gitlab onprem to gitlab.com)
In new repo (gitlab.com) I have modified .git/config and .gitmodules to reflect the new repo URI.
This change works great with git resource, it's cloned right. Also npm-cache resource is checked fine but when it executes get-resource it tries to fetch the old uri wich is not reachable.
Here is the processes running on the npm-cache-resource container when it's pointing to the old git uri
PID USER TIME COMMAND
1 root 0:00 /tmp/garden-init
6 root 0:00 bash /opt/resource/in /tmp/build/get
12 root 0:00 {in} /bin/bash /opt/resource/git/in /tmp/git-resource-destination.geNhmk
21 root 0:00 ssh-agent
110 root 0:00 {in} /bin/bash /opt/resource/git/in /tmp/git-resource-destination.geNhmk
114 root 0:00 git submodule update --init --no-fetch --recursive .contracts
115 root 0:00 {git-submodule} /bin/sh /usr/libexec/git-core/git-submodule update --init --no-fetch --recursive .contracts
138 root 0:00 {git-submodule} /bin/sh /usr/libexec/git-core/git-submodule update --init --no-fetch --recursive .contracts
139 root 0:00 {git-submodule} /bin/sh /usr/libexec/git-core/git-submodule update --init --no-fetch --recursive .contracts
140 root 0:00 git submodule--helper update-clone .contracts
141 root 0:00 /usr/libexec/git-core/git submodule--helper clone --path .contracts --name .contracts --url git@<redacted-onprem-gitlab>:lab/marketplace/contracts.git
142 root 0:00 /usr/libexec/git-core/git clone --no-checkout --separate-git-dir /tmp/git-resource-destination.geNhmk/.git/modules/.contracts -- git@<redacted-onprem-gitlab>:lab/marketplace/contracts.git /tmp/git-resource-destination.geNhmk/.contracts
143 root 0:00 /usr/bin/ssh git@<redacted-onprem-gitlab> git-upload-pack 'lab/marketplace/contracts.git'
Pipeline resources declaration
resources:
- name: source-code
type: git
check_every: 5m
source: &repo-source
uri: ((git-uri))
branch: develop
private_key: |
((git_ssh_privatekey))
- name: dependency-cache
type: npm-cache
source:
<<: *repo-source
paths:
- package.json
tasks declaration:
jobs:
- name: install dependencies
plan:
- get: source-code -> works fine
trigger: true
version: every
- get: dependency-cache -> tries to clone old git URI
getting git-resource works fine but getting npm-cache resource is pointing to old git repo but is configured in the same way than the git-resource using anchors.
Identity added: /tmp/git-resource-private-key (<redacted>@ip-10-1-28-86.eu-west-1.compute.internal)
Cloning into '/tmp/git-resource-destination.geNhmk'...
6c63a87 Merge branch 'develop' into feature/<redacted>-570
Submodule '.contracts' (git@<redacted-old-repo>/marketplace/contracts.git) registered for path '.contracts'
Cloning into '/tmp/git-resource-destination.geNhmk/.contracts'...
fatal: Could not read from remote repository.
Please make sure you have the correct access rights
and the repository exists.
fatal: clone of 'git@<redacted-old-repo>:<project>/marketplace/contracts.git' into submodule path '/tmp/git-resource-destination.geNhmk/.contracts' failed
Failed to clone '.contracts'. Retry scheduled
Cloning into '/tmp/git-resource-destination.geNhmk/.contracts'...
fatal: Could not read from remote repository.
Please make sure you have the correct access rights
and the repository exists.
fatal: clone of 'git@<redacted-old-repo>:<project>/marketplace/contracts.git' into submodule path '/tmp/git-resource-destination.geNhmk/.contracts' failed
Failed to clone '.contracts' a second time, aborting
Getting git-resource
Identity added: /tmp/git-resource-private-key (<redacted>@ip-10-1-28-86.eu-west-1.compute.internal)
Cloning into '/tmp/build/get'...
c017a54 Update .gitmodules to .contracts in gitlab.com
Submodule '.contracts' (git@<new-repo>:<project>-santander/shopping/contracts.git) registered for path '.contracts'
Cloning into '/tmp/build/get/.contracts'...
Submodule path '.contracts': checked out '1db867334ae3432ecd3f41e6549442effa28a606'
How can npm-cache-resource try to clone different repo than git-resource using the same parameters?
Hi,
I'm trying to implement this npm-cache-resource. I'm using private repository so that I added a key to it. This is the only difference between your example and my pipeline. However my pipeline get's stucked after everything is ok in the preparing build section. Also I tried out your example code and it's behaviour is the same. pending state stucked.
Do you have any advice or experience?
Thanks in advance
Zsolt
ps.: working behind proxy on company network.
I've followed the instructions, and while it works, at times builds fail with no output for the get cache.
Hi there, it seems like the docker image doesn't let installing https://www.npmjs.com/package/sharp
From their docs it seems like it needs to run
but something doesn't work quite right
Identity added: /tmp/git-resource-private-key (/tmp/git-resource-private-key)
Cloning into '/tmp/git-resource-destination.EgIMgN'...
git lfs fetch
git lfs checkout
434c9d5 revert all commits on npm-cache
git submodule foreach git lfs fetch && git lfs checkout
Identity added: /tmp/git-resource-private-key (/tmp/git-resource-private-key)
npm WARN notice [SECURITY] ws has the following vulnerability: 1 high. Go here for more details: https://nodesecurity.io/advisories?search=ws&version=3.0.0 - Run `npm i npm@latest -g` to upgrade your npm version, and then `npm audit` to get more info.
npm WARN notice [SECURITY] lodash has the following vulnerability: 1 low. Go here for more details: https://nodesecurity.io/advisories?search=lodash&version=4.11.1 - Run `npm i npm@latest -g` to upgrade your npm version, and then `npm audit` to get more info.
npm WARN notice [SECURITY] lodash has the following vulnerability: 1 low. Go here for more details: https://nodesecurity.io/advisories?search=lodash&version=3.10.1 - Run `npm i npm@latest -g` to upgrade your npm version, and then `npm audit` to get more info.
npm WARN notice [SECURITY] open has the following vulnerability: 1 critical. Go here for more details: https://nodesecurity.io/advisories?search=open&version=0.0.5 - Run `npm i npm@latest -g` to upgrade your npm version, and then `npm audit` to get more info.
npm WARN notice [SECURITY] debug has the following vulnerability: 1 low. Go here for more details: https://nodesecurity.io/advisories?search=debug&version=2.2.0 - Run `npm i npm@latest -g` to upgrade your npm version, and then `npm audit` to get more info.
npm WARN notice [SECURITY] https-proxy-agent has the following vulnerability: 1 high. Go here for more details: https://nodesecurity.io/advisories?search=https-proxy-agent&version=1.0.0 - Run `npm i npm@latest -g` to upgrade your npm version, and then `npm audit` to get more info.
npm WARN notice [SECURITY] lodash has the following vulnerability: 1 low. Go here for more details: https://nodesecurity.io/advisories?search=lodash&version=2.4.2 - Run `npm i npm@latest -g` to upgrade your npm version, and then `npm audit` to get more info.
npm WARN notice [SECURITY] tunnel-agent has the following vulnerability: 1 moderate. Go here for more details: https://nodesecurity.io/advisories?search=tunnel-agent&version=0.4.3 - Run `npm i npm@latest -g` to upgrade your npm version, and then `npm audit` to get more info.
> [email protected] install /tmp/git-resource-destination.EgIMgN/node_modules/sharp
> (node install/libvips && node install/dll-copy && prebuild-install) || (node-gyp rebuild && node install/dll-copy)
ERR! sharp Use with musl libc requires manual installation of libvips >= 8.6.1
ERR! sharp Please see http://sharp.pixelplumbing.com/page/install
make: Entering directory '/tmp/git-resource-destination.EgIMgN/node_modules/sharp/build'
TOUCH Release/obj.target/libvips-cpp.stamp
CXX(target) Release/obj.target/sharp/src/common.o
../src/common.cc:25:22: fatal error: vips/vips8: No such file or directory
#include <vips/vips8>
^
compilation terminated.
make: *** [sharp.target.mk:120: Release/obj.target/sharp/src/common.o] Error 1
make: Leaving directory '/tmp/git-resource-destination.EgIMgN/node_modules/sharp/build'
gyp ERR! build error
gyp ERR! stack Error: `make` failed with exit code: 2
gyp ERR! stack at ChildProcess.onExit (/usr/lib/node_modules/npm/node_modules/npm-lifecycle/node_modules/node-gyp/lib/build.js:258:23)
gyp ERR! stack at ChildProcess.emit (events.js:180:13)
gyp ERR! stack at Process.ChildProcess._handle.onexit (internal/child_process.js:209:12)
gyp ERR! System Linux 4.4.0-112-generic
gyp ERR! command "/usr/bin/node" "/usr/lib/node_modules/npm/node_modules/npm-lifecycle/node_modules/node-gyp/bin/node-gyp.js" "rebuild"
gyp ERR! cwd /tmp/git-resource-destination.EgIMgN/node_modules/sharp
gyp ERR! node -v v9.11.1
gyp ERR! node-gyp -v v3.6.2
gyp ERR! not ok
npm WARN optional SKIPPING OPTIONAL DEPENDENCY: [email protected] (node_modules/fsevents):
npm WARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for [email protected]: wanted {"os":"darwin","arch":"any"} (current: {"os":"linux","arch":"x64"})
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! [email protected] install: `(node install/libvips && node install/dll-copy && prebuild-install) || (node-gyp rebuild && node install/dll-copy)`
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the [email protected] install script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.
npm ERR! A complete log of this run can be found in:
npm ERR! /root/.npm/_logs/2018-06-15T12_01_10_166Z-debug.log
My project has a private git repo in package.json(E.g git+ssh://git.example.com/team/base.git)。
When Concourse run with this package.json, I got a error message in dev-cache
-> dev-module
:
dentity added: /tmp/git-resource-private-key (/tmp/git-resource-private-key)
Cloning into '/tmp/git-resource-destination.AniNbp'...
warning: templates not found /usr/share/git-core/templates
git lfs fetch
git lfs checkout
c378133 Merge branch 'develop' of team/app into develop
git: 'submodule' is not a git command. See 'git --help'.
Part of the pipeline.yml:
resource_types:
- name: yarn
type: docker-image
source:
repository: ymedlop/npm-cache-resource
tag: 8
resources:
- name: dev-module
type: yarn
source:
<<: *dev-source
yarn-support: true
paths:
- yarn.lock
- name: dev-repo
type: git
source: &dev-source
uri: ssh://[email protected]/team/app.git
branch: develop
private_key: ((repo-key))
jobs:
- name: dev-cache
plan:
- get: dev-repo
- get: dev-module
- name: dev-build
- get: dev-repo
trigger: true
passed: [dev-cache]
- get: dev-module
passed: [dev-cache]
......
......
Is there any way to get it works? thanks!
We are from here #51.
Hi,
I'm using the github-pullrequest-resource, however this resource picks up commits to master via the git-resource instead. Is there a way to make this cache from the pull request resource instead?
Hi, we get error on the log such as:
Command failed: autoreconf -ivf
/bin/sh: autoreconf: not found
at ChildProcess.exithandler (child_process.js:275:12)
at emitTwo (events.js:126:13)
at ChildProcess.emit (events.js:214:7)
at maybeClose (internal/child_process.js:925:16)
at Socket.stream.socket.on (internal/child_process.js:346:11)
at emitOne (events.js:116:13)
at Socket.emit (events.js:211:7)
at Pipe._handle.close [as _onclose] (net.js:554:12)
Command failed: autoreconf -ivf
/bin/sh: autoreconf: not found
at ChildProcess.exithandler (child_process.js:275:12)
at emitTwo (events.js:126:13)
at ChildProcess.emit (events.js:214:7)
at maybeClose (internal/child_process.js:925:16)
at Socket.stream.socket.on (internal/child_process.js:346:11)
at emitOne (events.js:116:13)
at Socket.emit (events.js:211:7)
at Pipe._handle.close [as _onclose] (net.js:554:12)
Shouldn't the build fail?
Would prefixing all the shell scripts with
set -euf -o pipefail
help?
The problem is that later on other (deploy) script fail for no good reason, clearly cos the cache isn't working well
We have provided the following registry params as per the documentation. We see the following error when the script gets executed -
/opt/resource/in: line 57: [: -z¡n: unary operator expected
Link to source - https://github.com/ymedlop/npm-cache-resource/blob/master/assets/in#L57
Registry params:
Any ideas on what to try?
Hi,
I was going over this code trying to figure out how to set this up so only packages under a certain scope are pulled from a private registry but all other packages come from another registry. Is this possible?
Normally my injected .npmrc file looks something like this:
# .npmrc
_auth = [...]
always-auth = true
email = [...]
@myscope:registry=http://[...]/foo # <--- Notice these two
registry = http://[...]/bar # <--- are different
//[...]/foo/:_password=[...]
//[...]/foo/:username=[...]
//[...]/foo/:email=[...]
//[...]/foo/:always-auth=true
And this works when I inject it manually... But I was hoping to use npm-cache-resource.
If this is possible, could you provide an example of how to do this?
Thank you
Can't authorize when using base64 encode user:password format with default npm registry https://registry.npmjs.org. It fails when it tries to install the first module "q" which is public.
I removed actual token when logging bug
Output:
Set token in our .npmrc file: XXXXXXXXXMYTOKEN
always-auth=true
_auth=XXXXXXXXXMYTOKEN
npm ERR! Linux 4.4.0-38-generic
npm ERR! argv "/usr/bin/node" "/usr/bin/npm" "install" "--quiet"
npm ERR! node v6.9.1
npm ERR! npm v3.10.8
npm ERR! code E401
npm ERR! Registry returned 401 for GET on https://registry.npmjs.org/q
npm ERR!
npm ERR! If you need help, you may report this error at:
npm ERR! https://github.com/npm/npm/issues
npm ERR! Please include the following file with any support request:
npm ERR! /tmp/git-resource-destination.ipkaFA/npm-debug.log
This requires an ssh key to be made available to npm.
Alpine node 8 has yarn already installed!!
I want to use it in a CI develop environment which is offline.
But now I have a refuse that in my opinion,this images can provide a container which has installed the global packages needed.So may I build the container with all the packages or should separate the folder?
We are coming from here #51. We have a problem to build this version:
Step 12/16 : RUN mkdir -p /opt/resource/git && wget https://github.com/concourse/git-resource/archive/master.zip -O /opt/resource/git/git-resource.zip && unzip /opt/resource/git/git-resource.zip -d /opt/resource/git && mv /opt/resource/git/git-resource-master/assets/* /opt/resource/git && rm -r /opt/resource/git/git-resource.zip /opt/resource/git/git-resource-master && sed -i '/git lfs/s/^/echo /' /opt/resource/git/in
---> Running in ccfda34cfd60
Connecting to github.com (140.82.113.4:443)
ssl_client: github.com: TLS connect failed
The current implementation of this resource assumes package.json
lives in the root directory (see https://github.com/ymedlop/npm-cache-resource/blob/master/assets/in#L30), but this is not always the case.
We have a npm project that lives in a subdirectory (since it's in a monorepo). Would you be interested in supporting an additional optional parameter that specifies where the package.json
lives?
I can draft a PR if you guys think this is interesting.
Hi, almost a year later I am still seeing weird behaviour with our pipelines.
I am attaching a pic of how most of our pipelines look like:
Does it look good to you?
I feel like often the test or deploy-dev tasks 'skips' the npm-cache and uses stale node modules.
the repo cache looks at both package and package-lock files
- name: npm-repo-cache
type: npm-cache
source:
<<: *repo-source
paths:
- package.json
- package-lock.json
First of all: thanks for this resource. It's been really helpful for us.
I'd like to re-use this resource in order to cache some other stuff, but I am having a hard time to understand how this resource actually works. I would really appreciate if you could explain it in a little more detail as to what actually happens when I am using this resource. For example: how is the actual workflow here, how do we have the actual cache, where is it stored etc?
Thanks again and I hope you can answer this. Sorry for me acting stupid here :)
Builds can even be faster if it would do npm ci
instead of npm install
. So this line
https://github.com/ymedlop/npm-cache-resource/blob/master/assets/in#L64
Should be changed, see background information at:
http://blog.npmjs.org/post/171556855892/introducing-npm-ci-for-faster-more-reliable
I'm trying to use the 'project-path' setting and it seems to ignore it and the checking never triggers. Here's the excerpt from my YAML file.
mhart/alpine-node
has various images for the versions of node available.
As you know, npm has different behaviour across versions.
Full install built with npm:
Hi,
Just trying this out. It appears that the branch specification is not being respected from the anchor - it seems always to pull dependencies from master, regardless of the specified branch.
Is this a known issue?
Thanks,
Tom
Bower support defaults to always attempt to run bower install. The conditional check for this feature has a bug.
https://github.com/ymedlop/npm-cache-resource/blob/master/assets/in#L81
if [ "BOWER_SUPPORT" != "false" ]; then
node_modules/.bin/bower install --allow-root >&2
mv bower_components "$DEST_DIR/"
fi
should be
if [ "$BOWER_SUPPORT" != "false" ]; then
node_modules/.bin/bower install --allow-root >&2
mv bower_components "$DEST_DIR/"
fi
If I understand correctly, this cache resource does checkout the source repo, run npm install and provides the node_modules folder. As far as I can see, this is only useful for avoiding to re-download node modules as part of the same pipeline execution.
However it does not cache npm modules in between pipeline executions, right?
It'd be awesome if we could have a way to incrementally cache the global npm/yarn cache and restore that before build execution. Do you see any way to make that happen?
My pipeline seems to get stuck in "Discovering any new versions of node_modules" where node_modules is the resource:
resource_types:
# https://github.com/ymedlop-sandbox/npm-cache-resource
- name: node_modules-cache-resource-type
type: docker-image
#source: {repository: ymedlop/npm-cache-resource, tag: latest}
source:
repository: {{docker-registry-npm-cache}}
tag: 'latest'
insecure_registries:
- {{docker-registry}}
resources:
#Git repo containing the client source:
- name: client-repo
type: git
source: &client-repo-source #YAML-Anchor for npm cache
uri: [email protected]:blablabla/client-devops.git
branch: frontend
private_key: {{private-key}}
# a resource caching the dependencies listed in the source repository
- name: node_modules
type: node_modules-cache-resource-type # as defined above
source:
<<: *client-repo-source # the source is the same as the corresponding git resource ...
paths: # ... except that it's only interested in files listing dependencies
- package.json
jobs:
#Cache client NPM packages:
- name: cache
plan:
- get: client-repo
trigger: true
- get: node_modules
Hi, thank you for providing this resource. It has worked great for our Concourse pipelines so far!
My question is does this support access to a bower_components
directory?
So currently I have a "postinstall": "node_modules/.bin/bower install --allow-root"
script located in my package.json which gets called after npm install is ran. I see the logs in my concourse npm-repo-cache job that it does run bower install and that it pulls down all my Angular dependencies.
I was hoping it would be as easy as doing something similar with node_modules "mv cache/node_modules src"
and I have tried "mv cache/bower_components src"
but it does not seem to work. Where does the cache folder exist exactly?
Hi,
Just a small issue, can you fix it so that we don't display the token in the output of this resource. Some pipelines are public and we shouldn't have the token displayed :)
When building a project with node-sass it throws an exception
Node Sass could not find a binding for your current environment: Linux 64-bit with Node.js 8.x
Found bindings for the following environments:
- Linux/musl 64-bit with Node.js 8.x
This usually happens because your environment has changed since running `npm install`.
Run `npm rebuild node-sass --force` to build the binding for your current environment.
To workaround this, I did a npm rebuild node-sass --force
before the npm run build
But in this case it would be preferable to just npm i && npm run build
There is another way to try?
This is in a way a follow up of #15. Yarn has a flag for yarn install
called --frozen-lockfile
which has the description from the docs:
Don’t generate a yarn.lock lockfile and fail if an update is needed.
In a sense it is similar to npm ci
as opposed to npm install
. It would be nice if this could be supported by the resource through yarn-frozen-lockfile
flag for the resource source config.
I am having some trouble getting a private Gemfury repository to cache the npm install.
What I am seeing is that the .npmrc
file contains only this
//npm-proxy.fury.io/USERNAME/:_authToken="XXXXXXXXXXXXXX"
Which does not work - when I run an npm config set registry XXXXXX
to get this...
//npm-proxy.fury.io/USERNAME/:_authToken="XXXXXXXXXXXXXX"
registry=https://npm-proxy.fury.io/USERNAME/
It works!
I am not sure of the impact to others of adding the npm config set registry
command in the in
asset but will fork this project and give it a go to see if it fixes and push back a pull request.
Just wondering if others have had similar issues?
I can't see an option to install private node modules. This would require an npm login before the npm install is carried out by the resource.
With the last concourse image version. I have to do a lot of changes to work on my localhost
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.