vercel / pkg Goto Github PK

Upon specifying -t node7-linux-armv7 as the only target on a MacOS host, pkg downloads the armv7 base binary, but then starts to (attempt to) build a macos-armv7 base binary from source. This is illogical and unnecessary. I stopped it after noticing what it was doing, so I don't know if it would subsequently attempt to make a win-armv7 binary as well.
Transcript of attempted build:

$ pkg . --targets node7-linux-armv7
> Fetching base Node.js binaries to: ~/.pkg-cache
  fetched-v7.6.0-linux-armv7   [====================] 100%
  fetched-v7.6.0-macos-armv7   [====================] 100%
> Error! 404 Not Found
  https://github.com/zeit/pkg-fetch/releases/download/v1.9/uploaded-v1.9-node-v7.6.0-macos-armv7
> Asset 'v1.9/uploaded-v1.9-node-v7.6.0-macos-armv7' not found by direct link
> Not found in GitHub releases:
  {"tag":"v1.9","name":"uploaded-v1.9-node-v7.6.0-macos-armv7"}
> Building base binary from source:
  built-v7.6.0-macos-armv7
> Cloning Node.js repository from GitHub...
  git                          [========            ] 40%^C

Update: it does not attempt to make a macos-armv7 target when running on an actual linux-armv7 host (i.e., a Raspberry Pi 3). So it's not using the default linux,macos,win targets, but apparently trying to build a host-native version as well as the specified target.
Update 2: Upon attempting a node6-macos-x64 build on the armv7 host, the same pattern followed where pkg attempted to build a host-native version of the target arch (in this case, node6-linux-x64) but crashed in the attempt:

$ pkg . -t node6-macos-x64
> Fetching base Node.js binaries to: ~/.pkg-cache
  fetched-v6.10.0-macos-x64    [====================] 100%
  fetched-v6.10.0-linux-x64    [====================] 100%
/home/pi/.pkg-cache/v1.9/fetched-v6.10.0-linux-x64: 1: /home/pi/.pkg-cache/v1.9/fetched-v6.10.0-linux-x64: cannot create $�x@��x@@%@@@@@�@@@@L�kL�k �k����(s� 0�k0��0��0@@DDP�td�O住住����Q�td/lib64/ld-linux-x86-64.so.2GNUGNU�N�ޞ:^�m�zw���3�m���A�3G
       : Directory nonexistent
/home/pi/.pkg-cache/v1.9/fetched-v6.10.0-linux-x64: 1: /home/pi/.pkg-cache/v1.9/fetched-v6.10.0-linux-x64: �ELF: not found
/home/pi/.pkg-cache/v1.9/fetched-v6.10.0-linux-x64: 4: /home/pi/.pkg-cache/v1.9/fetched-v6.10.0-linux-x64: Syntax error: ")" unexpected
> Error! Error: Not able to compile for 'x64' here
    at Socket.<anonymous> (/home/pi/.npm/lib/node_modules/pkg/lib-es5/producer.js:43:25)
    at emitOne (events.js:90:13)
    at Socket.emit (events.js:182:7)
    at onwriteError (_stream_writable.js:313:10)
    at onwrite (_stream_writable.js:331:5)
    at WritableState.onwrite (_stream_writable.js:89:5)
    at fireErrorCallbacks (net.js:448:13)
    at Socket._destroy (net.js:486:3)
    at WriteWrap.afterWrite (net.js:769:10)

Scoped modules (particularly modules containing slash) are handled incorrectly by pkg

I used the express example of the pkg project to test the animate.css module I used in my project, installed by: npm i animate.css -S, and when I ran the command: pkg package.json, it came out the result:

> Error! Unexpected character '@' (1:0)
  /home/test/pkg-master/examples/express/node_modules/animate.css/animate.css

Maybe it isn't the place to ask question, but I tried according to the readme of pkg， and with no luck. It didn't work, whether I modified the package.json by adding the assets part or other way, Does anybody know how to solve this, tell me please, thx.

#60

Practically, the only time you want to distribute a commercial app and use some sort of code protection is with desktop apps. So it would make a lot of sense to combine this with Electron. Are there any plans for that?

Hi, all.

I think that zeit's pkg is cool solution to everyone who want to simplify their nodejs resources to one executable.
Thanks for efforts of pkg maintainers in zeit :)

I found fs.readFile with { encoding: 'utf8' } as options can't read anything in /snapshot, but
but in fs.readFile with only "utf8" string as options, there is not such a problem.
It seems that in executable, fs.readFile does not handle encoding attribute in option of object type.

Interestingly, fs.readFileSync works well in both case.

The following is test code.

const path = require('path')
const fs = require('fs')

const meta = path.join(__dirname, './meta.txt')

let syncdata = fs.readFileSync(meta, { encoding: 'utf8' })
console.log('readFileSync case with { encoding: \'utf8\'} option: ', syncdata)

syncdata = fs.readFileSync(meta, 'utf8')
console.log('readFileSync case with \'utf8\' option: ', syncdata)

fs.readFile(meta, { encoding: 'utf8' }, (err, data) => {
  if (err) {
    console.log(err)
    return 
  }
  console.log('readFile case with { encoding: \'utf8\'} option: ', data)
})

fs.readFile(meta, 'utf8', (err, data) => {
  if (err) {
    console.log(err)
    return 
  }
  console.log('readFile case with \'utf8\' option: ', data)
})

the result is the following.

readFileSync case with { encoding: 'utf8'} option: # this is test text
readFileSync case with 'utf8' option: # this is test text
readFile case with 'utf8' option: # this is test text
{ Error: ENOENT: no such file or directory, open '/snapshot/home/nemoux/nemosnow/snow-apps/pkgsnapshot-test/meta.txt'
errno: -2,
code: 'ENOENT',
syscall: 'open',
path: '/snapshot/home/nemoux/nemosnow/snow-apps/pkgsnapshot-test/meta.txt' }

I executed the following command after extracting the following.

$ pkg index.js -t latest-linux-x64

pkgsnapshot-test.zip

Is it a bug or right behavior?

Trying to pack this project https://github.com/jbdemonte/Retro-Manager with this commande:

pkg package.json --options expose-gc --targets node7-linux-armv7 --out-dir build

I'm getting this error:

FATAL ERROR: Malloced operator new Allocation failed - process out of memory

Error! Error: /home/pi/.pkg-cache/v1.9/fetched-v7.6.0-linux-armv7 failed with code null
at ChildProcess. (/usr/lib/node_modules/pkg/lib-es5/producer.js:39:25)
at emitTwo (events.js:106:13)
at ChildProcess.emit (events.js:194:7)
at maybeClose (internal/child_process.js:899:16)
at Process.ChildProcess._handle.onexit (internal/child_process.js:226:5)

/Users/igorklopov/Work/ace-client/node_modules/async-retry/package.json

Hi!

I'm currently looking at various node packagers, and it's very nice to see that yours supports FreeBSD (unlike nar which only supports building executables for Linux, Darwin and Solaris — the ones with node binaries published by the node project itself).

However… FreeBSD's native package manager is literally called pkg ;)

Apologies if this is the wrong place to post this, but I couldn't find a more relevant repo.

I was browsing your site and came across this:

This reads "Singe-Command compiler" instead of the newer description "Package your Node.js project into an executable".

I cloned https://github.com/zeit/github-projects and verified that the latest description is coming through. Just wanted to let you know you may want to refresh your cache, if this is still powering your OSS page.

I don't see any license attached to this repository; could you please add an open source license (like the MIT license) so we can be sure we can use this legally?

Thanks in advance.

If I use pkg to make a CLI tool that manipulates files, then it's possible that users might try to use it on a /snapshot directory on their filesystem. (rare but possible, and for a CLI tool, I'd need to cover all cases) How do I read and write to the host filesystem's /snapshot directory with a pkg-bundled executable? I couldn't find this in the documentation, but maybe I missed the section that explains this.

Example:

cd /snapshot
vim index.js
my-pkg-bundled-utility uploadFileToWebsite index.js

Hi, I am cross-compiling for arm6 or arm7 on 64-bit Ubuntu host.

Regardless of if I use armv6 or armv7, and regardless of if I use node6 or node7, I always get the following error:

/home/msolters/.pkg-cache/v1.9/fetched-v6.10.0-linux-armv7: 1: /home/msolters/.pkg-cache/v1.9/fetched-v6.10.0-linux-armv7: Syntax error: word unexpected (expecting ")")
> Error! Error: Not able to compile for 'armv7' here
    at Socket.<anonymous> (/home/msolters/.nvm/versions/node/v6.10.0/lib/node_modules/pkg/lib-es5/producer.js:43:25)
    at emitOne (events.js:96:13)
    at Socket.emit (events.js:188:7)
    at onwriteError (_stream_writable.js:346:10)
    at onwrite (_stream_writable.js:364:5)
    at WritableState.onwrite (_stream_writable.js:90:5)
    at fireErrorCallbacks (net.js:468:13)
    at Socket._destroy (net.js:509:3)
    at WriteWrap.afterWrite (net.js:803:10)

electron-packager add resources to generated Windows executables (like company, version, copyright, icon, etc) by using https://github.com/electron/node-rcedit.
That could be cool to add this feature.

I guess readme.md file name change to README.md.
Please check.

Hello!

I was wondering if there was a way to ignore a specific missing file.
Summary:
pkg tries to resolve all path.join calls and we can be punished for a missing file that might not be important. A .pkgignore file might make sense for this?

Long version:
Here's my use case because maybe there's another/cleaner way of fixing this.
When I try to build my binaries I get this error:

> Error! Cannot stat, ENOENT
  /Users/zmarouf/my_module/node_modules/tabtab/.completions/cache.json
  The file was required from '/Users/zmarouf/my_module/node_modules/tabtab/src/cache.js'

This is the problematic code:

var cachefile = path.join(__dirname, '../.completions/cache.json');
...
this._cache = exists(cachefile) ? require(cachefile) : { timestamp: Date.now(), cache: {} };

My hacky way of fixing this right now is to just touch the missing file but that's—well— ugly and not very practical.

What do you think? What are your suggestions on this? :)
I did a quick search in the issues for a similar problem but it didn't yield anything.
If there's already an open/closed discussion about this then could you please direct me to it?

Thank you in advance for your time!

My app uses the Google Sheets API.

pkg generates the binaries ok but throws the following error when I run that binary:

/snapshot/Users/conoroneill/gitwork/yunmai-data-extract/node_modules/googleapis/lib/googleapis.js:62
fs.readdirSync(path.join(__dirname, '../apis')).forEach(function (file) {
   ^

Error: Directory '/Users/conoroneill/gitwork/yunmai-data-extract/node_modules/googleapis/apis' was not included into executable at compilation stage. Please recompile adding it as asset or script.
    at error_ENOENT (evalmachine.<anonymous>:0)
    at readdirFromSnapshot (evalmachine.<anonymous>:0)
    at Object.fs.readdirSync (evalmachine.<anonymous>:0)
    at Object.<anonymous> (/snapshot/Users/conoroneill/gitwork/yunmai-data-extract/node_modules/googleapis/lib/googleapis.js:62:4)
    at Module._compile (module.js:570:32)
    at Module._compile (evalmachine.<anonymous>:0)
    at Object.Module._extensions..js (module.js:579:10)
    at Module.load (module.js:487:32)
    at tryModuleLoad (module.js:446:12)
    at Function.Module._load (module.js:438:3)

I've tried many variations on

 "pkg": {
    "assets": "node_modules/googleapis/apis/**/*"
  }

But when I use --debug, I don't see the files being added. Am I missing something obvious in the path or glob?

I cannot package this binary into my app, because the binary cannot be codesigned by XCode.
This works on other binaries and files, so there must be an issue specifically around the binaries that pkg creates.

To reproduce, create a simple index.js with console.log('Hello world').

pkg -t node7-darwin index.js -o test

The resulting binary cannot be code signed.

$ codesign -vvvv --force --sign '<apple developer identity>' ./test
./test: main executable failed strict validation

Testing with the vue wrapper of quill editor, while the vue-quill-editor contains a vue component in the require expression, so the pkg run out with the result below:

> Error! Unexpected token (1:0)
  /home/test/jzver/node_modules/vue-quill-editor/src/editor.vue

and the index.js file of the vue-quill-editor just like:

/**
 * Vue-Quill-Editor
 * @author Surmon.me
 */

window.Quill = require('quill/dist/quill.js')
var quillEditor = require('./editor.vue')
var VueQuillEditor = {
  Quill: Quill,
  quillEditor: quillEditor,
  install: function(Vue) {
    Vue.component('quill-editor', quillEditor)
  }
}

module.exports = VueQuillEditor

Does it just add the editor.vue file to assets of package.json could solve this problem? I tried but failed, maybe I was wrong about the way to use the assets in package.json file. Any suggestion would be appreciated, thx a lot.

TLDR: I found Enclose, tried it and got an error. Found an issue report over there suggesting to use this instead. But I get the exact same message with Pkg.

Note: For the record, I do actually want to replace image-size with something that lets me do true image comparisons (currently I'm using the MD5 file hash, which is getting messy).

My console output is as follows:

D:\dev-sandbox\spotlight-extractor>pkg -t node6-win index.js
> Fetching base Node.js binaries to: ~/.pkg-cache
  fetched-v6.10.0-win-x86      [====================] 100%
> Warning Cannot resolve ''./types/' + type'
  D:\dev-sandbox\spotlight-extractor\node_modules\image-size\lib\index.js
  Use a string literal as argument for 'require', or leave it
  as is and specify the resolved file name in 'scripts' option.
> Warning Cannot resolve ''./types/' + type'
  D:\dev-sandbox\spotlight-extractor\node_modules\image-size\lib\detector.js
  Use a string literal as argument for 'require', or leave it
  as is and specify the resolved file name in 'scripts' option.

I have a few other projects and ideas that I want to package for distribution, but they're not yet ready for that just yet.. So for the sake of understanding this issue I'll ask my question using this project as reference for now.

Obviously modifying the library that causes the problem is one option, but is that the best way to handle this situation? Is there anything else that can be done to work around this?

A quick support - I am doing a test compile. Here is my package .json script I have added.

"pkg": {
    "scripts": "build/**/*.js",
    "assets": "views/**/*"
  }

Here is the folder structure

When I create a .exe it does create a exe but it gives an error saying this:

Error: File or directory 'C:\SGems\fullstack\5\views\index.html' was not included into executable at compilation stage. Please recompile adding it as asset or script.
at error_ENOENT (evalmachine.:0:0)
at findNativeAddonForStat (evalmachine.:0:0)
at statFromSnapshot (evalmachine.:0:0)
at Object.fs.stat (evalmachine.:0:0)
at SendStream.sendFile (C:\snapshot\SGems\fullstack\5\node_modules\send\index.js:729:6)
at SendStream.pipe (C:\snapshot\SGems\fullstack\5\node_modules\send\index.js:603:8)
at sendfile (C:\snapshot\SGems\fullstack\5\node_modules\express\lib\response.js:1057:8)
at ServerResponse.sendFile (C:\snapshot\SGems\fullstack\5\node_modules\express\lib\response.js:418:3)
at evalmachine.:0:0
at Layer.handle [as handle_request] (C:\snapshot\SGems\fullstack\5\node_modules\express\lib\router\layer.js:95:5)

Also, is there a way to keep the server running in quiet mode? Any options?

Old enclose had --loglevel param where you could have disabled warning level log messages for example. I can't seem to find this in the new version. Was it removed for some reason?

In some places I am not using string literals with require calls, but including those files using scripts config option. Hence those warnings aren't really interesting to me and I am well aware of the functionality.

I packaged my project, and ran into an issue that inhibits me from successfully running the output binary. One package I use heavily relies on parent-module, which in turn uses callsites, which is what causes the error. Honestly, I was going to be very impressed if it was able to deal with the parent-module module :). Here's the error I get when running the binary.

$ pkg --targets node6-macos-x64 .
... Packages, with some "Cannot resolve" warnings which I've added to `scripts` ...

$ ./my-project-macos
/snapshot/my-project/node_modules/callsites/index.js:1
(function (exports, require, module, __filename, __dirname) { module.exports = function() {throw new Error("Pkg: 'callsites' package is temporarily unsupported. Please create a github issue.")};
                                                                                                 ^

Error: Pkg: 'callsites' package is temporarily unsupported. Please create a github issue.
    at module.exports (/snapshot/my-project/node_modules/callsites/index.js:1:98)
    at module.exports (/snapshot/my-project/node_modules/parent-module/index.js:5:15)
    at Object.exports.using (/snapshot/my-project/node_modules/haute-couture/lib/index.js:19:39)
    at Object.console./snapshot/my-project/lib/index.js.dev (evalmachine.<anonymous>:0)
    at Module._compile (evalmachine.<anonymous>:0)
    at Object.Module._extensions..js (module.js:579:10)
    at Module.load (module.js:487:32)
    at tryModuleLoad (module.js:446:12)
    at Function.Module._load (module.js:438:3)
    at Module.require (module.js:497:17)

It instructed me to make a github issue, so here I am! Let me know if I can be useful to help solve the issue, or if I can provide any more information. Thanks!

Could you highlight in a nutshell how pkg is different?
https://github.com/nexe/nexe

so far i'm seeing Nexe rebuilding node with source, and pkg inlining the source into a new binary (thus not requiring a node rebuild)

Thanks!

Does pkg supersede enclose? There appears to be a relationship between the two projects, but I see that the github repo for enclose hasn't been updated in a while. Also pkg appears to be fully open source, whereas enclose is (partially?) proprietary.

Would be useful to have an explainer on this somewhere to reduce confusion.

Also, wow this project is cool. 😄

I use zeromq and grpc module in my project, while I tried to use pkg to bundle my project, it gave the warning like below:

> Warning Cannot resolve 'binding_path'
  /home//test/jzver/node_modules/grpc/src/node/src/grpc_extension.js
  Use a string literal as argument for 'require', or leave it
  as is and specify the resolved file name in 'scripts' option.
> Warning Cannot include native addon into executable.
  /hometest/jzver/node_modules/zeromq/build/Release/zmq.node
  The addon file must be distributed with executable.

The grpc_extension.js looks like:

var binary = require('node-pre-gyp/lib/pre-binding');
var path = require('path');
var binding_path =
    binary.find(path.resolve(path.join(__dirname, '../../../package.json')));
var binding = require(binding_path);

module.exports = binding;

While the grpc module should be compile to node C++ addon with the grpc_extension.js, and to the zeromq module, when the compile job is done, it produce the zmq.node to be used in node project. I don't know how to handle node modules like these by pkg.
And my node version is: 6.10.0
I used the command like this: pkg -t node6-linux ./server/index.js
Thanks for amazing work with pkg

It would be nice to be able to specify that you want all targets built that your current system is capable of building.

In the README you mention that scripts that are added as scripts in the config are compiled using the V8 compiler. Is that also true for regular dependencies that are found from the entry point?

And does this compilation prevent source code from being accessible in the package?

My use case here is, that I have a private key string in my source code, but when I ran strings on the resulting executable, the string was still there in plain text.

Thanks!

Running pkg . with a package.json containing no target (i.e. expecting default/current version):

  ,"pkg":{"scripts":"engine/**/*.js","assets":"engine/**/*"}
  ,"bin":"engine/app.js"

Returns Error! No available node version satisfies 'node5'

Trying with different targets specified such as "node5.0" or "node5-win" or "v5.0.0" or "latest-v5.x" or returns the following:

Error! No available node version satisfies 'node5.0'
Error! No available node version satisfies 'node5'
Error! Unknown token 'v5.0.0' in 'v5.0.0'
Error! Unknown token 'v5.x' in 'latest-v5.x'

I'm pretty sure there was a version of Node 5 downloaded when installing pkg, but I can't find where it was placed and there doesn't appear to be a command to list available Node versions / valid targets?

Hi sorry for asking this, I just want to double check before I get all excited.
So I have a node 6.x app running in production and my app was the sole reason my custom had to install node in the first place.

Could I uninstall node 6 on the server, develop the next iteration in say node 7 and pkg it to an executable express server? Would that actually work?

Trying to pack a project which includes jade (pug), these errors are triggered:

Error! Unexpected token (16:0)
../node_modules/jade/.npmignore
Error! Unexpected token (1:4)
../node_modules/jade/History.md
Error! Unexpected token (1:5)
../node_modules/jade/LICENSE

Build cmd: pkg package.json --options expose-gc --targets node7-linux-armv7 --out-dir build

jade dependency in package.json: "jade": "^1.11.0",

When I have a plain JS class and try to turn it into a string, my program hangs. I'm on a Mac, on node v6. I think the demo below will provide enough information to get started.

Contents of test.js

'use strict';

const arrow = () => null;
const Class = class {};

console.log('arrow');
console.log(typeof arrow);
console.log(arrow);
console.log(arrow.toString());

console.log('Class');
console.log(typeof Class);
console.log(Class);
console.log(Class.toString());

console.log('end');

Demo

$ pkg --targets node6-macos-x64 test.js

$ ./test 
arrow
function
[Function: arrow]
function arrow() { [native code] }
Class
function
[Function: Class]
^C # Program hung here!

$ node -v
v6.4.0

$ node test.js
arrow
function
[Function]
() => null
Class
function
[Function]
class {}
end

I would like to try, uh ... I still do not use github, thanks.

As I understand this is a rebranding of the enclose module? Would love to know how this happened btw.

But, in original documentation, here – http://enclosejs.com/#compilation-srsly it says that resulting binary will be at least 2x slower, is that still the case? And if so, can you mention that in the readme? It would be better I think to also document how this sort of measurements was performed.

As well as any other possible drawbacks, besides performance and size

I'd like to add a little test coverage shield to the README using coveralls.io, e.g.:

Right now, coveralls.io only has access to next.js and micro, see: https://coveralls.io/github/zeit

To unblock: The ziet organisation has to grant read permission on the public zeit/pkg repo

We need to avoid this using a config property within package.json (should hold an array of targets).

The canonical default executable is start script in the package.json file, that by default is server.js file, not bin entry since you can define several binaries in a single package.

I suppose running the binary instead of node with js files will be slow as v8 makes many optimizations which may not be possible at compile time. Are their any statistics of how slow this compiled code runs in comparison to the default interpretted (JIT) way running node apps

Error happend in compilation of simple index.js with few functions.
`C:\Users\Kradre\Desktop\mqttBroker>pkg index.js

Targets not specified. Assuming:
node4-linux-x64, node4-macos-x64, node4-win-x64
C:\Users\Kradre\AppData\Roaming\npm\node_modules\pkg\lib-es5\producer.js:89
return Buffer.alloc(padding);
^
TypeError: Buffer.alloc is not a function
at paddingBuffer (C:\Users\Kradre\AppData\Roaming\npm\node_modules\pkg\lib-es5\producer.js:89:17)
at C:\Users\Kradre\AppData\Roaming\npm\node_modules\pkg\lib-es5\producer.js:25:48
at MultiStream._next (C:\Users\Kradre\AppData\Roaming\npm\node_modules\pkg\node_modules\multistream\index.js:81:27)
at ReadStream.onEnd (C:\Users\Kradre\AppData\Roaming\npm\node_modules\pkg\node_modules\multistream\index.js:119:10)
at ReadStream.g (events.js:260:16)
at emitNone (events.js:72:20)
at ReadStream.emit (events.js:166:7)
at endReadableNT (_stream_readable.js:913:12)
at nextTickCallbackWith2Args (node.js:442:9)
at process._tickCallback (node.js:356:17)`

It would be awesome to set up github releases for the project in order to work well with other package managers (like homebrew and such).

[root@iZ94hmrh5tzZ ra-broker]# pkg index.js

Targets not specified. Assuming:
node7-linux-x64, node7-macos-x64, node7-win-x64
Fetching base Node.js binaries to: ~/.pkg-cache
fetched-v7.6.0-linux-x64 [====================] 100%
Error! connect ETIMEDOUT 54.231.33.195:443
Asset 'v1.9/uploaded-v1.9-node-v7.6.0-linux-x64' not found by direct link
fetched-v7.6.0-linux-x64 [====================] 100%
Error! connect ETIMEDOUT 54.231.33.195:443

Hi! Just done a quick test on windows builds. Bundle size can be reduced from 24,964 KB to 6,646 KB (3.75x smaller) with doing simple xz compression. This may even help more when having more npm dependencies. Adding this option may be super useful for small and embedded CLI/Webservers :) I've just opened this discussion before going further to know others idea on this.

UPDATE: To be clear, this topic and what i am talking about is SFX packages not js minification & optimizing.