nodejs / api Goto Github PK
View Code? Open in Web Editor NEWAPI WG
API WG
Recently, I am interesting in abstracing JS API, and have some general idea after comparing APIs between v8, jsc, chakra and spidermonkey. Hence, I have some issues about JS API(not include node-api temporarily).
Reasons;
There are alternative solutions (including the one we use with jxcore native interface).
Abstracting data types alone doesn't cover all the type related future VM / version changes ( typeof differences / multi context instanceof reference problems etc) However, IMHO it is an important actor and decisive to all the other definitions.
WARNING OBSOLETE
We've been making some progress on the ABI stable API (will provide an update in WG meeting #22)
This is to capture the current view of some of the major milestones for PoC:
webworker-threads
Date/Time: Thursday, Nov 19, 4pm eastern - world time
Doodle for picking your availability: http://doodle.com/poll/zdcb77922sqq8sca
Agenda and minutes can be collected in here and copied into this repo later: https://docs.google.com/document/d/1gyz0yaKtOW97bPcXOG2mgzpKy5LxFKWezWSzxc_-YQA/edit?usp=sharing
Previous minutes: https://github.com/nodejs/api/blob/master/wg-meetings/2015-10-23.md
Hangout on air for active participants: https://plus.google.com/hangouts/_/hoaevent/AP36tYcI7IrAtU4xkIGtK9DgI70NhIemFWx8-d1y2enY-ksZb4UtDg?hl=en
YouTube movie for observers and saved recording: http://www.youtube.com/watch?v=3rszjGYpbyM
Hangout on air event for more details: https://plus.google.com/events/cv3h505lip7aqvil4n5ri0kl8e8
Agenda discussion can happen in this thread.
(format copied from nodejs/Release#38)
(instructions for creating Hangouts On Air on the nodejs/node wiki)
Apologies for what follows as it's a bit of a (long) braindump.
Please feel free to close if I have missed the intention/direction of the project.
TL;DR; Will this project support browser based node implementations?
So myself and a couple of others have kicked of a small project called https://github.com/chromiumify
The idea is to martial all the packages that implement the node api in Chrome Apps as a coherent node runtime.
Currently we have implementations of http, dgram, net, fs and serialport (see https://github.com/chromiumify/project-list#project-list) and I thought it might be useful to share some initial thoughts.
Having the Sync calls in core is a problem as you can't implement them if you are in a pure JS environment.
e.g. The Chrome App environment has a large set of comparable calls for an implementation of
an 'os' module but this is not feasible as the chrome api is async
See (https://github.com/No9/chrome-os/blob/master/chrome.js#L38) for a sketch of the issue.
My initial thought here is to suggest that the test suite for each module should be broken into sync and async
i.e. ('os-sync-tests') and ('os-tests')
This approach also highlights that all sync calls such as the os.cpu()
should also have corresponding async calls specified in the tests.
The async shim could be implemented in userland for node.js but would give alternative runtime implementers options.
One of the largest amounts of time spent in the chrome-fs port https://github.com/No9/chrome-fs port is migrating the tests
This is because most of the tests in core rely on the sync api and or process.exit and static fixture folders.
It would be great if these dependencies could be removed in the case of sync and process.exit and removing the need for a fixtures folder
I have ported some of the fs tests to this approach
https://github.com/No9/chrome-fs/tree/master/test/simple
While it's time consuming it's should be possible.
It would be nice if the test suits could output an API coverage document something along the lines of https://github.com/No9/chrome-fs#api-status (?) so it's easy for folks to see what is and isn't available from that implementation.
The node browser projects (chromiumify and browserify) are more loosely coupled allowing core APIs to be swapped out according to needs so this would be great to have for end users.
Are there any ideas for the implementation of the test suite/JSAPI specification?
My hunch was that each module would mirror the abstract-leveldown approach https://github.com/Level/abstract-leveldown but that is very presumptuous as there are probably other ways to go about this too.
Anyway hope some of this is useful.
Moved to here: nodejs/abi-stable-node#1 as it belongs in that repo now that is exists under github/nodejs.
Here is a comparison of runtime of leveldown's performance on its benchmarks and tests (total wall clock time) between the V8 API implementation and the NAPI implementation.
This is using x64 release builds of node.js and leveldown, running on
Node.js and leveldown are built from these commits:
Each test was run three times.
Raw data here https://gist.github.com/ianwjhalliday/236bdb53448a372536793580c0882197
Averaged results:
Perf Test | leveldown-v8 | leveldown-napi | Delta |
---|---|---|---|
db-bench.js | 61 sec | 62 sec | 0% |
write-random.js | 170 sec | 170 sec | 0% |
write-sorted.js | 95 sec | 100 sec | 5% |
tests | 30 sec | 66 sec | 120% |
db-bench.js
and write-random.js
appear to perform equally well, while write-sorted.js
appears to have become slightly slower. The test suite is taking significantly longer, over twice as long.
These are interesting results that suggest we are correct to believe that performance is only hindered in very frequent calls from JavaScript code into native module code. I have not verified but suspect that the benchmark tests are exercising LevelDOWN's internals and LevelDB itself, rather than LevelDOWN's API layer.
So in the case of the benchmarks it appears the overhead of NAPI is insignificant relative to the workload LevelDB is handling, whereas in the case of the tests NAPI is significant presumably because the tests are focused on exercising the API that LevelDOWN exposes.
We currently know of two areas where our NAPI prototype has room to improve:
FunctionTemplate
optimizationnapi_create_string
, napi_create_error
, napi_throw
)Next thing I will do is whip up an API for creating a constructor with methods using a v8::FunctionTemplate
properly and see how that changes these numbers. I expect this will make a large difference. Next I'll add an API to create and throw a new error from a text message in one API call. I expect this to have a minor to nil effect on performance but will try it since it will be easy and quick. Finally after that if there is still a gap I will do profiling to see where time is being spent.
I will also get timing numbers for x86 release builds sometime this week.
One of the actions from the vm face-to-face was to bring this group up to speed on what was discussed and next steps. I agreed with Trevor to schedule the meeting.
@nodejs/api
Thursday April 21 @ 11 AM EST
Doodle poll (closed) http://doodle.com/poll/2dsp52h95yuh2ch3C
link for participants: https://plus.google.com/hangouts/_/hoaevent/AP36tYeCz7oV47m-jWBHlXU4M0F3uO1MIXt0COGLd7TRpPRw3lDV_g?hl=en
For those who just want to watch: http://www.youtube.com/watch?v=B6O3nbvkGHc
Events page: https://plus.google.com/events/cdd094nj844cq8ssn5ut4jdtl8s
Google doc for minutes: https://docs.google.com/document/d/1EJUdX_2hndr6k3E5mWOyrk8R-06iBCd48LRuVNlc2qg/edit
nbind is a C++ template based bindings generator with an API designed around C++ standard library types. That way it avoids JavaScript engine details and already the same code can be compiled to native Node.js addons and asm.js code that can be imported using Node.js require
or inside a browser with most loaders.
The next planned step is to serialize function arguments in a common binary format and compile the addon to a .so
, .dll
or .dylib
so it can provide a stable ABI within a single platform (all versions of Node.js and Electron). Emscripten's embind uses a similar serialization approach that works in practice.
nbind also generates TypeScript definitions from the C++ code and has tools for managing binding.gyp
files. An addon looks for example like:
class VectorTest {
public:
static std::vector<int> getInts() {
std::vector<int> a {{ 13, 21, 34 }};
return(a);
}
};
NBIND_CLASS(VectorTest) {
method(getInts);
}
For example there's a RisingStack tutorial for Node.js addons using NAN, and you can compare its example to an nbind-based version.
I'm really interested to see what others are coming up with. nbind is one possible way to provide a stable ABI (in the future) across all Node.js versions, and a stable API (already today) across all JavaScript implementations on all platforms in general.
Repo in which we are currently working: https://github.com/nodejs/abi-stable-node/
enhancement proposal with overview: nodejs/node-eps#20
See #25 for similar runs/data on windows
Steps followed:
Summary:
The differences in the benchmarks are quite small, and we probably need to do more to reduce variation on the machine and do more runs to consider the deltas valid.
Don't have times for running the tests as there is a failure in the ABI mode.
Perf Test | leveldown-v8 | leveldown-napi | Delta |
---|---|---|---|
db-bench.js | 72.67 sec | 75 sec | +3.21 % |
write-random.js | 193 sec | 195.7 sec | +1.4% |
write-sorted.js | 119.3 sec | 117.67 sec | -1.37% |
tests | - | - | -% |
I note the last meeting is April 5 and 6 but the last wg-meetings record is 2015-11-19.
Our group is really interested in the Native API. Node/V8 versions change much, so native module developers need to rewrite their code once the Node/v8 apis change. And multi-vms is also a question.
Can I know more about the progress? particularly about Native API.
Thanks very much.
The current API for native modules exposes the entire v8 API to native module developers. Even if NAN is used to insulate module code from the v8 API changes, it doesn't do anything to shield it from changes in the Node ABI, which in turn would require recompiling the module.
I think that an FFI-based approach has the potential of providing what's needed to implement the vast majority of native modules out there. The idea is to be able to marshal only basic types back and forth between JavaScript and C/C++. Since those types can hopefully be defined in standard terms, and they don't expose any engine-specific features or implementation details, the interface can stay consistent across engine versions and even across multiple engines. Native modules would have to be rewritten to expose their functionality through this marshaling layer, and they wouldn't have access to the v8 constructs. Note that arbitrary JavaScript objects would probably not be supported across the interface, because they risk exposing engine-specific implementation details. Instead, native modules following this model will likely need a JavaScript portion, to map the JavaScript-style API defined by the module to calls into the native portion that use only simple types.
I am hypothesizing that the vast majority of modules could be rewritten using this approach and that the only ones that couldn't are the ones that are designed to expose engine specific features (e.g. v8-profiler). Those will naturally need to continue to support specific engines, and be exposed to changes in the engine.
There is a widespread perception in the community that an FFI solution would be too slow to be of general use. I think the cause of this perception might be that the node-ffi module is known to introduce a lot of overhead. I haven't had a chance to study the node-ffi implementation, but I am guessing that it is using a reflection-based approach to do the marshaling, which may be the cause of the overhead. A template-based approach was suggested by @geoffkizer at nodejs/nan#349 (comment) which showed that the overhead can be very small. My experience with other platforms that use this approach (i.e. .NET) also leads me to believe that the overhead can be reasonable and that the approach should be feasible.
I am raising this issue so that at least we don't dismiss this possibility. It would be useful for this group to prove or disprove whether this can be an effective solution.
One of the open questions in my mind would be how to support the array/buffer type in a portable and performant way.
/cc @robpaveza (Chakra)
Hi folks,
@stefanmb, @mhdawson, and I have been experimenting with a prototype VM agnostic and ABI stable API for interfacing with JavaScript in native modules and we would like to test on real world native modules to reveal any performance bottlenecks or issues.
We suspect modules that expose a "chatty" API (frequent calls back and forth between JavaScript code and the module's native code) may reveal performance issues.
Does anyone know of good example modules that we can test with?
Modules that also have established benchmarks would be great so that we can avoid spending time on creating new benchmarks.
@aruneshchandra suggested leveldown as a good example, we are looking at it now to get started.
The concepts behind this group seemed to be well received at the collaborator summit. I'm not sure if this group has enough members yet, but maybe setting up a meeting would help it start moving along, even if it was only once a month.
One of the important parts for Node.js API will be the JavaScript engine interface. Before putting up an initial document for it, what necessary steps should be taken?
The questions I have;
Date/Time: Wed, 2015-10-23, 1PM pacific - world time
Agenda and minutes can be collected in here and copied into this repo later:
https://docs.google.com/document/d/1KgWQsUUeUbo8ip4G6q5Qaxo4sqYMxL8UJZ1CM231Wds/edit?usp=sharing
Previous minutes: none?
Hangout on air for active participants: https://plus.google.com/hangouts/_/hoaevent/AP36tYfRZEnX97TP3o2X3GG1LodWHmIy9EdVspVSbRWh34pHHV18eA?hl=en
YouTube movie for observers and saved recording: http://www.youtube.com/watch?v=fHU4L6Cl90M
Hangout on air event for more details: https://plus.google.com/events/cptau2lkifq0earrgidrmj623p4
Agenda discussion can happen in this thread.
(format copied from nodejs/Release#38)
(instructions for creating Hangouts On Air on the nodejs/node wiki)
Date/Time: TBD
Availablity in doodle: http://doodle.com/poll/8xmb563nez93e5vy
Agenda and minutes can be collected in here and copied into this repo later:
https://docs.google.com/document/d/1qPwK4Tzw1GWK8BqK4NLSbO5BgfhZPLACQ3hZJ6KznJw/edit?usp=sharing
Previous minutes: https://github.com/nodejs/api/blob/master/wg-meetings/2015-11-19.md
Hangout on air for active participants: https://plus.google.com/hangouts/_/hoaevent/AP36tYfKgTx6_mn_8maAK-TCeK63b8NoJ4EXvun7vmwBQsRooLZbIw?hl=en
YouTube movie for observers and saved recording: youtube: http://www.youtube.com/watch?v=Q27BSWI-soA (delayed for a minute or so)
Hangout on air event for more details: https://plus.google.com/events/civo3ec55ssvjrau0djvhvkk25s (moderator must start here to start recording meeting)
Agenda discussion can happen in this thread.
(format loosely copied from nodejs/Release#38)
(instructions for creating Hangouts On Air on the nodejs/node wiki)
AFAIK the initial coverage of the API is native C, C++ addon. What is the expectation for using this API internally ?
We've been making progress on the PoC for a new API, having gotten the Nan examples and some other modules to build/run with the API. (Some additional details can be found here: #22)
We have updated the PR for the eps - nodejs/node-eps#20 to reflect the work so far.
Having it as a PR that has not landed makes it more difficult to collaborate on updates. Would like to get CTC input on if it can land and if not what needs to happen before it can.
The work so far for the PoC has been in a branch on a personal fork of the node and nan-addon-examples repos:
https://github.com/ianwjhalliday/node/tree/api-opaque-prototype
https://github.com/ianwjhalliday/node-addon-examples/tree/api-opaque-prototype
Would like to get CTC input on what would be most appropriate.
I'd like to move this to a more visible location. Options include:
All, I would like to invite members of the Google V8 team, Microsoft Chakra team, @nodejs/ctc and @nodejs/api WG to a sit down face-to-face at IBM's Foster City location to discuss the technical details around supporting multiple JavaScript VMs in Node core. Getting everyone together for a day or two can help us identify and hammer out many of the various details and issues and determine the best way forward. I have created a doodle poll with some suggested dates for getting together. Please let me know what dates would work best. We'd like to get everyone in the same room but I can arrange a remote connection for anyone unable to attend in person.
http://doodle.com/poll/649rfsayknyqxwy2
If any of these days do not work, let me know and I'll add some additional options.
Update: Looks like April 5th-6th is the winner.
Thank you!
During the API creation process we should take WebAssembly into consideration in order to future proof what is being done now. One example is the fact that wasm
will allow fine grain control over how threads are spawned.
I'm currently a member of the WebAssembly community group, but could use help making sure our progress takes advantage of the work being done there.
Is this the new "home" for the Addon API working group?
https://nodejs.org/en/about/working-groups/ links it to https://github.com/nodejs/nan
@nodejs/api
Meeting set for Thursday July 14 at 12 pm EST.
link for participants: https://hangouts.google.com/hangouts/_/hoaevent/AP36tYenublbryPz0-q_Sjl0OZA599uDMCKR69JlkxDTeW-5BqXy2A?eid=100598160817214911030&hl=en&authuser=0
For those who just want to watch:http://www.youtube.com/watch?v=NJZ2TNul9To
Events page: https://plus.google.com/events/cvkrhrj8hfabfuho1tg5kvmeh5o
Google doc for minutes: https://docs.google.com/document/d/1iQD8JsCa-8_YJnpbd8Y2RTW7ELzY3dER8R9pzhC2fxc/edit
Links to the code shown in the meeting:
I see that ianwjhalliday/node.git is doing the work for C API.
we are also working on the project to add a C++ vm independent interface for v8 native api.
We may want to add these feature to native api:
We have done some work now: https://github.com/alibaba/node (branch jsni-v1.0).
There are a few of test cases under node/deps/jsni/.
The work is not all completed yet, there may be some deficiencies。But we want to do something to promote the project of the vm-neutral native API.
Any ideas and suggestions?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.