Any chances for node.js bindings? about link-grammar HOT 23 CLOSED

Someone would need to do the work ... patches are welcome. --linas

On Sun, Aug 2, 2015 at 3:30 AM, Gleki Arxokuna [email protected]
wrote:

It's been a while since the issue dijs/link-grammar#5
dijs/link-grammar#5 was opened so any plans
on making a node.js module for the master version of the library?

—
Reply to this email directly or view it on GitHub
#161.

from link-grammar.

FWIW I believe that LG version 5.3.2 now resolves all Apple MacOSX build problems that were blocking a port of the node.js module.

from link-grammar.

Compiling and linking the LG lib and link-parser to .js files according to the directions were straightforward and didn't produce any error or warning. I also produced a link-parser.html file.

However, when I tried to load link-parser.html" in Chrome and got the error: link-parser.js:1 GET http://localhost/test/link-parser.wasm 404 (Not Found)`
So I guess I need to read some more about how to actually use it.

from link-grammar.

[...] it seems apache cannot access things out of the vhost root

It was not apache problem but the general inability of JavaScript code to access the local file system by default. The Enscripten documentation tells how to enable file reading from JavaScript by packaging them beforehand or by mounting a specially JavaScript filesystem.

For just testing link-parser.js using en/corpus-basic.batch, I compiled it as follows:

emcc -O3 link-parser.bc PATHTO/.libs/liblink-grammar.so -o link-parser.js -s ALLOW_MEMORY_GROWTH=1 --embed-file en

The en at the end is a stripped down data/en directory, after all the unneeded files have been erased (especially 4.0.dict.m4). This is because --embed-file is not efficient (it adds the files to link-parser.js). From the batch files, I left corpus-basic.batch.

(The use of ALLOW_MEMORY_GROWTH=1 is a must, else it aborts on no memory for sentences with more than a few words (but there is still enough memory for "This is a test" ...).)

I ran it as follows:
node link-parser.js

For unknown reason, it has a terminal interaction problem even if configured with --disable-editline:
It needs ^D after each RETURN. However, it is still functional with that problem.

For this test, I used this input:
!file en/corpus-basic.batch
(ENTER ^D ...).

It took 4.13 second, when the same normal run takes almost exactly 2 seconds (1.97).
So it is slower only by a factor of 2 (and maybe less, I didn't consider the startup time).

(When I tried to package data/en for web access, the page load failed with Uncaught RangeError: Source is too large. This was before I discovered the need for ALLOW_MEMORY_GROWTH=1. But I didn't continue exploring that.)

To sum up:

@zenflow wrote:

Would the js build be part of the build of this repository? Or would we need to create another repository to pull in the source code from this one, compile it to js, and publish as a package on the npm registry?

@zenflow,
There is only a need for a build script, a sample JavaScript code and a README.
I think there will be no problems to store them in the current repository.
It can be a good idea to add in that README instructions for mounting the local file system for node and from the web.

If you share the code I would be happy to work on this with you (maybe this week when I have a bit more time)

Ok, let's work together on that.
But I will leave the main work for you, and continue for now with the LG lib development.

from link-grammar.

from link-grammar.

@lagleki On the above note about getting the build script right.. I am getting a mystifying OSError: [Errno 2] No such file or directory error from a Python script when I try to emconfigure configure --disable-sat-solver --disable-java-bindings --disable-python-bindings.. you didn't get this? I am getting it in the Travis CI environment as well as in WSL (Windows Subsystem for Linux) Ubuntu on my local Windows machine. Can you tell what I'm doing wrong here (Travis job log here)?

I myself didn't get errors when running emconfigure.
Maybe the configure script is somehow not executable (chmod +x is needed) or the current directory is not in the path (i.e. ./configure is needed)? Try adding "ls -l configure" to find out.

from link-grammar.

I see there 4 warnings that I don't like.
I will fix them.

Which errors have you encountered?

from link-grammar.

@linas Instead of providing node.js bindings to a binary via FFI (foreign function interface), like dijs/link-grammar does, would it be possible to compile link-grammar into pure JavaScript using emscripten?

Emscripten is an Open Source LLVM to JavaScript compiler. Using Emscripten you can:

Compile C and C++ code into JavaScript
Compile any other code that can be translated into LLVM bitcode into JavaScript.
Compile the C/C++ runtimes of other languages into JavaScript, and then run code in those other languages in an indirect way (this has been done for Python and Lua)!

I think that would be better because the package would be more portable, and could even be used in web applications. Also I think it may be less complicated to use.

from link-grammar.

@ampli What did you do to compile link-parser? If you share the code I would be happy to work on this with you (maybe this week when I have a bit more time)

Would the js build be part of the build of this repository? Or would we need to create another repository to pull in the source code from this one, compile it to js, and publish as a package on the npm registry?

from link-grammar.

What did you do to compile link-parser?

Something like:
(Note: It is not an actual script, it is just an approximate copy/paste from my screen, also with cd commands and/or full paths omitted.)

source PATHTO/emsdk_env.sh
emconfigure  configure --disable-sat-solver --disable-java-bindings --disable-python-bindings
emmake make
emcc -O3 PATHTO/liblink-grammar.so -o PATHTO/liblink-grammar.js
file PATHTO/.libs/link-parser # result: LLVM IR bitcode
mv link-parser link-parser.bc
emcc -O3 link-parser.bc PATHTO/liblink-grammar.js -o link-parser.js
emcc -O3 link-parser.bc PATHTO/liblink-grammar.js -o link-parser.html

I also tried to run it with this html code. Even though both scripts load a .js file, I got the said error on missing .wasm file, which I still need to resolve (I didn't have time to further investigate it).

In addition, there is a need to write a sample JavaScript code to directly use the LG library.

from link-grammar.

emcc -O3 link-parser.bc PATHTO/liblink-grammar.js -o link-parser.js
emcc -O3 link-parser.bc PATHTO/liblink-grammar.js -o link-parser.html

Bad reconstruction of shell-history commands - it was actually something like:

emcc -O3 link-parser.bc PATHTO/liblink-grammar.so -o link-parser.js
emcc -O3 link-parser.bc PATHTO/liblink-grammar.so -o link-parser.html

from link-grammar.

I solved the .wasm' problem, and the JavaScript-compiled link-parser is able to run from the browser.
The following generates also link-parser.wasm:
emcc -O3 PATHTO/link-parser.bc PATHTO/liblink-grammar.so -s WASM=1 -o link-parser.html

However, the JS link-parser couldn't find the dict.
You can give it arguments by editing link-parser.html: After Module=, I added
arguments:['PATHTO/data/en','-v=4'], (just before preRun:[]).
However, it seems apache cannot access things out of the vhost root, and the JS link-parser is running with current directory / but cannot access full paths ending in its vhost directory.

I will continue to struggle with that letter and will post the solution (unless you save me time by posting a solution before that...).

from link-grammar.

Performance within a factor of two is actually quite impressive.

from link-grammar.

@ampli @zenflow. any news?

from link-grammar.

@lagleki not yet, but I will carve out some time for this (and I'm really looking forward to it)

from link-grammar.

One question: Does this project currently manage publishing packages to any package registries (e.g. for Python, pypi, or for Java, Maven Repository)?

We should ideally be publishing the WASM in a package to npm (javascript/web package registry). This is super valuable to make it easily accessible within the JavaScript ecosystem. The WASM would even be automatically available on the unpkg.com CDN.

Assuming that the answer to my above question is "no", then after the build script is in this repo, I could create another repo for the npm package, that builds (pulling everything from this repo) and publishes the wasm in the normal npm package way. I can set up Travis CI to run nightly to check for any new release tags in this repo, and if there are any, build and publish them.

Does this make sense and sound reasonable to everyone?

from link-grammar.

Another thought: The npm package should not attempt to provide any kind of high-level API for this library, but should just publish the WASM build artifact along with some instructions & examples for using it (which probably should mirror the instructions/examples in this repo). With the WASM being published as a package on npm (call it link-grammar-wasm), it will be easy for other packages to build on top to provide a high-level API. This will allow multiple competing implementations and forks, without the burden of a complicated and less portable build process for them. An application could even choose to use none of the available high-level APIs (using it's own, or none).

from link-grammar.

Does this project currently manage publishing packages to any package registries

No.

create another repo

That's fine, but why do you need a different repo? What problem does that solve?

Oh, OK, I think I understand. It would contain the compiled code. OK.

high-level API

What do you imagine to be a "high-level API"? Something like RelEx? Something else?

Oh, never mind, I think I understand. If by "low level" you mean the FFI C interfaces, then, yes, I imagine the corresponding js API would be quite ugly. It seems to me that it would be very important to provide a nice, pretty js API, and surely that cannot be that hard. Nor do I imagine it would be that controversial, so I can't imagine why there would be a need for "competing implementations".

from link-grammar.

@linas

That's fine, but why do you need a different repo? What problem does that solve?
Oh, OK, I think I understand. It would contain the compiled code. OK.

No, it wouldn't contain the compiled code.. and IMO generally compiled code should only be included when publishing the package, but not in source code management (SCM)

What problem it solves? I guess none, if you are interested in starting to use a CI/CD (continuous integration/continuous delivery) service like TravisCI with this repo, and have it manage publishing to a package registry for the first time! I would be happy to help with that! The big bonus from this would be the CI part where all checks and tests are run automatically on code we push, and results show up in Pull Requests to help us avoid merging branches that broke something. See the checks in my PR in my fork of this repo, click "Show all checks". You can see two CI jobs (appearing twice each for some reason): the "PR" build check (executes the job on Travis with the result of merging the PR branch into the master branch) and the "branch" or " push" build check (executes the job on travis with the exact code from the branch). You can see the Travis job log by clicking click "details" of "continuous-integration/travis-ci/pr" or "continuous-integration/travis-ci/push".

I will open a PR to start us in this direction (set up Travis), though I still haven't been able to get the build script right, so I will not include that part.

@ampli On the above note about getting the build script right.. I am getting a mystifying OSError: [Errno 2] No such file or directory error from a Python script when I try to emconfigure configure --disable-sat-solver --disable-java-bindings --disable-python-bindings.. you didn't get this? I am getting it in the Travis CI environment as well as in WSL (Windows Subsystem for Linux) Ubuntu on my local Windows machine. Can you tell what I'm doing wrong here (Travis job log here)?

What do you imagine to be a "high-level API"? Something like RelEx? Something else?
Oh, never mind, I think I understand. If by "low level" you mean the FFI C interfaces, then, yes, I imagine the corresponding js API would be quite ugly. It seems to me that it would be very important to provide a nice, pretty js API, and surely that cannot be that hard. Nor do I imagine it would be that controversial, so I can't imagine why there would be a need for "competing implementations".

Yes, I am expecting the FFI C interface will be nasty to use in JavaScript. I unfortunately haven't been successful in compiling this project to WASM or asm.js to see and play around with that interface, so I don't really know what there is to deal with there yet. But I can imagine some things that may be controversial, or just that we don't get right the first time and need to iterate on. For example: Should it use the WASM or the asm.js? How to import the WASM binary data in a way that works for both browser and nodejs (this can be complicated)? How to include the languages data in data/ in a way that works for browser+node and ideally allows users to load just the needed data? Should we augment the POJO (plain old javascript object) data structures by making them with Object-Oriented objects? Could we improve on the naming of some set of enums or constants? Probably lots more.

If we publish packages that contain just the bare compiled assets (the WASM & the asm.js) and have a separate project (separate repo, separate package) to provide the [high-level] JS API, developers can easily fork the JS API and work on it (and maybe contribute back a PR) because it doesn't require them to work on a Linux machine set up with all the system requirements of this project plus emscripten plus system dependencies of emscripten. (I generally work on Windows btw). Also upgrades (breaking or backwards compatible) in the JS API wouldn't require you to advance version number of this project.

from link-grammar.

CircleCI

OK. Don't care. LG is a small enough project that this will have very little impact.

CICD

No. Users consistently fumble up the dependencies, and its a pain-in-the-neck to debug by email. I always strongly recommend that only the official tarballs ever be deployed.

Many WASM/asm.js questions

No opinion. Outside my area of expertise. Except for the remark about users not being able to manage dependencies - the easier for users to install/configure, the better.

advance version number of this project.

I assume js has it's own way of managing version numbering, independent of what LG is doing.

from link-grammar.

@ampli Yup, that was the problem. I changed emconfigure configure to emconfigure ./configure and that command is working now: 5c5685a job log

Thank you!

edit: Well with that change the command worked completely on my local machine at least. Now seeing a new error in Travis

from link-grammar.

Well, I just took Richard van der Dys node.js code and just slammed it into this repo, and then fixed up the stale/obsolete API calls. It took just over an hour. It works, and apsses unit tests. See pull req #1053

There's a lot of commentary above that I don't understand, so I won't close this issue yet; someone knowledgable about javascript should review and confirm and clean up as needed.

from link-grammar.

Closing. The node.js bindings have now been available for 15-odd months. If there are new issues, please open a bug report!

from link-grammar.