Can this library be used in node.js? about swiftlatex HOT 21 OPEN

Thanks, I was now trying the solution you mentioned yesterday, but I haven't succeeded yet, mainly due to my lacking knowledge of how the typescript build system works.

from swiftlatex.

from swiftlatex.

from swiftlatex.

Hi Mohamed,

I will take some time to have a look at this issue. Will keep you posted when it is done.

Cheers
Elliott

from swiftlatex.

On my 2nd try, I think I made some progress:

node index.js
Engine compilation start
Start downloading texlive file https://texlive2.swiftlatex.com/xetex/10/swiftlatexxetex.fmt
Engine compilation finish 7827.116400003433
RuntimeError: Aborted(Assertion failed: allocate no longer takes a type argument)
    at abort (/home/ducaale/swiftlatex-experiment/xetex.wasm/swiftlatexxetex.cjs:1:34670)
    at assert (/home/ducaale/swiftlatex-experiment/xetex.wasm/swiftlatexxetex.cjs:1:24486)
    at allocate (/home/ducaale/swiftlatex-experiment/xetex.wasm/swiftlatexxetex.cjs:1:25758)
    at kpse_find_file_impl (/home/ducaale/swiftlatex-experiment/xetex.wasm/swiftlatexxetex.cjs:1:5185)
    at _kpse_find_file_js (/home/ducaale/swiftlatex-experiment/xetex.wasm/swiftlatexxetex.cjs:1:101967)
    at wasm://wasm/00bc1c92:wasm-function[277]:0x9a139
    at wasm://wasm/00bc1c92:wasm-function[268]:0x99524
    at wasm://wasm/00bc1c92:wasm-function[259]:0x7d944
    at invoke_v (/home/ducaale/swiftlatex-experiment/xetex.wasm/swiftlatexxetex.cjs:1:115551)
    at wasm://wasm/00bc1c92:wasm-function[288]:0x9a972
Aborted(Assertion failed: allocate no longer takes a type argument)

Here is what I did so far:

1. Clone SwiftLaTeX and remove everything except xetex.wasm
2. Use https://github.com/developit/web-worker in place of Worker (update ENGINE_PATH to start with ./ and pass it as new URL(ENGINE_PATH, import.meta.url) to the Worker constructor)
3. Replace XeTeXEngine.tsx with a version compiled by typescript-playground to sidestep tsc issues
4. Polyfill XMLHttpRequest in pre.js using https://github.com/driverdan/node-XMLHttpRequest
5. (optional I think) update extensions to be either .cjs or .mjs. In my case, I am using es-modules so I had to rename swiftlatexxetex.js to swiftlatexxetex.cjs
6. Compile the native code by following https://github.com/SwiftLaTeX/SwiftLaTeX#compile-engines

from swiftlatex.

from swiftlatex.

I think that would be okay if the user is able to compile it in their own worker.

from swiftlatex.

I had a go. (checkout the nodejs branch), currently there is a tricky question. In browser, xmlhttprequest can be set to truly synchronus. While nodejs doesnt seem to support that. This breaks the file fetching module.

from swiftlatex.

We could do a sync fetch now by using execSync(), spawning another node process or even curl to do the fetch. Will come back to it.

from swiftlatex.

I would be spectacular if SwiftLaTeX ran in node.js, and we could publish a npm-package.
@elliott-wen what is the current progress on the nodejs-branch? Are you working on this? Do you need some help?

from swiftlatex.

@dreampulse @ducaale

Yes. I am working on the branch. It is a sorta work but a bit messy.
You can check out the noderunner.js in pdftex.wasm directory. It generates a latex format file without issues.

This is the changes I have made so far. I learned a lot from Ducaale's experience.

1. Changed worker to worker_threads.
2. Use Axios to replace xmlhttprequests and use execsync to emulate synchronous behavior.

Issues:

1. Console.log is working in a weird way. It only pumps out data when the engine stops. Weird.
2. ... messy code

Also, I have no idea how the npm package releasing works. Any help would be very appreciated

from swiftlatex.

Thanks for the update. I was about to recommend Emscripten's Asyncify feature to wrap async js functions but it is good that Axios supports making synchronous HTTP requests. I think it would also be helpful to use the web-workers library so the same code works in both node js and the browser.

By the way, where can I find the latest changes? I wanted to investigate the issues you mentioned but the node js branch is still using XMLHttpRequest instead of Axios.

Also, I have no idea how the npm package releasing works. Any help would be very appreciated

Although my knowledge about Wasm/Emscripten is very limited, I am happy to help with anything related to npm or GitHub actions.

from swiftlatex.

I am currently struggling with this error. Is there a way to know where abort is being called from the native side?

Engine compilation start
Start downloading texlive file https://texlive2.swiftlatex.com/xetex/10/swiftlatexxetex.fmt
Engine compilation finish 8259.939300000668
CompileResult {
  pdf: undefined,
  status: -254,
  log: 'SwiftLaTeX Engine Loaded\n' +
    'This is XeTeX, Version 3.14159265-2.6-0.999992 (XeTeX WebAssembly 0.3.0) (preloaded format=swiftlatexxetex)\n' +
    'Engine crashed'
}
RuntimeError: Aborted(native code called abort())
    at abort (/home/ducaale/swiftlatex-experiment/xetex.wasm/swiftlatexxetex.cjs:1:34645)
    at _abort (/home/ducaale/swiftlatex-experiment/xetex.wasm/swiftlatexxetex.cjs:1:97377)
    at <anonymous>:wasm-function[270]:0x99755
    at <anonymous>:wasm-function[259]:0x7da0a
    at invoke_v (/home/ducaale/swiftlatex-experiment/xetex.wasm/swiftlatexxetex.cjs:1:115526)
    at <anonymous>:wasm-function[288]:0x9a972
    at compileLaTeX (<anonymous>:wasm-function[289]:0x9aa14)
    at /home/ducaale/swiftlatex-experiment/xetex.wasm/swiftlatexxetex.cjs:1:35221
    at compileLaTeXRoutine (/home/ducaale/swiftlatex-experiment/xetex.wasm/swiftlatexxetex.cjs:1:1756)
    at self.onmessage (/home/ducaale/swiftlatex-experiment/xetex.wasm/swiftlatexxetex.cjs:1:3769)
Aborted(native code called abort())

from swiftlatex.

from swiftlatex.

I have done some experiments with Rollup + comlink to see if workers can be used in a library that targets both Node and browsers. The next step is to see how Wasm/Emscripten fits in this setup.

https://github.com/ducaale/workers-rollup-experiment

from swiftlatex.

from swiftlatex.

No worries, I have now managed to set up emscripten in the workers-wasm-rollup-experiment project. The next thing will be to try to apply the learnings from that project into SwiftLaTeX.

from swiftlatex.

FYI, i created some local changes. Most notably the ability to run compilation within containers (using earthly).

See:
BjoernAkAManf@ade77fa

Compilation of the typescript is not great, but produces the same file currently pushed to master.

I think this is might be a better foundation to allow for both node and browser based environments.
Ideally SwiftLatex does not rely on either nodejs or the browser in particular and is able to be used in any wasm environment.

from swiftlatex.

Hi Björn,
Thanks so much. I strongly agree with you that the current build system needs some renovation.
And your earthly script for dvipdfmx.wasm looks pretty neat. We can merge your change if you make it work for pdftex.wasm and xetex.wasm as well.

Regarding the typescript, we can definitely remove it.

from swiftlatex.

I did some experiments in this branch where:

• The comlink library is used to simplify the web worker communication.
• package.json is added so SwiftLaTeX can be published as an npm library (tested locally with Vite and it worked!).
• Removed TypeScript to simplify the build step (as an alternative, either JSDoc or .d.ts could be provided for consumers of the library).

Currently, I am planning to apply the same changes to pdftex.wasm or maybe try to share the js files to avoid any duplication.

By the way, I was wondering what dvipdfm.wasm is used for and if it needs to be included in the final npm package?

from swiftlatex.

Hi @ducaale and @elliott-wen , Did you guys manage to solve this? If yes, your help is appreciated. I did exactly what @ducaale did, and I am also having same issues. self is not defined when I try to import and run engine.loadEngine() from the PdfTeXEngine.js. It seems like self refers to window object from browser. How can I get this working?

Steps I did -

Extracted SwiftLatex
Moved the directory inside my project
Created index.jsx and imported PdfTeXEngine from the directory.
index.jsx looks like below

  const PdfTexEngine = require("./lib/SwiftLaTeX/PdfTeXEngine.js");
  
  async function main() {
    const pdftex = new PdfTexEngine();
    const engine = pdftex.loadEngine();
    console.log(engine);
  }

main();

I also had issues with 'worker' which I removed by const { Worker } = require("node:worker_threads"); inside PdfTeXEngine

from swiftlatex.