[BUG] js_of_ocaml is excessively memory hungry about js_of_ocaml HOT 20 CLOSED

#1614 has been merged. Reopen if you still have issues

from js_of_ocaml.

• --no-inline is no longer necessary (since js_of_ocaml.5.7.0).
• dealing with debug info seems to be slow in your use case. Removing --source-map should speed up your build. I'll try to investigate.
• adding --disable globaldeadcode should give you a good speedup as well.

With the change mentioned, I the following for ExtractionJsOfOCaml/bedrock2_fiat_crypto.js

112.97user 1.26system 1:54.24elapsed 99%CPU (0avgtext+0avgdata 4581104maxresident)k
0inputs+21904outputs (0major+1275732minor)pagefaults 0swaps

from js_of_ocaml.

with #1614, one no longer need to disable globaldeadcode, at the cost of extra memory usage.

78.16user 1.85system 1:20.13elapsed 99%CPU (0avgtext+0avgdata 6034576maxresident)k
0inputs+19696outputs (0major+1631738minor)pagefaults 0swaps

I'll try to improve #1614

from js_of_ocaml.

I've updated #1614, I now get

73.86user 1.00system 1:14.87elapsed 99%CPU (0avgtext+0avgdata 4581396maxresident)k
0inputs+19696outputs (0major+1209270minor)pagefaults 0swaps

I still need to investigate the sourcemap issue.

Can you test #1614 and confirm it solves part of your issue ?

from js_of_ocaml.

@hhugo Regarding this, I have been working on the sourcemap slowness and will open a PR today.

from js_of_ocaml.

@JasonGross, any luck with #1614 ?

from js_of_ocaml.

I have not had a chance to try it, but if it works on your end, I don't see why it would be any different on GitHub Actions. (The files I linked to are the ones I actually use, not simplified examples .). But I can set up GHA to use the PR. Should I just clone the repo and opam pin add . on that branch?

from js_of_ocaml.

#1617 may help too if the memory consumption happens to be in sourcemaps. I find that it nearly halves the peak memory usage when linking JSOO using itself.

from js_of_ocaml.

@JasonGross, I think you can just do

opam pin add js_of_ocaml-compiler https://github.com/ocsigen/js_of_ocaml.git#speedup

from js_of_ocaml.

I've set it up on CI:
pre (specifically here)
post speedup (mit-plv/fiat-crypto#1922) (#1614) (still in progress)
post optim_sourcemap_link (mit-plv/fiat-crypto#1923) (#1617) (still in progress)

from js_of_ocaml.

It looks both #1614 and #1617 make both the run time and the peak memory usage worse. I haven’t worked on #1614 but that surprises me a lot in the case #1617. Are these tests runnable on Linux? I may try to inspect them locally.

from js_of_ocaml.

Not all CI jobs have been updated.

Here is what I see for #1614

1m28.92s | 4461616 ko | ExtractionJsOfOCaml/with_bedrock2_fiat_crypto.js

from js_of_ocaml.

And for #1617

4m47.42s | 5253024 ko | ExtractionJsOfOCaml/with_bedrock2_fiat_crypto.js

from js_of_ocaml.

compared to

6m08.88s | 7720996 ko | ExtractionJsOfOCaml/with_bedrock2_fiat_crypto.js

from js_of_ocaml.

@OlivierNicole, I would expect your PR to only affect separate compilation during the link step but I don't think separate compilation is involved here. What part of your PR would improve the situation during whole program compilation ?

from js_of_ocaml.

Not all CI jobs have been updated.

Here is what I see for #1614

1m28.92s | 4461616 ko | ExtractionJsOfOCaml/with_bedrock2_fiat_crypto.js

I didn’t quite follow which of the many jobs to inspect to find the info, but I trust that your numbers are right.

@OlivierNicole, I would expect your PR to only affect separate compilation during the link step but I don't think separate compilation is involved here. What part of your PR would improve the situation during whole program compilation ?

I’m honestly not sure. Looking into it.

from js_of_ocaml.

I switched to `Stringlit  and Yojson.Raw (rather than `String and Yojson.Basic) because it saves a non-negligible amount of time on the parsing and the writing of the mappings fields of source maps (essentially, Yojson.Basic.to_string (`String s) checks for special characters or Unicode code points in the string, which takes a suprising amount of time and is unnecessary on mappings since they contain only base64 numbers, commas and semicolons.

I’m not sure it explains it all though. Trying to profile locally.

from js_of_ocaml.

Are these tests runnable on Linux?

Yes. The cheapest way to run them is to download any of the artists labeled ExtractionJsOfOCaml-source* from our CI. These artifacts contain a handful of self-contained .ml files, the ones that we want to turn into .js files. I gave the flags I use in the initial post.

The expensive way to run the tests is to clone the repo, do opam install coq, and then run something like make js-of-ocaml

from js_of_ocaml.

I can’t reproduce a significant difference in terms of run time nor profile between master and #1617. The time spent on source maps is negligible compared to the time spent optimizing. I’m starting to suspect that the CI run times have a huge variance.

from js_of_ocaml.

P.S. I’ve done the test on with_bedrock2_fiat_crypto. Peak memory usage is not significantly affected, either.

from js_of_ocaml.