simonw / download-esm Goto Github PK

View Code? Open in Web Editor NEW

84.0 4.0 3.0 17 KB

Download ESM modules from npm and jsdelivr

License: Apache License 2.0

Python 100.00%

download-esm's Introduction

download-esm

Download ESM modules from npm and jsdelivr

See download-esm: a tool for downloading ECMAScript modules for background on this project.

Installation

Install this tool using pip:

pip install download-esm

Warning

This is alpha software. It works for downloading Observable Plot, but has not been tested against many other packages yet.

Your help welcome in testing this further!

Usage

To download Observable Plot and all of its dependencies as ECMAScript modules:

download-esm @observablehq/plot

This will download around 40 .js files to the current directory.

To put them in another directory, add that as an argument:

download-esm @observablehq/plot ./js

Each file will have any import and export statements rewritten as relative paths.

You can then use the library in your own HTML and JavaScript something like this:

<div id="myplot"></div>
<script type="module">
import * as Plot from "./js/observablehq-plot-0-6-6.js";
const plot = Plot.rectY(
    {length: 10000}, Plot.binX({y: "count"}, {x: Math.random})
).plot();
const div = document.querySelector("#myplot");
div.append(plot);
</script>

Development

To contribute to this tool, first checkout the code. Then create a new virtual environment:

cd download-esm
python -m venv venv
source venv/bin/activate

Now install the dependencies and test dependencies:

pip install -e '.[test]'

To run the tests:

pytest

download-esm's People

Contributors

Stargazers

Watchers

Forkers

alphapapa pauliojanpera jraedisch

download-esm's Issues

Consider Skypack CDN

Skypack (previously Pika) seems well aligned with your needs and may make it simpler to download more packages, especially ones also loading CSS, Icons or WSAM files to function.

Load optimized npm packages with no install and no build tools
https://www.skypack.dev/

The pinned URL's feature in particular would reduce the number of files downloaded.

Consider replacing jsDelivr with cjstoesm or something similar

I really like the idea of download-esm but after trying it out there is one thing bothering me: The downloaded files are minified. I tried to figure out how to get jsDelivr to produce the esm version of a module without also minification it but so far I haven't found a way to make that happen.

I thought it would be as simple as using the content of the file referenced by the path returned from extract_original_file but it turns out that in many cases that refers to the cjs module. That is, many modules don't have esm compatible code but an esm version is entirely synthesized by jsDelivr.

Since I couldn't figure out how to get the un-minified esm version of a cjs package from jsDeliver, I looked into other ways to do it and I found https://github.com/wessberg/cjstoesm which seems like it could work on an arbitrary npm package and produce plain JavaScript without minification.

Not sure if this is interesting to anyone else or if minified versions of dependencies are what you prefer. I'm going to try my hand at writing a tool similar to download-esm that is essentially just a wrapper around cjstoesm.

I guess I'm curious if there is interest in developing download-esm into something slightly more complex or whether I should just release a separate tiny tool that does this very similar but slightly different thing.

Consider importmaps support

https://web.dev/import-maps-in-all-modern-browsers/

It might be possible to have it rewrite modules it downloads to use this syntax, and to spit out an import maps block to make that work.

Initial design

Inspired by this comment by @mbostock:

observablehq/plot#1496 (comment)

I think maybe the answer here is that someone should write a “downloader” tool that downloads the compiled ES modules from jsDelivr (or other CDN) and rewrites the import statements to use relative paths. Then you could just download this URL

https://cdn.jsdelivr.net/npm/@observablehq/plot/+esm

and you’d get the direct dependencies

https://cdn.jsdelivr.net/npm/[email protected]/+esm https://cdn.jsdelivr.net/npm/[email protected]/+esm https://cdn.jsdelivr.net/npm/[email protected]/+esm

and the transitive dependencies and so on as separate files.

Why not esbuild?

This is a neat tool! But I think you can get everything you want with more standard tools, if I understand this right.

I initially thought the idea was that you wanted to preserve the original source code of your dependencies so you could edit it later if need be. But I see that in fact you're using jsdeliver's +esm mode, which is already running a minifier as well as a tool to convert cjs to esm, which is somewhat fragile but is the only reason this works at all. So you've already given up on keeping the original source code.

If you don't have that constraint, the way a JS developer would solve this problem is by bundling. For example, here's a complete workflow using esbuild:

Install node (which will include npm)
echo '{}' > package.json'
npm install @observablehq/plot
echo 'export * from "@observablehq/plot"' | npx esbuild --bundle --format=esm > observable-plot.js - note that this will download and run esbuild; you can also get the esbuild binary yourself and plop it somewhere on your PATH and drop the npx part
in your script, import * as Plot from "./observable-plot.js"

That will produce a single ES module named observable-plot.js which includes all of your dependencies, and which exports the things that @observablehq/plot does. It will also be a lot more readable than the things you get off of jsdeliver because it's not minified, though if you actually want minification you can pass --minify to esbuild.

This has a few advantages: 1) doesn't rely on jsdelivr and whatever opaque transformations it is doing; 2) produces a single file, which is better for the network and also can be smaller because it can do better dead-code elimination; 3) does not require python.

I know you said you didn't want a build step, but this is a run-once step, exactly the same way that running download-esm would be; you don't need to re-run when making changes. The output is a JS file you can drop on your server, load from your HTML, and code against for as long as you'd like.

Make this robust in the face of more packages

So far I've tested this on exactly one package: @observablehq/plot.

It would be good to test this on some other packages too! Help very much appreciated here.

Any packages that don't work can be used to expand the test suite. It's all in triple-string-literals in https://github.com/simonw/download-esm/blob/0.1a0/tests/test_download_esm.py at the moment, but it would make sense to turn those HTTPX fixtures into actual files on disk as it expands.