Code Monkey home page Code Monkey logo

self-hosted-shared-dependencies's Introduction

self-hosted-shared-dependencies

A tool for self hosting shared dependencies from npm

Motivation

To share dependencies between microfrontends with SystemJS, you need a URL reachable by the browser for each shared dependency. Using popular CDNs such as jsdelivr.net, unpkg.com, and cdnjs.com is the easiest way to do this, but requires that you rely on a third party service. For some organizations, self-hosting the dependencies is required for security or other reasons.

The self-hosted-shared-dependencies project generates a directory of static frontend assets that can be hosted on a server or CDN of your choosing. The assets are generated upfront, so that the server does not have to do anything more than serve static files. An advantage of generating all files upfront is improved performance, availability, and scalability, since global object stores (such as AWS S3, Digital Ocean Spaces, or GCP Storage) generally are really good at that.

Comparison to other tools

Bundlers like webpack, rollup, etc do not produce separate files for each dependency, by default. Additionally, they generally do not create separate files for different versions of dependencies.

Tools like jspm, snowpack, and vite can do this but often convert the packages to ESM format which is not usable by SystemJS.

esm-bundle libraries produce SystemJS versions of npm packages, but there are only a few dozen libraries available.

Using a forked version of unpkg generally requires running a live server in production which makes calls to the npm registry as it receives requests from users, which is nice because you don't have to specify which packages you're using but also potentially worse for availability, performance, and scalability.

Installation

npm install --save-dev self-hosted-shared-dependencies

yarn add --dev self-hosted-shared-dependencies

pnpm install --save-dev self-hosted-shared-dependencies

# Global installation (optional)

npm install --global self-hosted-shared-dependencies

yarn global add self-hosted-shared-dependencies

pnpm install --global self-hosted-shared-dependencies

Requirements

self-hosted-shared-dependencies requires NodeJS@>=14 (uses ES modules and nullish coalescing operator)

Usage

It's recommended to run self-hosted-shared-dependencies during the CI/CD build and deploy process of a repository called shared-dependencies within your organization. It will generate a static directory of frontend assets, and optionally a Dockerfile for self-hosting the frontend assets. The easiest way to accomplish this is often to add to your npm-scripts in your project's package.json:

{
  "scripts": {
    "build-shared-deps": "shared-deps build shared-deps.conf.mjs"
  }
}

package.json

For simpler use cases, self-hosted-shared-dependencies can read the "dependencies" section of your project's package.json and determine which packages to download. The main limitation of this approach is that you cannot provide package and version specific configuration to control which folders are included in the final output.

To build from package.json, add the --usePackageJSON CLI flag

shared-deps build --usePackageJSON
// Or if you're using an npm-script to build, add the flag to your package.json
{
  "scripts": {
    "build-shared-deps": "shared-deps build --usePackageJSON"
  }
}

Then the "dependencies" in your package.json will be used to determine which versions to include. For example, the code below will result in all React 17 versions being included:

// In your package.json
{
  "dependencies": {
    "react": "^17.0.0"
  }
}

When using the package.json, you do not need to create a shared-deps.conf.mjs file. However, you may combine --usePackageJSON with a config file, if desired, as long as you don't specify packages in the config file (as packages and usePackageJSON are mutually exclusive options).

Config File

For full configuration options, create a shared-deps.conf.mjs file:

// shared-deps.conf.mjs

/**
 * @type {import('self-hosted-shared-dependencies').BuildOpts}
 */
const config = {
  // Required if not using package.json, a list of npm package versions to include in the output directory
  packages: [
    {
      // Required. The name of the package to include
      name: "react",

      // Optional. A list of glob strings used to determine which files within
      // the package to include in the build. By default, all files are included.
      // See https://www.npmjs.com/package/micromatch for glob implementation
      // Note that package.json and LICENSE files are always included.
      include: ["umd/**"],

      // Optional. A list of glob strings used to determine which files within
      // the package to exclude from the build. By default, no files are excluded.
      // See https://www.npmjs.com/package/micromatch for glob implementation
      // Note that package.json and LICENSE files are always included.
      exclude: ["cjs/**"],

      // Required. A list of semver ranges that determine which versions of the
      // npm package should be included in the build.
      // See https://semver.npmjs.com/ for more details
      versions: [
        // When the version is a string, the package's include and exclude lists
        // are applied
        ">= 17",

        // When the version is an object, the version's include and exclude lists
        // take priority over the package's include and exclude lists
        {
          version: "16.14.0",
          include: ["umd/**", "cjs/**"],
        },
      ],
    },
  ],

  // Optional, defaults to false
  // When true, will parse the package.json file and use the
  // dependencies as the package list
  usePackageJSON: false,

  // Optional, defaults to "npm"
  // Change the name of the output directory where the static assets
  // will be placed. The outputDir is resolved relative to the CWD
  outputDir: "npm",

  // Optional, defaults to false
  // When true, the outputDir will be deleted at the beginning of the build
  clean: false,

  // Optional, defaults to false.
  // When true, a Dockerfile will be created in your static directory.
  // The Dockerfile uses nginx:latest as its base image
  generateDockerfile: false,

  // Optional, defaults to building all packages (no skipping)
  // When provided, this allows you to do incremental builds where
  // the build first calls out to your live server hosting your
  // shared dependencies to decide whether it needs to rebuild
  // the package. This is a performance optimization that makes the
  // build faster. For each package version, it will check
  // <skipPackagesAtUrl>/<packageName>@<version>/package.json to
  // see if it needs to build the package version or not
  skipPackagesAtUrl: "https://cdn.example.com/npm/",

  // Optional, defaults to {}.
  // When provided, this allows you to configure the behavior of npm-registry-fetch,
  // such as providing username, password, or token to access private npm packages.
  // See https://github.com/npm/npm-registry-fetch#-fetch-options for documentation
  registryFetchOptions: {
    username: "test",
    password: "test",
    token: "test",
    registry: "https://registry.npmjs.org/",
  },

  // Optional, defaults to "debug". Must be one of "debug", "warn", or "fatal"
  // This changes the verbosity of the stdout logging
  logLevel: "warn",

  // Optional, defaults to true. This is a safeguard against the clean operation deleting important directories accidentally, by forcing them to be absolute paths. To disable that behavior, set to false.
  absoluteDir: true,
};

export default config;

Now you can run npm run build to generate the output directory.

Once you have the output directory, you can run npx http-server npm to start up a server that hosts the files. In CI processes, usually the output directory is uploaded to a live server as part of a deployment.

Example output

Here's an example showing the file structure created by running shared-deps build

npm
npm/Dockerfile
npm/[email protected]
npm/[email protected]/LICENSE
npm/[email protected]/umd
npm/[email protected]/umd/react.production.min.js
npm/[email protected]/umd/react.development.js
npm/[email protected]/umd/react.profiling.min.js
npm/[email protected]/package.json
npm/[email protected]
npm/[email protected]/LICENSE
npm/[email protected]/umd
npm/[email protected]/umd/react.production.min.js
npm/[email protected]/umd/react.development.js
npm/[email protected]/umd/react.profiling.min.js
npm/[email protected]/package.json
npm/[email protected]
npm/[email protected]/LICENSE
npm/[email protected]/umd
npm/[email protected]/umd/react.production.min.js
npm/[email protected]/umd/react.development.js
npm/[email protected]/umd/react.profiling.min.js
npm/[email protected]/package.json
npm/[email protected]
npm/[email protected]/LICENSE
npm/[email protected]/umd
npm/[email protected]/umd/react-dom-server.browser.development.js
npm/[email protected]/umd/react-dom.production.min.js
npm/[email protected]/umd/react-dom.profiling.min.js
npm/[email protected]/umd/react-dom-test-utils.production.min.js
npm/[email protected]/umd/react-dom.development.js
npm/[email protected]/umd/react-dom-server.browser.production.min.js
npm/[email protected]/umd/react-dom-test-utils.development.js
npm/[email protected]/package.json

Docker

To host the output directory in a server running in a docker container, set the generateDockerfile option to true. That will produce an npm/Dockerfile file which you can use to create an image and run containers.

To test the docker container, run the following:

# assumes that your outputDir is set to "npm"

# build the image
docker build npm -t shared-deps

# run the image as a container, exposing it to your host computer's port 8080
docker run --name shared-deps -d -p 8080:80 shared-deps

# verify that you can retrieve one of the built files
curl http://localhost:8080/npm/[email protected]/umd/react.production.min.js

# shut down the container
docker stop shared-deps

CLI

The CLI has the following flags:

shared-deps build shared-deps.conf.mjs --clean --outputDir npm --generateDockerfile --skipPackagesAtUrl https://cdn.example.com/npm/ --logLevel warn

Javascript API

You may also use this project via javascript. Note that it is published as an ES module so you must use import or import() to use it, you cannot use require().

import { build } from "self-hosted-shared-dependencies";

build({
  // This object is the same as the object exported from the Config File above
  packages: [
    {
      name: "react",
      include: ["umd/**"],
      exclude: ["cjs/**"],
      versions: [
        ">= 17",
        {
          version: "16.14.0",
          include: ["umd/**", "cjs/**"],
        },
      ],
    },
  ],
  usePackageJSON: false,
  outputDir: "npm",
  clean: false,
  generateDockerfile: false,
  skipPackagesAtUrl: "https://cdn.example.com/npm/",
  logLevel: "warn",
}).then(
  () => {
    console.log("Finished!");
  },
  (err) => {
    console.error(err);
    process.exit(1);
  }
);

self-hosted-shared-dependencies's People

Contributors

joeldenning avatar jsmapr1 avatar miguel-silva avatar mstergianis avatar prof-schnitzel avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar

self-hosted-shared-dependencies's Issues

Default npm registry is always used

I am using a private registry which I enabled with the command npm config set registry <url-to-registry>. The self-hosted-shared-dependencies tool always seems to use registry.npmjs.org regardless of what is configured locally. Is it possible to make the tool respect the configured npm registry?

[Question] Do private package registry download is working?

Hi folks!

After some long debugging I was able to retrieve the metadata from the private package registry:

 registryFetchOptions: {
        "//registry.npmjs.org/@mycompany/mypackage:_authToken": "npm_my-auth-token"
    },

// OR

 registryFetchOptions: {
       forceAuth: {
         token: "npm_my-auth-token"
      }
    },

these are the ways that I found to allow successfully retrieve the package metadata. But it fails when trying to download the tar file. Reviewing the code I can't see in this line

that you authenticate the request to download the tar file properly from the private registry.

Just wondering if this is something that happens in another place or is not working at all at this version (2.0.0).

Regards,
Leonardo Monge GarcΓ­a.

Dependencies with a slash (foo/bar) do not work

When putting a package with a slash into the shared-deps.conf.mjs file (like @company/my-lib), the tool crashes with an exception, since it does not create subdirectories recursively. It needs to create first @company and then my-lib as a subdirectory, but this won't happen because the recursive option of the command fs.mkdir is not used.

Per version include/exclude not working

Include/exclude per version is not working.
From README

{
      versions: [
        // When the version is a string, the package's include and exclude lists
        // are applied
        ">= 17",

        // When the version is an object, the version's include and exclude lists
        // take priority over the package's include and exclude lists
        {
          version: "16.14.0",
          include: ["umd/**", "cjs/**"],
        },
      ],
}

The problem is at self-hosted-shared-dependencies.js in lines 310 and 310

In that context matchedVersion is always a string as it comes from npm's metadata. Line 247

Would it also be possible to use the files from node_modules?

Hey there, I'm wondering why we are downloading things from npm "again".

Especially in the case of using the usePackageJSON option, after installing the dependencies for a project, all those dependencies are already available in the local file system.

What would it take to just use those files?
Or is there a good reason not to do that?

Is there support for private packages from npm?

Many organizations have private packages hosted on the public registry. I think npm-registry-fetch has a way to handle that. But I'm not sure if this package does so. It would be an easy addition with options, but if there is already a way what would it be?

Fetching of dist archive for private packages is broken

Hi! πŸ‘‹

Firstly, thanks for your work on this project! πŸ™‚

I've noticed that lib was only using npm fetch to fetch metadata, but tarball should also be fetched using npmFetch so that private packages are working? It also slightly simplifies code (in my patch i assume i am not handling errors)

Here is the diff that solved my problem:

diff --git a/node_modules/self-hosted-shared-dependencies/lib/self-hosted-shared-dependencies.js b/node_modules/self-hosted-shared-dependencies/lib/self-hosted-shared-dependencies.js
index c09225d..8879a03 100644
--- a/node_modules/self-hosted-shared-dependencies/lib/self-hosted-shared-dependencies.js
+++ b/node_modules/self-hosted-shared-dependencies/lib/self-hosted-shared-dependencies.js
@@ -335,27 +335,8 @@ export async function build({
         });
 
         const tarballUrl = metadata.versions[matchedVersion].dist.tarball;
-        const requestStream = (
-          tarballUrl.startsWith("http://") ? http : https
-        ).request(tarballUrl);
 
-        requestStream.on("response", (responseStream) => {
-          responseStream.pipe(untarStream);
-        });
-
-        requestStream.on("timeout", () => {
-          reject(
-            Error(
-              `Request timed out to download tarball for ${p.name}@${matchedVersion}`
-            )
-          );
-        });
-
-        requestStream.on("error", (err) => {
-          reject(err);
-        });
-
-        requestStream.end();
+        npmFetch(tarballUrl, npmFetchOptions).then((res) => res.body.pipe(untarStream)).catch(reject);
 
         untarStream.on("end", () => {
           resolve();

Not compatible with windows

The line import(path.resolve(process.cwd(), configFile)) in bin/self-hosted-shared-dependencies-cli.js is not compatible with windows since you need correct URLs for the import command that start with file://.

`npmRegistry` is not passed in to `npm-registry-fetch`

It appears npmRegistry doesn't do anything. Running with and without the option gave me the same error for a private package.

--> @vmware/some-private-lib
HttpErrorGeneral: 404 Not Found - GET https://registry.npmjs.org/@vmware/some-private-lib - Not found
    at <project-dir>/node_modules/npm-registry-fetch/check-response.js:134:15
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at async buildPackage (file:///<project-dir>/node_modules/self-hosted-shared-dependencies/lib/self-hosted-shared-dependencies.js:235:18)
    at async build (file:///<project-dir>/node_modules/self-hosted-shared-dependencies/lib/self-hosted-shared-dependencies.js:208:20)
    at async file:///<project-dir>/node_modules/self-hosted-shared-dependencies/bin/self-hosted-shared-dependencies-cli.js:29:9 {
  headers: [Object: null prototype] {
    date: [ 'Wed, 11 Aug 2021 20:46:42 GMT' ],
    'content-type': [ 'application/json' ],
    'content-length': [ '21' ],
    connection: [ 'keep-alive' ],
    'expect-ct': [
      'max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"'
    ],
    vary: [ 'Accept-Encoding' ],
    server: [ 'cloudflare' ],
    'cf-ray': [ '67d450987ea7cab4-YYZ' ],
    'x-fetch-attempts': [ '1' ]
  },
  statusCode: 404,
  code: 'E404',
  method: 'GET',
  uri: 'https://registry.npmjs.org/@vmware/some-private-lib',
  body: { error: 'Not found' },
  pkgid: 'some-private-lib'
}

Getting error "unable to open X server"

I followed the instructions in the README.md and used the same content for the file shared-deps.conf.mjs. When I execute the command npm run build-shared-deps I get the following error:

import-im6.q16: unable to open X server `' @ error/import.c/ImportImageCommand/358.
import-im6.q16: unable to open X server `' @ error/import.c/ImportImageCommand/358.
import-im6.q16: unable to open X server `' @ error/import.c/ImportImageCommand/358.
/node_modules/.bin/shared-deps: 5: /node_modules/.bin/shared-deps: Syntax error: "(" unexpected

I am using node with version 14.16.1.

You can also reproduce this issue by executing docker run -it node:14 bash and copying all scripts from README.md. Or by just cloning polyglot-microfrontends/shared-dependencies and running npm run build.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    πŸ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. πŸ“ŠπŸ“ˆπŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❀️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.