Code Monkey home page Code Monkey logo

yarnpnp2nix's Introduction

yarnpnp2nix

Yet another way of packaging Node applications with Nix. Unlike alternatives, this plugin is built for performance (both speed and disk usage) and aims to be unique with the following goals:

  • NPM dependencies should be stored in the /nix/store individually rather than in a huge "node_modules" derivation. Zip files should be used where possible to work well with Yarn PNP
  • Rebuilding when just changing source code should be fast as dependencies shouldn't be fetched
  • Adding a new dependency should just fetch that dependency rather than fetching or linking all node_modules again
  • Build native modules (e.g canvas) once, and once they are built they shouldn't be built again across packages
  • Unplugged/native modules should have their outputs hashed to try and enforce reproducibility
  • When using workspaces, adding a dependency in another package (modifying the yarn.lock file) in the workspace shouldn't cause a different package to have to be rebuilt
  • devDependencies shouldn't be included as references in the final runtime derivation, only dependencies

Usage

Requires a Yarn version > 4 project using PnP linking (the default). Zero installs are not required, so it's recommended to just use the global cache when developing your project rather than storing dependencies in your repo.

yarnpnp2nix is currently tested with Yarn version 4.0.1.

  • Install Yarn plugin in your project:

    yarn plugin import https://github.com/madjam002/yarnpnp2nix/raw/master/plugin/dist/plugin-yarnpnp2nix.js
    
  • Run yarn to make sure all packages are installed and to automatically generate a yarn-manifest.nix for your project.

  • Create a flake.nix if you haven't already for your project, add this repo (yarnpnp2nix) as an input (e.g yarnpnp2nix.url = github:madjam002/yarnpnp2nix;)

  • See test/flake.nix for an example on how to create Nix derivations for Yarn packages using mkYarnPackagesFromManifest.

Quick examples

Setting a build command for a package:

mkYarnPackagesFromManifest {
    yarnManifest = import ./workspace/yarn-manifest.nix;
    packageOverrides = {
        "my-package@workspace:packages/my-package".build = ''
            // Any custom build logic here
        '';
    };
};

Fixing a hash mismatch:

mkYarnPackagesFromManifest {
    yarnManifest = import ./workspace/yarn-manifest.nix;
    packageOverrides = {
        "my-package@workspace:packages/my-package".outputHash = "sha512-4pNZfI6GbsEsBySIs+gK98AGZhWf9QZ3SLytsWIzLnCeJYt2ma6qVK5Gk4TSHsUOmSjqUX8seBCKBBL7f1pvTQ==";
    };
};

Other notes

Known caveats:

  • Initial build can be a bit slow as each dependency is a separate derivation and needs to be fetched from the package registry. Still, a sample project with a couple of thousand dependencies (!!) only takes a couple of minutes to build all of the dependency derivations. Make sure you have enough parallelism in your nix build (either using the -j argument or setting your Nix config appropriately)

  • Patched packages with localy stored patches need to have their revision strings updated to use relative paths instead of the default ~/ prefix

    For example, to fix a revision of a package stored in packages/my-package (relative to the workspace root), update the revision as such:

- "next": "patch:next@npm%3A13.2.1#~/.yarn/patches/next-npm-13.2.1-585715321e.patch",
+ "next": "patch:next@npm%3A13.2.1#../../.yarn/patches/next-npm-13.2.1-585715321e.patch",

Possible future improvements:

  • When adding a Yarn package, copy it straight into the nix store rather than a Yarn cache in the users home directory
  • ...and run postinstall/install builds from within Nix by defaulting Yarn to --ignore-scripts

License

Licensed under the MIT License.

View the full license here.

yarnpnp2nix's People

Contributors

frantisekhanzlikbl avatar madjam002 avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar

yarnpnp2nix's Issues

Working with workspaces containing multiple applications\services

Hi @madjam002

I understand that you are using this plugin with some large workspaces.

Would you mind sharing how you handle building sub-projects within a workspace? Do you just build them all in one massive derivation based on top level yarn.lock? Or do you maintain individual lock files for sub-projects which allow them to be built in isolation?

Thanks!

simplify some of the nix code

would you be open to a PR simplifying some of the nix code, to make it more readable?

For example:

_outputHash = if builtins.hasAttr "outputHash" packageManifest && packageManifest.outputHash != null then packageManifest.outputHash else null;

could be rewritten to:

 _outputHash = packageManifest.outputHash or null;

same thing could be applied to most cases where hasAttr is used in the above file

Parse yarn.lock file directly

Investigate whether we can just parse yarn.lock instead of relying on yarn-manifest.nix.

The manifest would still be a useful optional addon for automatically calculated outputHashes for unplugged modules.

Most info is there in the lock file, dependencies, package names and references, but the difficult bits will probably be:

  • Working out Yarn virtual packages (for peer dependencies, see https://yarnpkg.com/advanced/lexicon#virtual-package), we rely on virtual packages quite a bit. I don't see any easy way of doing this without hacks like import-from-derivation, as virtual package info isn't found in the yarn lockfile.
    Perhaps we first investigate removing virtuals from the manifest file and calculating them at runtime in generate-pnp-file and generate-lockfile, and then we're better positioned to parse the lockfile directly.

  • devDependencies... we'd probably have to source in package.json's of local packages into Nix and then read those to see which dependencies are devDependencies

  • bin paths, we can get this from the lock file, but we should source in the package.json for local packages and read the bin from there so that the lock file doesn't need to be regenerated when adding a new bin during local development

  • shouldBeUnplugged this is a real PITA, I don't think there's any way to avoid this one, so we'll still need the manifest for this and perhaps if the manifest is not provided (as we want to make it optional) then a side effect is any unplugged packages won't be unplugged, perhaps this isn't such a big deal

Cleanup plugin codebase

The plugin/sources/index.ts source code is just thrown together so it is rather messy and even fails typechecking in most places.

TypeError: S.getCustomDataKey is not a function

I'm trying to package this application, but running yarn in the root I get this error:

➤ YN0001: TypeError: S.getCustomDataKey is not a function
    at /home/turing/src/actual/actual-server/.yarn/plugins/@yarnpkg/plugin-yarnpnp2nix.cjs:118:8117
    at Array.map (<anonymous>)
    at afterAllInstalled (/home/turing/src/actual/actual-server/.yarn/plugins/@yarnpkg/plugin-yarnpnp2nix.cjs:118:8080)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async Na.triggerHook (/home/turing/src/actual/actual-server/.yarn/releases/yarn-3.4.1.cjs:392:3295)
    at async je.install (/home/turing/src/actual/actual-server/.yarn/releases/yarn-3.4.1.cjs:442:4457)
    at async /home/turing/src/actual/actual-server/.yarn/releases/yarn-3.4.1.cjs:499:12044
    at async Ge.start (/home/turing/src/actual/actual-server/.yarn/releases/yarn-3.4.1.cjs:409:2387)
    at async cu.execute (/home/turing/src/actual/actual-server/.yarn/releases/yarn-3.4.1.cjs:499:11944)
    at async cu.validateAndExecute (/home/turing/src/actual/actual-server/.yarn/releases/yarn-3.4.1.cjs:345:664)
➤ YN0000: Failed with errors in 1m 18s

I know very little of yarn, let alone yarn 3, so please forgive if I'm doing something silly :)

Can you advise?

infinite recursion encountered

I've tried migrating my react project with workspaces to yarnpnp2nix, but encountered error during package building with nix build
`

   … while evaluating attribute 'unplugPhase' of derivation 'babel-helper-compilation-targets-7.23.6-aa6f07f088'

     at /nix/store/mmihlxna7x7wryngb0h47n0ssbravph0-source/lib/mkYarnPackage.nix:309:9:

      308|
      309|         unplugPhase =
         |         ^
      310|           # for debugging:

   … from call site

     at /nix/store/mmihlxna7x7wryngb0h47n0ssbravph0-source/lib/mkYarnPackage.nix:153:30:

      152|
      153|       createLockFileScript = mkCreateLockFileScript_internal {
         |                              ^
      154|         inherit packageRegistry;

   … while calling 'mkCreateLockFileScript_internal'

     at /nix/store/mmihlxna7x7wryngb0h47n0ssbravph0-source/lib/mkYarnPackage.nix:67:5:

       66|   mkCreateLockFileScript_internal =
       67|     {
         |     ^
       68|       packageRegistry,

   … while evaluating derivation 'browserslist-4.22.3-c3c1809375'
     whose name attribute is located at /nix/store/80v3x99d9cl7h9fbhqrpajwg4vjyxg6y-source/pkgs/stdenv/generic/make-derivation.nix:348:7

   … while evaluating attribute 'unplugPhase' of derivation 'browserslist-4.22.3-c3c1809375'

     at /nix/store/mmihlxna7x7wryngb0h47n0ssbravph0-source/lib/mkYarnPackage.nix:309:9:

      308|
      309|         unplugPhase =
         |         ^
      310|           # for debugging:

   … from call site

     at /nix/store/mmihlxna7x7wryngb0h47n0ssbravph0-source/lib/mkYarnPackage.nix:153:30:

      152|
      153|       createLockFileScript = mkCreateLockFileScript_internal {
         |                              ^
      154|         inherit packageRegistry;

   … while calling 'mkCreateLockFileScript_internal'

     at /nix/store/mmihlxna7x7wryngb0h47n0ssbravph0-source/lib/mkYarnPackage.nix:67:5:

       66|   mkCreateLockFileScript_internal =
       67|     {
         |     ^
       68|       packageRegistry,

   … while evaluating derivation 'update-browserslist-db-1.0.13-ea7b8ee24d'
     whose name attribute is located at /nix/store/80v3x99d9cl7h9fbhqrpajwg4vjyxg6y-source/pkgs/stdenv/generic/make-derivation.nix:348:7

   … while evaluating attribute 'unplugPhase' of derivation 'update-browserslist-db-1.0.13-ea7b8ee24d'

     at /nix/store/mmihlxna7x7wryngb0h47n0ssbravph0-source/lib/mkYarnPackage.nix:309:9:

      308|
      309|         unplugPhase =
         |         ^
      310|           # for debugging:

   … from call site

     at /nix/store/mmihlxna7x7wryngb0h47n0ssbravph0-source/lib/mkYarnPackage.nix:153:30:

      152|
      153|       createLockFileScript = mkCreateLockFileScript_internal {
         |                              ^
      154|         inherit packageRegistry;

   … while calling 'mkCreateLockFileScript_internal'

     at /nix/store/mmihlxna7x7wryngb0h47n0ssbravph0-source/lib/mkYarnPackage.nix:67:5:

       66|   mkCreateLockFileScript_internal =
       67|     {
         |     ^
       68|       packageRegistry,

   error: infinite recursion encountered

   at /nix/store/80v3x99d9cl7h9fbhqrpajwg4vjyxg6y-source/lib/customisation.nix:250:7:

      249|       drvPath = assert condition; drv.drvPath;
      250|       outPath = assert condition; drv.outPath;
         |       ^
      251|     };`

I suspect that it has something to do with how the yarn-manifest.nix is generated, because of code that looks like a cyclic dependency ( packages."browserslist@npm:4.22.3 referring to update-browserslist-db@npm:1.0.13 and update-browserslist-db@npm:1.0.13 back to former)

"update-browserslist-db@npm:1.0.13" = {
    name = "update-browserslist-db";
    reference = "npm:1.0.13";
    linkType = "HARD";
    outputName = "update-browserslist-db-1.0.13-ea7b8ee24d";
    outputHash = "";
    shouldBeUnplugged = true;
    flatName = "update-browserslist-db";
    languageName = "node";
    scope = null;
    descriptorRange = "npm:^1.0.13";
    checksum = "10c0/e52b8b521c78ce1e0c775f356cd16a9c22c70d25f3e01180839c407a5dc787fb05a13f67560cbaf316770d26fa99f78f1acd711b1b54a4f35d4820d4ea7136e6";
    bin = {
      "update-browserslist-db" = "cli.js";
    };
    dependencies = {
      "escalade" = packages."escalade@npm:3.1.2";
      "picocolors" = packages."picocolors@npm:1.0.0";
    };
    packagePeers = [
      "@types/browserslist"
      "browserslist"
    ];
    };
    "update-browserslist-db@virtual:c3c1809375389507a1aeedf534fd79f857bb3c64ecc9e006768cadbfe62986d5f0c6d60929598771fa39a935d74fd2f97a9ee5a236e30da17e2b3a480f1ecae0#npm:1.0.13" = {
    name = "update-browserslist-db";
    reference = "virtual:c3c1809375389507a1aeedf534fd79f857bb3c64ecc9e006768cadbfe62986d5f0c6d60929598771fa39a935d74fd2f97a9ee5a236e30da17e2b3a480f1ecae0#npm:1.0.13";
    canonicalPackage = packages."update-browserslist-db@npm:1.0.13";
    dependencies = {
      "browserslist" = packages."browserslist@npm:4.22.3";
      "escalade" = packages."escalade@npm:3.1.2";
      "picocolors" = packages."picocolors@npm:1.0.0";
    };
};
and
"browserslist@npm:4.22.3" = {
    name = "browserslist";
    reference = "npm:4.22.3";
    linkType = "HARD";
    outputName = "browserslist-4.22.3-c3c1809375";
    outputHash = "";
    shouldBeUnplugged = true;
    flatName = "browserslist";
    languageName = "node";
    scope = null;
    descriptorRange = "npm:^4.22.2";
    checksum = "10c0/5a1f673ce0d6e61a68369835a6b66e199669bde02c3bed5ec51e77598d8daafd91719dba55b15af2021b9ad0bbaa94951fd702eb71087449eb28be8002815ece";
    bin = {
      "browserslist" = "cli.js";
    };
    dependencies = {
      "caniuse-lite" = packages."caniuse-lite@npm:1.0.30001585";
      "electron-to-chromium" = packages."electron-to-chromium@npm:1.4.665";
      "node-releases" = packages."node-releases@npm:2.0.14";
      "update-browserslist-db" = packages."update-browserslist-db@virtual:c3c1809375389507a1aeedf534fd79f857bb3c64ecc9e006768cadbfe62986d5f0c6d60929598771fa39a935d74fd2f97a9ee5a236e30da17e2b3a480f1ecae0#npm:1.0.13";
    };
};

Does anyone have idea how to fix it?

don't use FOD for unplugged packages

as it leads to hash mismatch errors due to inability to predict the hash of a FOD for all platforms and hence bad user experience

FODs are also not rebuilt when their inputs change and so breaking changes can slip in if FOD with given hash is already ix nix store (or cache)

I'm happy to work on this, but would be great to know if there was any particular reason it was done this way or if you foresee any potential blockers.

related

btw. https://github.com/stephank/yarn-plugin-nixify (or rather my fork of it) correctly handles unplugged packages so it is definitely possible to implement

Hash mismatch for node-gyp

Trying to set up yarnpnp2nix in https://github.com/niklaskorz/puredit/tree/yarnpnp2nix. There, building nix build .#example leads to the following error:

error: hash mismatch in fixed-output derivation '/nix/store/lrn7xkr2d0d0ikkiix534zqgs54274v0-node-gyp-9.3.1-43540bab9c.drv':
         specified: sha512-Rq/Fv8Ya1YML4urA/z6/BEmHhMBTqGFKD+1P/u7f4RuqEUiulhyyMkylVqHPp7mlBRobRZEwiPosvDZDo7qSVw==
            got:    sha512-050SzpW2p/TcC3FmCzOVwPGmRUxVQXTdylIroIB2yJ3Q9Z6TXiiKOeSZvIWwfQtsNrCKd2eRvyShZPxSiGiNGg==
error: 1 dependencies of derivation '/nix/store/i2h53i1j6165skmmrfmnlm8f9p77ydii-puredit-example-0.0.0-eb98272acc.drv' failed to build

The system is aarch64-darwin.

./runTests.sh fails with hash mismatch

error: hash mismatch in fixed-output derivation '/nix/store/6hhh94ldl0555ddh9w5vlw8p6v4nn8z3-esbuild-linux-64-0.15.10-702e43a31d.drv':
         specified: sha512-H6P3xJuvxQ6JN1xAWDjZGRlHCgBju4iJZlvlhrUnDsqD1ZLZhd7gKYuN+ALWIRW+zxVqYAN9rLTKKZE5SECjXg==
            got:    sha512-DN7uyk6C+SwXhiz7do0pu8+naQBURMYqDNb8EFzdH34fkeqz2dCiT88mxJV14y6hRIGHWbukh5KEgWLxH9AvLQ==
error: 1 dependencies of derivation '/nix/store/8gvrlx95892qnbv343077ym7g6p36hfr-testb-0.0.0-819008c351.drv' failed to build

btw. the script assumes jq is installed, which was not the case on my machine

would be better to nixify runTest as well :)

Executing yarn scripts as part of `build` fails

When using yarn inside a build script, e.g.

"testb@workspace:packages/testb" = {
  build = ''
    yarn build # defined in package.json as `node build.js`
  '';
};

The following error occurs:

Internal Error: testb@workspace:.: This package doesn't seem to be present in your lockfile; run "yarn install" to update the lockfile

Incompatible with Nix 2.19

As of Nix 2.19, leading dots are not allowed in store path names: NixOS/nix#9095

This cause issues with the pnp loader:

error: store path 'vpl7h5psym1p892kk0k1xagwn48490ja-.pnp.loader.mjs' starts with illegal character '.'

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.