0x80 / isolate-package Goto Github PK

Currently, yarn lockfiles are copied to the isolated package as-is. However, I think there's an easy solution. Running yarn install in the isolated package will prune unused entries from the lockfile. I believe the resulting lockfile will pin the dependencies of the isolated package to the same dependencies that were pinned in the monorepo. It will also pin the file: dependencies as expected, so there shouldn't be any issue installing the package with --frozen-lockfile in a deploy environment.

It's important that yarn doesn't consider the isolated package as a workspace, but otherwise it seems sound.

Does this seem sensible?

I'll test this in my own repo by running yarn install in a script after isolate. My use case is also Firebase functions.

In situations where the lockfile isn't working, and fallback to NPM is not an option (PNPM) we still want an escape hatch.

Hey. So perhaps this is a misconfiguration on my part, but firebase code reloading is not working for me when isolate-package is involved.

Excerpt from firebase.json:

{
  "functions": [
    {
      "source": "./isolate",
      "runtime": "nodejs20",
      "codebase": "default",
      "ignore": [
        "node_modules",
        ".git",
        "firebase-debug.log",
        "firebase-debug.*.log"
      ],
      "predeploy": [
        "npm run build",
        "npm run isolate"
      ]
    }
  ]
}

So, npm run build causes tsc to emit into ./dist and then npm run isolate bundles that all up into ./isolate. So far so good, the emulators boot up fine and my functions handle requests as expected.

If I now make some changes and run npm run build, obviously nothing will change in the emulators as they are not watching that directory. Then, if I then run npm run isolate again, I would expect the emulators to see the changes and cycle in the new code - but it does not. At this point I have to manually kill and restart the emulators to see the new code changes take effect.

I believe the cause of this behaviour is firebase-tools using chokidar to watch the ./isolate/dist directory. Every time the isolate command is run, it removes the ./isolate directory entirely before replacing it with the new isolated bundle. The removal of the ./isolate/dist directory seems to cause chokidar to stop tracking changes, even though a new ./isolate/dist directory is created shortly after.

There are a couple of unanswered issues in the chokidar issue tracker that talk about this behaviour, so I don't expect it be resolved there or in firebase-tools any time soon. And since this seems to be an issue because of the way the isolate command works, I suspect it would be easiest to fix here - probably by not deleting isolate or isolate/dist. The dist part is specific to my project too I suspect (derived from main in package.json?), so that would need generalising too.

I suppose one other workaround would be to remove isolate from firebase.json entirely, and only use it when deploying but that feels like more of a hack and would mean I was testing with code that wasn't necessarily identical to that which would be in the isolated bundle.

I'm running isolate in a yarn (v1) workspace monorepo, and getting Error: Cannot find module '@pnpm/logger'. It seems this is a peer dependency of @pnpm/lockfile-file, which is a dependency of isolate-package.

It's not listed as a dependency in this repo, so it never ends up being installed in mine.

It seems like a simple fix to add @pnpm/logger@^5.0.0 as a dependency in this repo.

Hello,
I'm trying to setup isolate in my monorepo setup. (turbo + pnpm)

I have root level firebase.json

....
  "functions": [
    {
      "predeploy": ["turbo build --filter api", "isolate"],
      "source": "apps/api/isolate",
      "codebase": "api",
      "runtime": "nodejs18"
    }
  ]
}

And isolate.config.json:

{
  "targetPackagePath": "apps/api",
}

That apps/api have only 1 dependency packages/mypackage and in package.json I'm using "mypackage": "workspace:*".

I checked Prerequisites in the README and made sure that all good.

However when I try to deploy functions from root I got this error:

firebase deploy -P firebase-project --only functions:api:myFunc
...
✔  functions: Finished running predeploy script.
i  functions: preparing codebase api for deployment
i  functions: ensuring required API cloudfunctions.googleapis.com is enabled...
i  functions: ensuring required API cloudbuild.googleapis.com is enabled...
i  artifactregistry: ensuring required API artifactregistry.googleapis.com is enabled...
✔  artifactregistry: required API artifactregistry.googleapis.com is enabled
✔  functions: required API cloudbuild.googleapis.com is enabled
✔  functions: required API cloudfunctions.googleapis.com is enabled

Error: Could not detect language for functions at /path_to_monorepo/apps/api/isolate

And in apps/api/isolate
I see: __tmp/mypackage-0.0.0.tgz

I wonder what I have missed?

In a monorepo typically only the root manifest contains a definition for the package manager used. It would be good to copy this field to the target package manifest so that the isolated output adopts the same version.

During the Rush support refactoring I messed something up in building the packages registry.

In my expressjs server that I'm using with isolate to deploy on GAE I do imports like

import app from "@/app"

with in the tsconfig:

"@/*": "./src/*"

but after compiling, and running isolate, I have this bug when deploying:

2023-12-01 13:17:04 default[20231201t131511]  Error: Cannot find module '@/app'
2023-12-01 13:17:04 default[20231201t131511]  Require stack:
2023-12-01 13:17:04 default[20231201t131511]  - /workspace/dist/server.js
2023-12-01 13:17:04 default[20231201t131511]      at Module._resolveFilename (node:internal/modules/cjs/loader:1077:15)
2023-12-01 13:17:04 default[20231201t131511]      at Module._load (node:internal/modules/cjs/loader:922:27)

is there a way for this to work?

here is my tsconfig.json (the lib-gouach-db-schemas is my local monorepo lib that I'm including in the project)

{
  "references": [{ "path": "../lib-gouach-db-schemas/tsconfig.cjs.json" }],
  "compilerOptions": {
    "target": "ES2020",
    "module": "CommonJS",
    "moduleResolution": "Node",
    "esModuleInterop": true,
    "allowSyntheticDefaultImports": true,
    "strict": true,
    "sourceMap": true,
    "allowJs": true,
    "skipLibCheck": true,
    "lib": ["esnext.asynciterable"],
    "outDir": "dist",
    "rootDir": "src",
    "baseUrl": ".",
    "paths": {
      "@/*": ["./src/*"],
      "@gouach/db-schemas/*": ["../lib-gouach-db-schemas/dist/cjs/*"]
    },
    "pretty": true,
    "composite": false
  },
  "compileOnSave": true,
  "include": ["src", "src/**/*.js", "src/**/*.ts"]
}

Hello everyone, I am just reporting this behavior and asking if this is expected.

This is happening for every transitory dependency, is this normal since hoisted is enabled?

After upgrading to pnpm v9, the isolate script is now failing. Here's what I see in the logs:

i  functions: Start isolating the source folder...
info Generating PNPM lockfile...
i  functions: +++ Failed to isolate: Failed to generate lockfile: value.startsWith is not a function

I know that the lockfile format has changed in PNPM v9. Could it be related to that?

Hi @0x80,

first, thanks a lot for all the time you're spending on this issue.

I have a Rush mono-repo (though I think this shouldn't make a difference) and I'm currently replacing my version of bundling dependencies using your firebase-tools-with-isolate.

I have the following directory structure:

The directory packages/apps/backend contains a set of Firebase functions I want to deploy. The content of my firebase.json is the following:

{
"functions": [
  {
    "source": "packages/apps/backend",
    "codebase": "platform-backend",
    "runtime": "nodejs20",
    "ignore": [".rush", "node_modules", "src", ".env*", "*.log", "tsconfig.json", "webpack.config.js"],
    "isolate": true
  }
 ]
}

When running firebase deploy, I get the following error:

It seems to me that it's trying to isolate the root directory instead of the backend directory. It should be looking for a tsconfig.json in packages/apps/backend. Is there anything I'm missing?

I tried putting the isolate.config.json file in the backend directory (same error) as well as in the root directory (throws error since there's no package.json in the root dir)

Any clue on why the following is occuring?

Failed to load function definition from source: FirebaseError: Failed to find location of Firebase Functions SDK. Please file a bug on Github

This is when I try to run with emulators.

My setup:

firebase.json

...
"functions": {
  "source": "./apps/functions/isolate",
  "predeploy": ["turbo build", "pnpm isolate"],
  "runtime": "nodejs18"
},
...

Note: I have my firebase config files at the root of my monorepo

isolate.config.json

{
  "targetPackagePath": "./apps/functions"
}

apps/functions/package.json

...
"engines": {
  "node": "18"
},
"main": "./dist/index.js",
"files": [
  "dist"
],
"scripts": {
  "build": "tsc"
},
devDependencies: {
  ...
  "isolate-package": "^1.3.3"
}
...

apps/functions/src/index.ts

import { initializeApp } from 'firebase-admin/app'
import { onWordCreated } from './words'

initializeApp({
  projectId:
    process.env.GCLOUD_PROJECT === 'demo-words' ? 'demo-words' : undefined
})

export { onWordCreated }

When I try firebase deploy --only functions, I also get:

Running command: npx isolate npm ERR! could not determine executable to run

I encountered deployment issues with npm due to the package-lock.json file.

Console messages indicated missing packages within the package-lock.json. Though excluding this file from the configuration allowed for smooth deployment, I wanted to retain it.

To identify the problematic package, I began removing packages individually. For me, the issue stemmed from firebase-functions-test—an old package I no longer utilized. After uninstalling it, the deployment went smoothly.

If you're facing a similar issue, I recommend removing packages individually to identify the problematic one.

I hope future versions of isolate-package become universally compatible. Until then, this method might be a workaround.

Lastly, if you pinpoint a problematic package, please share it here to aid others in faster troubleshooting!

I'm not sure if this is an issue with this library or me not using it correctly, but my .env variables are not getting deployed along with everything else.

Other than that, everything is working great and there are no error messages or anything. Just silently not picking up the contents of .env.prod.

Can we have a folder with an example(s)?

The list of packages from the root lockfile remains untouched. This does not have have an effect on the number of installed dependencies, but for large projects this can be quite a bit of unnecessary data.

We could run pnpm dedupe to slim it down, or find another way to remove the unwanted packages from the list.

If choosing dedupe it would probably be wise to make it a configurable option, as dedupe does more than just removing the excess packages from the list.

See mono-ts#9

Any idea where I can look for cause of this error?

I'm using isolate-package in a pnpm monorepo.

When I'm isolating the functions, the isolated pnpm-lock.yaml includes all patchedDependencies from the root pnpm-lock.

When deploying the functions to GCP gen2 cloud functions, the builder is installing with 'pnpm install --frozen-lockfile' which results in an error because of the patchedDependencies.

It would be great if there was a built in way to exclude the patchedDependencies (in my case, this patch is from my frontend code which has no business breaking my backend). Right now I had to run a custom node script to remove the patched deps after isolate.

I'm trying to deploy to firebase, but it fails if there are pnpm overrides in the root package.json.

ERR_PNPM_LOCKFILE_CONFIG_MISMATCH  Cannot proceed with the frozen installation. The current "overrides" configuration doesn't match the value found in the lockfile

To workaround this, I've duplicated the overrides in each apps package.json, but this does give the warning The field "pnpm" was found in /XXX/apps/my-app/package.json. This will not take effect. You should configure "pnpm" at the root of the workspace instead.

It would be awesome if isolate-package could copy the pnpm.overrides from the root package.json to the isolate/package.json file, so I don't need to manually copy them.

Finally, thanks @0x80 for building isolate-package 🙌 🙌

I have this object as my workspaces setting in my root package.json

"workspaces": { "packages": [ "apps/*", "packages/*" ], "nohoist": [ "**/ts-node" ] },

This causes an error on this line: https://github.com/0x80/isolate-package/blob/main/src/helpers/create-packages-registry.ts#L61, as flatMap is not defined on an object.

I have fixed my issue by providing it trough setting workspacePackages in the config manually. Thanks a lot for the project!

You rock!

First of all, thank you for your work on this awesome project.

I'm trying to use my local package which is a Prisma (DB) client so I can use it in my firebase function (I'm starting based on this T3-Turbo template).

When npx isolate is run as part of the deploy, it copies over my package exactly as it is in my typescript project (it copies index.ts with import statements). However I believe I need my shared package to be transpiled to commonJS for it to run in the firebase cloud environment and it doesn't seem like there's a way to do this.

The error I get is SyntaxError: Cannot use import statement outside a module which seems to from db/index.ts where I have the line import { PrismaClient } from "@prisma/client"

Do you have any idea why this would happen?

Here's the last stage of my deploy where it fails:

debug Linking dependency @acme/db to file:packages/db
debug Copied lockfile to /Users/jjdevereux/code/reasily-monorepo/packages/functions/isolate/package-lock.json
debug Deleting temporary directory (root)/packages/functions/isolate/__tmp
debug Stored isolate output at /Users/jjdevereux/code/reasily-monorepo/packages/functions/isolate
info Isolate completed
✔  functions: Finished running predeploy script.
i  functions: preparing codebase default for deployment
i  functions: ensuring required API cloudfunctions.googleapis.com is enabled...
i  functions: ensuring required API cloudbuild.googleapis.com is enabled...
i  artifactregistry: ensuring required API artifactregistry.googleapis.com is enabled...
✔  functions: required API cloudbuild.googleapis.com is enabled
✔  artifactregistry: required API artifactregistry.googleapis.com is enabled
✔  functions: required API cloudfunctions.googleapis.com is enabled

Error: Failed to load function definition from source: Failed to generate manifest from function source: SyntaxError: Cannot use import statement outside a module
npm ERR! Lifecycle script `deploy` failed with error: 
npm ERR! Error: command failed 
npm ERR!   in workspace: [email protected] 
npm ERR!   at location: /Users/jjdevereux/code/reasily-monorepo/packages/functions 

Hi @0x80, I wonder if that could be a possibility, I am currently using isolate through cli, but I am not able to pass down any arg.

In case it makes sense, I am willing to help.

Thanks!

Hi, i'm trying to get away from manually copying local dependencies in a firebase functions monorepo and came across this tool which seems to abstract away the work really nicely, thanks for your efforts 👍.

My issue is that copying the lock file in isolate makes the firebase build fail with the following message.

npm ERR! `npm ci` can only install packages when your package.json and package-lock.json or npm-shrinkwrap.json are in sync. Please update your lock file with `npm install` before continuing.

I tried doing a clean install with no luck. Is this something you can help with? For now, i have it working with the excludeLockfile flag on, but would like to steer away as the deps tree grows.

My environment (created lock file here, and matched it in gcp build)

❯ node -v
v16.19.1
❯ npm -v
8.19.4

I'm having trouble getting this working within a project that uses npm workspace. I have a workspace defined in a package.json file at the root:

  ...
  "workspaces": [
    "packages/*",
    "firebase/*"
  ],
  ...

And in the firebase directory I have a functions subdirectory (containing my firebase.json file). And inside packages there is a directory containing a typescript library called shared-code which is declared as a dependency in the package.json file in the functions.

root
  -- tsconfig.json
     package.json (defines workspaces "firebase" and "packages")
     node_modules

  -- firebase
    -- functions
      -- firebase.json
         package.json (with dependency file:../../shared-code)
         <-- no node_modules! -->

  -- packages
    -- shared-code
      -- package.json
         tsconfig.json
         node_modules

When I run npm install in the root of the entire project, the result is that a node_modules directory gets created at the root of the project, BUT there is no node_modules directory at all inside my functions workspace.

So when I run firebase deploy ... (inside firebase/functions) it fails immediately because there is no node_modules dir present. In my case with Cannot find local ESLint!.

@0x80 Is there a solution?

Isolate seem to be not copying the .env file. (but it copies .env.local and .env.staging in my case)

According to firebase docs,

If you need an alternative set of environment variables for your Firebase projects (such as staging vs production), create a .env.<project or alias> file and write your project-specific environment variables there. The environment variables from .env and project-specific .env files (if they exist) will be included in all deployed functions.

(Also see the tabled example from the docs to see how the envs are inherited)

In short, .env is a valid env config for firebase functions.

Hi!

I'm experiencing an issue.

I have a dependency graph that looks like this:

functions <- utils package.
utils package <- other_shared_package.

Running isolate succesfully puts both packages side by side in a folder structure like:

./isolate
--./dist
--./packages
----./utils
----./other_shared_package

Inspecting the package.json of these packages in the isolate folder I see that the utils package has this dep (isolate/packages/utils/package.json):

"dependencies": {
   "other_shared_package": "file:packages/other_shared_package",
}

This causes an error when running yarn install:
"Package "other_shared_package" refers to a non-existing file '"[long_monorepo_path]/isolate/packages/utils/packages/other_shared_package"'."

So it looks like yarn thinks that the file should be relative to the utils folder. Suspecting this, I tried to modify the dependency to be

"dependencies": {
    "other_shared_package": "file:../other_shared_package",
}

Which resulted in a succesful install!

So it seems like the isolate process is not compatible with yarn (I'm running yarn v.1.22.17)?

I'm also experiencing the same issue when trying to deploy to the firestore function service (they try to install using yarn v1.22.19), but debugging there takes a lot longer than debugging locally.

Thanks again for the package. I'm going to see if I can maybe force firestore to install with npm and hopefully that can serve as a temp fix!

Hi @0x80, thanks for investigating this problem and creating this package.

I tried to use it with Firebase Functions and a "shared-library" within a PNPM monorepo, but I've stumbled on the following issue when running ISOLATE_CONFIG_LOG_LEVEL=debug npx isolate:

debug Attempting to load config from /<path>/monorepo/firebase-project/functions/isolate.config.json
debug Using configuration: {
  buildDirName: undefined,
  includeDevDependencies: false,
  isolateDirName: 'isolate',
  logLevel: 'info',
  targetPackagePath: undefined,
  tsconfigPath: './tsconfig.json',
  workspacePackages: undefined,
  workspaceRoot: '../..'
}
error SyntaxError: Expected double-quoted property name in JSON at position 283
    at JSON.parse (<anonymous>)
    at readTypedJson (file:///<path>/monorepo/node_modules/.pnpm/[email protected]/node_modules/isolate-package/src/utils/json.ts:14:21)
    at getBuildOutputDir (file:///<path>/monorepo/node_modules/.pnpm/[email protected]/node_modules/isolate-package/src/helpers/get-build-output-dir.ts:21:22)
    at start (file:///<path>/monorepo/node_modules/.pnpm/[email protected]/node_modules/isolate-package/src/index.ts:46:26)

Note: I have installed the packaged under the Firebase functions folder.

Can't isolate packages

NPM log

1 verbose cli [
1 verbose cli   '/Users/mac/.nvm/versions/node/v14.21.3/bin/node',
1 verbose cli   '/Users/mac/.nvm/versions/node/v14.21.3/bin/npm',
1 verbose cli   'pack',
1 verbose cli   '--pack-destination',
1 verbose cli   '/Users/mac/Projects/suscripciones/apps/functions/isolate/__tmp'
1 verbose cli ]
2 info using [email protected]
3 info using [email protected]
4 verbose npm-session 3bb98f59d50c53a9
5 silly fetchPackageMetaData error for /Users/mac/Projects/suscripciones/apps/functions/isolate/__tmp Could not install from "../../apps/functions/isolate/__tmp" as it does not contain a package.json file.
6 verbose stack Error: ENOENT: no such file or directory, open '/Users/mac/Projects/suscripciones/apps/functions/isolate/__tmp/package.json'
7 verbose cwd /Users/mac/Projects/suscripciones/packages/api
8 verbose Darwin 20.5.0
9 verbose argv "/Users/mac/.nvm/versions/node/v14.21.3/bin/node" "/Users/mac/.nvm/versions/node/v14.21.3/bin/npm" "pack" "--pack-destination" "/Users/mac/Projects/suscripciones/apps/functions/isolate/__tmp"
10 verbose node v14.21.3
11 verbose npm  v6.14.18
12 error code ENOLOCAL
13 error Could not install from "../../apps/functions/isolate/__tmp" as it does not contain a package.json file.
14 verbose exit [ 1, true ]

isolate.config.json

package '@repo/backend' has conflicts in the following paths:
/Users/kevin/foo/packages/backend
/Users/kevin/foo/apps/gen2/isolate/packages/backend

Struggling with logging/debugging

Do I need to manually delete the isolate folder before each call to isolate?

PNPM used partially and then NPM used to pack...

$ npx isolate
debug Using isolate-package version 1.13.2
debug Found tsconfig at: ./tsconfig.json
debug Workspace root resolved to /Users/kevin/foo
debug Isolate target package (root)/apps/gen2
debug Isolate output directory (root)/apps/gen2/isolate
debug Detected package manager pnpm 8.15.7
debug Detected pnpm packages globs: [ 'apps/*', 'packages/*', 'services/*' ]
debug Registering package ./apps/gen2
debug Registering package ./packages/typescript-config
debug Registering package ./packages/firebase-admin
debug Registering package ./packages/eslint-config
debug Registering package ./packages/connectrpc
debug Registering package ./packages/common
debug Registering package ./packages/backend
debug Using PNPM pack instead of NPM pack
debug Packed (temp)/repo-backend-0.0.0.tgz
debug Packed (temp)/repo-common-0.0.0.tgz
debug Unpacking (temp)/repo-backend-0.0.0.tgz
debug Unpacking (temp)/repo-common-0.0.0.tgz
debug Moved package files to (isolate)/packages/common
debug Moved package files to (isolate)/packages/backend
Error: Command failed: npm pack --pack-destination "/Users/kevin/foo/apps/gen2/isolate/__tmp"

    at ChildProcess.exithandler (node:child_process:422:12)
    at ChildProcess.emit (node:events:517:28)
    at maybeClose (node:internal/child_process:1098:16)
    at Process.ChildProcess._handle.onexit (node:internal/child_process:303:5)

Declared using pnpm but then uses npm pack.

Trying to introduce into firebase monorepo, but need to keep the firebase.json at the root and use codebase for different functions services.

Using your demo mono-ts repo as an example, I have attempted to integrate my own library package and deploy firebase functions. On deploy, it makes the isolate folder but as soon as firebase analyses the folder it gives me a ERR_MODULE_NOT_FOUND for one of the external packages in the library package.

info Isolate completed at /mono-ts/services/fns/isolate
✔  functions: Finished running predeploy script.
i  functions: preparing codebase fns for deployment
i  functions: ensuring required API cloudfunctions.googleapis.com is enabled...
i  functions: ensuring required API cloudbuild.googleapis.com is enabled...
i  artifactregistry: ensuring required API artifactregistry.googleapis.com is enabled...
✔  functions: required API cloudfunctions.googleapis.com is enabled
✔  functions: required API cloudbuild.googleapis.com is enabled
✔  artifactregistry: required API artifactregistry.googleapis.com is enabled
i  functions: Loading and analyzing source code for codebase fns to determine what to deploy
Serving at port 8449

Error [ERR_MODULE_NOT_FOUND]: Cannot find package 'ssh2-sftp-client' imported from /mono-ts/services/isolate/dist/index.js
Did you mean to import [email protected]/node_modules/ssh2-sftp-client/src/index.js?
    at new NodeError (node:internal/errors:406:5)
    at packageResolve (node:internal/modules/esm/resolve:789:9)
    at moduleResolve (node:internal/modules/esm/resolve:838:20)
    at defaultResolve (node:internal/modules/esm/resolve:1043:11)
    at ModuleLoader.defaultResolve (node:internal/modules/esm/loader:383:12)
    at ModuleLoader.resolve (node:internal/modules/esm/loader:352:25)
    at ModuleLoader.getModuleJob (node:internal/modules/esm/loader:228:38)
    at ModuleWrap.<anonymous> (node:internal/modules/esm/module_job:85:39)
    at link (node:internal/modules/esm/module_job:84:36) {
  code: 'ERR_MODULE_NOT_FOUND'
}

Ran pnpx isolate and getting the following error:

.../Library/pnpm/store/v3/tmp/dlx-4070 | +1 +
 ERR_PNPM_DLX_NO_BIN  No binaries found in isolate
.../Library/pnpm/store/v3/tmp/dlx-4070 | Progress: resolved 1, reused 0, downloaded 0, added 0

Running pnpm isolate or npx isolate

error Error: Failed to detect package manager
at detectPackageManager (file:///ROOT/node_modules/.pnpm/[email protected]/node_modules/isolate-package/src/helpers/detect-package-manager.ts:19:9)
at start (file:///ROOT/node_modules/.pnpm/[email protected]/node_modules/isolate-package/src/index.ts:59:26)

Folder structure:

apps
└─ native
   ├─ package.json
   └─ .eslintrc.js
packages
└─ eslint-config-custom
   ├─ index.js
   └─ package.json
functions
├─ lib/index.js
├─ firebase.json
├─ tsconfig.json
└─ package.json

Thanks for this great package @0x80!

Unfortunately I keep having a critical error when deploying using Github actions.
Locally isolate-package is working like a charm. However, using Github CI crashed because it cannot file a packages (.tgz) file.

@packages/server:deploy: Error: ENOENT: no such file or directory, open 'api-4.1.1.tgz' @packages/server:deploy:  ELIFECYCLE  Command failed with exit code 1. @packages/server:deploy: ERROR: command finished with error: command (/my/private/path) pnpm run deploy exited (1)

I am kinda stuck here. Would you know what is going on?

//isolate.config.json { "buildDirName": "./", "excludeLockfile": true }

We are trying to train people to use pnpm, and one of my devops guys inserted a script that intercepts npm (even though it's installed). That results in this:

(venv) gitpod /workspace/costvine/jspackages/server (dev) $ isolate
error The response from pack could not be resolved to an existing file: /workspace/costvine/jspackages/server/isolate/__tmp/Use pnpm instead of npm :)
Error: ENOENT: no such file or directory, open '/workspace/costvine/jspackages/server/isolate/__tmp/Use pnpm instead of npm :)'

I know npm pack will probably work just as well here, but I'm thinking maybe it would be cleaner to use pnpm pack in this case.

This is really just a prompt for possible discussion. I'm going to try to work around it. I ran into similar problems with the Firebase emulator, but it still worked in our pnpm monorepo, with only a bundler, despite tripping over the fake npm command. I've left the stub for npm in there partly just to see what's calling it, which is more things than I expected.

But I don't want Firebase deploy using npm to install my dependencies, and I need that lockfile to make absolutely sure it builds what we tested. If I could just generate that pruned lockfile, I could probably do the rest with esbuild. There's an old project in the pnpm repository to generate stand-alone lock files, but it's deprecated, and didn't work for me, for different reasons. The instructions say to use pnpm deploy instead, but I can't figure out any way to get that to generate a lock file.

I have a yarn 3.x monorepo with functions as a one of the workspaces located at apps/x-api.
x-api has a local workspace dependency installed from packages/utils.
Trying to just run run isolate results in a type Error in isoalte.

Steps:

1. install

cd apps/x-api
yarn add isolate-package

2. run isolate yarn isolate or npx isolate

Error

error TypeError: packagesGlobs.flatMap is not a function
    at createPackagesRegistry (<my project>/node_modules/isolate-package/src/helpers/create-packages-registry.ts:61:6)
    at start (<my project>/node_modules/isolate-package/src/index.ts:80:34)

It seems that cloud-run deployment from source can not (yet) handle the packageManager field correctly. It fails when setting it to pnpm 9.

Add a flag omitPackageManager to not include it in the output manifest.

Build failed: installing pnpm dependencies: %!w(*buildererror.Error=&{  2 2 8622936f ERR_PNPM_OUTDATED_LOCKFILE  Cannot install with "frozen-lockfile" because pnpm-lock.yaml is not up to date with package.json

Note that in CI environments this setting is true by default. If you still need to run install in such cases, use "pnpm install --no-frozen-lockfile"

As a temporary workaround there is configuration option excludeLockfile, which will exclude the lockfile from the isolate output, but as a result the deployment would be possibly non-deterministic.

In order to solve problem described in #56, I built a script to handle my isolation process. However, TypeScript refuses to import the isolate-package types.

Here is the full error:

Could not find a declaration file for module 'isolate-package'. '[workspace]/node_modules/.pnpm/[email protected]/node_modules/isolate-package/dist/index.mjs' implicitly has an 'any' type.
  There are types at [workspace]/apps/api/node_modules/isolate-package/dist/index.d.ts', but this result could not be resolved when respecting package.json "exports". The 'isolate-package' library may need to update its package.json or typings.ts(7016)

I am using isolate-package v1.10.1.

import { execSync } from 'node:child_process';
import { readFile, writeFile } from 'node:fs/promises';
import { isolate } from 'isolate-package';
import { rimraf } from 'rimraf';
import { build as tsup } from 'tsup';
import tsupConfig from '../tsup.config';

const isolateDirName = 'isolate';

await rimraf(['dist', isolateDirName]);
await tsup(tsupConfig);
await isolate({
  isolateDirName,
});

// Copy the overrides from the root workspace to the isolate workspace
const rootPackageJson = JSON.parse(await readFile('../../package.json', 'utf-8'));
const isolatePackageJson = JSON.parse(await readFile('package.json', 'utf-8'));
isolatePackageJson.pnpm = { ...isolatePackageJson.pnpm, overrides: rootPackageJson.pnpm.overrides };

await writeFile('isolate/package.json', JSON.stringify(isolatePackageJson, null, 2));
execSync('pnpm install --lockfile-only', { cwd: isolateDirName, stdio: 'inherit' });
execSync('zip -r ../dist/function.zip ./*', { cwd: isolateDirName, stdio: 'inherit' });

await rimraf([isolateDirName]);

{
  "extends": "@tsconfig/strictest/tsconfig.json",
  "compilerOptions": {
    "outDir": "dist",
    "strict": true,

    "allowJs": false,
    "checkJs": false,
    "noEmit": true,
    "allowImportingTsExtensions": true,

    "forceConsistentCasingInFileNames": true,
    "verbatimModuleSyntax": true,
    "esModuleInterop": true,
    "resolveJsonModule": true,
    "isolatedModules": true,
    "allowSyntheticDefaultImports": true,

    "exactOptionalPropertyTypes": false,

    "target": "ES2022",
    "module": "ESNext",
    "moduleResolution": "Bundler",

    "paths": {
      "@/*": ["./src/*"]
    }
  },
  "include": ["**/*.ts"],
  "exclude": ["node_modules"]
}

The fix is fairly easy: add explicit types to the package.json exports:

   "type": "module",
   "types": "./dist/index.d.ts",
   "exports": {
-    ".": "./dist/index.mjs"
+    ".": {
+      "import": "./dist/index.mjs",
+      "types": "./dist/index.d.ts"
+    }
   },
   "files": [
     "dist",

Happy to submit a PR.

First off, thank you for this cool tool. ⭐
I thought I could give some feedback here. (Also I thought like this should be a part of turborepo!)

Support for default firebase init generated folder structure.

The firebase init command generates a folder structure similar to:

firebase
├── functions/
└── firebase.json

The functions source will go into functions dir.

The most simplest approach I found for adopting this into a monorepo (I'm using pnpm+turbo) is copy this firebase directory into mono/apps.

mono
├── apps
│      ├── web
│      ├── firebase  <-----------------(the firebase directory copied)
│      └── my-another-app
├── packages/
└── pnpm-workspace.yml

And my pnpm-workspace.yml looked like:

packages:
  - "packages/*"
  - "apps/*"
  - "apps/firebase/functions"
  - "!**/test/**"

Unfortunately this setup seem to be not compatible with isolate. As in the docs of isolate, nested packages are not supported. But the firebase directory is not a package (it doesn't have a package.json, but firebase/functions is). I don't know if we call this a nested package, but isolate couldn't out-of-the-box support firebase init folder structure in a monorepo.

So tried to flat it out, copied the content for mono/firebase/functions into mono/firebase including the package.json. And it worked and I did a real deployment ⚡ .

But. I had to fiddle around the .env management & firebase emulator setup. Previously during development, firebase functions emulator reloads the changed functions automatically after the tsc build. Now I lost that, even after adding the isolate after the tsc. So now every time I have to stop the emulator, build and tsc ,start again to reflect the changes locally. This was a little painful.

So it would be really cool, if isolate support the native firebase folder structure so that devs feel home at the firebase directory.

Currently npm pack is used always in the last instance

With PNPM it doesn't work, and modern yarn already falls back to NPM, so that leaves classic yarn and that seems to work fine, so we don't need the option

My project has a shared database package that uses Prisma. Prisma generates a client for interacting with your database based on your schema, and the generated files are put into node_modules/@prisma/client. The prisma generate command needs to be run on the target platform. Most guidance is to run this command using a postinstall script within package.json (example).

However, isolate-package omits out scripts from the generated package.json files here, so this approach currently won't work.

Is it possible to add an option to preserve (specific) scripts?

For now I'm working around this with a script that appends the postinstall script to the generated package.json after running isolate.

Thank you for creating this project!

I'm working with a standard turbo/yarn workspaces setup as can be found in https://github.com/vercel/turbo/tree/main/examples/with-tailwind and want to use isolate-package to isolate apps/functions (firebase functions setup as outlined in the docs here).

Even after I gave it several tries and went over my whole setup to make sure I'm in line with both the standard turbo/yarn workspaces as well as isolate-package setup, I do not manage though to understand what's going wrong when running isolate, so let me share some code here.

First the actual error which occurs when I run isolate. The problem occurs with the common package (the only package that is needed from ./packages/*) and it seems as the path includes some parts twice with a weird cut or missing piece where usually is a whitespace (in Own Projects):

⋊> ~/L/D/O/l/a/functions on main ⨯ isolate                                                                                                                                     20:02:31
debug Running isolate-package version 1.3.3
debug Found tsconfig at: ./tsconfig.json
debug Workspace root resolved to /Users/<username>/Own Projects/<project>
debug Isolate target package (root)/apps/functions
debug Isolate output directory (root)/apps/functions/isolate
debug Cleaned the existing isolate output directory
debug Detected package manager yarn 3.6.4
debug Override workspace packages via config: packages/*,apps/*
debug Registering package ./packages/ui-interactive
debug Registering package ./packages/ui
debug Registering package ./packages/tsconfig
debug Registering package ./packages/tailwind-config
debug Registering package ./packages/eslint-config-custom
debug Registering package ./packages/common
debug Registering package ./apps/web
debug Registering package ./apps/functions
error Error: Command failed: npm pack --pack-destination /Users/<username>/Own Projects/<project>/apps/functions/isolate/__tmp
npm WARN Ignoring workspaces for specified package(s) 
npm ERR! code ENOENT
npm ERR! syscall open
npm ERR! path /Users/<username>/Own Projects/<project>/packages/common/Projects/<project>/apps/functions/isolate/__tmp/package.json
npm ERR! errno -2
npm ERR! enoent ENOENT: no such file or directory, open '/Users/<username>/Own Projects/<project>/packages/common/Projects/<project>/apps/functions/isolate/__tmp/package.json'
npm ERR! enoent This is related to npm not being able to find a file.
npm ERR! enoent 

npm ERR! A complete log of this run can be found in:
npm ERR!     /Users/<username>/.npm/_logs/2023-10-21T18_02_37_406Z-debug-0.log

    at ChildProcess.exithandler (node:child_process:419:12)
    at ChildProcess.emit (node:events:513:28)
    at maybeClose (node:internal/child_process:1091:16)
    at Socket.<anonymous> (node:internal/child_process:449:11)
    at Socket.emit (node:events:513:28)
    at Pipe.<anonymous> (node:net:322:12)

... the invalid path is the following: /Users/<username>/Own Projects/<project>/packages/common/Projects/<project>/apps/functions/isolate/__tmp/package.json which is a weird combination of /Users/<username>/Own Projects/<project>/packages/common/ and Projects/<project>/apps/functions/isolate/__tmp/package.json where the latter path lacks the beginning and the Own in Own Projects

Now let me share my setup:

./package.json:

{
  "private": true,
  "workspaces": [
    "apps/*",
    "packages/*"
  ],
  "scripts": {
    "build": "turbo build",
  },
  "prettier": "@vercel/style-guide/prettier",
  "devDependencies": {
    "@types/node": "^20.8.7",
    "@types/react": "^18.2.29",
    "@types/react-dom": "^18.2.14",
    "prettier": "^3.0.3",
    "prettier-plugin-tailwindcss": "^0.5.3",
    "tsconfig": "*",
    "turbo": "latest",
    "typescript": "^5.2.2"
  },
  "packageManager": "[email protected]"
}

apps/functions/package.json:

{
  "name": "functions",
  "version": "0.0.0",
  "scripts": {
    "build": "tsc -b tsconfig.build.json",
  },
  "engines": {
    "node": "18"
  },
  "main": "lib/index.js",
  "dependencies": {
    "common": "*",
    "firebase-admin": "^11.8.0",
    "firebase-functions": "^4.3.1"
  },
  "devDependencies": {
    "firebase-tools": "latest",
    "isolate-package": "^1.3.3",
    "typescript": "^4.9.0"
  },
  "private": true
}

apps/functions/isolate.config.json:

{
  "workspaceRoot": "../..",
  "workspacePackages": ["packages/*", "apps/*"],
  "logLevel": "debug"
}

apps/functions/tsconfig.json:

{
  "compilerOptions": {
    "module": "commonjs",
    "noImplicitReturns": true,
    "noUnusedLocals": true,
    "outDir": "lib",
    "sourceMap": true,
    "strict": true,
    "target": "es2017"
  },
  "compileOnSave": true,
  "include": [
    "src",
    "./*.d.ts"
  ]
}

apps/functions/tsconfig.build.json:

{
  "extends": "./tsconfig.json",
  "compilerOptions": {
    "rootDir": "src"
  },
}

packages/common/package.json:

{
  "name": "common",
  "version": "0.0.0",
  "private": true,
  "license": "MIT",
  "exports": {
    ".": "./dist"
  },
  "types": "./dist/index.d.ts",
  "files": [
    "dist"
  ],
  "scripts": {
    "build": "tsup",
    "dev": "tsup --watch",
    "lint": "eslint src/",
    "typecheck": "tsc --noEmit"
  },
  "dependencies": {
    "react": "^18.2.0",
    "react-dom": "^18.2.0"
  },
  "devDependencies": {
    "react": "^18.2.0",
    "tsconfig": "*",
    "tsup": "^6.1.3",
    "typescript": "^5.1.6"
  }
}

packages/common/tsconfig.json:

{
  "extends": "tsconfig/react-library.json",
  "include": ["."],
  "exclude": ["dist", "build", "node_modules"]
}

packages/common/tsup.config.ts:

import { defineConfig, Options } from 'tsup';

export default defineConfig((options: Options) => ({
  treeshake: true,
  splitting: true,
  entry: ['src/**/*.ts'],
  outDir: 'dist',
  format: ['esm'],
  dts: true,
  minify: true,
  clean: true,
  external: ['react'],
  ...options,
}));

In create-packages-registry.ts, the registry is constructed which indexes package manifests by their name. However, the registry object is built using lodash.set, which parses the key as a recursive path when it includes periods, so that package name "@mylib/a.b" results in { "@mylib/a": { "b": { ... }}}. Later, when "@mylib/a.b" is looked up in the registry, it's not found resulting in the error error TypeError: Cannot read properties of undefined (reading 'rootRelativeDir')

It appears that the extra behavior of lodash.set is not needed. If so, the following change fixes this issue:

.reduce<PackagesRegistry>(
    (acc, info) => {
      if (info) {
        acc[info.manifest.name] = info;
      }
      return acc;
    },
    {}
  );

Hi! First of all, thank you for putting this together! It looks like a very useful tool, especially when it comes to using Firebase with monorepos.

I'll be honest, this is the first time I'm working with Firebase, so the experience there isn't much, but I do have a good understanding of Turborepo monorepos.

I've read both your blog post and the readme in this repo and, even though I do understand the problem, I'm having a hard time using this to set up Firebase for my needs.

Imagine the following example monorepo:

turbo.json
package.json
apps/
    nextjs/
        package.json
        src/
    another/
        package.json
        src/
packages/
    api/
        package.json
        src/
   ui/
       package.json
       src/
tooling/
    firebase/
      firebase.json
      package.json

Assuming the scenario described above, I'd like to be able to:

• deploy the apps/nextjs app to Firebase (hosting + functions)
• deploy the packages/api package to Firebase (this package contains our backend API in form of cloud functions)
• tooling/firebase simply contains emulator configuration and possibly other firebase config files to be used across the repo (basically Firebase config that is not deployed, per se)

Is this scenario a possibility? How would I go about configuring this? I'm assuming this means multiple "target packages"?

Thanks in advance for the help!

P.S.: I'm using pnpm, if that matters

I use Windows, and I often have processes running that have open files in node_modules which prevent it from being moved. Rather than always killing those processes, I did some experimentation with a fork of isolate-package. Interestingly, skipping the move altogether seems to work fine. Arborist found the dependent packages, even though they was several levels above the isolate folder.

Could we add a configuration option which disables the move?

Thanks!

Have you managed to get this working when packages are not compiled themselves, but instead have main and types pointing directly to src/index.ts?

This style is championed by Turbo Repo on their blog and in the docs for internal packages. The point being we don't need to separately build the package each time. We import it as pure TypeScript and then the app does the transpiling when needed.

I prefer it because I can skip the build step for internal packages, changes appear instantly throughout the monorepo, and it allows for the customization of the tsconfig.json file for each individual app, as opposed to having to having a more generic tsconfig.json version for the packages that aims to cater to all the apps.

But when working with isolate package I cannot get it working because I think it requires that packages are built before they are isolated. Instead I need the internal packages added, and then build everything.

Is that possible?

Currently scripts in the package manifests of internal dependencies are included. If they contain a "prepare" script it will get executed on install, which typically triggers a build of that package, which then fails because it requires dev dependencies.

Dependencies should already have been built before isolate happens. All scripts should be discarded probably.

Thanks for making this package because it's been sorely needed for a long time for FIrebase Functions deployment. I have most of it working based on your instructions, but yarn install --immutable is breaking on deployments.

Error

In the below example, @arc/feature-flag is a workspace that was relocated to functions/isolate/modules/feature-flag from modules/feature-flag by yarn isolate. The error shows up when we attempt to deploy to firebase functions.

Logs from Google Cloud Build triggered by Firebase deploy:

INFO 2023-08-21T22:16:21.352014524Z Step #2 - "build": Installing Yarn v3.6.1
INFO 2023-08-21T22:16:21.352389680Z Step #2 - "build": 2023/08/21 22:16:21 [DEBUG] GET https://repo.yarnpkg.com/3.6.1/packages/yarnpkg-cli/bin/yarn.js
INFO 2023-08-21T22:16:21.616034608Z Step #2 - "build": DEBUG: Setting environment variable PATH=/layers/google.nodejs.yarn/yarn_engine/bin:/layers/google.nodejs.runtime/node/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
INFO 2023-08-21T22:16:21.660331912Z Step #2 - "build": WARNING: Skipping adding auth token to /www-data-home/.yarnrc.yml. Unable to find Artifact Registry in /workspace/.yarnrc.yml.
INFO 2023-08-21T22:16:21.660349522Z Step #2 - "build": --------------------------------------------------------------------------------
INFO 2023-08-21T22:16:21.660351700Z Step #2 - "build": Running "yarn install --immutable"
INFO 2023-08-21T22:16:22.540249844Z Step #2 - "build": ➤ YN0000: ┌ Resolution step
INFO 2023-08-21T22:16:23.150742724Z Step #2 - "build": ➤ YN0013: │ @arc/feature-flag@file:./modules/feature-flag#./modules/feature-flag::hash=7122b8&locator=%40arc%2Ffunctions%40workspace%3A. can't be found in the cache and will be fetched from the disk
INFO 2023-08-21T22:16:23.310145669Z Step #2 - "build": ➤ YN0000: └ Completed in 0s 769ms
INFO 2023-08-21T22:16:23.518625341Z Step #2 - "build":
INFO 2023-08-21T22:16:23.519048711Z Step #2 - "build": ➤ YN0000: ┌ Post-resolution validation
INFO 2023-08-21T22:16:23.519058767Z Step #2 - "build": ➤ YN0000: │ @@ -217,10 +217,9 @@
INFO 2023-08-21T22:16:23.519060244Z Step #2 - "build": ➤ YN0000: │ linkType: hard
INFO 2023-08-21T22:16:23.519062243Z Step #2 - "build": ➤ YN0000: │
INFO 2023-08-21T22:16:23.519063426Z Step #2 - "build": ➤ YN0000: │ "@arc/feature-flag@file:./modules/feature-flag::locator=%40arc%2Ffunctions%40workspace%3A.":
INFO 2023-08-21T22:16:23.519072723Z Step #2 - "build": ➤ YN0000: │ version: 1.0.0
INFO 2023-08-21T22:16:23.519751074Z Step #2 - "build": ➤ YN0028: │ - resolution: "@arc/feature-flag@file:./modules/feature-flag#./modules/feature-flag::hash=7b4bc8&locator=%40arc%2Ffunctions%40workspace%3A."
INFO 2023-08-21T22:16:23.519999320Z Step #2 - "build": ➤ YN0028: │ - checksum: 9697a40672390184a2b0cc3d42f437a1e56e9e63b3fc24658793cdacec3681b95a373d13bc856b90b1cb9c2ca7cb337b4d2916c9cdaae5ddf6ae808e8fdec42f
INFO 2023-08-21T22:16:23.520005499Z Step #2 - "build": ➤ YN0028: │ + resolution: "@arc/feature-flag@file:./modules/feature-flag#./modules/feature-flag::hash=7122b8&locator=%40arc%2Ffunctions%40workspace%3A."
INFO 2023-08-21T22:16:23.520006863Z Step #2 - "build": ➤ YN0000: │ languageName: node
INFO 2023-08-21T22:16:23.520298259Z Step #2 - "build": ➤ YN0000: │ linkType: hard
INFO 2023-08-21T22:16:23.520474017Z Step #2 - "build": ➤ YN0000: │
INFO 2023-08-21T22:16:23.520479605Z Step #2 - "build": ➤ YN0000: │ "@arc/functions@workspace:.":
INFO 2023-08-21T22:16:23.520945843Z Step #2 - "build": ➤ YN0000: │
INFO 2023-08-21T22:16:23.521661955Z Step #2 - "build": ➤ YN0028: │ The lockfile would have been modified by this install, which is explicitly forbidden.
INFO 2023-08-21T22:16:23.521782341Z Step #2 - "build": ➤ YN0000: └ Completed in 0s 205ms
INFO 2023-08-21T22:16:23.522244724Z Step #2 - "build": ➤ YN0000: Failed with errors in 0s 987ms

Setup

• Yarn v3.6.1 with multiple workspaces.
• firebase.json configured explicitly to ignore .yarn/cache and node_modules because including them would violate the 100MB Firebase Function zip file limit.

isolate.config.json:

{
  "includeDevDependencies": true,
  "targetPackagePath": "./functions",
  "workspaceRoot": "./"
}

firebase.json:

{
  "functions": {
    "source": "functions/isolate",
    "ignore": [
      "**/.yarn/cache/**",
      "**/.yarn/install-state.gz",
      "**/.git",
      "**/.runtimeconfig.json",
      "**/firebase-debug.log",
      "**/firebase-debug.*.log",
      "**/node_modules/**"
    ]
  }
}

Pre-deploy steps in CI job

yarn isolate
cp -r .yarn ./functions/isolate/.yarn
cp .yarnrc.yml ./functions/isolate/.yarnrc.yml
cd ./functions/isolate

# On hindsight this probably does not do anything because Cloud Build explicitly 
yarn config set enableImmutableInstalls falseruns yarn install --immutable
yarn install --mode=update-lockfile 

yarn firebase deploy -P $FIREBASE_PROJECT --only functions --force --debug

Very cool!

I have the same issue where I tried to isolate my local monorepo package manually and bundle them with webpack to deploy an expressjs API using some local monorepo shared libs on Google App Engine

how would I use isolate-package to do the same?

Instead of using JSON it would be better to use a JS/TS file and define the config using a typed function.