Threads massively slows down linting / re-linting about eslint-webpack-plugin HOT 19 CLOSED

@eek I will try to reproduce as well in the morning.

Great.

I've also tested it now and what @imsandez reported here - facebook/create-react-app#10145 (comment) is true.

if you create a new react app:

npx create-react-app test-app --template typescript

(I've created a TS one to be closer to what I use)

I takes between 3-6s to compile a change in App.tsx (closer to 3s rather than 6s) but once you downgrate the eslint-webpack-plugin by adding resolutions at the end of package.json:

  "resolutions": {
    "react-scripts/eslint-webpack-plugin": "2.3.0"
  }

and yarn again. (don't know if npm supports resolutions)

Compile will take maximum 0.5s.

  "eslintConfig": {
    "parser": "@typescript-eslint/parser",
    "extends": [
      "airbnb-typescript",
      "plugin:@typescript-eslint/eslint-recommended"
    ],
    "parserOptions": {
      "project": "./tsconfig.json",
      "ecmaVersion": 8,
      "ecmaFeatures": {
        "experimentalObjectRestSpread": true
      }
    },
    "plugins": [
      "@typescript-eslint",
      "import",
      "react"
    ],
    "env": {
      "browser": true
    },
    "globals": {
      "gtag": true,
      "isNaN": true
    },
    "settings": {
      "import/resolver": {
        "node": {
          "extensions": [
            ".js",
            ".jsx",
            ".ts",
            ".tsx"
          ],
          "moduleDirectory": [
            "node_modules",
            "src/"
          ]
        }
      }
    }
  },

on the blank project it always takes 6s. Once I downgrade to the previous eslint-webpack-plugin it takes 0.5s tops.

from eslint-webpack-plugin.

Using the defaults provided by npx create-react-app test-app --template typescript I'm getting 1-2ms recompile time editing App.tsx. Adding your lint config makes it substantially slower and exposes what the slow down is.

The jest-worker doesn't auto-warm up the pool. So until you hit max-pool size each lint request is paying the eslint init cost. After each compile, we cleanup the pool. So, each recompile rebuilds the pool... and since incremental edits often are one file... that takes a pool init + module init + ESLint setup AND run... So, my first stab at this is to disable the pool after the first build. All rebuilds would be on the main process and not respin the pool.

I can go further to preserve the pool (and warmup) such that re-compiles don't start over. Dispatching seems to eat ~150-200ms. So recompiles will always be slower by that much minimum. Would some lint configs warrant always run off thread? or would just running lint on the main thread after the first compile be sufficient? @ricardogobbosouza @eek @alexander-akait

from eslint-webpack-plugin.

@jsg2021
In my view the problem is here, I think
https://github.com/webpack-contrib/eslint-webpack-plugin/blob/master/src/getESLint.js#L58
Is carrying the eslint twice

from eslint-webpack-plugin.

@ricardogobbosouza I think we should set threads: false by default

from eslint-webpack-plugin.

This makes sense on large projects.

from eslint-webpack-plugin.

@ricardogobbosouza I think we should set threads: false by default

I agree, should be an option for people that really need it.

For now, I also don't see a lot of improvements in our build time - it's 5 seconds faster with threads than without threads enabled - 64.66s vs 69.05s (and we have 297 files and 7440 3rd party files), but having 10s+ delay of each file change is too much.

I've also upgraded from node v14 to node v15 to see if there's a difference, but there wasn't one.

from eslint-webpack-plugin.

Just out of curiosity, what is your cpu? I imaging form hardware to hardware this will vary. The report should be only issued once per compile and will take the same amount of time threaded or not.

The linting ... reading/parsing/validating happens in workers.

I'm quite surprised the recompile time you're seeing. I haven't noticed that locally. Is your cpu under heavy load?

from eslint-webpack-plugin.

and we have 297 files and 7440 3rd party files), but having 10s+ delay of each file change is too much.

I sure hope you aren't linting third party code. A 10s delay sounds like a problem and is not expected behavior.

I maintain a very large project that has thousands of files (ignoring third party files) and editing and recompiling is nearly imperceptible. So I'm curious what could be different in your config?

from eslint-webpack-plugin.

I've added before this line:

eslint-webpack-plugin/src/index.js

Line 91 in 46fc07a

lint(file);

Your time measurement here is measuring time between invocations... which is not really measuring the speed of this block. Webpack is calling this repeatedly on all the files in the graph. It only makes it into that block when we see a file we want to lint. And then, we kick off an asynchronous call to have it linted, but return synchronously to not block the compiler.

The report starts at then end of compiling and waits for the lint queue to finish. (this will take some time depending on how many files are still being worked on... similar to how long running eslint command line on your project)

from eslint-webpack-plugin.

@ricardogobbosouza I think we should set threads: false by default

it may be a bug. it shouldn't ever be slower. It turns itself off if there are 2 or less cores reported. on 4+ core systems it should be a net speed up.

from eslint-webpack-plugin.

and we have 297 files and 7440 3rd party files), but having 10s+ delay of each file change is too much.

I sure hope you aren't linting third party code. A 10s delay sounds like a problem and is not expected behavior.

I maintain a very large project that has thousands of files (ignoring third party files) and editing and recompiling is nearly imperceptible. So I'm curious what could be different in your config?

I'm definitely not linting 3rd party code (especially since I don't have this problem with threads: off)

On just a file I've added some timings above, it takes 3ms to lint but then it takes awaitReport around 11-16s to complete.

I don't have a slow machine. It's a Dell XPS 9560 - i7-7700HQ, 32gb RAM, and an NVME SSD (on Windows). And the same problem happens to my colleague as well, who runs a Dell Area 51 laptop with (i9-10980HK I think) on PopOS

Since this also happens to a lot of people that run react-create-app, definitely a bug somewhere, probably related to the config file or linting rules?

This guy says it's the same issue on blank projects as well - facebook/create-react-app#10145 (comment)

I will give it a try and see if I have the same issue on a newly created React project and come back.

from eslint-webpack-plugin.

@eek I will try to reproduce as well in the morning.

from eslint-webpack-plugin.

I agree that it should be deactivated by default, in most projects that are usually small, it does not make sense to have it activated.

We forgot to put this option in the documentation 😟
And a note recommending to be activated for large projects

WTDY @alexander-akait ?

from eslint-webpack-plugin.

I agree that it should be deactivated by default, in most projects that are usually small, it does not make sense to have it activated.

We forgot to put this option in the documentation 😟
And a note recommending to be activated for large projects

WTDY @alexander-akait ?

@ricardogobbosouza, I also understand what @jsg2021 says as well. If it wouldn't be for this weird bug that's affecting recompile time, it would be a performance improvement for all projects.

from eslint-webpack-plugin.

@jsg2021 just running the lint on the main thread after the first compile I think it's sufficient.

Updated now to 2.4.1 and things are no longer slow 🙌 thanks! 😁

from eslint-webpack-plugin.

my first stab at this is to disable the pool after the first build. All rebuilds would be on the main process and not respin the pool

After pr #55, we can feel free to set threads: true ? For a large project, linting files through jest-worker in the first build is meaningful. Any suggestion?

from eslint-webpack-plugin.

I think best value is false right now, in future we can revisit it

from eslint-webpack-plugin.

The main problem is initial workers start and we should not ending threads in watch mode, I think we need webpack API for this

from eslint-webpack-plugin.

my first stab at this is to disable the pool after the first build. All rebuilds would be on the main process and not respin the pool

After pr #55, we can feel free to set threads: true ? For a large project, linting files through jest-worker in the first build is meaningful. Any suggestion?

we didn't take it away. it's just not default. you may turn it on in your config. that's what i have done for my project.

from eslint-webpack-plugin.