modclean / modclean Goto Github PK

View Code? Open in Web Editor NEW

392.0 7.0 15.0 162 KB

Remove unwanted files and directories from your node_modules folder

License: MIT License

JavaScript 100.00%

disk-space enterprise cli enterprise-environment benchmark huge javascript cleanup clean cleaner

modclean's Introduction

ModClean

Remove unwanted files and directories from your node_modules folder

This documentation is for ModClean 2.x which requires Node v6.9+, if you need to support older versions, use ModClean 1.3.0 instead.

ModClean is a utility that finds and removes unnecessary files and folders from your node_modules directory based on predefined and custom glob patterns. This utility comes with both a CLI and a programmatic API to provide customization for your environment. ModClean is used and tested in an Enterprise environment on a daily basis.

Why?

There are a few different reasons why you would want to use ModClean:

Commiting Modules. Some environments (especially Enterprise), it's required to commit the node_modules directory with your application into version control. This is due to compatibility, vetting and vunerability scanning rules for open source software. This can lead to issues with project size, checking out/pulling changes and the infamous 255 character path limit if you're unlucky enough to be on Windows or SVN.
Wasted space on your server. Why waste space on your server with files not needed by you or the modules?
Packaged applications. If you're required to package your application, you can reduce the size of the package quickly by removing unneeded files.
Compiled applications. Other tools like, NW.js and Electron make it easy to create cross-platform desktop apps, but depending on the modules, your app can become huge. Reduce down the size of the compiled application before shipping and make it faster for users to download.
Save space on your machine. Depending on the amount of global modules you have installed, you can reduce their space by removing those gremlin files.
and much more!

The 🍰 is a lie, but the Benchmarks are not.

How?

New! In ModClean 2.0.0, patterns are now provided by plugins instead of a static patterns.json file as part of the module. By default, ModClean comes with modclean-patterns-default installed, providing the same patterns as before. You now have the ability to create your own patterns plugins and use multiple plugins to clean your modules. This allows flexibility with both the programmatic API and CLI.

ModClean scans the node_modules directory of your choosing, finding all files and folders that match the defined patterns and deleting them. Both the CLI and the programmatic API provides all the options needed to customize this process to your requirements. Depending on the number of modules your app requires, files can be reduced anywhere from hundreds to thousands and disk space can be reduced considerably.

(File and disk space reduction can also be different between the version of NPM and Operating System)

IMPORTANT This module has been heavily tested in an enterprise environment used for large enterprise applications. The provided patterns in modclean-patterns-default have worked very well when cleaning up useless files in many popular modules. There are hundreds of thousands of modules in NPM and I cannot simply cover them all. If you are using ModClean for the first time on your application, you should create a copy of the application so you can ensure it still runs properly after running ModClean. The patterns are set in a way to ensure no crutial module files are removed, although there could be one-off cases where a module could be affected and that's why I am stressing that testing and backups are important. If you find any files that should be removed, please create a pull request to modclean-patterns-default or create your own patterns plugin to share with the community.

Removal Benchmark

So how well does this module work? If we npm install sails and run ModClean on it, here are the results:

All tests ran on macOS 10.12.3 with Node v6.9.1 and NPM v4.0.5

Using Default Safe Patterns

modclean -n default:safe or modclean

	Total Files	Total Folders	Total Size
Before ModClean	16,179	1,941	71.24 MB
After ModClean	12,192	1,503	59.35 MB
Reduced	3,987	438	11.88 MB

Using Safe and Caution Patterns

modclean -n default:safe,default:caution

	Total Files	Total Folders	Total Size
Before ModClean	16,179	1,941	71.24 MB
After ModClean	11,941	1,473	55.28 MB
Reduced	4,238	468	15.95 MB

Using Safe, Caution and Danger Patterns

modclean --patterns="default:*"

	Total Files	Total Folders	Total Size
Before ModClean	16,179	1,941	71.24 MB
After ModClean	11,684	1,444	51.76 MB
Reduced	4,495	497	19.47 MB

That makes a huge difference in the amount of files and disk space.

View additional benchmarks on the Wiki: Benchmarks. If you would like to run some of your own benchmarks, you can use modclean-benchmark.

Install

Install locally

npm install modclean --save

Install globally (CLI)

npm install modclean -g

Read the CLI Documentation

Read the API Documentation

Read the Custom Patterns Plugin Documentation

Issues

If you find any bugs with either ModClean or the CLI Utility, please feel free to open an issue. Any feature requests may also be poseted in the issues.

License

ModClean is licensed under the MIT license. Please see LICENSE in the repository for the full text.

modclean's People

Contributors

Stargazers

Watchers

Forkers

iwege cmcgheit shananholm everydev jehy akshaybheda bachp 94yifanwu circle9r i-fork s100 paulgain jtrinklein fgyweb

modclean's Issues

--ignore option does not work

I'm trying to use this in a monorepo with symlinked packages. In my package.json I have my monorepo packages set with file: links. In node_modules these are symlinked to the path in the monorepo, outside the node_modules path. I'm using this inside of Docker, but I don't think that should affect the pattern matching or ignore. Docker might affect the symlinks though.

I tried --ignore="directory-name" and --ignore="@mynamespace/mypackage" and it still removed files from those packages which should have been ignored.

I tried --follow-symlinks=false and that didn't stop it, but I'm also not sure how that option is supposed to work.

Maybe I'm using it wrong, in which case maybe the docs need to be more clear.

v3 release with updated dependencies?

Looks like v3-dev has been used for years with no major issues. Maybe it is time to cut a release?!

Support for `modclean.json`

modclean-cli should really support reading it's settings from a modclean.json file, so that we can use this package declaratively.

Reduce dependencies

The biggest goal for Modclean 3.0.0 is to reduce dependencies across the board. I will either remove dependencies altogether, write custom functionality within this module, or find better dependencies.

Undefined reference - pkg in modclean.js on v3.0.0-alpha.5

v3.0.0-alpha.5 fails to run in my docker container with the following error.

Step 25/32 : RUN npm install modclean -g && modclean -r
[main] ---> Running in f335624c34e4
[main] /usr/local/bin/modclean -> /usr/local/lib/node_modules/modclean/bin/modclean.js
[main] + [email protected]
[main] added 41 packages in 0.701s
[main] /usr/local/lib/node_modules/modclean/bin/modclean.js:56
[main] chalk.gray(' Version ' + pkg.version) + "\n"
[main] ^
[main] ReferenceError: pkg is not defined
[main] at new ModClean_CLI (/usr/local/lib/node_modules/modclean/bin/modclean.js:56:38)
[main] at Object. (/usr/local/lib/node_modules/modclean/bin/modclean.js:280:1)
[main] at Module._compile (module.js:652:30)
[main] at Object.Module._extensions..js (module.js:663:10)
[main] at Module.load (module.js:565:32)
[main] at tryModuleLoad (module.js:505:12)
[main] at Function.Module._load (module.js:497:3)
[main] at Function.Module.runMain (module.js:693:10)
[main] at startup (bootstrap_node.js:188:16)
[main] at bootstrap_node.js:609:3

My node version is 8.11.1

Remove "semver" from danger patterns

semver is a popular module for checking package versions www.npmjs.com/package/semver and removing is will obviously do harm - and I don't think it can be useful.

Removed module uglify-save-license

I ran into a problem running my Gulp scripts after using modclean -n safe. This is because the patterns.json has *license* under the safe patterns. My current Gulp flow however includes uglify and also a module called uglify-save-license.

Now my Gulp task breaks with the error Error: Cannot find module 'uglify-save-license'.
Might be a difficult thing to solve since license files never have an extension.

What would you recommend? Should this be in the patterns.json or is it possible to add an exception via the CLI? (haven't found it in the docs).

additionalPatterns does not work

If I call modclean -t --patterns="default:*" -rPt it will find some amount of files. However if I run
modclean --patterns="default:*" --additional-patterns="*.h,*.c,*.cpp,*.cc,*.hpp,*.gypi,*.mk,*.Makefile,*.a,*.o,**/node_modules/*/build" -rPt
no files at all will be found.

Wrong repository link on npmjs

Github link on https://www.npmjs.com/package/modclean points to "https://github.com/ModClean/modclean/blob/3.0.0-dev" which gives 404.

case-sensitive is always active

I thought the existing of the command line option --case-sensitive implies case-insensitive is active by default but it is not true. And in my opinion / observation case-insensitive is actually helpful / desired.

Fails to install on Vagrant Windows host

vagrant@debian-testing-amd64:~/sync$ sudo npm install modclean
npm ERR! Linux 3.16.0-4-amd64
npm ERR! argv "node" "/usr/bin/npm" "install" "modclean"
npm ERR! node v0.12.2
npm ERR! npm v2.8.4
npm ERR! path ../rimraf/bin.js
npm ERR! code EROFS
npm ERR! errno -30

npm ERR! rofs EROFS, symlink '../rimraf/bin.js'
npm ERR! rofs This is most likely not a problem with npm itself
npm ERR! rofs and is related to the file system being read-only.
npm ERR! rofs
npm ERR! rofs Often virtualized file systems, or other file systems
npm ERR! rofs that don't support symlinks, give this error.

vagrant@debian-testing-amd64:/sync$ npm --version
2.8.4
vagrant@debian-testing-amd64:/sync$ node --version
v0.12.2

history* pattern

Hello! Thanks for your script, I find it very useful especially now that I'm using it while building an Electron app. Anyway I have this small problem with the React Router library, since they have a file named History.js, and so after running modclean the entire app breaks.

I understand that "history" is a common name for changelog files, but could the rule be changed so it delete the file only if its extension is something different from js? Thank you.

safe pattern *.map causes removal of lodash.map package

The whole lodash.map folder gets removed because of safe rule *.map. Maybe matching this rule against files only would help rather than removing the rule

Number of removed files by pattern

If would be interesting to show the number of removed files by pattern. Some of the "safe" patterns aren't that safe, if from these proposed statistics I can see that 90% of the savings come from removing *.md files or something I would just use that pattern.

CLI doesn't remove a typescript file

I've noticed that there are many *.ts (typescript) files remained in node_modules even after running this CLI tool.

BTW, This is awesome utility tool!

Don't follow symlinks

Currently running modclean when npm link cause deletion in the linked packages.
We should have a way to ignore symlinks

Use exact version for modclean-patterns-default

If someone is using modclean in production without package-lock, then updating transitive dependency modclean-patterns-default may cause sudden harm with no obvious reasons.

I think that version of modclean-patterns-default should only be updated with updating modclean version and should be exact.

Provide an option to disable progress bar

I am using modclean with quite some dependencies and it removes 3384 files. I run that in a build environment with non-interactive console (Jenkins) and the progress bar indicating the removal of every file adds 3384 lines to the build console output. This is not very convenient for me.

Please provide a command line option that will prevent modclean from outputting progress indicator.

Possible issue with default patterns and "license" keyword

I have been using Modclean ^3.0.0-beta.1 as part of a AWS Lambda packaging process to remove excess junk from the final .zip file prior to uploading to AWS for deployment.

However, at times, I've been chasing a weird problem where the .zip is uploaded with a damaged copy of the AWS Node.js SDK package. After stepping through each task on my build, I found out Modclean is the most likely culprint.

File Tree Excerpt Before:

-rw-rw-r-- 1 bamboo bamboo    8023 Oct 26  1985 kinesisvideomedia.d.ts
-rw-rw-r-- 1 bamboo bamboo     643 Oct 26  1985 kinesisvideomedia.js
-rw-rw-r-- 1 bamboo bamboo  149363 Oct 26  1985 kms.d.ts
-rw-rw-r-- 1 bamboo bamboo     541 Oct 26  1985 kms.js
-rw-rw-r-- 1 bamboo bamboo   86733 Oct 26  1985 lambda.d.ts
-rw-rw-r-- 1 bamboo bamboo     922 Oct 26  1985 lambda.js
-rw-rw-r-- 1 bamboo bamboo  121591 Oct 26  1985 lexmodelbuildingservice.d.ts
-rw-rw-r-- 1 bamboo bamboo     655 Oct 26  1985 lexmodelbuildingservice.js
-rw-rw-r-- 1 bamboo bamboo   29960 Oct 26  1985 lexruntime.d.ts
-rw-rw-r-- 1 bamboo bamboo     592 Oct 26  1985 lexruntime.js
-rw-rw-r-- 1 bamboo bamboo   30490 Oct 26  1985 licensemanager.d.ts
-rw-rw-r-- 1 bamboo bamboo     620 Oct 26  1985 licensemanager.js
-rw-rw-r-- 1 bamboo bamboo  237289 Oct 26  1985 lightsail.d.ts
-rw-rw-r-- 1 bamboo bamboo     583 Oct 26  1985 lightsail.js
-rw-rw-r-- 1 bamboo bamboo  124033 Oct 26  1985 machinelearning.d.ts
-rw-rw-r-- 1 bamboo bamboo     758 Oct 26  1985 machinelearning.js
-rw-rw-r-- 1 bamboo bamboo   14752 Oct 26  1985 macie.d.ts
-rw-rw-r-- 1 bamboo bamboo     555 Oct 26  1985 macie.js
-rw-rw-r-- 1 bamboo bamboo   15449 Oct 26  1985 marketplacecommerceanalytics.d.ts
-rw-rw-r-- 1 bamboo bamboo     716 Oct 26  1985 marketplacecommerceanalytics.js
-rw-rw-r-- 1 bamboo bamboo    6279 Oct 26  1985 marketplaceentitlementservice.d.ts
-rw-rw-r-- 1 bamboo bamboo     711 Oct 26  1985 marketplaceentitlementservice.js

Running Modclean as follows: node_modules/.bin/modclean --run --modules-dir=build/out/node_modules

Output:

MODCLEAN  Version 3.0.0-beta.1

Deleting files, please wait...
Deleting empty directories, please wait...

FILES/FOLDERS DELETED
    Total:    1713
    Skipped:  0
    Empty:    626

File Tree Excerpt After:

-rw-rw-r-- 1 bamboo bamboo    8023 Oct 26  1985 kinesisvideomedia.d.ts
-rw-rw-r-- 1 bamboo bamboo     643 Oct 26  1985 kinesisvideomedia.js
-rw-rw-r-- 1 bamboo bamboo  149363 Oct 26  1985 kms.d.ts
-rw-rw-r-- 1 bamboo bamboo     541 Oct 26  1985 kms.js
-rw-rw-r-- 1 bamboo bamboo   86733 Oct 26  1985 lambda.d.ts
-rw-rw-r-- 1 bamboo bamboo     922 Oct 26  1985 lambda.js
-rw-rw-r-- 1 bamboo bamboo  121591 Oct 26  1985 lexmodelbuildingservice.d.ts
-rw-rw-r-- 1 bamboo bamboo     655 Oct 26  1985 lexmodelbuildingservice.js
-rw-rw-r-- 1 bamboo bamboo   29960 Oct 26  1985 lexruntime.d.ts
-rw-rw-r-- 1 bamboo bamboo     592 Oct 26  1985 lexruntime.js
-rw-rw-r-- 1 bamboo bamboo  237289 Oct 26  1985 lightsail.d.ts
-rw-rw-r-- 1 bamboo bamboo     583 Oct 26  1985 lightsail.js
-rw-rw-r-- 1 bamboo bamboo  124033 Oct 26  1985 machinelearning.d.ts
-rw-rw-r-- 1 bamboo bamboo     758 Oct 26  1985 machinelearning.js
-rw-rw-r-- 1 bamboo bamboo   14752 Oct 26  1985 macie.d.ts
-rw-rw-r-- 1 bamboo bamboo     555 Oct 26  1985 macie.js
-rw-rw-r-- 1 bamboo bamboo   15449 Oct 26  1985 marketplacecommerceanalytics.d.ts
-rw-rw-r-- 1 bamboo bamboo     716 Oct 26  1985 marketplacecommerceanalytics.js
-rw-rw-r-- 1 bamboo bamboo    6279 Oct 26  1985 marketplaceentitlementservice.d.ts
-rw-rw-r-- 1 bamboo bamboo     711 Oct 26  1985 marketplaceentitlementservice.js

Note how licensemanager.js gets tossed, and unfortunately the AWS SDK fails to load because files are missing.

I looked at the default safe patterns, and it does not seem like the license term is treated with a wildcard, and yet it appears any file with the word license as part of the name get tossed.

Am I misunderstanding something?

Write tests

Need to write tests to ensure all functionality works correctly going forward.

empty directories do not get removed

I run the command
node_modules/modclean/bin/modclean.js -n default:safe
three times and this is my results:
###########FIRST TIME#############

✔ Found 9001 files to remove
[==============================] 100% (9001/9001) 0.0s

✔ Found 6661 empty directories to remove
[==============================] 100% (6661/6661) 0.0s

FILES/FOLDERS DELETED
Total: 15662
Skipped: 0
Empty: 6661

###########SECOND TIME#############

✔ Found 0 files to remove
✔ Found 6545 empty directories to remove
[==============================] 100% (6545/6545) 0.0s

FILES/FOLDERS DELETED
Total: 6545
Skipped: 0
Empty: 6545

###########THIRD TIME#############

✔ Found 0 files to remove
✔ Found 6545 empty directories to remove
[==============================] 100% (6545/6545) 0.0s

FILES/FOLDERS DELETED
Total: 6545
Skipped: 0
Empty: 6545

As you can see the empty folders still are showing up after multiple times running. It should show 0 empty after the first run correct?

Does this package can be used to clean bower components?

Minimize files in product mode

I am using electron, mode_modules is too big

Don't clean directories directly below any node_modules directory

modclean removes the well-known history package. This problem is similar to the one @phips28 encountered in ModClean/modclean-patterns-default#1.

My suggestion is to not delete (unless empty) directories that are children of a node_modules directory. This catches all situations in which a package matches a pattern.

--ignore option does not work

--ignore option does not work, at least for "*.sh" pattern.

Reproduce:

mkdir test
cd test
mkdir node_modules
cd node_modules/
mkdir test_module
cd test_module/
touch smth.sh
cd ../../
modclean --patterns="default:safe,default:caution,default:danger" --run --ignore="*.sh" --verbose --test

Expected not to remove smth.sh, actual result:

MODCLEAN Version 2.1.2

RUNNING IN TEST MODE
When running in test mode, files will not be deleted from the file system.

EVENT> start
VERBOSE>
{
"cwd": "/home/jehy/tmp/test/node_modules",
"patterns": [
"default:safe",
"default:caution",
"default:danger"
],
"additionalPatterns": [],
"ignorePatterns": [
"*.sh"
],
"noDirs": false,
"ignoreCase": true,
"dotFiles": true,
"modulesDir": "node_modules",
"removeEmptyDirs": true,
"errorHalt": false,
"test": true,
"followSymlink": false
}
Searching for files in /home/jehy/tmp/test/node_modules...
EVENT> files
Found 1 files/folders to remove

EVENT> process
VERBOSE> DELETED (1/1) test_module/smth.sh
EVENT> finish
VERBOSE> FINISH Deleted 1 files/folders of 1
EVENT> complete

FILES/FOLDERS DELETED
Total: 1
Skipped: 0
Empty: 0

Action required: Greenkeeper could not be activated 🚨

🚨 You need to enable Continuous Integration on all branches of this repository. 🚨

To enable Greenkeeper, you need to make sure that a commit status is reported on all branches. This is required by Greenkeeper because it uses your CI build statuses to figure out when to notify you about breaking changes.

Since we didn’t receive a CI status on the greenkeeper/initial branch, it’s possible that you don’t have CI set up yet. We recommend using Travis CI, but Greenkeeper will work with every other CI service as well.

If you have already set up a CI for this repository, you might need to check how it’s configured. Make sure it is set to run on all new branches. If you don’t want it to run on absolutely every branch, you can whitelist branches starting with greenkeeper/.

Once you have installed and configured CI on this repository correctly, you’ll need to re-trigger Greenkeeper’s initial pull request. To do this, please delete the greenkeeper/initial branch in this repository, and then remove and re-add this repository to the Greenkeeper App’s white list on Github. You'll find this list on your repo or organization’s settings page, under Installed GitHub Apps.

feature request: add --force/-f or similar

This wold allow users to run using CI with no interaction by skipping the warning/confirmation request.

Refactor to async/await

In version 3.0.0, Modclean will be refactored to use async/await and promises instead of traditional callbacks. This should help speed up the process, reduce code complexity, and stay in line with the latest Node releases.

Cli use throws promisify error

Hello

I have been unable to get the CLI to work
I recieve the following error details when run the modclean command:

const glob = util.promisify(require('glob'));
                  ^

TypeError: util.promisify is not a function
    at Object.<anonymous> (C:\Users\Tommy\AppData\Roaming\nvm\v7.7.4\node_modules\modclean\lib\modclean.js:15:19)
    at Module._compile (module.js:571:32)
    at Object.Module._extensions..js (module.js:580:10)
    at Module.load (module.js:488:32)
    at tryModuleLoad (module.js:447:12)
    at Function.Module._load (module.js:439:3)
    at Module.require (module.js:498:17)
    at require (internal/module.js:20:19)
    at Object.<anonymous> (C:\Users\Tommy\AppData\Roaming\nvm\v7.7.4\node_modules\modclean\bin\modclean.js:11:18)
    at Module._compile (module.js:571:32)

I currently use Node v7.7.4

Steps to reproduce

run npm i -g modclean
run modclean

Maximum call stack size exceeded

/c/v/w/frontend master ❯ modclean --test                                                                                                                            ✭ ✱ ◼

MODCLEAN v1.2.7

RUNNING IN TEST MODE
When running in test mode, files will not be deleted from the file system.

> Found 2927 files/folders to remove

Cleanup Progress [||||||||||||---------------------------] 32% (934/2927)/home/rush/.nvm/versions/node/v4.4.2/lib/node_modules/modclean/node_modules/colors/lib/extendStringPrototype.js:70
      'indexOf', 'lastIndexof', 'length', 'localeCompare', 'match', 'replace', 'search', 'slice', 'split', 'substring',
                                                              ^

RangeError: Maximum call stack size exceeded
    at String.valueOf (native)
    at Object.stylize (/home/rush/.nvm/versions/node/v4.4.2/lib/node_modules/modclean/node_modules/colors/lib/colors.js:55:33)
    at String.magenta (/home/rush/.nvm/versions/node/v4.4.2/lib/node_modules/modclean/node_modules/colors/lib/extendStringPrototype.js:58:21)
    at log (/home/rush/.nvm/versions/node/v4.4.2/lib/node_modules/modclean/bin/modclean.js:230:29)
    at ModClean.<anonymous> (/home/rush/.nvm/versions/node/v4.4.2/lib/node_modules/modclean/bin/modclean.js:166:13)
    at emitOne (events.js:77:13)
    at ModClean.emit (events.js:169:7)
    at ModClean._deleteFile (/home/rush/.nvm/versions/node/v4.4.2/lib/node_modules/modclean/index.js:239:14)
    at /home/rush/.nvm/versions/node/v4.4.2/lib/node_modules/modclean/index.js:210:43
    at ModCleanCLI.options.process (/home/rush/.nvm/versions/node/v4.4.2/lib/node_modules/modclean/bin/modclean.js:96:42)

Doesn't work as global

After doing npm install -g modclean and running modclean it throws

env: node\r: No such file or directory

Latest release removes semver

Hey,

we are using modlean for quite some time, unfortunately the latest release (1.2.6) started to break our docker builds/node api:

Error: Cannot find module 'semver'
    at Function.Module._resolveFilename (module.js:326:15)
    at Function.Module._load (module.js:277:25)
    at Module.require (module.js:354:17)
    at require (internal/module.js:12:17)
    at Object.<anonymous> (/opt/api/node_modules/loopback-boot/lib/executor.js:2:14)
    at Module._compile (module.js:410:26)
    at Module._extensions..js (module.js:417:10)
    at Object.require.extensions.(anonymous function) [as .js] (/opt/api/node_modules/babel-core/lib/api/register/node.js:214:7)
    at Module.load (module.js:344:32)
    at Function.Module._load (module.js:301:12)
module.js:328
    throw err;

This is because the semver safe pattern was added. Would it be feasible to add a CLI option for specifying a JSON list of patterns to exclude, for instance --ignore=keep_us.json, or even .modcleanignore?
That way we could keep a small ignore json in our repository.

Remove .bin from pattern

Please remove .bin from patterns, a lot of projects use the .bin created by npm from local module binaries for different npm scripts, examples being running tests with jasmine, linters like jshint, jscs.