Code Monkey home page Code Monkey logo

node-klaw-sync's Introduction

node-klaw-sync

npm Package Build Status windows Build status JavaScript Style Guide Known Vulnerabilities

klaw-sync is a Node.js recursive and fast file system walker, which is the synchronous counterpart of klaw. It lists all files and directories inside a directory recursively and returns an array of objects that each object has two properties: path and stats. path is the full path of the file or directory and stats is an instance of fs.Stats.

Install

npm i klaw-sync

Usage

klawSync(directory[, options])

  • directory <String>
  • options <Object> (optional)
    • nodir <Boolean> default: undefined
      • return only files (ignore directories).
    • nofile <Boolean> default: undefined
      • return only directories (ignore files).
    • depthLimit: <Number> default: -1
      • the number of times to recurse before stopping. -1 for unlimited.
    • fs: <Object> default: graceful-fs
      • custom fs, useful when mocking fs object.
    • filter <Function> default: undefined
      • function that gets one argument fn({path: '', stats: {}}) and returns true to include or false to exclude the item.
    • traverseAll <Boolean> default: undefined
      • traverse all subdirectories, regardless of filter option. This can be useful when you have a filter function and still want to traverse all subdirectories even if your filter function doesn't pass for some directories.
  • Return: <Array<Object>> [{path: '', stats: {}}]

Examples

const klawSync = require('klaw-sync')

const paths = klawSync('/some/dir')
// paths = [{path: '/some/dir/dir1', stats: {}}, {path: '/some/dir/file1', stats: {}}]

catch error

const klawSync = require('klaw-sync')

let paths
try {
  paths = klawSync('/some/dir')
} catch (er) {
  console.error(er)
}
console.dir(paths)

files only

const klawSync = require('klaw-sync')

const files = klawSync('/some/dir', {nodir: true})
// files = [{path: '/some/dir/file1', stats: {}}, {path: '/some/dir/file2', stats: {}}]

directories only

const klawSync = require('klaw-sync')

const dirs = klawSync('/some/dir', {nofile: true})
// dirs = [{path: '/some/dir/dir1', stats: {}}, {path: '/some/dir/dir2', stats: {}}]

ignore hidden directories

const path = require('path')
const klawSync = require('klaw-sync')

const filterFn = item => {
  const basename = path.basename(item.path)
  return basename === '.' || basename[0] !== '.'
}

const paths = klawSync('/some/dir', { filter: filterFn})

filter based on stats

Here traverseAll option is required since we still want to read all subdirectories even if they don't pass the filter function, to see if their contents do pass the filter function.

const klawSync = require('klaw-sync')

const refTime = new Date(2017, 3, 24).getTime()
const filterFn = item => item.stats.mtime.getTime() > refTime

const paths = klawSync('/some/dir', { traverseAll: true, filter: filterFn })

Run tests

lint: npm run lint

unit test: npm run unit

lint & unit: npm test

benchmark: npm run benchmark

Performance compare to other similar modules

Running some benchmark tests on these modules:

(as of Jan 25, 2017) klaw-sync is the fastest module!

results (tested on Ubuntu 18.04, Intel(R) Core(TM) i7-2630QM CPU @ 2.00GHz, 8 CPUs, 8g RAM, node v10.9.0)
Running benchmark tests..

root dir length: 1110
walk-sync x 80.71 ops/sec ±1.42% (72 runs sampled)
klaw-sync x 160 ops/sec ±1.17% (79 runs sampled)
Fastest is klaw-sync

root dir length: 11110
walk-sync x 7.55 ops/sec ±3.39% (23 runs sampled)
klaw-sync x 14.95 ops/sec ±0.27% (40 runs sampled)
Fastest is klaw-sync

root dir length: 111110
walk-sync x 0.63 ops/sec ±6.92% (6 runs sampled)
klaw-sync x 1.22 ops/sec ±0.96% (7 runs sampled)
Fastest is klaw-sync

License

Licensed under MIT

node-klaw-sync's People

Contributors

geelik avatar jskrzypek avatar manidlou avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar

node-klaw-sync's Issues

Confusing description of traverseAll default value

Package version: 6.0.0

The README reads:

  • traverseAll <Boolean> default: true
    • traverse all subdirectories, regardless of filter option. (When set to true, traverseAll produces similar behavior to the default behavior prior to v4.0.0. The current default of traverseAll: false is equivalent to the old noRecurseOnFailedFilter: true).

First, it says it's true by default.

Then it says "The current default of traverseAll: false" implicating it's false by default.

In the code it is in fact undefined, which is equivalent to false in this case.

If the parameter is not present, klaw-sync behaves as if it was false.

What is the intended default value of this parameter, true or false?
true seems more reasonable to me, but regardless, I believe default value should be indicated clearly in the docs and correspond to the actual behavior.

I also find it confusing that sister module node-klaw does not have this option at all.

Use lstatSync instead of statSync

Hi! 👋

Firstly, thanks for your work on this project! 🙂

Today I used patch-package to patch [email protected] for the project I'm working on.

This fixes #17.

Here is the diff that solved my problem:

diff --git a/node_modules/klaw-sync/klaw-sync.js b/node_modules/klaw-sync/klaw-sync.js
index 74b87c6..6e7689a 100644
--- a/node_modules/klaw-sync/klaw-sync.js
+++ b/node_modules/klaw-sync/klaw-sync.js
@@ -13,7 +13,7 @@ function klawSync (dir, opts, ls) {
   const paths = opts.fs.readdirSync(dir).map(p => dir + path.sep + p)
   for (var i = 0; i < paths.length; i += 1) {
     const pi = paths[i]
-    const st = opts.fs.statSync(pi)
+    const st = opts.fs.lstatSync(pi)
     const item = {path: pi, stats: st}
     const isUnderDepthLimit = (!opts.rootDepth || pi.split(path.sep).length - opts.rootDepth < opts.depthLimit)
     const filterResult = opts.filter ? opts.filter(item) : true

This issue body was partially generated by patch-package.

Reading files only with depthLimit:0 also reads directories.

example:
const files = klawSync(os.homedir()+'/Documents', {nodir: true, depthLimit: 0});

with above line i get the file and directory names. But dir names are just truncated file path in children.
result-> file.txt, dirname, dirname2
directory names are actually pointing to the files in child directories but truncated.
i.e -> dirname/file-in-child.txt

noDir is not working

I am using version 6.0.0 and it seems that noDir is not working.

const path = require('path');
const klawSync = require('klaw-sync');

const tmplDir = path.join(__dirname, 'generators/react/app/templates/');

// get all templates (no directories)
const templates = klawSync(tmplDir, { noDir: true });

templates.forEach((template) => {
  const tmplPath = template.path;
  console.log(template.stats.isFile() ? 'f' : 'd', tmplPath);
});

Output shows both files and directories. Am I doing something wrong?

.asar files are treat like folders

OS: Windows 10.

Unfortunetly klaw-sync treat .asar folders like foders and list's all files that are inside.

this is very problematic, because thouse files are not accesible, example:
electron.asar\browser\api\app.js
electron.asar\browser\api\auto-updater\auto-updater-native.js

loading all files with certain name

After switching to klaw, I seem to be having some issue loading only files with a certain name.

Say I want to recursively load only files named package.json. It seems like I could do something like:

var klawSync = require('klaw-sync')
var paths = klawSync('/some/dir', {ignore: '!**/package.json'})

I have also tried several other variations of the ignore pattern but nothing seems to match properly.

I am probably missing something obvious... Any ideas?

Exception thrown when directory contains broken symlink.

Right now, the code uses statSync, but this attempts to read through symlinks, and will throw ENOENT for any broken symlinks.

IMO, this call should be lstatSync as it gives users more control (and also doesn't throw exceptions if the link is broken).

I found this will using a library that depends on klawSync, I wasn't aware that there was a broken symlink in the directory, but nonetheless I don't think the library should assume that users always want to read through symlinks. It would be better to let the user library break and they can deal with it accordingly.

EPERM dir or file error

When the path is D:\ or C:\ on Windows, had the error which operation not permitted.
Maybe should to have options for ignore the error and keep to walking directory.

{ Error: EPERM: operation not permitted, stat 'D:\\System Volume Information'
    at Object.statSync (fs.js:850:3)
    at Object.statSync (D:\...\node_modules\graceful-fs\polyfills.js:295:24)
    at klawSync (D:\...\node_modules\klaw-sync\klaw-sync.js:16:24)
....
}

depthLimit option is ignored when filter option is set

Hi,

When using this options, I can only access the level 0 (default) of the directory been read.

I want to match any "/folder/subfolder" that match my regexp (link: https://regex101.com/r/EYiftm/1).

The regexp seem to be correct, but when I walk the root folder with the option "deepthLimit:1" (so to get the folder and subfolder names in the path) and the filter option, I only get the folder in deep 0.

If I disable the filteroption, I get the folder/subfolder as expected.

depthLimit: 1 get ignored when using with filter option.

function filter_yearFolder(item){
    return /\/\d{4}\/.+$/.test(item.path);
}

let dirs = klawSync("/Users/user1/work", {nofile: true, depthLimit: 1, filter: filter_yearFolder})
 console.log(dirs);

Removing filter option let depthLimit:1work as expected (but sans any filtering)

let dirs = klawSync("/Users/user1/work", {nofile: true, depthLimit: 1})
 console.log(dirs);

Best regards!!

filter function still called with directories when nodir: true

I just updated to version 4 (was on version 2).

I noticed that with the option nodir: true, the library was actually looking only at the files, which is expected, but unfortunately it does so only at depth 0, and never goes inside of the directories.
It is a bit of a problem, because if one wants to find specific files (using a filter function), you will need instead to do something like:

const path = require('path');
const klawSync = require('klaw-sync');

function filterFn(item) {
  return item.stats.isDirectory() || /[some regex]/.test(path.basename(item.path));
}

const paths = klawSync([FOLDER_PATH], { filter: filterFn });

and then you will need to loop on the paths one more time to filter the directories out yourself.

const files = paths.filter(function(item) {
  return item.stats.isFile();
});

It would great if the default with nodir: true was to traverse all the directories, and call the filter function only for the files.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.