danielstjules / jsinspect Goto Github PK
View Code? Open in Web Editor NEWDetect copy-pasted and structurally similar code
License: MIT License
Detect copy-pasted and structurally similar code
License: MIT License
Hey @danielstjules I needed to make some changes to be able to fix build breaks asap, so I created a fork at:
https://github.com/detect-copy-paste/detect-copy-paste and invited you to it.
Hi,
You've done a very great work on jsinspect, it's a great, great tool.
If I may suggest a little modification : the possibility to use jsinspect programmatically, without the CLI-interface, by exposing the lib/inspector.js
file to the main
property of the package.json file, and maybe integrate a callback support for the results, allowing (and facilitate) the use and integration of jsinspect in other tools, and tasks runners like grunt or gulp, by example, and with the promise mecanisms like async.
Regards,
E.g. I have the code:
function sum(a, b) {
return a + b;
}
I copy it to the new file - copy-paste is detected. If I add export
before the function, copy-paste is not detected anymore
Would be nice to highlight the identifiers/literals that are the only difference between the snippets, when one uses the -I
or -L
options, in a similar manner to a color diff.
They would be helpful in refactoring since I'd have an hint on what should be a parameter of the newly created function.
Have you thought about it?
jsinspect fails when code is using anonymous ES6 class: https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Operators/class
To reproduce:
var Foo = class {
constructor() {}
bar() {
return "Hello World!";
}
};
var instance = new Foo();
instance.bar(); // "Hello World!"
Foo.name;
[TypeError: Cannot read property 'type' of null]
0.11.4 is reporting the following duplication even though our threshold is set to 40 in the jsinspectrc. It wasn't reporting this false positive in 0.9:
Match - 2 instances
./App/Containers/GuestJoinLayout.js:168,168
</Animatable.View>
./App/Containers/SignInLayout.js:160,160
</Animatable.View>
------------------------------------------------------------
The relevant part of GuestJoinLayout.js:
render () {
return (
<View style={Styles.container}>
<View style={Styles.content}>
<Animatable.View ref='content' animation='fadeInUp' duration={2000} ease='ease-in'>
<LinearGradient
start={{x: 0.0, y: 0.0}} end={{x: 0.0, y: 1.0}}
locations={[0, 1]}
colors={['#074595', '#6589A4']}
style={Styles.signInContainer}
>
{this.renderTitle()}
{this.renderMeetingId()}
{this.renderPasscode()}
{this.renderUsernameInput()}
{this.renderDropdown()}
{this.renderJoinButton()}
{/* this.renderSignIn() */}
</LinearGradient>
</Animatable.View>
</View>
</View>
)
}
and SignInLayout.js:
render () {
return (
<Animatable.View ref='container' style={Styles.container} easing='ease-in'>
<LinearGradient
start={{x: 0.0, y: 0.0}} end={{x: 0.0, y: 1.0}}
locations={[0, 1]}
colors={['#074595', '#6589A4']}
style={Styles.blurOverlay}>
<View style={Styles.content}>
{this.renderLogo()}
{this.renderErrorMessage()}
{this.renderUsernameInput()}
{this.renderPasswordInput()}
{this.renderNextButton()}
{this.renderSignInOptions()}
{this.renderSignUp()}
</View>
</LinearGradient>
</Animatable.View>
)
}
https://github.com/danielstjules/jsinspect/blob/master/lib/reporters/default.js#L60
chalk.grey is unavailable method and change name chalk.gray
https://github.com/chalk/ansi-styles/blob/master/index.js#L25
must fix it
jsinspect 0.8.0 (windows)
jsinspect -i --ignore "public|test|\/js" -m 3
gives
error: unknown option '-m'
According to the documentation -m
sets min matches.
I have a Node.js CLI utility written with tj/commander.js, which consist of several executable files without extension, with the form:
#!/usr/bin/env node
var program = require('commander');
...
These are ignored by jsinspect and I would like to have an option to include them, maybe even manually inside an array, something like:
{
"threshold": 30,
"identifiers": true,
"ignore": "test|spec",
"reporter": "default",
"suppress": 100,
"include": ['myScript1', 'myScript2', 'src/myScript3']
}
Currently, it is possible for a configuration error or file system (e.g. file not found) errors to have the same exit code as if the tool found offending matches. Is it possible for the tool to return a negative result to indicate an unrecoverable error and a positive result to indicate that the tool ran without error but found problems? This should make it easier for users integrating the tool to distinguish problems with an environment or script vs search results from the tool.
Similar to istanbul it would be nice to skip sections with code inserts. In my angularjs app many of the modules start with a similar 'imports' section and they are all stating they are duplicates. Yes they are, but I accept these sections and would like to focus on the meat and potatoes
Add more reporters, perhaps JSON and TAP, which seem to be popular in CI environments
Hello,
I work on a personal project and use jsinspect.
I wrote my own node modules, when using jsinspect it ignore my folder app/node_modules.
How can ignore only root node_modules directory and still checking my custom modules ?
Best regards,
__
We have this in our .jsinspectrc:
{
"threshold": 40,
"identifiers": false,
"matches": 3,
"ignore": "Test.js|Spec.js|Colors.js|Images.js", // used as RegExp,
"jsx": true,
"reporter": "default",
"suppress": 100
}
But are getting reports like this after upgrading to 0.11.4 even though 'matches' is set to 3:
Match - 2 instances
./App/Components/InMeeting/CallControlHeader.js:41,43
<View style={Styles.center}>
{ !videoMuted && <ControlButton icon={Images.camera} title={'VIDEO ON'} onPress={videoMute} /> }
{ videoMuted && <ControlButton icon={Images.cameraOff} title={'VIDEO OFF'} onPress={videoUnmute} off /> }
./App/Containers/DashboardLayout.js:118,120
<View style={Styles.topControls}>
{ !videoMuted && <ControlButton icon={Images.camera} title={'VIDEO ON'} onPress={videoMute} /> }
{ videoMuted && <ControlButton icon={Images.cameraOff} title={'VIDEO OFF'} onPress={videoUnmute} off /> }
------------------------------------------------------------
Adding an option to get word-diff between sources parts would be wonderful to spot exactly what changed. See the --word-diff
option of git diff
or the dwdiff command.
Ignoring white space (like the git diff -w
option) would a easier and less powerfull alternative, but still a great addition.
Those ASTs are pretty big 😅
In some libraries like Polymer (and maybe other Web Components solutions as well) JS mostly lives in inline script
tags. We are able to use eslint-plugin-html for such cases. Are there any plans to add similar support to jsinspect
?
A pmd-cpd reporter would be nice. I could hook this up to Jenkins and have it create graphs overtime of violations.
Looking at a few outputs from other tools that I use, it looks like it needs to be in the format:
<?xml version="1.0" encoding="utf-8"?>
<pmd-cpd>
<duplication lines="2" tokens="38">
<file path="file/path1.js" line="1373"/>
<file path="file/path2.js" line="1389"/>
<codefragment>
XML Escaped Code Fragment
</codefragment>
</duplication>
</pmd-cpd>
A code example
let b;
[, b] = [4, 5]
The Syntax is valid, and Babylon parses it correctly.
However, running it through the inspector, I get the following error:
TypeError: Cannot read property 'type' of null
at Function.getChildren (/Users//node_modules/jsinspect/lib/nodeUtils.js:98:24)
at dfs (/Users//node_modules/jsinspect/lib/nodeUtils.js:47:17)
at Array.forEach (native)
Other than that, the new version looks great!
e.g.,
jsinspect -i -matches 3 ./path/to/src
Then show the instances which match at least 3 times.
Our angular app includes tests in the folders. I would like to ignore these files from being inspected.
jsinspect -t 40 -i ./app --ignore *-test
I want to build a webpack loaded for this. What do you think? Maybe a plugin would be more appropriate?
Hi,
Currently it's only possible to select one of the build in reporters.
It would be great if custom reporter could be specified via CLI or .jsinspectrc
config file. It would enable the creation of custom reporters (in separate npm packages)... like it is possible e.g. in JSCPD or ESLint.
In this context, it would probably also make sense to expose the base reporter in the lib/reporters/index.js
file to simplify writing custom reporters...
Is it possible to not suppress diff at all?
Do you have plans on that?
And if not, do you mind when I wrap it?
identifiers, literals, and colors can't be set from the config file.
Because:
--no-
flag is set to true
by default, and the code only replaces undefined
values.It would be useful to have an extension by Atom
I have this code:
export * as util from './util';
Checking of this files throws the type of null
error
I fixed it by refactoring:
import * as util from './util';
export { util };
Anyway it will be great to fix it
Executing the following command works wonderfully for including .jsx
files:
jsinspect --jsx ./src/js
When I remove the --jsx
switch from here to the .jsinspectrc
file, the .jsx
files are no longer included. Here's the config file I was using:
{
"identifiers": true,
"ignore": "tests",
"jsx": true,
"reporter": "default",
"suppress": 100,
"threshold": 30
}
This feature doesn't seem to be supported in the config file. Is there any interest in supporting it? If so, I can work on a PR.
When I run $ jsinspect <path>
directly from the command line, it works as expected. However when I run it as an npm script ($ npm run jsinspect
)
"scripts": {
"jsinspect": "jsinspect src/"
},
and matches are found, error is thrown:
1 match found across 2 files
npm ERR! Darwin 15.5.0
npm ERR! argv "/Users/aborowski/.nvm/versions/node/v6.3.1/bin/node" "/Users/aborowski/.nvm/versions/node/v6.3.1/bin/npm" "run" "jsinspect"
npm ERR! node v6.3.1
npm ERR! npm v3.10.3
npm ERR! code ELIFECYCLE
npm ERR! [email protected] jsinspect: `jsinspect src/`
npm ERR! Exit status 1
Hi,
jsinspect produces false positives for a check with the linked 2 files, when the -i parameter is NOT set. I used the node level 30, 40 Looks like for function arguments that are callback functions the function code is not checked.
best regards Armin
https://github.com/agienger/issue_files/blob/master/jsinspect/callBack1.js
https://github.com/agienger/issue_files/blob/master/jsinspect/callBack2.js
Can jsinspect return an exit code other than 0 when duplicate code is found? I'm trying to use jsinspect as part of a git hook and I'm going to stop the commit/push if it finds anything. Right now, the exit code is always 0 (unless there is an error) and so I have to do some string parsing of the output.
See discussion in stefanjudis/grunt-jsinspect#29
Starting point would be here: https://github.com/danielstjules/jsinspect/blob/master/lib/reporters/base.js#L16
I have a Windows 10 machine with 8 gb of RAM. First time I run jsinspect in one of my projects and I get this:
$ jsinspect
[
<--- Last few GCs --->
208383 ms: Scavenge 1402.4 (1457.2) -> 1402.4 (1457.2) MB, 15.8 / 0 ms (+ 2.0 ms in 1 steps since last GC) [allocation failure] [incremental marking delaying mark-sweep].
210220 ms: Mark-sweep 1402.4 (1457.2) -> 1401.4 (1456.2) MB, 1836.9 / 0 ms (+ 4.0 ms in 2 steps since start of marking, biggest step 2.0 ms) [last resort gc].
211992 ms: Mark-sweep 1401.4 (1456.2) -> 1400.9 (1457.2) MB, 1771.8 / 0 ms [last resort gc].
<--- JS stacktrace --->
==== JS stack trace =========================================
Security context: 000003A02DCB4639 <JS Object>
1: finishNode [C:\Users\miparnisari\AppData\Roaming\npm\node_modules\jsinspect\node_modules\acorn\dist\acorn_loose.js:~647] [pc=00000259F86AE55F] (this=000001850C2CBAB1 <a LooseParser with map 0000035629E4CA89>,node=0000010720CDDF69 <a Node with map 0000035629E4D6E9>,type=000000B68F9CC089 <String[16]: MemberExpression>)
2: parseSubscripts [C:\Users\miparnisari\AppData\Roaming\npm\node...
FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - process out of memory
I am using these settings:
{
"threshold": 30,
"identifiers": true,
"matches": 3,
"ignore": "Test.js|Spec.js",
"jsx": false,
"reporter": "json",
"suppress": 100
}
EDIT: I used WindDirStat to know how many JS files there are: 23,540. It's a lot but I'm assuming that we are not scanning the files within .git or node_modules, right?
test1.js
(function () {
test1 = new Class({
build: function () {
this.options.header = this.options.title;
this.options.subtitle = this.options.preSubtitle;
this.options.main_class = this.options.mainClass;
this.options.wrapper_class = this.options.wrapperClass;
this.options.loading_class = this.options.loadingClass;
this.options.id = this.options.ID;
this.options.close_class = this.options.closeClass;
this.options.download = this.options.download;
this.options.downloadclass = this.options.downloadClass;
this.options.caputured_text = this.options.capturedText;
this.options.captured_date = new Date().format(this.options.titleDateFormat);
this.options.view_options_class = this.options.viewOptionsClass;
this.options.view_options = this.options.viewOptions;
this.options.show_status_label = this.options.showStatusLabel;
this.options.show = this.options.show;
this.options.hide = this.options.hide;
this.parent();
}
});
test2.js
(function () {
A = new Class({
initialize: function () {
o = o || {};
this.options = Global.options || {};
this.dialog = null;
this.request = null;
this.pollIntervalMax = 10000;
this.pollIntervalIncrement = 1000;
this.pollInterval = 1000;
this.pollToken = null;
this.pollIndex = 1;
this.elementId = 'reputations';
this.element = null;
this.messages = {
requestNoSuccess: 'request_no_success',
};
this.responseElements = {};
this.responseElements2 = {};
this.item = null;
this.requestsPending = [];
this.ResponseCount = 0;
},
});
}());
If you run jsinspect -i -t 34 test1.js test2.js, it will find a match. But with the identifier flag turned on, it don't think it should have matched them.
I believe the problem is that you don't have an identifierHandler for an ExpressionStatement so it is just comparing the fact that there are 16 ExpressionStatements in both files. If I add an identifierHandler for an ExpressionStatement (by checking the node type of the left node and using its identifierHandler), it stops matching the two.
class Inspector extends EventEmitter {
^^^^^
SyntaxError: Block-scoped declarations (let, const, function, class) not yet supported outside strict mode
at exports.runInThisContext (vm.js:53:16)
at Module._compile (module.js:387:25)
at Object.Module._extensions..js (module.js:422:10)
at Module.load (module.js:357:32)
at Function.Module._load (module.js:314:12)
at Module.require (module.js:367:17)
at require (internal/module.js:20:19)
at Object.<anonymous> (/usr/local/lib/node_modules/jsinspect/bin/jsinspect:9:17)
at Module._compile (module.js:413:34)
at Object.Module._extensions..js (module.js:422:10)
NodeJS version : v5.11.1
In some cases, the parser is not able to parse a file. Yet, it's very helpful to get the duplications report for other files.
An example: parsing a project with this line: var mode = 0700
, throws an error SyntaxError: Invalid number (46:30)
. The notation 0700 is actually in use, for instance here.
Before yesterday's changes, it was able to process the project as the parser was Acorn-loose (error-tolerant parser). The current parser, Babylon, doesn't seem to have this option.
Maybe JSInspect can have a CLI option of tolerant mode, and when that option is enabled, it'll continue to parse other files and ignore the failing ones (or emit an 'error' event with the parser's error message). What do you think?
Hi, Daniel,
I have just tried this on 2 projects, got identical error block like this:
/usr/local/lib/node_modules/jsinspect/lib/inspector.js:10
class Inspector extends EventEmitter {
^^^^^
SyntaxError: Block-scoped declarations (let, const, function, class) not yet supported outside strict mode
at exports.runInThisContext (vm.js:53:16)
at Module._compile (module.js:374:25)
at Object.Module._extensions..js (module.js:417:10)
at Module.load (module.js:344:32)
at Function.Module._load (module.js:301:12)
at Module.require (module.js:354:17)
at require (internal/module.js:12:17)
at Object. (/usr/local/lib/node_modules/jsinspect/bin/jsinspect:9:17)
at Module._compile (module.js:410:26)
at Object.Module._extensions..js (module.js:417:10)
Not sure if this rule is over-enforced here. I have worked on 2 large existing ES6 projects. ES6 modules and classes are strict by default. Any chance you can take a look at this? Do you have a configuration entry that will suppress this error? Or a workaround?
Thanks!
William
Extract configuration file handling from https://github.com/danielstjules/jsinspect/blob/master/bin/jsinspect#L32 and allow setting it via command line.
Options could be -c
and --config
.
Somewhat related to stefanjudis/grunt-jsinspect#10
console out too much,how to save a file reporter
I plan in creating a Vscode extension that provides in editor feedback for copy pasted code. The only problem i have is that i need to process code with Flowtype type annotations in it. I have been using flowparser package with great success previously since it produces output compatible with most of the other parsers and also works fine with tools such as Recast, Jscodeshift etc.
Wondering if there is an easy way to make the parser configurable via options ? thanks.
Came across it at anandthakker/doiuse#1
Seems rather trivial in comparison to the current default format, to add support for this format style.
Format specified https://www.gnu.org/prep/standards/html_node/Errors.html
Let's see if I can get some additional functionality in master before tagging it
is a good feature ignore the files and directories in .gitignore
file ?
normally .gitignore have files/directories like coverage, lib-cov, results, etc
Hey hey,
just had a look on how to wrap jsinspect with a grunt task.
In my mind, it would be much cleaner (and easier) if jsinspect would expose a binary and additionally a module, which makes it easier for other projects to use.
Usually the binary is only for parsing arguments and handing it over to the module.
E.g. a grunt task could spawn and child process using the binary, but an implemented module would be great.
Please let me know, what you think of this.
In the default reporter when there are 2 instances of a match and the --no-diff flag is on, you can't see where the problem is. Occurrences are not listed.
Hi,
Is there anyway to ignore files based on a path.
I usually have my tests under test directory, it would be great if I can do something like this in .jsinspectrc
{
"ignore": "test/"
}
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.