Code Monkey home page Code Monkey logo

dustmite's Introduction

DustMite

testcodecov

DustMite is a general-purpose data reduction tool.

Its main use is to reduce D programs into minimal examples exhibiting some behavior (usually a compiler bug).

For documentation, see the GitHub wiki: https://github.com/CyberShadow/DustMite/wiki

DustMite was created by Vladimir Panteleev, and is distributed under the Boost Software Licence, version 1.0.

dustmite's People

Contributors

cybershadow avatar ibuclaw avatar jmdavis avatar john-colvin avatar mrmonday avatar nordlow avatar thewilsonator avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

dustmite's Issues

A new switch `--strip-unittests`

This is an enhancement request.

Before the start of dustmite, if we already know the unittest blocks not affect to the test script result, dustmite can strip off unittest blocks and version(unittest) declarations in preparation, similar to --strip-comments switch.

Pros: It can speed up dustmiting process in specific case.

When I tried to use dustmite a failure of minimal script with Phboos code, although the script compilation had not required -unittest switch, it had consume much times to remove unused unittest blocks. I think that usually big projects contain many unittest blocks, so the proposed feature would become useful in many cases.

reduction should attempt to remove function parameters

Many times dustmite stops making progress because a certain function takes arguments that prevents further reductions.

How about attempting to remove ith parameter to a function declared in the working set, and adjusting the callers as needed.

Add --version

Would be nice, there seems to be far more mites floating around than their should be.

Issue with dmd color output

Depending whether color output is on or off in dmd (default is on), there is different text output. E.g.

void main()
{
    int i;
    e = 3;
}

Color on:

sample.d(4): Error: undefined identifier e

Color off:

sample.d(4): Error: undefined identifier `e`

Therefore if you copy the text (from colored output) in the clipboard and then search via dustmite, you will not get any result. I always fall into this trap. You may add this info to your wiki.

(I also created a dmd ticket, to solve this issue, but it was closed as won't fix: https://issues.dlang.org/show_bug.cgi?id=20788)

--obfuscate should obfuscate AFTER reduction (as an option)

it would be nice to option to combine --obfuscate and reduction to make reductions more unique across problems.
Obfuscate would be run on the reduced output only (for speed), and would be deterministic, so different people running into same compiler bug from their own different programs would be more likely to end up with word-for-word identical reductions, make it especially useful to find your bug in bugzilla.
This could even be a default.

Furthermore, it would be nice if dustmite attempted to reduct strings to "" (again for generating as identical as possible outputs)

Information about strategies

Dustmite has multiple different strategies, but no information about which would be better in what circumstances.

Enhancement Request: Ordered Reduction

DustMite would be a lot more useful if there were a way to specify the ordering of the reduction, or if it detected dependencies between files by building an import graph and reduced files that depend on a lot of other files but have no files depending on them first. I imagine the latter would be hard to implement, but the former might not be.

Motivation: I'm trying to reduce a compiler segfault in Plot2kill, a plotting library that depends on gtkD. This is an incredibly slow process. I want to reduce Plot2kill first so that it depends on as little of gtkD as possible, and then reduce gtkD once the first pass through it will likely allow a lot of code to be removed. I'm aware that this can be done by hand or with a hand-written script, but it would be a lot easier if supported directly.

Possible interface: A command line argument called --reduce-first that takes the name of a file or directory to fully reduce before touching any others. If multiple --reduce-first arguments are specified, the first one specified is reduced first, then the second, etc.

Help text not shown

The help text is not shown if dustmite is called with zero arguments. A RangeError is thrown instead (dustmite.d:75).

I think that's because line 54 in dustmite.d is wrong:
if (args.length == 0 || args.length>3)
should likely be
if (args.length == 1 || args.length>3)
(as args[0] is present most of the time and is the program name/path)

Omitting -q in grep can help on Windows.

Hi Vladimir,

I am making my first experiences with dustmite in a hunt to reduce an ICE, on Windows. I have two cents to share on this invaluable tool!

The error looks like this:

---
ERROR: This is a compiler bug.
Please report it via https://issues.dlang.org/enter_bug.cgi
with, preferably, a reduced, reproducible example and the information below.
DustMite (https://github.com/CyberShadow/DustMite/wiki) can help with the reduction.
---
DMD v2.088.0-dirty
predefs   Have_pias_d Have_derelict_gl3 Have_mir_algorithm Have_pascal2d_epcompat Have_derelict_util Have_mir_core DigitalMars Windows CRuntime_Microsoft CppRuntime_Microsoft LittleEndian D_Version2 all D_InlineAsm D_InlineAsm_X86 X86 Win32 assert D_ModuleInfo D_Exceptions D_TypeInfo D_HardFloat
binary    C:\D\dmd2\windows\bin\dmd.exe
version   v2.088.0-dirty
config    C:\D\dmd2\windows\bin\sc.ini
DFLAGS    -IC:\D\dmd2\windows\bin\..\..\src\phobos -IC:\D\dmd2\windows\bin\..\..\src\druntime\import -L/OPT:NOICF
---

object.Error@(0): Access Violation
----------------
0x006526B7
0x0065A151
0x00633651
0x006F6C18
0x0063438B
0x006F6C18
0x0063438B

Following the instructions on the wiki I was using a script like this:

dmd [...]  2>&1 | grep -qF "ERROR: This is a compiler bug"

But for each successful case, this caused the exception message at the end to appear in a popup box that had to be manually closed.

Just now I discovered that simply leaving out the q grep option causes the popups to stay away, and dustmite is happily reducing my code unattended. Output from dustmite is as before, there is no output from grep that leaks through to the console.

Thank you and be well,
Bastiaan.

Protected unittest moved to different file

The fix of issue #60 seems to have caused a regression (or showed up another bug):

test.d:

import sub.sub;

auto pp(int a, int b)
{
    return a^^b;
}

// DustMiteNoRemoveStart
unittest
{
    assert(pp(3,4) == 81);
}
// DustMiteNoRemoveStop

And in sub/sub.d:

module sub.sub;

called with dustmite . "dmd -unittest -main -i -run test.d.

This time, the unittest is moved to sub/sub.d, while test.d is empty.

Reduce false-positive results by replacing code with whitespace?

I believe currently Dustmite deletes code when it does its reduction? But that means we cannot have a test condition like grep -F module.d(145): Error: No property info for 'abcd' and instead have to search for something like: grep -F "Error: No property info for 'abcd' without the line number, which will bring out more false positives during reduction.

If instead Dustmite replaced code it wants to delete with whitespace characters, then the line numbers would be preserved, and we could have less false-positives this way.

What do you think?

Source Reductions Prio List

I'm missing a way to hint dustmite about files it should start reducing first to quickly reduce the time it takes to run the command to be reduced.

In-place reduction

Some environments make it very difficult to make the test command position-independent. For such cases, in-place reduction is desired:

  • Load all data into memory
  • Use given directory as the test directory
  • Still save the current best result to a .result directory
  • Move the .result directory to the initial directory upon completion?

CC @yebblies

Edit: maybe merge this with --reduce-only

DustMiteNoRemoveStart/Stop ignored

I noticed, that dustmite sometimes removes code, which is marked not to be removed. Small testcase:

test.d:

import sub.sub;

// DustMiteNoRemoveStart
unittest
{
}
// DustMiteNoRemoveStop

And in sub/sub.d:

module sub.sub;

called with dustmite . "dmd -main -unittest -run test.d sub/sub.d"

I tried this with several versions (*) of dustmite, including the latest (selfcompiled, linux). According to the dump the correct Entieties are marked as "noRemove", but are removed anyway.

(*) By the way dustmite --version doesn't work, nor does dustmite -V (a Phobos bug?).

Internal test fails

$ dmd @cmd.txt 2>&1 | grep -qF "as parameter to non-global template isEqual(alias T)"
$ echo $?
0
$ dustmite . 'dmd @cmd.txt 2>&1 | grep -qF "as parameter to non-global template isEqual(alias T)"'
None => No
object.Exception@DustMite/dustmite.d(201): Initial test fails
----------------
dustmite() [0x45f3c4]
dustmite() [0x45eef6]
dustmite() [0x45f414]
dustmite() [0x45eef6]
dustmite() [0x45eeb2]
dustmite() [0x45ecff]
/usr/lib/libc.so.6(__libc_start_main+0xf5) [0x7ff8a17cbbc5]
----------------

Ignore .DS_Store file on OS X

Mac's Finder (file browser) automatically adds .DS_Store files to browsed folders. So there may be a .DS_Store file in the folder that dustmite is going to run on. dustmite then errors with:

Suspicious file found: crash2/.DS_Store
You should use a clean copy of the source tree.
If it was your intention to include this file in the file-set to be reduced,
re-run dustmite with the --force option.

Perhaps dustmite could just ignore .DS_Store, or files starting with . in general?

Access is denied

In using dustmite on windows I am getting the following error message consistently (but after a variable number of reductions):

std.file.FileException@std\file.d(507): Attempting to rename file start.reduced.
inprogress to start.reduced: Access is denied.

I have yet to determine the cause of this, but the highly alarming thing is the test.reduced directory disappears.

I recommend introducing a nested scope into safeSave:

void safeSave(string savedir)
{
    auto tempdir = savedir ~ ".inprogress";
    {
        scope(failure) safeRmdirRecurse(tempdir);
        if (exists(tempdir)) safeRmdirRecurse(tempdir);
        save(nullReduction, tempdir);
        if (exists(savedir)) safeRmdirRecurse(savedir);
    }
    rename(tempdir, savedir);
}

with the idea being if rename fails, tempdir should not be removed so that the user will at least have something left to work with.

Somehow when I ran the patched dustmite, it resolved the issue. No idea what's going on here.

Win32: Failed compilation on 2.054, 2.055, 2.056, 2.057

2.054:

dmd dsplit.d dustmite.d
dustmite.d(257): template argument expected following !

After I fix that I get:

dmd dsplit.d dustmite.d
Assertion failure: '!vthis->csym' on line 719 in file 'glue.c'
abnormal program termination

2.055:

dmd dsplit.d dustmite.d
Assertion failure: '!vthis->csym' on line 713 in file 'glue.c'
abnormal program termination

2.056

dmd dsplit.d dustmite.d
Notice: As of Phobos 2.055, std.regexp has been deprecated. It will
 be removed in February 2012. Please use std.regex instead.
Assertion failure: '!vthis->csym' on line 713 in file 'glue.c'

2.057 (beta):

dmd dsplit.d dustmite.d
Notice: As of Phobos 2.055, std.regexp has been deprecated. It will
 be removed in February 2012. Please use std.regex instead.
D:\DMD\dmd2\windows\bin\..\..\src\phobos\std\file.d(3710): Error: f
unction std.file.isDir is deprecated

The phobos bug was recently fixed in dlang/phobos@7774df3 , however even with -d workaround:

dmd -d dsplit.d dustmite.d
Notice: As of Phobos 2.055, std.regexp has been deprecated. It will be removed in February
 2012. Please use std.regex instead.
_error_ 009F3C1C
ty = 35
Assertion failure: '0' on line 145 in file 'mtype.c'

Dustmite generates wrong-code when reducing and then freezes

Found this while reducing a dscanner -inline compilation error(https://github.com/roman-d-boiko/Dscanner).

I've got the test-case and results here (win32):

[email protected]:AndrejMitrovic/Bugs.git

In dscan.test\codegen.d you'll see it reduced a class definition to:

: TrieNode!(K, V)

So the name is missing. There's also a log.txt, and you can reproduce the whole thing again by running run_dustmite.bat (but also do a rdmd runme.d beforehand to remove one of the dialog boxes). There's actually two sets of dialog boxes, one is triggered at the start, and one starts showing up after a few minutes of reduction.

Works with dub

Trying to get DustMit works with dub, but never manage :
Here is the command

[llaine@thinkpad-llaine Workspace] $ cd admin
[llaine@thinkpad-llaine admin] $  dub dustmite --compiler-regex="Internal Error" --bare 'dub build -b release'
Failed to load package in /home/llaine/Workspace/admin/.dub/: The package "name" field is missing or empty.
Destination path must not be a sub directory of the tested package!

OR

[llaine@thinkpad-llaine Workspace] $ dub dustmite admin/ --compiler-regex="Internal Error" 'dub build -b release'
Expected destination path.

any ideas where it might comes from?

module merging

I often end up with code like this after reduction:

// a.d
void foo();

// b.d
import a;
void bar() { foo(); }

where if I manually merge the two files and remove the import, the test still passes. Is there some way dustmite could perform this reduction? It can make a huge difference in big projects

dustmite sometimes can get into infinite loop

On OSX, I was testing this code to see if I could reduce it:

https://issues.dlang.org/show_bug.cgi?id=14587

My script to test looked like this:

#!/bin/sh
dmd app.d
if ./app
then
    exit 1
fi
exit 0

I didn't think to check the result of dmd to return failure if dmd fails to compile. But the result of:

dustmite src ../testit.sh

results in an infinite loop:

############### ITERATION 0 ################
============= Depth 0 =============
[  0.0%] Remove [] => Yes
============= Depth 1 =============
############### ITERATION 1 ################
============= Depth 0 =============
[  nan%] Remove [] => Yes (cached)
============= Depth 1 =============
############### ITERATION 2 ################
============= Depth 0 =============
[  nan%] Remove [] => Yes (cached)
============= Depth 1 =============
############### ITERATION 3 ################
============= Depth 0 =============
[  nan%] Remove [] => Yes (cached)
============= Depth 1 =============
############### ITERATION 4 ################
============= Depth 0 =============
[  nan%] Remove [] => Yes (cached)
============= Depth 1 =============
...

While clearly I didn't do my test right, I don't think dustmite should get into an infinite loop here.

hashing/cache is too slow

Dustmite spends way to much time computing MD5 sums only to find very few cache hits.
I'm currently reducing a dmd bug on vibe.d's source code w/ gdb as test runner (so it's expensive), and dustmite ss still a lot faster w/o cache.
One could try to compute hashes hierarchically, so one wouldn't to rehash a function that is used verbatim.

Please remove or replace non-free files

Hi!
The inclusion of non-free code pieces currently prevents Debian and other distributions from including Dustmite into their repositories.
The problematic files are https://github.com/CyberShadow/DustMite/blob/master/tests/lines/src.dump and https://github.com/CyberShadow/DustMite/blob/master/tests/lines/src/alice30.txt

[3]  Pay a trademark license fee to the Project of 20% of the
     net profits you derive calculated using the method you
     already use to calculate your applicable taxes.  If you
     don't derive profits, no royalty is due.  Royalties are
     payable to "Project Gutenberg Association / Illinois
     Benedictine College" within the 60 days following each
     date you prepare (or were legally required to prepare)
your annual (or equivalent periodic) tax return.

Since this imposes restrictions on how the software is used and prevents us from bundling it with Debian (as then people can not sell copies of Debian anymore), we currently can not include Dustmite (or need to remove the problematic files).

See https://www.debian.org/social_contract#guidelines for what makes free software free.

It would be awesome if you could replace the text with a different one under a completely free license.
Thanks!

DMD 2.072.1 throws linker error

dustmite.o:dustmite.d:_D8dustmite12__ModuleInfoZ: error: undefined reference to '_D8splitter12__ModuleInfoZ'
dustmite.o:dustmite.d:function _Dmain: error: undefined reference to '_D8splitter13splitterNamesyAAa'
dustmite.o:dustmite.d:function _Dmain: error: undefined reference to '_D8splitter13splitterNamesyAAa'
dustmite.o:dustmite.d:function _Dmain: error: undefined reference to '_D8splitter8optimizeFC8splitter6EntityZv'
dustmite.o:dustmite.d:function _D8dustmite4mainFAAyaZ14parseSplitRuleMFNaNfAyaZS8splitter9ParseRule: error: undefined reference to '_D8splitter13splitterNamesyAAa'
dustmite.o:dustmite.d:function _D8dustmite4mainFAAyaZ14parseSplitRuleMFNaNfAyaZS8splitter9ParseRule: error: undefined reference to '_D8splitter13splitterNamesyAAa'
dustmite.o:dustmite.d:function _D8dustmite4mainFAAyaZ9__lambda5MFZv: error: undefined reference to '_D8splitter9loadFilesFKAyaS8splitter12ParseOptionsZC8splitter6Entity'
dustmite.o:dustmite.d:function _D8dustmite9obfuscateFbZv: error: undefined reference to '_D8splitter88__T10parseSplitS693std10functional47__T3notS36_D3std5ascii10isAlphaNumFNaNbNiNfwZbZ3notZ10parseSplitFAyaZAC8splitter6Entity'
dustmite.o:dustmite.d:function _D8dustmite14applyReductionFKS8dustmite9ReductionZv: error: undefined reference to '_D8splitter6Entity7__ClassZ'
dustmite.o:dustmite.d:function _D8dustmite14applyReductionFKS8dustmite9ReductionZv: error: undefined reference to '_D8splitter6Entity6__ctorMFAyaAC8splitter6EntityAyaZC8splitter6Entity'
dustmite.o:dustmite.d:function _D8dustmite14applyReductionFKS8dustmite9ReductionZv: error: undefined reference to '_D8splitter8optimizeFC8splitter6EntityZv'
dustmite.o:dustmite.d:function _D8dustmite20applyReductionToPathFAyaS8dustmite9ReductionZAya: error: undefined reference to '_D8splitter88__T10parseSplitS693std10functional47__T3notS36_D3std5ascii10isAlphaNumFNaNbNiNfwZbZ3notZ10parseSplitFAyaZAC8splitter6Entity'
dustmite.o:dustmite.d:_D29TypeInfo_S8splitter9ParseRule6__initZ: error: undefined reference to '_D8splitter9ParseRule9__xtoHashFNbNeKxS8splitter9ParseRuleZm'
dustmite.o:dustmite.d:_D29TypeInfo_S8splitter9ParseRule6__initZ: error: undefined reference to '_D8splitter9ParseRule11__xopEqualsFKxS8splitter9ParseRuleKxS8splitter9ParseRuleZb'
dustmite.o:dustmite.d:_D27TypeInfo_xC8splitter6Entity6__initZ: error: undefined reference to '_D8splitter6Entity7__ClassZ'
dustmite.o:dustmite.d:_D27TypeInfo_AC8splitter6Entity6__initZ: error: undefined reference to '_D8splitter6Entity7__ClassZ'
collect2: error: ld returned 1 exit status
Error: linker exited with status 1

seems more like a bug in DMD?

Detect test scripts not marked executable

A common mistake for people infrequently using DustMite is to try using a shell script as the test command and forgetting to mark it as executable (Unix).

It happens frequently enough that special-casing this seems useful (detecting whether the test command is a file and emitting a specialised error message instead of the default "initial test failed").

Feature Request: timeout for test command

When reducing a bug, I noticed DustMite might produce code that deadlocks.
Even though it's possible to implement a timeout for a command in a bash script, these solutions are usually complicated and bug prone.

As a solution to this problem DustMite could accept an optional timeout argument. If the test command then hadn't finished in the specified time, DustMite would kill the test command and consider this test as failed.

Offer separate examples for testing segfaults.

I spent some time figuring this out, but apparently processes (on Linux) won't output anything to stdout when they segfault. The obvious exception is that they will if stdout is a terminal device, which allows you to usually /see/ the output. But it won't happen in scripts or pipes because the output buffers won't be flushed if the program segfaults. (See this article for details.) This means that the 2>&1 | grep -qF "blah" idiom doesn't work for catching segfaults. I had to write my test case like this instead:

testit.sh:

dmd main.d ...args...
if [ $? -eq 139 ]; then
        exit 0
else 
        exit 1
fi

Segfaulting programs do seem to return an error code of 139, so it can be caught that way. Once I discovered this, I was able to make DustMite do its dance and summon a reduced test case.

[question] How to use DustMiteNoRemoveStart/Stop?

struct SList
{
    // ...
    struct Range
        // ...
        // DustMiteNoRemoveStart
        void popFront()
        {
            // DustMiteNoRemoveStart
            assert(!empty, "SList.Range.popFront: Range is empty");
            _head = _head._next;
            // DustMiteNoRemoveStop
        }
        // DustMiteNoRemoveStop
        // ...
    }
    // ...
}

How do I prevent DustMite from removing code from popFront? What I've tried doesn't seem to work.

Fuzzer

DustMite's splitter should allow nice fuzzing, e.g. by replacing a random node with another random node from elsewhere in the code.

C++ support

Do you have some tips for creating a C++ parser for DustMite?
I could really use a usable reduction tool for a clang ICE.

Stable branch?

Hello.
Seems there's only master branch, but this tool is widely used.
Maybe there's a reason to create a stable branch, even if it will not differ from master most of the time?

Assert expressions shouldn't be reduced

Dustmite incorrectly reduces assert expressions in code such as

module faulty;

auto identity(T)(T x)
{
    return x;
}

auto square(T)(T x)
{
    const y = identity(x);
    if (x == 4)
    {
        return y*y - 1;         // bug
    }
    else
    {
        return y*y;
    }
}

@safe pure nothrow unittest
{
    assert(square(2) == 4);     // ok
    assert(square(3) == 9);     // ok
    assert(square(4) == 16, "Some specific failure in " ~ square.stringof); // triggers bug
}

into something like

auto square(T)(T x)
{
}

unittest
{
    assert(4 == 16, "Some specific failure in " ~ square.stringof); // triggers bug
}

when called to grep for failure string

Some specific failure in square(T)(T x)

Is this easy to fix?

Invalid UTF-8 sequence on recent master

The following file:

unittest
{
    auto t = "DåbDÅBdÅb";
}

gets dustmite to spit:

std.utf.UTFException@/home/dmitry/dmd2/linux/bin64/../../src/phobos/std/utf.d(1113): Invalid UTF-8 sequence (at index 1)
----------------
./dustmite(pure @trusted dchar std.utf.decode!(immutable(char)[]).decode(ref immutable(char)[], ref ulong)+0xc2) [0x4b25be]
./dustmite(pure @property @safe dchar std.array.front!(immutable(char)).front(immutable(char)[])+0x25) [0x4f9f3d]
./dustmite(pure @safe bool std.algorithm.startsWith!("a == b", immutable(char)[], char).startsWith(immutable(char)[], char)+0x3c) [0x4ea0d8]
...

regardless of test command.

Works fine with 9abec53
but not with latest master. So apparently the latest fix did something with decoding...

Removing unused imports

Improvement proposal

Right now is seems that the only way to know which imports are unused is to try manually to remove them. I think making it a known feature by only trying to change import lines could be a valuable improvement.

better heuristics to propose candidate reducations (to improve reduction speed)

during reduction, there are lots of obvious cases that take a while (in terms of iteratiosn) to remove, eg:
symbols not referenced.

these cases could be detected (using either a d parser or even using purely lexical analysis / word tokenization) to prioritize these reductions to be attempted before others

eg:
in following case, prioritize removing
int unused_variable=0; or void unused_function(){....} to be attempted before the other ones

void fun(){
bar;
}

void bar();

int unused_variable=0;

void unused_function(){....}

likewise with unused modules, etc

feature request: file merging as simplification

It'd be nice if dustmite simplifies imports to get a single file in the end (or reduce #files as much as possible), as follows:

example:
command:
dustmite dir 'rdmd --force fun1.d 2>&1|grep -F "Error: found"'
input:

//cat fun1.d
import std.stdio;
import fun2;

void main(){
    fun;

}

//cat fun2.d
import std.stdio;
void fun(){
aaa
}
------

output:

//cat fun1.d
import fun2;

//cat fun2.d
void (){
aaa
}

desired output:

//cat fun1.d
void (){
aaa
}

DustMite should attempt to autodetect error patterns

It's often a little difficult to write the correct matcher for an error and it would be nice if dustmite automatically recognized the most common errors.

https://github.com/CyberShadow/DustMite/wiki/Useful-test-scripts

So for example if I run dustmite src dmd foo.d and get an assertion in dmd then dustmite should reduce matching a segfault of dmd and grepping for the assertion.
Likewise a compiler segfault without output should be matched as such.
And for crashing applications (dustmite src rdmd foo) it should attempt to extract the function where an error occurred.

dustmite improvements

quoting from newsgroup post from you:

It should work for simple cases (when the program can compile after all files' contents is moved into one single file). The obstacles with making it more flexible are that 1) to merge every pair of files, it would have to try N*(N-1) combinations; 2) it doesn't know which file is the "main" program file (the file which must be present for the test script to succeed), if there is one.

A)
for 1, you could attempt to only merge a into b, where b is the only module that imports a.

B)
For 2, you can just provide a flag to dustmite:
dustmite -main=main.d ....
if no flag is specified, you may even just not attempt to do any file merging.
As a user of dustmite, i'd rather take the pain to specify more things to help dustmite than to have dustmite do a suboptimal reduction which i'll have to reduce. Specifying main file is very easy. There could be more flags to help reduction.

C)
in my experience the test.reduced files are left identical to source during reduction and only when process completes it shows the reduced files. Is there a way to have the reduced files being indeed the current best reduced files during reduction process ?
I maybe be wrong about that one, maybe it was due to case D below.

D)
i had a very long running dustmite which eventually consumed all my memory (i have 16GB) ; i had to kill the process. Upon killing the test.reduced didn't reduce anything despite going through many iterations. When i kill dustmite at any time in the middle, i would expect to have a well defined state where the test.reduced do satisfy the dustmite condition. Is that supposed to be the case? There are easy ways to make this fail safe (eg saving a tar file of last known good test.reduced)

E)
is it 100% safe to use dustmite on a source tree (ie it will not affect it, even the .o files)? wasn't 100% sure. right now i always made a copy before doing any reduction, in part because it contained some .o files that dustmite complained about.

F)
can it also (optionally) reduce phobos modules? right now it seems to leave them untouched. this would help with debugging phobos issues.

G)
usuall i have to write:
-Ipwd in command line flag passed to dustmite to make sure it will refer to the correct include directory in the test.reduced files. This is a bit restrictive (eg restricts from where you can run the command). Can we have instead:
dustmite -dir_reduced=my_dir_reduced -before=dir_base=my_dir_source -after=dir_base=my_dir_reduced '-I$dir_base $dir_base/main.d'

Generate and Use Coverage at each run

There is already

  --coverage DIR     Load .lst files corresponding to source files from DIR

. Why doesn't dub dustmite generate new coverage information "at each call" to dub run and use that via loadCoverage when deciding which code to remove prior to subsequent call?!!! Wouldn't this enable amazingly faster reduction speed for D especially in the reduction of dependencies.

I guess most of this logic should be implemented in dub.

Ping, @CyberShadow.

See https://github.com/CyberShadow/DustMite/blob/master/dustmite.d#L2405

Update dustmite in tools repo

For the upcoming 2.067 release it would be nice to also update dustmite, because I just hit a bug (ignored SIGINT) that is already fixed here ;).

multicore option

Is it possible to make dustmite use multi-core when available? For some projects, a lot of time ends up being spent on only a single core (link, test execution), and it seems like one could get a significant speedup if we added a parameter to specify number of concurrent processes to run.

It seems like a lot of time is spent trying out modifications that don't result in the desired output, and is then discarded. Modification attempts like this could be run in N threads, with a join after each iteration, where you could see which were successful, and then start all threads again from the new starting-point..

Doable?

Specify file list

A --reduce-only switch would be handy as an easier-to-use alternative to the current ways of reducing only a few files.

Alternatively, add a wiki article which makes this task easier.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.