Add support for the predicates:

• "is/is not... strictly positive"
• "is/is not... smaller than"

TTC: 1 week

Of the method org.apache.commons.math3.genetics.CycleCrossover.crossover(Chromosome first, Chromosome second), the comment if the chromosomes are not an instance of AbstractListChromosome is translated as (args[0] instanceof AbstractListChromosome) == false. Instead, the expected output is (args[0] instanceof AbstractListChromosome) == false || (args[1] instanceof AbstractListChromosome) == false.

On randoop branch, when run with the enum org.apache.commons.math3.exception.util.LocalizedFormats as the target class, get the following

Exception in thread "main" java.lang.NullPointerException
        at org.toradocu.translator.JavaElementsCollector.collect(JavaElementsCollector.java:48)
        at org.toradocu.translator.Matcher.subjectMatch(Matcher.java:36)
        at org.toradocu.translator.ConditionTranslator.translatePropositions(ConditionTranslator.java:201)
        at org.toradocu.translator.ConditionTranslator.processTag(ConditionTranslator.java:408)
        at org.toradocu.translator.ConditionTranslator.translate(ConditionTranslator.java:37)
        at org.toradocu.Toradocu.main(Toradocu.java:145)

Happens also for:
org.apache.commons.math3.ode.sampling.StepNormalizerBounds (also an enum)

Toradocu currently do not support subjects in comments implicitly referring to more than one Java code element.

To see an example of this behavior look at the test case org.toradocu.PrecisionRecallCommonsMath3.cycleCrossoverTest (method crossover(Chromosome first, Chromosome second)). The comment is "if the chromosomes are not an instance of AbstractListChromosome", the extracted subject is "chromosomes" and it is correctly translated to the two parameters of the method. The method org.toradocu.translator.ConditionTranslator.translatePropositions requires an explicit clue (like the presence of the word "both") to consider more than one code element matching the subject.
Currently Toradocu considers only one of the two subjects.

Test case: org.toradocu.PrecisionRecallCommonsMath3.cycleCrossoverTest.crossover(Chromosome first, Chromosome second)
Comment: "if the chromosomes are not an instance of AbstractListChromosome"
Expected Output: (args[0] instanceof AbstractListChromosome) == false || (args[1] instanceof AbstractListChromosome) == false
Actual Output: (args[0] instanceof AbstractListChromosome) == false

Unable to procces conjunction of conditions

What was the expected outcome

For the giving throws tag,

@throws IllegalArgumentException if myvar1 is less than 1 and myvar2 is less than 3

The outcome should be: "condition": "args[0] <1 && args[1]<3"

What happened instead?

Toradocu wasn't able to create the condition properly

Actual condition: "args[0]<1"

AJ already identified where's the problem:

Text: "if myvar1 is less than 0 and myvar2 is less than 3"
Expected: "myvar1<0 && myvar2<3"
Actual: "myvar1<0"
The original text gets preprocessed to "if myvar1 is INEQUALITY_0 and myvar2 is INEQUALITY_1".
The problem is with the semantic graph.

Semantic graph:
-> INEQUALITY_0/NN (root)
  -> countMap/NN (nsubj)
  -> is/VBZ (cop)
  -> and/CC (cc)
  -> size/NN (conj:and)
  -> INEQUALITY_1/JJ (acl:relcl)
    -> is/VBZ (cop)

Only "myvar1" has a subject relation. As a result, the SentenceParser does not attempt to create a proposition with "myvar2" as the subject.

Recreate the problem:

Create a class with the @throws tag with two conditions joined by a conjunction

System details

I'm using the devel branch of the project with the .jar that gradlew shadowJar produces.

Toradocu generates the following condition method

// double org.apache.commons.math3.analysis.differentiation.DerivativeStructure.getPartialDerivative(int[] orders)
    public static boolean m12_p0(org.apache.commons.math3.analysis.differentiation.DerivativeStructure receiver12_p0, int[] orders) {
        // @param orders derivation orders with respect to each variable (if all orders are 0, the value is returned) ==> args[0]==0
        return orders == 0;
    }

for org.apache.commons.math3.analysis.differentiation.DerivativeStructure.getPartialDerivative

The @param comment indicates that the condition should be orders[i]==0 for all i, but the condition is testing if the whole array is zero.

This relates somewhat to issue #65 where Toradocu is also generating a condition where the types are not compatible.

Correct translation of arithmetic expressions made with symbols

What was the expected outcome

For the giving throws tag,

@throws IllegalArgumentException if b>1

The outcome should be: "condition": "args[0]>1"

What happened instead?

The outcome of the program is "condition": " "

It happens always. Even if we change the spaces between the number, the variable and the comparator or the length of the variable.

Recreate the problem

Create a class with the @throws tag with the following characteristics:

@param {varname}
@throws {Exception} if {varname} {arithmetic comparator} {number}

Then, create the method as specified in the javadoc.

Example:

/**
     *
     * @param b
     * @throws IllegalArgumentException if b<1
      */   
    public int testMethod1  (int b) throws IllegalArgumentException{       
        if(b<1) throw new IllegalArgumentException();
               return b;
    }

System details

I'm using the devel branch of the project with the .jar that gradlew shadowJar produces.

Those are the .java, the output in .json and the log of the console whne I execute Toradocu
Files.tar.gz

Toradocu currently ignores {@code} tags that should be used to match Java code elements during the condition translation phase.

Example.
Comment: if the number of sample points {@code n} is negative.
Expected condition: args[3]<0.
Actual condition: args[0]<0.

Solution.
Modify the matching phase to take advantage of {@code} tags or parameter descriptions in @param comments. In this way n can be directly matched with args[3], instead of being wrongly translated as args[0].

Toradocu currently cannot translate properly the keyword this - i.e. the Java reference to the current object.

To see an example of this behaviour, take a look at the project “FreeCol”: https://github.com/fenyo1/FreeCol.git

In the class net.sf.freecol.common.model.Player the method setStance takes a Player object as first
argument and must throw an exception “if player is null or this”. Toradocu correctly identifies the player as the subject and produces the condition args[0]==null , but the complete version should be args[0]==null || args[0]==this.

To reproduce this example, you can execute the following commands:

• FreeCol is an ant project, so you can build it by executing the command ant from the root folder, where build.xml is located. This will create the FreeCol.jar
  (To run the command, ant must be installed in your system; ant is also integrated in the Eclipse IDE, if you prefer)

• Now that FreeCol.jar has been created you can compile the test suite:

javac -cp FreeCol.jar:test/lib/junit.jar:src:test/src test/src/net/sf/freecol/common/model/PlayerTest.java

• And then, run it; you should get OK (5 tests):

 java -cp test/lib/junit.jar:test/src:src:FreeCol.jar org.junit.runner.JUnitCore net.sf.freecol.common.model.PlayerTest

• In the end, run Toradocu (specify correctly the path to your own .jar):

java -jar toradocu-1.0-all.jar \
--target-class net.sf.freecol.common.model.Player \
--test-class net.sf.freecol.common.model.PlayerTest \
--source-dir src \
--class-dir build \
--aspects-output-dir aspects

You can see the relevant part of the output also in the PDF file attached: output_setStance.pdf

Comment: "if player is null or this"
Expected Output: (args[0]==null || args[0]==this)
Actual Output: args[0]==null

Toradocu is generating Randoop specifications that don't use Class.forName() format for member class types when describing methods/constructors.

In particular, the parameter type of the method org.apache.commons.collections4.comparators.FixedOrderComparator.setUnknownObjectBehavior(org.apache.commons.collections4.comparators.FixedOrderComparator.UnknownObjectBehavior)
is given incorrectly as
org.apache.commons.collections4.comparators.FixedOrderComparator.UnknownObjectBehavior.
and instead needs to be
org.apache.commons.collections4.comparators.FixedOrderComparator$UnknownObjectBehavior.

There is code that translates the declaring class of a method, but the parameter types are not modified.

What was the expected outcome

For the giving throws tag,

@throws IllegalArgumentException if myvar and myvar2 are negative/positive/true/false/null

The outcome should be:

"comment": "if myvar and myvar2 are negative",
"condition": "args[0]<0 && args[1]<0"

"comment": "if myvar and myvar2 are positive",
"condition": "args[0]>0 && args[1]>0"

"comment": "if myvar and myvar2 is negative",
"condition": ""

"comment": "if myvar and myvar2 is positive",
"condition": ""

"comment": "if myvar and myvar2 is true",
"condition": ""

"comment": "if myvar and myvar2 is false",
"condition": ""

"comment": "if myvar and myvar2 is null",
"condition": ""

What happened instead?

"comment": "if myvar and myvar2 are negative",
"condition": ""

"comment": "if myvar and myvar2 are positive",
"condition": ""

"comment": "if myvar and myvar2 is negative",
"condition": "args[0]<0 && args[1]<0"

"comment": "if myvar and myvar2 is positive",
"condition": "args[0]>0 && args[1]>0"

"comment": "if myvar and myvar2 is true",
"condition": "args[0]==true && args[1]==true"

"comment": "if myvar and myvar2 is false",
"condition": "args[0]==false && args[1]==false"

"comment": "if myvar and myvar2 is null",
"condition": "args[0]==null && args[1]==null"

The type of the variables in the true/false cases, is boolean, in the null case, is Integer, and in the positive/negative, just int.

I'm not a native, so, maybe my English is not good enough to identify if those cases are right or wrong. But I think that, if I'm correct, the program is not able to differentiate between the singular and the plural in a sentence.

But it's kind of confusing, because with the null case, Toradocu creates both conditions, with plural and singular.

System details

I'm using the devel branch, recently pulled and rebuilt.

Toradocu generates method/constructor specifications using args[<index>] pattern to refer to parameters.
This is due to an implementation detail that should not influence the generated specifications.
Specifications should be generated using the real parameter names, to be more readable and have more sense for developers.

Currently, in Toradocu we need to convert args[<index>] pattern to real names to make specifications compatible with other tools like Randoop. Using real parameter names would simplify the export of generated specifications to format not using the unconventional Toradocu format.

This relates somewhat to issue #67 (because it deals with array length) and issue #56 (because the entity is ambiguous).

For Apache Commons Math DividedDifferenceInterpolator, have a couple of methods where get condition methods like this

    // double[] org.apache.commons.math3.analysis.interpolation.DividedDifferenceInterpolator.computeDividedDifference(double[] x,double[] y)
    public static boolean m1_t1(org.apache.commons.math3.analysis.interpolation.DividedDifferenceInterpolator receiver1_t1, double[] x, double[] y) {
        // @throws org.apache.commons.math3.exception.NumberIsTooSmallException if the number of points is less than 2. ==> args[0]<2
        return x < 2;
    }

I haven't looked at the class, but I think the parameters are a parallel array of "points" and so the condition would be expressed on the length of either array.

Program ends with NullPointerException at FieldRotation Method

This is the command I run:

#!/bin/bash

FOLDERDIR=/home/sergio/Repositories/toradocu
PACKAGEANDCLASS=org.apache.commons.math3.geometry.euclidean.threed.FieldRotation

rm /home/sergio/Toradocu/CommonsMath3/$PACKAGEANDCLASS.json

/usr/bin/java -jar /home/sergio/Repositories/toradocu/build/libs/toradocu-1.0-all.jar \
--target-class $PACKAGEANDCLASS \
--source-dir $FOLDERDIR/src/test/resources/src/commons-math3-3.6.1-src/commons-math3-3.6.1-src/src/m$
--class-dir $FOLDERDIR/src/test/resources/bin/commons-math3-3.6.1.jar \
--oracle-generation false \
--condition-translator-output /home/sergio/Toradocu/CommonsMath3/$PACKAGEANDCLASS.json \
--debug

This is the log: (I skip the part where Toradocu identifies the methods with documentation due to its length)

done [0.7 sec].
[main] TRACE org.toradocu.translator.ConditionTranslator - Translated proposition (axis norm, is zero) as: args[0]==0
[main] TRACE org.toradocu.translator.ConditionTranslator - Identifying propositions from: "if the axis norm is zero" in FieldRotation(org.apache.commons.math3.geometry.euclidean.threed.FieldVector3D axis,org.apache.commons.math3.RealFieldElement angle,org.apache.commons.math3.geometry.euclidean.threed.RotationConvention convention)
[main] TRACE org.toradocu.translator.ConditionTranslator - Translated proposition (axis norm, is zero) as: args[0]==0
[main] TRACE org.toradocu.translator.ConditionTranslator - Identifying propositions from: "if the matrix is not a 3X3 matrix, or if it cannot be transformed into an orthogonal matrix with the given threshold, or if the determinant of the resulting orthogonal matrix is negative" in FieldRotation(org.apache.commons.math3.RealFieldElement[][] m,double threshold)
[main] ERROR org.toradocu.translator.Matcher - Could not load method/constructor from DocumentedMethod org.apache.commons.math3.geometry.euclidean.threed.FieldRotation.FieldRotation(org.apache.commons.math3.RealFieldElement[][] m,double threshold)
Exception in thread "main" java.lang.NullPointerException
    at org.toradocu.translator.Matcher.extractCodeElements(Matcher.java:251)
    at org.toradocu.translator.Matcher.subjectMatch(Matcher.java:46)
    at org.toradocu.translator.ConditionTranslator.translatePropositions(ConditionTranslator.java:198)
    at org.toradocu.translator.ConditionTranslator.translate(ConditionTranslator.java:61)
    at org.toradocu.Toradocu.main(Toradocu.java:132)

The class I run is in org.apache.commons.math3.geometry.euclidean.threed.FieldRotation

Steps to reproduce:

1. Download the zip package of the latest Toradocu version from here.
2. Build Toradocu: ./gradlew build.

Expected result:
Build completes correctly.

Actual result:
Build fails with the error ln: .git/hooks/pre-commit: No such file or directory.

Additional information:
The downloaded zip doesn't contain any .git folder. In the build file we try anyway to install the pre-commit hook. We should not try to install the pre-commit hook if there is no .git directory (i.e., the current directory is not a git project).

Toradocu produces wrong translations for comments that are inherited in overriding methods.
Consider the method org.apache.commons.math3.complex.Complex#divide(Complex).
Toradocu extracts the following tag:

{
  "exception": {
    "qualifiedName": "org.apache.commons.math3.exception.MathArithmeticException",
    "name": "MathArithmeticException",
    "isArray": false
  },
  "codeTags": [
    "a"
  ],
  "comment": "if a is zero",
  "kind": "THROWS",
  "condition": "target.add()==0 || target.abs()==0 || target.tan()==0"
}

The translation is wrong. The problem is due to the fact that the comment written by developers if a is zero has a meaning when applied to the supertype method. The supertype method is overridden and the parameter, originally named a in the supertype, is renamed to divisor in the subtype. Since Toradocu uses the parameter list of the method in the subtype, there is no match between a and divisor. As a consequence, the translated condition is wrong.

Feature request: instanceof

I want to add the feature of the comparator instanceof into the rest of the comparators.

An example of its behavior can be something like this:

With the throws tag:
" @throws IllegalArgumentException if myvar is not an instance of BinaryMutation "

the output for the condition of Toradocu could be like this:

condition: "args[0] instanceof BinaryMutation"

Running Toradocu on org.apache.commons.collections4.bidimap.TreeBidiMap with --export-conditions (on randoop branch) get

    // void org.apache.commons.collections4.bidimap.TreeBidiMap.checkNonNullComparable(java.lang.Object o,org.apache.commons.collections4.bidimap.TreeBidiMap.DataElement dataElement)
    public static boolean m51_t0(org.apache.commons.collections4.bidimap.TreeBidiMap target, java.lang.Object o, org.apache.commons.collections4.bidimap.TreeBidiMap.DataElement dataElement) {
        // @throws java.lang.NullPointerException if o is null ==> args[0]==null
        return o == null;
    }

where checkNonNullComparable() is a private method of the class.

Though, the condition class doesn't compile because org.apache.commons.collections4.bidimap.TreeBidiMap.DataElement is package private.

Running the randoop branch on org.apache.commons.collections4.IterableUtils yields the following errors:

[main] WARN org.toradocu.translator.SentenceParser - Unable to identify subjects in "the Iterable to search .".
Exception in thread "main" java.lang.NullPointerException
        at org.toradocu.translator.JavaElementsCollector.collect(JavaElementsCollector.java:35)
        at org.toradocu.translator.Matcher.subjectMatch(Matcher.java:36)
        at org.toradocu.translator.ConditionTranslator.translatePropositions(ConditionTranslator.java:203)
        at org.toradocu.translator.ConditionTranslator.processTag(ConditionTranslator.java:410)
        at org.toradocu.translator.ConditionTranslator.translate(ConditionTranslator.java:39)
        at org.toradocu.Toradocu.main(Toradocu.java:145)

Unable to process comparison between variables

What was the expected outcome

for the giving @throws tag,

@throws IllegalArgumentException if myvar is not equal to myvares (Both boolean variables)

The outcome should be: " condition:"args[0] != args[1]" "

What happened instead

The .json condition's part was empty.

The log at the execution of the program is:

[main] TRACE org.toradocu.translator.ConditionTranslator - Identifying propositions from: "if myvar is not equal to myvares" in TestMethod6(boolean myvar,boolean myvares)
[main] TRACE org.toradocu.translator.ConditionTranslator - Failed predicate translation for: (myvar, not (is myvares))

Recreate the problem:

Create a class with the @throws tag with a comparison between two variables

System details

I'm using the last developer version of the program.

NullPointerException at JavaElementsCollector

What was the expected outcome

Given the arguments for toradocu

--source-dir /home/sergio/Repositories/toradocu-dev/src/test/resources/src/guava-19.0-sources/  \
--class-dir /home/sergio/Repositories/toradocu-dev/bin/guava-19.0.jar \
--oracle-generation false \
--condition-translator-output /home/sergio/Toradocu/Ordering.json \
--debug

The expected outcome was the .json file with the analysis for that class.

What happened instead

Toradocu broke at JavaElementsCollector with the following output

Exception in thread "main" java.lang.NullPointerException
at org.toradocu.translator.JavaElementsCollector.collect(JavaElementsCollector.java:35)
at org.toradocu.translator.Matcher.subjectMatch(Matcher.java:36)
at org.toradocu.translator.ConditionTranslator.translatePropositions(ConditionTranslator.java:203)
at org.toradocu.translator.ConditionTranslator.processTag(ConditionTranslator.java:410)
at org.toradocu.translator.ConditionTranslator.translate(ConditionTranslator.java:37)
at org.toradocu.Toradocu.main(Toradocu.java:142)

What to do to reproduce the error

Have the toradocu repository up to date, then, execute toradocu at the class Ordering.java, from the Guava library, at com/common/collect/, as specified here:

--source-dir /home/sergio/Repositories/toradocu-dev/src/test/resources/src/guava-19.0-sources/  \
--class-dir /home/sergio/Repositories/toradocu-dev/bin/guava-19.0.jar \
--oracle-generation false \
--condition-translator-output /home/sergio/Toradocu/Ordering.json \
--debug

Change the route as needed.

For the classes

package pkg;

public class SuperClass {
  /**
   * @param count  the object count, must be positive
   */
  public void methodWithOverride(int count) {}

  /**
  * @param count  the something count, must be positive
  */
  public void methodWithoutOverride(int count) {}
}

and

package pkg;

public class SubClass extends SuperClass {
  /**
  * @param count the count of things, must be less than 3
  */
  @Override
  public void methodWithOverride(int count) {
    super.methodWithOverride(count);
  }
}

Toradocu generates the condition count > 0 for SuperClass.methodWithOverride(count), and
count < 3 for SubClass.methodWithOverride(count).
This, technically, is correct in general since the overriding method could change the behavior, even it it would be bad code. But, in this case the overriding method calls super.methodWithOverride(count) where count must be positive, and so the condition should be count > 0 && count < 3.

Unable to process a class without package

What was the expected outcome

For the giving throws tag in the class without package,

@throws IllegalArgumentException if text is null

The outcome should be: "condition": "args[0] == null"

What happened instead?

Toradocu wasn't able to read the throws tag in the class

The log was:

[main] DEBUG org.toradocu.Toradocu - Methods with Javadoc documentation found in source code: []

Recreate the problem:

Create a class with the @throws tag with the characteristics that you desire, but it has to be without a package

System details

I'm using the devel branch of the project with the .jar that gradlew shadowJar produces.

The script that I'm executing is:

!/bin/bash

CLASS=inCorrectFormat
FOLDERDIR=/home/sergio/Repositories/toradocu-testing

rm /home/sergio/Toradocu/$CLASS.json

/usr/bin/java -jar /home/sergio/Repositories/toradocu-dev/build/libs/toradocu-dev-1.0-devel-all.jar
--target-class $CLASS
--source-dir $FOLDERDIR/src
--class-dir $FOLDERDIR/bin
--oracle-generation false
--condition-translator-output /home/sergio/Toradocu/$CLASS.json
--debug

For Apache Commons Math class StepFunction Toradocu generates the condition method:

    // org.apache.commons.math3.analysis.function.StepFunction.StepFunction(double[] x,double[] y)
    public static boolean m0_t3(org.apache.commons.math3.analysis.function.StepFunction receiver0_t3, double[] x, double[] y) {
        // @throws org.apache.commons.math3.exception.DimensionMismatchException if x and y do not have the same length. ==> (args[0].length) == false && (args[1].length) == false
        return (x.length) == false && (y.length) == false;
    }

The actual condition should be x.length == y.length.

Other cases:

    // org.apache.commons.math3.analysis.integration.gauss.GaussIntegrator.GaussIntegrator(double[] points,double[] weights)
    public static boolean m0_t1(org.apache.commons.math3.analysis.integration.gauss.GaussIntegrator receiver0_t1, double[] points, double[] weights) {
        // @throws org.apache.commons.math3.exception.DimensionMismatchException if points and weights don't have the same length ==> (args[0].length) == false && (args[1].length) == false
        return (points.length) == false && (weights.length) == false;
    }

IllegalArgumentException while reading Javadoc on class RealVector from CommonsMath3

I was running Toradocu at this class, when suddenly the execution stopped with this log:

[main] TRACE org.toradocu.translator.ConditionTranslator - Identifying propositions from: "if the index is not valid." in getEntry(int index)
done [0.4 sec].
[main] DEBUG org.toradocu.translator.StanfordParser - Input sentence: the index is not valid.
Semantic Graph:
-> valid/JJ (root)
  -> index/NN (nsubj)
    -> the/DT (det)
  -> is/VBZ (cop)
  -> not/RB (neg)
  -> ./. (punct)

Exception in thread "main" java.lang.IllegalArgumentException: Method 'unmodifiableRealVector' has 1 parameters. Expected 0
    at org.toradocu.translator.StaticMethodCodeElement.<init>(StaticMethodCodeElement.java:38)
    at org.toradocu.translator.Matcher.extractCodeElements(Matcher.java:310)
    at org.toradocu.translator.Matcher.subjectMatch(Matcher.java:49)
    at org.toradocu.translator.ConditionTranslator.translatePropositions(ConditionTranslator.java:200)
    at org.toradocu.translator.ConditionTranslator.translate(ConditionTranslator.java:61)
    at org.toradocu.Toradocu.main(Toradocu.java:132)

The script I'm running is:

#!/bin/bash

FOLDERDIR=/home/sergio/Repositories/toradocu
PACKAGEANDCLASS=org.apache.commons.math3.linear.RealVector

rm /home/sergio/Toradocu/CommonsMath3/$PACKAGEANDCLASS.json

/usr/bin/java -jar /home/sergio/Repositories/toradocu/build/libs/toradocu-1.0-all.jar \
--target-class $PACKAGEANDCLASS \
--source-dir $FOLDERDIR/src/test/resources/src/commons-math3-3.6.1-src/src/main/java \
--class-dir $FOLDERDIR/src/test/resources/bin/commons-math3-3.6.1.jar \
--oracle-generation false \
--condition-translator-output /home/sergio/Toradocu/CommonsMath3/$PACKAGEANDCLASS.json \
--debug

I was debugging this with Eclipse, and it seems that at [Matcher, 301], when we do the
if (classMethod.getParameterCount() == 0)
it does not pass, because it has parameters, but the error that it gives us is inconsistent and it should get inside this if.

At this moment of the execution, the classMethod var contains the unmodifiableRealVector information, and there's a null value on parameters.

Running (randoop branch) on org.apache.commons.collections4.map.PassiveExpiringMap gives

    // long org.apache.commons.collections4.map.PassiveExpiringMap.validateAndConvertToMillis(long timeToLive,java.util.concurrent.TimeUnit timeUnit)
    public static boolean m8_p1(org.apache.commons.collections4.map.PassiveExpiringMap target, long timeToLive, java.util.concurrent.TimeUnit timeUnit) {
        // @param timeUnit the unit of time for the timeToLive parameter, must not be null. ==> (target.wait()==null) == false
        return (target.wait() == null) == false;
    }

where condition should be on timeUnit instead of target.wait().

Unextracted throws tag when a class is inside another class

I was testing the class org.apache.commons.math3.fitting.GaussianFitter when I realized that the expected output does not contain the methods that I wanted to debug. So, when I read the expected output file, I see that the methods for the class ParameterGuesser, that it's contained in the GaussianFitter class, are not extracted.

I don't know if I have to specify that I want to analyze that class with Toradocu in order to obtain those throw tags extracted.

this is how I call Toradocu:

#!/bin/bash

FOLDERDIR=/home/sergio/Repositories/toradocu
PACKAGEANDCLASS=org.apache.commons.math3.fitting.GaussianFitter

rm /home/sergio/Toradocu/CommonsMath3/$PACKAGEANDCLASS.json

/usr/bin/java -jar /home/sergio/Repositories/toradocu/build/libs/toradocu-1.0-all.jar \
--target-class $PACKAGEANDCLASS \
--source-dir $FOLDERDIR/src/test/resources/src/commons-math3-3.6.1-src/commons-math3-3.6.1-src/src/m$
--class-dir $FOLDERDIR/src/test/resources/bin/commons-math3-3.6.1.jar \
--oracle-generation false \
--condition-translator-output /home/sergio/Toradocu/CommonsMath3/$PACKAGEANDCLASS.json \
--debug

I add also the .json that I obtain, so you can see that there's no methods from the ParameterGuesser class in the file.

 /**
         * Interpolates using the specified points to determine X at the
         * specified Y.
         *
         * @param points Points to use for interpolation.
         * @param startIdx Index within points from which to start the search for
         * interpolation bounds points.
         * @param idxStep Index step for searching interpolation bounds points.
         * @param y Y value for which X should be determined.
         * @return the value of X for the specified Y.
         * @throws ZeroException if {@code idxStep} is 0.
         * @throws OutOfRangeException if specified {@code y} is not within the
         * range of the specified {@code points}.
         */
        private double interpolateXAtY(WeightedObservedPoint[] points,

This is an example of a method in the file of the GaussianFitter that's inside the ParameterGuesser class that does not appear in the .json

org.apache.commons.math3.fitting.GaussianFitter.json.tar.gz

Toradocu throws a ParserException on the package-info.java files in Apache Commons Collections.
For instance, for org.apache.commons.collections4.package-info.java that consists of just

package org.apache.commons.collections4;

results in

com.github.javaparser.ParseException: Encountered unexpected token: "-" "-"
    at line 2, column 53.

Was expecting one of:

    "<"
    "extends"
    "implements"
    "{"
        at com.github.javaparser.ASTParser.generateParseException(ASTParser.java:9791)
        at com.github.javaparser.ASTParser.jj_consume_token(ASTParser.java:9650)
        at com.github.javaparser.ASTParser.ClassOrInterfaceBody(ASTParser.java:858)
        at com.github.javaparser.ASTParser.ClassOrInterfaceDeclaration(ASTParser.java:500)
        at com.github.javaparser.ASTParser.TypeDeclaration(ASTParser.java:421)
        at com.github.javaparser.ASTParser.CompilationUnit(ASTParser.java:217)
        at com.github.javaparser.InstanceJavaParser.parse(InstanceJavaParser.java:92)
        at com.github.javaparser.JavaParser.parse(JavaParser.java:173)
        at org.toradocu.Toradocu.convertConditionsToJava(Toradocu.java:312)
        at org.toradocu.Toradocu.main(Toradocu.java:203)

These are not interesting files as input, but Toradocu should do nothing rather than fail.

Toradocu produces a wrong translation for comments like if <array name> has length 0.
From this comment, Toradocu identifies the predicate has length. This leads the condition translator to translate the predicate as .length instead of .length==0.

In order for Randoop to be able to use reflection to recreate types (especially member types), they need to be given in the format expected by Class.forName(), which is produced by Class<>.getName().

For instance, Running Toradocu over JGraphT (https://github.com/randoop/jgrapht/tree/master/jgrapht-core)
generates a JSON file with the member type org.jgrapht.experimental.dag.DirectedAcyclicGraph.Visited given in that format where the Class.forName() format would be org.jgrapht.experimental.dag.DirectedAcyclicGraph$Visited

When given

/**
 * @returns true if x is positive, false otherwise 
 */
public static boolean isPositive(int x) {
  return x > 0;
}

Toradocu is generating the condition x==false ? result == true instead of x > 0 ? result == true : result == false.

If the @returns-clause is returns true if x is positive, then it does return x > 0 ? result == true.

Running the master branch on com.google.common.collect.Ordering

gives a NullPointerException

The logout is

[main] TRACE org.toradocu.translator.ConditionTranslator - Identifying propositions from: "if the parameters are not mutually comparable under this ordering." in min(T a,T b) Exception in thread "main" java.lang.NullPointerException at org.toradocu.translator.JavaElementsCollector.collect(JavaElementsCollector.java:35) at org.toradocu.translator.Matcher.subjectMatch(Matcher.java:36) at org.toradocu.translator.ConditionTranslator.translatePropositions(ConditionTranslator.java:239) at org.toradocu.translator.ConditionTranslator.translate(ConditionTranslator.java:68) at org.toradocu.Toradocu.main(Toradocu.java:137)

At PrecisionRecallCommonsMath3#testArithmeticUtils, given the comment:

Empty @throws condition. Comment: if e < 0.

The expected is "args[1]<0"

But received empty condition instead.

-> INEQUALITY_0/JJ (root)
-> e/SYM (dep)
-> is/VBZ (cop)
-> ./. (punct)

This is the semantic graph. As you can see, e is a dep for the Stanford Parser, not a noun, therefore, there's no subject we can match.

Using Guava clone (https://github.com/randoop/guava) to generate conditions for com.google.common.base.SmallCharMatcher get the following error

[main] ERROR org.toradocu.util.Reflection - Unable to load class com.google.common.base.CharMatcher.NamedFastMatcher. Check the classpath.
Exception in thread "main" java.lang.NullPointerException
        at org.toradocu.extractor.DocumentedMethod.getExecutable(DocumentedMethod.java:263)
        at org.toradocu.Toradocu.convertConditionsToJava(Toradocu.java:305)
        at org.toradocu.Toradocu.main(Toradocu.java:243)

The classpath is the Guava build script generate jar file, so it should be OK.

The input class is declared as

final class SmallCharMatcher extends NamedFastMatcher

Perhaps the issue is that com.google.common.base.CharMatcher.NamedFastMatcher is package private.

From the target method plume.RegexUtil#asRegex. Toradocu is not able to translate the comment "if argument is not a regex".

Solution: Consider static methods of the target object class as possible match of the predicate. Also, include methods (both static and non-static) that take as input the types compatible with the parameters or the target object.

TTC: 1 week

Running Randoop on org.apache.commons.collections4.ClosureUtils with --export-conditions (on randoop branch) gives the following condition-method

    // org.apache.commons.collections4.Closure org.apache.commons.collections4.ClosureUtils.ifClosure(org.apache.commons.collections4.Predicate predicate,org.apache.commons.collections4.Closure trueClosure,org.apache.commons.collections4.Closure falseClosure)
    public static boolean m12_p1(org.apache.commons.collections4.ClosureUtils target, org.apache.commons.collections4.Predicate predicate, org.apache.commons.collections4.Closure trueClosure, org.apache.commons.collections4.Closure falseClosure) {
        // @param trueClosure the closure called if the predicate is true ==> args[0]==true
        return predicate == true;
    }

where predicate has type org.apache.commons.collections4.Predicate and cannot be converted to boolean. So, translation of "if predicate is true" to predicate == true is incorrect.
Similarly, for statement "if predicate is false" for the falseClosure parameter.

If you look at the code these param-tags are not preconditions anyway, they describe the behavior of the Closure object that the method creates.

When a Type or Interface is involved in a predicate, Toradocu doesn't always translate it correctly.

For example, in org.apache.commons.collections4.collection.UnmodifiableBoundedCollection, the constructor unmodifiableBoundedCollection(Collection<? extends E> coll) reports the following comment in the Javadoc:

• @throws IllegalArgumentException if coll is not a {@code BoundedCollection}

In this case, Toradocu produces an empty condition, because the predicate is BoundedCollection doesn't mean anything to it.

A possible solution could be exploiting the @code tag in order to find if the tagged element is a Type, or an Interface, etc.
Obviously we cannot be sure that such elements will always be tagged with @code: maybe a more general solution could work better.

Toradocu execution on net.sf.freecol.FreeCol terminates with the following error:

java.lang.NoClassDefFoundError: org/apache/commons/cli/ParseException

	at java.lang.Class.getDeclaredMethods0(Native Method)
	at java.lang.Class.privateGetDeclaredMethods(Class.java:2701)
	at java.lang.Class.getDeclaredMethods(Class.java:1975)
	at org.toradocu.extractor.DocumentedMethod.getExecutable(DocumentedMethod.java:263)
	at org.toradocu.translator.JavaElementsCollector.collect(JavaElementsCollector.java:47)
	at org.toradocu.translator.Matcher.subjectMatch(Matcher.java:36)
	at org.toradocu.translator.ConditionTranslator.translatePropositions(ConditionTranslator.java:204)
	at org.toradocu.translator.ConditionTranslator.processTag(ConditionTranslator.java:430)
	at org.toradocu.translator.ConditionTranslator.translate(ConditionTranslator.java:39)
	at org.toradocu.Toradocu.main(Toradocu.java:163)
	[...]
Caused by: java.lang.ClassNotFoundException: org.apache.commons.cli.ParseException
	at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
	at java.net.FactoryURLClassLoader.loadClass(URLClassLoader.java:814)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
	... 38 more

This might be related to #79.

Input: "@param predicates the the predicates to check, cloned, not null"
Expected output: (args[0]==null) == false
Actual output: (args[0]==null) == false || (target.getPredicates()==null) == false

From the comment Toradocu extracts a proposition with subject predicates and predicate not is null.
Then Toradocu picks as matching code elements the method parameter predicates and the method getPredicates in the target class.
Toradocu would use only one of the two (according to a defined priority) if the subject is singular, but in this case predicates is plural.
Then Toradocu retains both identified code elements and merge the two translations in
(args[0]==null) == false || (target.getPredicates()==null) == false.
Notice that || (or) is the conjunction used when no other conjunction is specified.

One possibility to solve this is to look at the verb and check if it is a third person singular verb, and in that case treat the subject as a singular noun.

For the @param in the following class

package pkg;

public class Counter {
  /**
  * @param count  the count of all things wonderful, must be positive but less than 10
  */
  void wonderfulCount(int count) {}
}

The condition should be count >0 && count < 10, but Toradocu is generating count > 0 || count < 10.

Are there examples where the correct interpretation of "but" is not a conjunction?

In Toradocu the input sentence shape is <= 0 or scale is <= 0. is preprocessed into shape is INEQUALITY_0 or scale is INEQUALITY_1.
The semantic graph produced by the Stanford parser is the following:

-> INEQUALITY_0/NN (root)
  -> shape/NN (nsubj)
  -> is/VBZ (cop)
  -> or/CC (cc)
  -> scale/NN (conj:or)
  -> INEQUALITY_1/JJ (acl:relcl)
    -> is/VBZ (cop)
  -> ./. (punct)

Instead, we expect a graph like the following:

-> INEQUALITY_0/JJ (root)
  -> shape/NN (nsubj)
  -> is/VBZ (cop)
  -> or/CC (cc)
  -> INEQUALITY_1/JJ (conj:or)
    -> scale/NN (nsubj)
    -> is/VBZ (cop)
  -> ./. (punct)

The difference in the graphs is due to the different POS tagging of the inequalities placeholders.
I suggest to modify the current proposition extractor to:

1. Preprocess-text like we currently do.
2. Ad-hoc POS-tag the inequalities placeholder words as JJ (adjective) or another tag that make sense and cause the Stanford Parser to behave correctly.
3. Build propositions like we currently do.

More information about how to add custom POS tags can be found here: https://nlp.stanford.edu/software/parser-faq.shtml#f

The following Randoop specification is being generated by Toradocu

  {
    "operation": {
      "classname": "org.apache.commons.collections4.BoundedMap",
      "name": "isFull",
      "parameterTypes": []
    },
    "identifiers": {
      "parameters": [],
      "receiverName": "target",
      "returnName": "result"
    },
    "throwsSpecifications": [],
    "postSpecifications": [
      {
        "property": {
          "conditionText": "result==true",
          "description": "true if the map is full"
        },
        "description": "return true if the map is full",
        "guard": {
          "conditionText": "target.isFull()",
          "description": ""
        }
      }
    ],
    "preSpecifications": []
  }

This is the condition "if target.isFull() then target.isFull()". I don't expect this to cause problems for Randoop, but it certainly isn't very interesting.

Running Toradocu over org.apache.commons.math3.analysis.polynomials.PolynomialFunctionLagrangeForm

yields

[main] WARN org.toradocu.translator.SentenceParser - Unable to identify a predicate (governor = points) in "interpolating points ."
[main] WARN org.toradocu.translator.SentenceParser - Unable to identify a predicate (governor = values) in "function values at interpolating points ."
[main] WARN org.toradocu.translator.SentenceParser - Unable to identify a predicate (governor = ascertain) in "when the activated method itself can ascertain that a precondition specified in the API expressed at the level of the activated method has been violated ."
[main] WARN org.toradocu.translator.SentenceParser - Unable to identify a predicate (governor = specified) in "when the activated method itself can ascertain that a precondition specified in the API expressed at the level of the activated method has been violated ."
[main] WARN org.toradocu.translator.SentenceParser - Unable to identify a predicate (governor = expressed) in "when the activated method itself can ascertain that a precondition specified in the API expressed at the level of the activated method has been violated ."
[main] WARN org.toradocu.translator.SentenceParser - Unable to identify a predicate (governor = value) in "the function value ."
[main] WARN org.toradocu.translator.SentenceParser - Unable to identify subjects in "the degree of the polynomial .".
[main] WARN org.toradocu.translator.SentenceParser - Unable to identify subjects in "a fresh copy of the coefficients array .".
Exception in thread "main" java.lang.NullPointerException
        at java.util.Objects.requireNonNull(Objects.java:203)
        at org.toradocu.translator.Proposition.setTranslation(Proposition.java:88)
        at org.toradocu.translator.ConditionTranslator.translatePropositions(ConditionTranslator.java:304)
        at org.toradocu.translator.ConditionTranslator.processTag(ConditionTranslator.java:426)
        at org.toradocu.translator.ConditionTranslator.translate(ConditionTranslator.java:38)
        at org.toradocu.Toradocu.main(Toradocu.java:163)

The condition for
org.apache.commons.math3.analysis.differentiation.DerivativeStructure org.apache.commons.math3.analysis.differentiation.DerivativeStructure.divide(org.apache.commons.math3.analysis.differentiation.DerivativeStructure a)
assumes that zero translates to the corresponding int value.
However, the class has a zero object that is different from the int, and may be difficult to figure out from the class.

This is the generated condition method:

// org.apache.commons.math3.analysis.differentiation.DerivativeStructure org.apache.commons.math3.analysis.differentiation.DerivativeStructure.divide(org.apache.commons.math3.analysis.differentiation.DerivativeStructure a)
    public static boolean m22_t2(org.apache.commons.math3.analysis.differentiation.DerivativeStructure receiver22_t2, org.apache.commons.math3.analysis.differentiation.DerivativeStructure a) {
        // @throws org.apache.commons.math3.exception.MathArithmeticException if a is zero ==> args[0]==0
        return a == 0;
    }

When run on net.sf.freecol.client.gui.panel.InPortPanel of https://github.com/randoop/FreeCol
get the condition class

// net.sf.freecol.client.gui.panel.InPortPanel.InPortPanel(net.sf.freecol.client.gui.panel.PortPanel portPanel,java.lang.String name,boolean editable)
    public static boolean m0_p2(net.sf.freecol.client.gui.panel.PortPanel portPanel, java.lang.String name, boolean editable) {
        // @param editable Is this panel editable? ==> target.isEditable()
        return receiver0_p2.isEditable();
    }

that doesn't compile.

The issue is that constructors don't have receivers

From Apache Commons Math StepFunction, Toradocu generates condition method

  // org.apache.commons.math3.analysis.function.StepFunction.StepFunction(double[] x,double[] y)
    public static boolean m0_t2(org.apache.commons.math3.analysis.function.StepFunction receiver0_t2, double[] x, double[] y) {
        // @throws org.apache.commons.math3.exception.NoDataException if x or y are zero-length. ==> args[0]==0 || args[1]==0
        return x == 0 || y == 0;
    }

The condition is on the arrays instead of the length of the arrays, and should be x.length == 0 || y.length == 0

This exception occurred when running Toradocu on com.google.common.util.concurrent.Futures from https://github.com/randoop/guava/tree/master/guava:

java.lang.NullPointerException
        at com.sun.tools.javadoc.ClassDocImpl.importedClasses(ClassDocImpl.java:1111)
        at org.toradocu.extractor.JavadocExtractor.getExceptionName(JavadocExtractor.java:236)
        at org.toradocu.extractor.JavadocExtractor.extractThrowsTags(JavadocExtractor.java:325)
        at org.toradocu.extractor.JavadocExtractor.extract(JavadocExtractor.java:68)
        at org.toradocu.Toradocu.process(Toradocu.java:509)
        at org.toradocu.doclet.formats.html.HtmlDoclet.generateClassFiles(HtmlDoclet.java:182)
        at org.toradocu.doclet.internal.toolkit.AbstractDoclet.generateClassFiles(AbstractDoclet.java:192)
        at org.toradocu.doclet.internal.toolkit.AbstractDoclet.startGeneration(AbstractDoclet.java:137)
        at org.toradocu.doclet.internal.toolkit.AbstractDoclet.start(AbstractDoclet.java:82)
        at org.toradocu.doclet.formats.html.HtmlDoclet.start(HtmlDoclet.java:87)
        at org.toradocu.doclet.standard.Standard.start(Standard.java:39)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.sun.tools.javadoc.DocletInvoker.invoke(DocletInvoker.java:310)
        at com.sun.tools.javadoc.DocletInvoker.start(DocletInvoker.java:189)
        at com.sun.tools.javadoc.Start.parseAndExecute(Start.java:366)
        at com.sun.tools.javadoc.Start.begin(Start.java:219)
        at com.sun.tools.javadoc.Start.begin(Start.java:205)
        at com.sun.tools.javadoc.Main.execute(Main.java:162)
        at org.toradocu.Toradocu.main(Toradocu.java:102)

Currently, a comment (@throws) like "if any of the specified vertices is null" produces the following tree:

-> null/JJ (root)
  -> any/DT (nsubj)
    -> vertices/NNS (nmod:of)
      -> of/IN (case)
      -> the/DT (det)
      -> specified/VBN (amod)
  -> is/VBZ (cop)
  -> ./. (punct)

This example is taken from org.jgrapht.experimental.dag.DirectedAcyclicGraph .

As a result, with default distance threshold of 2, no subject match is found (thus, empty condition). Instead, with distance 3, the method wait() is matched as subject - but obviously it is not correct.

More acceptable subject in that comment could be: "any of the specified vertices", "specified vertices", or just "vertices". The expected condition should be: (args[0]==null || args[1]==null)

Running over net.sf.freecol.FreeCol from https://github.com/randoop/FreeCol get the following error:

Exception in thread "main" java.lang.NullPointerException
        at org.toradocu.translator.JavaElementsCollector.collect(JavaElementsCollector.java:49)
        at org.toradocu.translator.Matcher.subjectMatch(Matcher.java:36)
        at org.toradocu.translator.ConditionTranslator.translatePropositions(ConditionTranslator.java:204)
        at org.toradocu.translator.ConditionTranslator.processTag(ConditionTranslator.java:429)
        at org.toradocu.translator.ConditionTranslator.translate(ConditionTranslator.java:39)
        at org.toradocu.Toradocu.main(Toradocu.java:163)

If there is no .class file for the target class in --class-dir the following is output

[main] ERROR org.toradocu.util.Reflection - Unable to load class net.Connection. Check the classpath.
Exception in thread "main" java.lang.NullPointerException
	at org.toradocu.translator.ClassCodeElement.<init>(ClassCodeElement.java:22)
	at org.toradocu.translator.JavaElementsCollector.collect(JavaElementsCollector.java:30)
	at org.toradocu.translator.Matcher.subjectMatch(Matcher.java:36)
	at org.toradocu.translator.ConditionTranslator.translatePropositions(ConditionTranslator.java:201)
	at org.toradocu.translator.ConditionTranslator.processTag(ConditionTranslator.java:408)
	at org.toradocu.translator.ConditionTranslator.translate(ConditionTranslator.java:36)
	at org.toradocu.Toradocu.main(Toradocu.java:145)

Toradocu should exit after the error message unless there is a way to proceed without the Class<> object.