lukeanewton / tool-framework-to-measure-test-case-diversity Goto Github PK

Some of the objects like FileReader and FileWriter need to be changed because the names are already in use in java. Adding 'Service' to the end of some of these components is a good idea. For example, 'Reflector' can be changed to 'ReflectService' or 'ReflectingService' or something similar.

The user shall be able to set a configuration parameter that sets whether or not a thread pool is being used during comparisons.
The UI will need a new parameter (maybe a flag -tp for thread pool) that toggles this.
This could be saved to config.
Comparing service should change actions based on this parameter.

when generating javadoc for the system, many method headers were flagged as having missing or outdated information. Once the last couple issues are done, we should go through the system and make sure all of these are up to date, then update the javadoc in the documentation folder

@LukeANewton @EricRBedard
It looks like line 53 in InputParser.java uses String.strip() which appears to work in java 11+ but doesn't work with Java 10 and below.

I would recommend we keep building with Java 8 compatibility. Are we now going to use a different version of java?
Should we also figure out some build automation to ensure that a branch is passing tests and building before we can merge?

Write the AggregationMethod interface for use by the Comparator, and implement one simple aggregation method (eg. max, average) so the -a flag for compare instructions is functional

this should be done so that the end product comes with a whole host of possible aggregation methods. Refer to the research document for examples of methods that should be implemented

currently, if a user wanted to save results of their compare command to a file, they must include "-s " in the command. We want to make the specification of a filename optional, so that there are 3 choices for output:

1. "-s " will let the user save the results in a specified file
2. "-s" will let the use save the results in the default output file specified in the config object
3. omission of "-s" will only display the results to the console, and not save to any file

Currently, if the user specifies a file we fill the output filename field of the DTO with that filename. When the time comes to save results, we either save to the filename provided, or do not save if the filename is null. To achieve this new choice, an extra boolean field should be added into the CompareDTO. This new field would tell us whether to save results to a file or not, and the output file name being null would tell us to get the default filename from the config file.

In addition to changing the CompareDTO, this will require changes in the InputParser to make the filename specifiecation after "-s" optional, and changes in the Controller when writing results to a file.

From the meeting yesterday, Jan 13th, it has been recommended that we have the reflection service check the reflected class' interface. Testing will also have to be reopened to accommodate this.

Write the initial thread logic using ScheduledThreadPoolExecutor

It might be helpful to allow the option of a verbose mode, which displays more information about each phase of an operation.

From a debugging standpoint, this will increase observability so it will be easier to see what is happening in the system.

From a user standpoint, using the verbose mode could give more information about what goes wrong when a command entered is syntactically valid, but still fails

Only having one aggregation strategy isn't always enough. A user may want to aggregate the results in different formats at the same time. For example, printing out the mean, median and mode in a format that is machine-readable at the same time.

We have an idea of what the project structure should look like and what packages we want, so let's make some packages, provide feedback, and stick with it.

The system needs a module developed to read in system files. There exists a module called FileReader in the core package, but this module may require some additional features before it is complete.

Currently, when we print out the result of the similarities, we print out a single aggregate value.
We should be able to print out more than just a value. Printing out information like what parameters were chosen for this run and other aggregates would be beneficial. Perhaps we can print it out in human readable formats and machine readable formats.
New report formats should implement the AggregationStrategy interface.

A file writing module needs to be developed for when the user specifies that they want to save calculation results to a file. This module must be able to create files and write to them, but must also check if the file already exists and should get confirmation from the user if contents should be overridden, new values should be appended to the file, or if the write should be cancelled.

We have a chicken and egg problem with five tests that hit searchPackage() in the reflection service.

Three tests are in the controller test suite:
testAggregationHelp()
testPairwiseHelp()
testDataRepresentationHelp()

Two tests are in the reflection test suite:
testSearchPackage()
testSearchPackageWithNonClassFile()

After the target directory is cleaned, we have no jar, so searchPackage() won't find the jar file, failing the tests.

Steps to reproduce:
run a maven clean
run a maven test or package or install

A possible workaround is adding @ignore to the above tests then running maven package or never running maven clean ever again.

Fix options I can think of:

1. Removing these tests
2. Adding @ignore to these tests
3. Packaging the jar first, running the tests and then if they pass, repackaging the jar again, making sure the old one is overwritten otherwise delete the jar that didnt pass the tests
4. Use a framework with built in IoC like Spring.
   I think this can be solved with one dependency, 7 annotations and a call to spring for each of the reflection tests. Dont know about the first three though.

Note: When you already have the jar in the target directory, running mvn package will also run a clean, but the jar will still be read and tests will pass before the directory is fully cleaned. This is why we didn't have an issue before.

Move the listwise package into the comparison package and make a new package in the comparison package for the pairwise comparisons.

comparison
|
--- pairwise
--- listwise

this is something that we set out to do. Currently, the rounding parameters are included in the configuration file, but not yet used in the system.

There are some issues with specifying the delimiter in a compare command due to how String.split() works. split() always expects a regex, so any delimiter with a metacharacter that is treated as so.

For example, if I wanted to delimit my test cases with "*****" the user would need to type "*****" into the terminal, otherwise you would get an error. That is manageable, but it would be nice if you could explicitly choose whether or not your delimiter should be treated as a regex or literal.

We can use Pattern.quote(string) to force a string to be a literal for string.split(), but we require some way for the user to say whether they have a regex or literal delimiter. maybe use "-d" for literal delimiter, and "-rd" for regex delimiter? you could store the delimiter as normal in the DTO and add a field to specify whether the delimiter string is a regex or literal.

Development of a Controller so pieces of the system can begin to be brought together.

Compare commands must be stubbed until the Comparator is ready
Config commands must be stubbed until the Config file is ready
Exit command can immediately be implemented
Update and Help commands must be stubbed until the Reflector is ready

implementation of the text based UI. This module obtains user input and packages it into DTOs for use by the controller.

Consists of Console InputParser, and InvalidCommandException.

this should be done so that the system is deployed with a host of options for comparison metrics. Most of these algorithms can be found online, but may require small modifications to make them work for our system

we should take the pairing out of the Comparator and place it in its own object. This object would have two methods, one that would generate pairs from a single test suite (for diversity WITHIN a test suite), and a second method that would take 2 test suites and generate pairs (for diversity BETWEEN test suites)

The comparator then would only need to take pairs of test cases (pairs of data representations) and do comparisons between each.

We have some unit tests and a lot of integration tests. It would be good to have every service/component unit tested using something like Mockito.

now that all the functionality concerning the comparisons are completed, the case study files can be used to look at how execution time compares for the thread pool with different thread values and sequential execution.

This can be implemented as a script the just runs a command several times, each on different subsets of a large test suite, and logs the time taken by the operation.

this is of low priority to the actual software system, since performance is not a major concern for us, but gives us some nice graphs to use in the poster fair and final report

We need a config file that is loaded on startup by the controller. This file can also be edited via the command line, so it needs to be a format that can easily be loaded in and edited. This format will be JSON. Let's use Jackson to parse JSON - find the maven dependency.

Currently, the pairing service takes a long time to make pairs; much longer than the comparisons.

Suggested to do:

1. Make service threaded
2. Optimize the time complexity
3. Include this time in the progress bar (i.e. when we run this phase, the progress bar should be present)

The way we package our jar is currently lacking.

1. Dependencies are not included in our jar file. This means that gson isn't included, so the program won't start. We need to tell maven that we want to create a fat/uber jar using their shadow plugin so that the user only as to download and run the jar. Other packages like JUnit need to be excluded once we create the fat jar, but this is easy.

2. Our config.json file isn't being included in our jar. Moving this to the resources folder fixes this.

3. FileReader and FileWriter are not able to access jar files. Our config.json file is in the jar which is a compressed archive. The JVM runs this from the outside and FileReader/Writer are incapable to unpackaging a jar file to read contents. InputStream needs to be used for read and write operations from/to the config file instead. This will only impact the FileWriterService and FileReaderService used by the Controller and not testing because we can assume that tests are only ran in a development environment with an uncompressed file structure.

The reflector needs to be written to take the name of a class to use (assuming the class is in the project's class path) or a class' path to load and send back a new instance of the class to the caller.

as discussed with Cameron on Friday

some of the diversity metrics from the life sciences part of research are inherently non-pairwise. The metrics still move element-by-element through a collection, but compare every collection at once, instead of generating pairs of collections.

There is an easy way to do this that makes use of the infrastructure currently in place. Create an interface for these listwise metrics (eg. ListwiseStrategy). In the command pattern for the ComparisonService, add another command (ListwiseCommand) that takes a set of test cases and a ListwiseStrategy, and uses the ComparisonService code to execute the ListwiseCommand.

This will take some added code to the Controller and parser to let these commands be entered, but this first issue is just for adding the back-end code

changes to overall structure (new part in bottom-right)

order of onvoking services to achieve listwise command (same as pairwise, exception you dont need to generate pairs)

These objects encapsulate two functionalities of the system, parsing test cases into the system and performing pairwise comparisons. The DataRepresentation class performs the functionality for parsing test cases. The DataRepresentation and PairwiseComparison together provide functionality for perfoming pairwise comparisons.

This task contains a number of files:
DataRepresentation: an interface for concrete data representation classes to implement
CSV: an implementation of DataRepresentation for comma separated values
PairwiseComaprisonStrategy: an interface for pairwise comparisons to implement
JaccardIndex: an implementation of a PairwiseComparisonStrategy
CommonElements: another implementation of a PairwiseComparisonStrategy
Capstone: contains a main file to run some tests on the above files (part of a test driver along with FileReader)
FileReader: an INCOMPLETE version of the file reader that will be used in the system (part of a test driver along with Capstone)