Comments (6)
Hi Jose. Thank you for your kind words :)
To run TOGA on your project, you can use toga.py
in the root directory.
TOGA takes 2 arguments: an input data file and a metadata file.
The input file should be a csv with (focal_method
, test_prefix
) pairs.
The metadata should specify (project,bug_num,test_name,exception_bug,assertion_bug,exception_lbl,assertion_lbl,assert_err)
Rather than actually executing the test and oracles, we check the "expected" oracle in the metadata against the predicted oracle and report an evaluation of our technique. If you would like to execute your tests with TOGA oracles, the outputs are saved in predicted_oracles.csv
file upon running toga.py
.
If you don't want to construct such an elaborate metadata file (we understand, it's formatted for Defects4J bugs), you can pass in a dummy metadata file. For example, copy over the one in data/evosuite_reaching_tests
. The reported results from toga.py
won't be correct, but the predicted_oracles.csv
will be.
Hope this helps!
from toga.
Yes, it does help, thanks.
Could you please describe each column and/or the values each column might take? Thanks in advance.
- the input data file, i.e.,
focal_method
andtest_prefix
. - the metadata file, i.e.,
project
,bug_num
,test_name
,exception_bug
,assertion_bug
,exception_lbl
,assertion_lbl
, andassert_err
.
Some columns are self explanatory, for instance project
might be, e.g., Chart
or Math
, bug_num
might be 7
, and test_name
represents the name of the test case, but others, for instance, exception_bug
, assertion_bug
, exception_lbl
, assertion_lbl
, assert_err
are not that easy to understand. Also, what is the expected format of the focal_method
column? Same question for the test_prefix
column, is it a single string (perhaps without newlines) of the test case's source code?
from toga.
Apologies for the confusion.
inputs file:
-
focal_method
is a src of the method under test as a single string (newlines are ok). -
test_prefix
is the src of your test (newlines also ok).
meta file:
exception_bug
1/0 binary label. If you are testing exceptional behavior, entry should be 1.assertion_bug
1/0 binary label. Should be opposite ofexception_bug
. If you are testing non-exceptional behavior (assertion) entry should be 1.exception_lbl
True/False. True if an exception is expected and False otherwise.assertion_lbl
the assertion written in your test src as a string. If your test has multiple assertions split into multiple rows.assert_err
this is the error printed when the assertion is violated. This is something we used for defects4j testing. You can leave this as an empty string for testing on your own project.
from toga.
Hello @elizabethdinella,
I have managed to run TOGA but I have a few questions.
- Regarding the input file, could we give TOGA no focal_method and let it try its best or is the focal method 100% necessary?
- As for the meta file, you mention a dummy could be used. Could a dummy be a file with just commas (i.e., no data in the file)?
After running TOGA, a series of files are generated assertion_preds.csv
, assert_model_inputs.csv
, exception_preds.csv
, except_model_inputs.csv
, oracle_preds.csv
and results.csv
. I imagine the first 4 are used by the model for inference and, therefore, are not something we really need to understand.
Could you describe what the other two files are for and what each collumn represents?
oracle_preds.csv
file ->project
,bug_num
,test_name
,test_prefix
,except_pred
,assert_pred
results.csv
file ->project
,bug_num
,test_name
,exception_bug
,assertion_bug
,exception_lbl
,assertion_lbl
,assert_err
,id
,except_pred
,except_correct
,assert_pred
,assert_correct
,assert_bug_found
,except_bug_found
,expected_except_bug
,unexpected_except_bug
,bug_found
,tp
,fp
,tn
,fn
Thank you in advance,
Daniel
from toga.
Hi Daniel,
- Although we evaluated using the focal method signature and docstring, a focal method input is not necessarily required. Feel free to input the empty string or anything else.
- If you populate the meta file with empty strings, the models should still be invoked correctly and the
oracle_preds.csv
will be populated. Thetoga.py
script will crash before populating theresults.csv
file, but if you're only interested in seeing model outputs this should work fine :)
The oracle_preds.csv
file shows the outputs of our model(s) for each input sample. The important columns are except_pred
and assert_pred
. If except_pred
== 1, our system predicted that an exception is expected on the test_prefix
. If an exception is expected, assert_pred
will be empty. In the other case, where except_pred
== 0, the assert_pred
column will indicate the system's predicted assertion.
The results.csv
file is a bit more complicated. It stores the information necessary to compute if our predictions were correct. The important columns are except_correct
and assert_correct
. If these are both true, the model predicted the test oracle correctly. Other columns that might be useful are tp
, fp
, tn
, and fn
. These columns contain binary entries (1/0) based on the outputs to the model and the entries exception_bug
and assertion_bug
in the metadata input file.
from toga.
Hello @elizabethdinella
when i just change this
then run toga.py
the result of oracle_preds.csv just change to this
and miss the assert_pred
from toga.
Related Issues (4)
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from toga.