BOHR is a repository of heuristics for preparation (labeling, grouping, linking, filtering) of software engineering artifacts, e.g. commits, bug reports, code snippets, etc. Preparation of these artifacts is often required by researchers in the field of Software Engineering (SE) and Mining Software Repositories (MSR) to convert artifacts mined from software repositories such as GitHub, StackOverflow into datasets that can be used for empirical experiments and for training machine learning models. An example could be classifying commits mined from GitHub into bug-fixing and others in order to create a training dataset to train a machine-learning model on.
Preparing each artifact manually is expensive and does not scale well. Therefore BOHR offers an approach to define heuristics (short functions) to do the job automatically. Even though using heuristics is usually less accurate than letting experts analyse each artifacts, we claim that using a large number of diverse heuristics and combining them "smartly" can significantly reduce the noise compared to for example using one heuistic. The way heuristics are combined depends on the type of the task, but one of the most common way is to use the algorithm used by the snorkel, the state-of-the-art weak supervision tool.
Contents
- Examples of tasks and heuristics
- Main Concepts (maybe this is not needed in README, rather in the docs)
- BOHR workflow
- 1a. For the given task, pull existing heuristics developed by community
- 3 Check whether the existing labeled datasets are suitable for your purposes.
- 4. Label your own dataset with the default classifier.
- 5. Develop a new heuristic
- 6. Debug and tune the heuristic by checking its coverage and accuracy on a stand-alone test-set
- 7. Submit a pull request with the new heuristic to remote BOHR repository
- 8. Add a new task
- Installation
- Pre-prints and publications
One example is classifying commits mined from GitHub into "bugfix" and "non-bugfix", in order to create a training dataset to train a machine-learning model on.
# ... other imports from bohrapi.core import Heuristic from bohrlabels.core import OneOrManyLabels from bohrapi.artifacts import Commit from bohrlabels.labels import CommitLabel @Heuristic(Commit) def bugless_if_many_files_changes(commit: Commit) -> OneOrManyLabels: if len(commit.commit_files) > 15: return CommitLabel.NonBugFix else: return None
Important things to note:
- A heuristics is marked with the
Heuristicdecorator, and the artifact type to which it is applied is passed to it as a parameter; - The artifact instance is exposed to the heuristic as a function parameter; the properties of the artifact object can be used to implement the logic;
- The label assigned to the artifact by the heuristic is the result of the execution on the passed artifact object; the heuristic must assign one of the labels defined in the BOHR label hierarchy or
Noneif it abstains on the data point.
TBD: Insert somewhere later?
- Simplifies the process of adding new heuristics and evaluating their effectiveness;
- Labels the datasets registered with BOHR and automatically updates the labels once heuristics are added;
- Keeps track of heursitics used for each version of generated datasets and models, and, in general, makes sure they are reproducible and easily accessable by using DVC.
BOHR is a repository of heuristics, hence, a heuristic is a primary concept in BOHR. Sub-program (python function) that accepts an artifact or multiple artifacts of the same or different types.
Artifact is BOHR's abstraction that represents a software engineering artifact - a product of SE activities, e,g. code, commit, software project, software repository, issue report. Dataset is a collection of artifacts of the same type.
Task is an abstraction that describes the problem that a researcher is working on in terms of BOHR. The input and the output of the tasks are datasets. Task types are labeling, grouping, linking, filtering. The task is defined by specifying artifact type(s) heuristics are applied to, possible outputs of heuristics, strategy how heuristics aree combined, test datasets and metrics to use to evaluate the effectiveness of heuristics.
Experiment is an attempt to solve a task using a specific set of heuristics (and training data if needed).
bohr clone https://github.com/giganticode/bohr-workdir-bugginess
This will clone the so called BOHR working directory that corresponds to the <task> to <path>
Every task comes with a trained classifier and a default dataset labeled by this classifier. Check whether the default datasets suits your purposes.
cd bugginess-work-dir && bohr pull default
The path where dataset is load will be displayed.
$ bohr dataset add ~/new_commit_dataset.csv
$ bohr task add-dataset bugginess new_commit_dataset --repro
$ vi heuristics/commit_files.py
$ bohr repro
$ bohr upload
Label model is trained and metrics are calculated on stand-alone test set as a part of a CI-pipeline. If metrics has been improved, the new heuristic is added to BOHR, and is available for other researchers.
$ bohr task add tangled-commits \
... -l TangledCommit.NonTangled,TangledCommit.Tangled \
... --repro
TODO: add links to other repos
If you use BOHR in your research, please cite us:
@inproceedings{babii2021mining,
title={Mining Software Repositories with a Collaborative Heuristic Repository},
author={Babii, Hlib and Prenner, Julian Aron and Stricker, Laurin and Karmakar, Anjan and Janes, Andrea and Robbes, Romain},
booktitle={2021 IEEE/ACM 43rd International Conference on Software Engineering: New Ideas and Emerging Results (ICSE-NIER)},
pages={106--110},
year={2021},
organization={IEEE}
}
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent