This is a repository for calculating the
pip install wrandai
The following code demonstrates how to quantify the uncertainty of a Machine Learning benchmarking investigation. Here n_benchmark_tests represents the different metrics/datasets combinations that an algorithm can be tested on. The wrandai.w_randomness() function calculates the uncertainty due to the presence of testing on random seeds. The format for the absolute scores required to calculate the [benchmark_tests, seeds, algorithms]. The function assumes that a higher score is better when calculating ranking. Use return_ranks=True to return the average rank of each algorithm across all the benchmark tests and random seeds.
from wrandai import wrandai
import numpy as np
n_algorithms = 10
n_benchmark_tests = 20
n_random_seeds = 10
perfect_results = np.zeros((n_benchmark_tests, n_random_seeds, n_algorithms))
random_results = np.zeros((n_benchmark_tests, n_random_seeds, n_algorithms))
for test in range(n_benchmark_tests):
for seed in range(n_random_seeds):
interpolation_values = np.linspace(0, 1, n_algorithms).round(3)
perfect_results[test, seed, :] = interpolation_values
np.random.shuffle(interpolation_values)
random_results[test, seed, :] = interpolation_values
print(f"{random_results[0, :, 0]} best algorithm dependant on seed used [one test, all seeds, one algorithm]")
print(f'{perfect_results[0, :, 0]} best algorithm is best across all seeds [one test, all seeds, one algorithm]')
print(f"W_w Randomness Coefficient for Random Rankings: {wrandai.w_randomness(random_results, w_method='w_wasserstein')}")
print(f"W_w Randomness Coefficient for Perfect Rankings: {wrandai.w_randomness(perfect_results, w_method='w_wasserstein')}")
print(f"W_t Randomness Coefficient for Random Rankings: {wrandai.w_randomness(random_results, w_method='w_ties')}")
print(f"W_t Randomness Coefficient for Perfect Rankings: {wrandai.w_randomness(perfect_results, w_method='w_ties')}")
print(f"W Randomness Coefficient for Random Rankings: {wrandai.w_randomness(random_results, w_method='w_random_coeff')}")
print(f"W Randomness Coefficient for Perfect Rankings: {wrandai.w_randomness(perfect_results, w_method='w_random_coeff')}")Please cite this paper or this paper (or both) if you use this in your work:
@inproceedings{ugle2023leeney,
title={Uncertainty in GNN Learning Evaluations: The Importance of a Consistent Benchmark for Community Detection.},
author={Leeney, William and McConville, Ryan},
booktitle={Twelfth International Conference on Complex Networks \& Their Applications},
year={2023},
organization={Springer}
}
@article{leeney2023uncertainty2,
title={Uncertainty in GNN Learning Evaluations: A Comparison Between Measures for Quantifying Randomness in GNN Community Detection},
author={Leeney, William and McConville, Ryan},
journal={arXiv preprint arXiv:2312.09015},
year={2023}
}
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent