Comments (2)
ofrancon
commented on August 25, 2024
@EKMeyerson was suggesting computing the average rank across geos.
For instance:
predictor_1: ranks 3rd on Italy, 2nd on USA. Avg = 2.5
predictor_2: ranks 5th on Italy, 1st on USA. Avg = 3
=> predictor_1 ranks better than predictor_2.
I guess it's possible to compute a weighted average too, where the weight could be the population size of the geo.
from covid-xprize.
EKMeyerson
commented on August 25, 2024
We have decided to go with Mean Rank (as Olivier has described above) for the main quantitative metric. This will be computed both across all regions and across specialty regions.
We will compute additional metrics, like MAE, and MAE per 100K cases as additional data useful to the judges.
from covid-xprize.
Related Issues (20)
- Update predictor_robojudge notebook to consider only the fixed list of countries
- Update scenario_generator notebook to generate the official IP file
- Remove Turkmenistan and Brazil regions from list of countries and regions
- Add a main to scenario_generator
- Check PredictedDailyNewCases is a number
- Check NPI columns contain numbers in prescriptor_validation.py
- Add population estimates for Washington DC and Virgin Islands
- Check Date column contains dates
- Ground truth missing for Washington DC HOT 2
- Add an example prescription model output HOT 1
- Fix shortsighted scenario_generator tests HOT 1
- NEAT prescriptor sample failing HOT 1
- Provide a prescribe.py API
- Provide a prescriptor_submission_template notebook
- Provide the "standard" predictor to use for Phase 2 prescriptors evaluation
- Make prescriptors pip installable HOT 1
- Standard predictor is predicting from Jan 11, 2021 HOT 3
- Refactor notebooks to create `prescriptor_scoring` module
- Manage US Virgin Islands
- Specify list of countries for cost_generator
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from covid-xprize.