rte-france / challenge-roadef-2020 Goto Github PK

Dear challenge participants,

We trully apologize for publishing late the qualification results. They are now available in the challenge web site :
https://www.roadef.org/challenge/2020/en/qualifresult.php

Congratulations (again) Mirsad and Michel !

Note also that we decided to shift next steps of 2 weeks :
https://www.roadef.org/challenge/2020/en/calendrier.php

The challenge organizers.

None of the instances in the sets A, B, C have non-trivial lower bounds for the resources. Can we assume that for the X set of hidden instances? Or we should provide a solver capable of dealing with nontrivial lower bounds for the resources?

Dear organizers:
Some of us will be travelling for/after New Year's celebrations. I'm wondering if it could be possible to extend the deadline only a little in order to have a couple of days after returning home. For us it would be OK if the deadline for the qualification submission was extended e.g. to January 8 (the Friday after the current deadline).

Thanks in advance

Dear Roadef organisers,

Could you please advise if we are allowed to use the constraint programming optimizer included in IBM ILOG CPLEX Optimization Studio?

Thanks in advance.

Trudy

Hello!
We have some questions about the qualification evaluation.

1. Which version of Cplex is used for evaluation? Is it okay to use the new version of Cplex(20.1)?

2. Which version of python is used for evaluation?

3. When evaluating, should a single execution produce results for both 15 and 90-minute solutions? Or you will execute twice for 15 and 90-min solutions (with different time limits)?

Thank you!

Hello,
As I know that the risks calculation is based on both the starting time of the interventions and the scenario.
In A_07.json, Intervention_700 has delta 5.0 or 6.0.
line 5115, we have the followings,
"5":{
"1":[
921.55,
898.12,
2172.72,
865.58,
845.08,
890.96
],
"2":[
1071.57,
1044.32,
2526.42,
1006.49,
982.65,
1036.0
],
"3":[
1071.57,
1044.32,
2526.42,
1006.49,
982.65,
1036.0
],
"4":[
1071.57,
1044.32,
2526.42,
1006.49,
982.65,
1036.0
],
"5":[
1071.57,
1044.32,
2526.42,
1006.49,
982.65,
1036.0
]
},

Since there is a "5" as the object key and a "5" as a key in the object.
Please help me in clarifying if my understanding is correct.
1.) Is the outer "5" the starting time of an intervention?
2.) If 1.) holds true, we have to use delta = 5.0. Then why is the inner key from 1-5 but not 5 -9?

Thanks and Best Regards

Chriss

For those who haven't seen the update on the repository, B instances have been published, good luck!

I got the reason ...

The instructions indicate Euros. Is this thousands, millions, billions, etc.?

Hi!
An update of the quantile parameter is on going, it was the same for all instances and for those with very small number of scenarios, the 0.95 quantile was simply the maximum value of the distribution.
Manuel

What will be the requisites for a final submission to be considered within the open source category? If it relies on non open source solvers (like CPLEX or Gurobi), is it still considered open source if the code produced by the team is in a public repository?

How do we interpret the number of risk values?

Such as in A_02.json, we have risks in intervention with 6 values and 5 values. And in some other tasks with 4 values. Although we know the length of risk should be equal to the length of scenario_number, but both the index of the inner array and the number of values in the inner array are confusing. Would you mind providing explanation on this?

Thanks and Best Regards

Chriss IT. L.

Hi!
On some instances, it seems that many good solutions have almost the same objective value, and only differ due to different rounding in the computations. I don't see anything in the challenge's rules regarding close solutions: how would two solutions that differ by about 10e-8 be ranked?
Thanks,
Gabriel

Dears participants, we entering in the last phase : scientific price.

Every team qualified in semi-final can participate by sending us a paper to describe their "scientific contribution" to this challenge.

More info on : https://www.roadef.org/challenge/2020/en/scientificprize.php

Just to keep you inform, independently, after the submission deadline, we will launch a Call For Paper to a special issue in Journal of Heuristic related to the RTE Challenge open to everyone (no money to win !)

             The comittee

According to the schedule, set of C instances will be available on March 16. However, according the rules doc the set of C instances will we available after the end of the semi-final phase. Which one is correct? Also, what will be the protocol for the semi-final submission?

Thanks

I guess this comes a bit late, but I just realized that the format of the input files is somewhat problematic.
The additional white space is nice for readability in small example files but of no great use in larger files which aren't read by humans anyways.
The issue with the whitespace is that it causes unnecessarily large file sizes and consequentially also longer than necessary read-in times!
Both file size and read-in times can be reduced significantly (70-80%) by using input files for which all whitespace is removed.
The following script provides one way to accomplish this:

# compactify.py
import sys
import os
file_path = sys.argv[1]
path, filename = os.path.split(file_path)
name, extension = os.path.splitext(filename)
import json
with open(file_path, 'r') as f:
    data = json.load(f)
with open(os.path.join(path, name + '_compact' + extension), 'w') as f:
    json.dump(data, f, separators=(',', ':'))

e.g. with

for f in `ls A_set/*`;
do
  python compactify.py $f
done

I feel that it would be in everyone's interest, if you'd run the evaluation with the reduced-size input files.

Kind regards

Marco

The README file mentions two small functional examples that should be provided: exemple1.json and exemple2.json.

Is it possible to add these files?
(exemple1.json, exemple2.json, output1.txt, output2.txt)

I would like to use a random number generator in my solver, but I would like to use it with a fixed seed. Is this in accordance with the rules?

Dear competitors,
The final prize will be delivered at EURO2021 in Athens. If you want, you can send a small abstract (<1500 car.) to present your work in a dedicated stream : https://euro2021athens.com/abstract-submission/, the dead line has been extended to Sunday May 2, 2021.

Hi,

Given that IFORS 2020 is not happening this June, could we expect soon the results of the sprint?

Hello! I registered myself as a Junior team yesterday (Rinat Babichev BS 4 Innopolis University student, Russia) however, I still did not get any response in my mailbox
Can you please check?

Which Linux distribution will be used to run the experiments?
This information allows us to compile and test the program on the same platform as the competition.

Thanks!

Dear all competitors,
Semi-final results are available on the website, here is the link.
Congratulation to S34!

According to the rules we are allowed to implement our solvers relying on CPLEX, Gurobi and/or LocalSolver. Will we be allowed to use well known open source solvers such us GLPK, SCIP, CBC or Google's OR-Tools for the following phases?

Thanks!

I ask as it is not listed in the allowed solvers in the rules.
Is there any reason for this?

I suspect there is an off-by-one error in check_resources while printing violated constraints:

if worload > upper_bound + tolerance:    
    print('ERROR: Resources constraint 4.2 upper bound: Worload on Resource ' + resource_name + ' at time ' + str(time) + ' exceeds upper bound.'    
            + ' Value ' + str(worload) + ' is greater than bound ' + str(upper_bound) + ' plus tolerance ' + str(tolerance) + '.')    
    # Check min         
if worload < lower_bound - tolerance:    
    print('ERROR: Resources constraint 4.2 lower bound: Worload on Resource ' + resource_name + ' at time ' + str(time) + ' does not match lower bound.'    
            + ' Value ' + str(worload) + ' is lower than bound ' + str(lower_bound) + ' minus tolerance ' + str(tolerance) + '.')    

Suggestion

Change str(time) → str(time + 1) because time takes values in [0, T-1] but should be displayed in [1, T] to be consistent with check_exclusions and the input format.

In the challenge rules it's stated: "The computer that should be used to evaluate programs of candidates is a Linux OS machine with 2 CPU, 16GB of RAM."

Could you please give more details on the machine?
Is multi-threading allowed?

Dear Roadef organisers,

There hasn't been any update on the special issue in Journal of Heuristic related to the challenge. May I know when this special issue will be launched?

Regards,
Trudy

Hi!
I find a issue with the checker, I have the next solution for the instance A_09:

Intervention_257 14
Intervention_31 6
Intervention_652 10
Intervention_212 14
Intervention_580 14
Intervention_503 4
Intervention_552 2
Intervention_660 13
Intervention_309 9
Intervention_659 14
Intervention_680 15
Intervention_653 15
Intervention_259 8
Intervention_63 12
Intervention_345 13
Intervention_602 14
Intervention_90 1
Intervention_385 2

And the checker give me the next output:

Done
Loading solution from A_09_57.out...
Done
Checking constraints...
Checking schedule constraints...
Done
Checking resources constraints...
ERROR: Resources constraint 4.2 upper bound: Worload on Resource Resources_control at time 15 exceeds upper bound. Value 1.71 is greater than bound 0.7941176471 plus tolerance 1e-05.
ERROR: Resources constraint 4.2 upper bound: Worload on Resource Resources_control at time 16 exceeds upper bound. Value 1.0 is greater than bound 0.7941176471 plus tolerance 1e-05.
Done
Checking exclusions constraints...
Done
Done
Computing objectives values...
Computing risk...
Done
Computing mean risk...
Done
Computing Quantile...
Done
Done
Instance infos:
Interventions number: 18
Scenario numbers: 17
Solution evaluation:
Objective 1 (mean risk): 3048.5019215686275
Objective 2 (expected excess (Q0.5)): 0.32564705882352785
Total objective (alpha*mean_risk + (1-alpha)*expected_excess): 1524.4137843137255

But I check with and excel the days 15 and 16 and do not exceeds the upper bound,...

When running some tests, model for risk calculation of Intervention_77 on A_01.json was failing because of lack of data. Interestingly, the provided checker did not pick this up.....

    for i, additional_risk in enumerate(intervention_risk[time_plus_one][start_time]):
KeyError: '61'

according to the provided files

Intervention_77
start_time= 61
{
  "29": [
    1019.65
  ],
  "30": [
    1019.65
  ],
  "31": [
    1019.65
  ],
  "32": [
    1019.65
  ],
  "33": [
    1019.65
  ],
  "34": [
    1019.65
  ],
  "35": [
    1019.65
  ],
  "36": [
    1019.65
  ],
  "37": [
    1019.65
  ],
  "38": [
    1019.65
  ],
  "39": [
    1019.65
  ],
  "40": [
    1019.65
  ],
  "41": [
    1019.65
  ],
  "42": [
    1019.65
  ],
  "43": [
    1019.65
  ],
  "44": [
    1019.65
  ],
  "45": [
    1019.65
  ],
  "46": [
    1019.65
  ],
  "47": [
    1019.65
  ],
  "48": [
    1019.65
  ],
  "49": [
    1019.65
  ],
  "50": [
    1019.65
  ],
  "51": [
    1019.65
  ],
  "52": [
    1019.65
  ],
  "53": [
    1019.65
  ],
  "54": [
    1019.65
  ],
  "55": [
    1019.65
  ],
  "56": [
    1019.65
  ],
  "57": [
    1019.65
  ],
  "58": [
    1019.65
  ]
}

which proves the error stated above

It just happened and I used A_01.json, for A_03.json I did not notice anything.
Please update A_set or advise accordingly.

Happy to provide any feedback, as required.

Many Thanks,
Christos

The current code replaces the risk at time with the risk induced by the current intervention instead of adding the intervention's risk to the total risk. Hence, risk[time] is simply the risk of most recent intervention considered running at time.

I suggest the following change:

        for time in range(start_time_idx, start_time_idx + delta):
            for i, additional_risk in enumerate(intervention_risk[str(time + 1)][str(start_time)]):
                risk[time][i] += additional_risk

Hello,
Given my little experience on working with Linux environment, would it be possible for me to design a solution under windows OS?

Hello, I have two questions with regards to the program that we should submit for the qualification round.

1. Can it contain short instalation instructions? Such as

python3 -m venv venv
source venv/bin/activate
pip install -r requirements.txt
pip install .

2. Should the program bundle the problem files? Or can it contain instructions such as

extract A_set.7z to the data/ folder

Thanks!

The semi final phase will end on March 8 2021 and we will be competing on set B of instances. Are we allowed/requested to send a new version of the program before that deadline?

Thanks

In the Rules document, I found the statement : "In "Resources", "min" and "max" are ordered lists of length T".
Can we consider an intervention starting at different time with resources will have different max and min stated in the list?

Dear competitors
Instance C set is available in the repository.
Good luck!

The quantile computation in the checker is wrong and produces different results than thoses expected in the rules document.
Q0.50 of [2 20] is 2 (example in the rules document updated)
Q0.95 of [101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120] is 119