Evaluation schema - Optimization tracks
The following rules apply for optimization tracks:
- Each planner is given 30 mins to solve each problem. The time needed to solve each problem is not taken into account; what matters is only whether the problem was solved or not within 30 mins.
The quality of the resulting plans is an on/off criterion, since plans have to be optimal. The organizers will try to obtain optimal solutions for all problems, mainly through the development of specialized domain-specific solvers. If a planner produces at least one non-optimal plan in a given domain, all results of that planner in that domain are ignored, i.e., it is assigned a score of 0 for this domain. However, the organizers will try to give the participants a chance to repair the bug that caused the problem and rerun the planner on this domain, in so far as this is possible within the time constraints and the blind evaluation mode of the competition.
- So, the only thing that matters is the number of solved problems. All problems (either hard or easier) are considered equivalent. The winner of each category (sequential, temporal, net benefit) is simply the planner that solves most of the problems.