This page summarizes the rules for the deterministic part of IPC-2008. It is meant to be the central place where participants can find everything they need to know to build a good entry for the competition. In particular, you will find information or links about:
- the different tracks of the competition
- the modeling language used to represent competition domains and problems
- the domains and problems used in the competition
- evaluation:
- how the results are collected (including timeout and memory limits)
- how the planners are scored and ranked
- miscellaneous issues:
- availability of source code
- parallel/multi-core planning algoirthms
- external memory search
Other useful information such as deadlines, updates, and supporting documents/information can be found by following links from the competition homepage. If you still do not find the answer to your question, please use the mailing list to discuss the issue. Issues discussed on the mailing list which are of general importance will be reflected in the contents of this page.
Tracks
The IPC-2008 deterministic part is divided into three main tracks: sequential, temporal planning, and net-benefit. Each track in turn is divided into two subtracks: satisficing and optimal planning. Awards will be given to planners in each subtrack and not aggregated across several subtracks, so each subtrack essentially defines a competition of its own.
The six subtracks differ in which PDDL features are required of the participating planners, which objective function planners are expected to optimize, and whether suboptimal solutions are considered acceptable.
In each track, there are certain "core features" (e.g. durative actions in the temporal track) that are prerequisites for entering the track, and there are also optional features (e.g. ADL, derived predicates) that may appear in some of the domains. For planners that do not support the optional features, we will offer equivalent domain versions where these features have been compiled away. So it is enough to support the core features to fully participate in a given track. Note, however, that compiled problem formulations are often significantly larger than the original problem formulations, so it may be advantageous to support the optional features directly.
The following links contain a detailed description of each track:
Sequential: satisficing | optimization. The objective of this track is to minimize total action cost of a plan.
Temporal: satisficing | optimization. The objective of this track is to minimize plan makespan.
Net-benefit: satisficing | optimization. The objective of this track is to maximize total plan net benefit (i.e. total achieved goal utility - total action cost).
A couple of example domains for the three tracks described above can be found here.
Modeling Language
We introduce PDDL3.1 to model domains and problems in IPC-2008. The main extension of PDDL3.1 over the previous PDDL versions is the introduction of multi-valued state variables (object fluents). Please refer to the language page for details. Note that this page is continuously evolving; examples using the new languages for different tracks will be available around March 15.
Note that object fluents are an optional feature of all tracks, so they need not be supported by a planner to participate in the competition. However, in many cases object fluents may lead to a more natural problem representation than binary-only problem descriptions.
IPC-2008 Domains/Problems
Like previous competitions, we plan to introduce new domains and also re-use several domains from previous competitions. We are actively seeking interesting and challenging new domains for this competition. Therefore, if you know any problem that can potentially be a good competition domain, please send the suggestion by email to one of the organizers (Minh Do <minhdo AT parc DOT com>, Malte Helmert <helmert AT informatik DOT uni DASH freiburg DOT de>, Ioannis Refanidis <yrefanid AT uom DOT gr>). We prefer domains that are inspired by real-world applications, and also can be naturally represented (or closely approximated) by the PDDL fragment of any of the three main tracks of this competition.
Depending on the properties of each domain, a given domain may not be used in all three tracks. The problem set, which is likely to be randomly generated for each domain, can also be different between the two subtracks (satisficing and optimization). In particular, given that it is much harder to find proven optimal solutions than finding a satisficing solution, the problems used in the optimization subtracks are likely to be less complex than the ones used in the satisficing subtracks.
Evaluation
The biggest difference in the evaluation process for this competition is the change to blind evaluation. Thus, instead of having the competitors run their planners on problems provided (incrementally) by the organizers, we as the organizers will run all planners according to our schedule and only report the final results when the competition ends. The actual problem sets used in the competition will not be known to the competitors and will only be made public at the end of the IPC.
We will provide example problems for all tracks so that the participants can test their planners before submitting the final executable. We hope that the examples will cover all possible constraints that appear in the actual testing problems. However, if there is a problem running a participating planner on a given testing problem, we will try our best to work with the participants to figure out the issue. Please check the IPC-2008 homepage for information on when the example problems for all tracks will be available and when we will start running the submitted planners on testing problems. At the moment, we provide some PDDL3.1 examples on the PDDL 3.1 page.
Evaluation Criteria: Like the last several IPCs, we emphasize good plan quality and put less emphasis on solving time. If two plans are found within the time and memory limits, then a plan with better quality, as defined by the objective function of each track, will have a better score. Note that this is only applicable for the satisficing subtracks because plans in the optimization track will have equal quality. In the optimization subtracks, only the number of solutions found is relevant for the final planner score. The details on the evaluation criteria can be found here: satisficing tracks and optimization tracks.
Resources allocated for each planner run: Right now, the amount of time and memory allocated for each planner run is not finalized yet. Those numbers partly depend on how many planners enter the competition, how many domains/problems we will use for each track/sub-track, and the computing resources available to us at the competition time. At the moment, the numbers we have in mind for each run are: 30 minutes for each run and 2-3 GB RAM. There is a possibility that we will allow more time and/or memory for optimal planners, compared to satisficing planners.
Supporting Tools: We expect to be able to provide a limited set of supporting tools such as updated planning validation software for PDDL3.1 (VAL) from the Strathclyde Planning Group, PDDL 3.1 syntax checkers and standalone parsers (in a limited set of programming languages).
Miscellaneous Issues
Submission of source code: Unlike previous competition, we require all competitors to make public, through the IPC-2008 website, the source code of the version competing in the competition. This will encourage information sharing and allow independent evaluation and double checking of the competition results by the planning community.
Parallel/multi-core planning algorithms: There were some discussions regarding planners that use multi-core or parallel search algorithm to compete in the competition. While we indeed will use multi-core machines to run the competing planners, we will allocate a single core for each planner run. We believe that while moving to multi-core/parallel algorithm is inevitable given the current hardware trend, we are not ready for this IPC. There are a couple of reasons such as: (i) implementation efforts: if someone uses several cores, then the others need to do the same to stay competitive; we do not want to add additional engineering challenges to the extensions we have already decided on (e.g. blind evaluation, PDDL3.1); (ii) state of the art: we are not aware of many planners that at the moment use multi-core or parallel algorithms; (iii) limited CPU power that we currently have for the competition.
Using external memory: External memory search is another trend in the search community that can have a big impact on planner performance, when the amount of main memory is limited. We do not disallow planners that use external memory search in the competition. However, the I/O time needs to be counted against the total allocated time for each run (expected to be 30 minutes at the moment).