Submitting your planner

This page explains how to submit the planners to us, and how the planners will be invoked during the competition. Please pay close attention to this: due to the large number of participants, we will require that the rules on this page are followed strictly.

We remind you that the planner submission deadline is May 31st 2008. We follow the ICAPS rules regarding time zones: You submission is in time as long as there is some time zone in which it is still May 31st. In other words, the deadline is 11:59 PM UTC-10 (Honolulu time). The deadline is strict. No extensions will be granted.

Sending us your planner

Planners are submitted by email to Malte Helmert <helmert AT informatik DOT uni DASH freiburg DOT de>.

Example: You submit the planners "Find Plans Quickly" that participates only in the satisficing temporal track, and "Find Optimal Plans Slowly" that participates in both temporal tracks. Then your archive should contain the following directories:

tempo-sat-fpq
tempo-sat-fops
tempo-opt-fops

Compiling your planner

Example: You submitted an archive named fpq-fops.tar.bz2 that contains the "Find Plans Quickly" and "Find Optimal Plans Slowly" planners mentioned above. In that case, we will build your planners as follows:

tar xjf fpq-fops.tar.bz2
cd tempo-sat-fpq
./build
cd ..
cd tempo-sat-fops
./build
cd ..
cd tempo-opt-fops
./build
cd ..

Running your planner

Example: The executable for the "Find Plans Quickly" planner is called fpq and accepts the same command-line arguments as the FF planner. Then your plan script may look as follows:

#! /bin/bash
./fpq -o $1 -f $2
mv $2.soln $3

Special planner aspects

Some technical aspects of a planner may make the evaluation more complicated. Please check the following list to see if any of the following points applies to your planner. In this case, please clearly indicate this in the submission email, and give us the detailed information we need in order to run the planner.

Alternative domain formulations

Each evaluation domain for the competition will come in several alternative formulations that use different subsets of the optional features for that track (including, of course, formulations that use none of the optional features). For example, we will provide formulations with and without derived predicates, and formulations with and without object fluents. In some cases, it is clear which formulation is most appropriate for a planner (e.g., a planner that does not support derived predicates will only use formulations that don't make use of them). In other cases, the decision is not as clear. For example, a planner might support conditional effects, but still prefer domain formulations that don't use them in some cases.

In previous competitions, participants could manually choose which of the alternative formulations their planners would use in each domain. Due to the blind evaluation mode, this won't be possible this year. Instead, we will choose the "best" formulation for each planner in each domain by first using probing runs, where each planner attempts each formulation of each task in each domain with a reduced timeout of 1 minute. For each domain, we check for which of the formulations the planner achieved the best score in the probing runs, according to the evaluation criteria of the competition. This formulation is then considered the "best" formulation of that domain for that planner, and the results of the probing runs are thrown away. In the final evaluation (which then uses a timeout of 30 minutes), the planner will only be evaluated against the formulation selected earlier. (In case of ties, we will prefer formulations that use a larger subset of PDDL features, because most planners perform better if fewer features need to be compiled away.)

We reserve the right to run some planners against multiple formulations in the final evaluation and only use the best result for scoring, for example in cases where two planners achieve very similar scores or in cases where the probing runs produce very erratic results. In any case, formulations are chosen per-domain, not per-instance.

If you feel that this automated method won't lead to optimal selection of domain formulations for your planner, please bring up the issue in the submission email so that we can work out an alternative policy.

Bug fix policy

In some cases, we will offer the opportunity to fix bugs that arise during the evaluation period, but any changes after the submission deadline will be strictly limited to bugfixes only. We will use a diff tool to check that patches don't contain new features or parameter tuning, and will reject patches that don't look like minimal changes to fix bugs. It is your responsibility to provide patches that are easy to verify with a diff tool. We reserve the right to reject changes for which the only-bugfixes rule is unnecessarily hard to check (e.g. because you reformatted the whole code).

PlannerSubmission (last edited 2009-03-11 16:08:35 by MalteHelmert)