TUPLES’ members papers at the 2023 ICAPS Conference

Automatic Metamorphic Test Oracles for Action-Policy Testing AUTHORS: Jan Eisenhut, Álvaro Torralba, Maria Christakis and Joerg Hoffmann

Learned action policies π can be used to make real-time decisions in dynamic environments; one simply evaluates the policy on the current state in order to obtain the next action. Yet, this raises obvious concerns regarding potential policy “bugs”, that is, undesirable or even fatal policy behavior in particular situations. Testing – searching for bugs – is a natural paradigm to address these concerns.

A central component of action-policy testing are test oracles, which are responsible for recognizing states to be bugs. In the context of this paper, this means that, given a query state t, a test oracle attempts to establish that the behavior of π on t is sub-optimal. Recent work introduced metamorphic oracles realizing this by comparing the behavior of π on state pairs where one of the states is known to be easier to solve, i.e., if π performs better on a more difficult state s than on a simpler state t, then its behavior on t must be sub-optimal so that t must be a bug.

This paper shows how to automatically design such oracles based on simulation relations between states. It introduces two oracle families of this kind: first, morphing query states t to obtain suitable s; second, maintaining and comparing upper bounds on the cost of optimal plans for the states encountered during testing. Experiments show that these new oracles can find bugs much more quickly than the existing (search-based) alternatives and that the combination of the new oracles with search-based ones almost consistently dominates all other oracles.

You can view the paper here


The International Conference on Automated Planning and Scheduling (ICAPS) is the premier forum for exchanging news and research results on the theory and applications of intelligent and automated planning and scheduling technology. ICAPS 2023 is part of the ICAPS conference series. After three years of virtual events, ICAPS 2023 will be a physical conference again. ICAPS 2023 will be held July 8-13, 2023, in Prague, Czech Republic.


Latest News

Online collaboration is working fine but sometimes meeting face to face is really a good thing. So we did it for the third time in KU Leuven.

We are part of a broader community of research and development efforts dedicated to advancing AI in a responsible and human-centered manner.

FAI Group, part of the Saarland Informatics Campus, contribution to the TUPLES project and ongoing research efforts in the foundations of AI are significant steps forward in advancing our understanding of this technology and ensuring its safe and responsible use in the future.

We are very pleased to announce that our coordinator, Sylvie Thiebaux, is at the prestigious IJCAI23 conference both as Area Chair and as presenter of the paper "Formal Explanations of Neural Network Policies for Planning".