Alejandro Agostini

ICRA Conference 2025 Conference Paper

Bootstrapping Object-Level Planning with Large Language Models

• David Paulius
• Alejandro Agostini
• Benedict Quartey
• George Konidaris 0001

We introduce a new method that extracts knowledge from a large language model (LLM) to produce object-level plans, which describe high-level changes to object state, and uses them to bootstrap task and motion planning (TAMP). Existing work uses LLMs to directly output task plans or generate goals in representations like PDDL. However, these methods fall short because they rely on the LLM to do the actual planning or output a hard-to-satisfy goal. Our approach instead extracts knowledge from an LLM in the form of plan schemas as an object-level representation called functional object-oriented networks (FOON), from which we automatically generate PDDL subgoals. Our method markedly outperforms alternative planning strategies in completing several pick-and-place tasks in simulation. † † Project Website: https://davidpaulius.github.io/olpllm/

AIJ Journal 2019 Journal Article

Distributional semantics of objects in visual scenes in comparison to text

• Timo Lüddecke
• Alejandro Agostini
• Michael Fauth
• Minija Tamosiunaite
• Florentin Wörgötter

The distributional hypothesis states that the meaning of a concept is defined through the contexts it occurs in. In practice, often word co-occurrence and proximity are analyzed in text corpora for a given word to obtain a real-valued semantic word vector, which is taken to (at least partially) encode the meaning of this word. Here we transfer this idea from text to images, where pre-assigned labels of other objects or activations of convolutional neural networks serve as context. We propose a simple algorithm that extracts and processes object contexts from an image database and yields semantic vectors for objects. We show empirically that these representations exhibit on par performance with state-of-the-art distributional models over a set of conventional objects. For this we employ well-known word benchmarks in addition to a newly proposed object-centric benchmark.

AIJ Journal 2017 Journal Article

Efficient interactive decision-making framework for robotic applications

• Alejandro Agostini
• Carme Torras
• Florentin Wörgötter

The inclusion of robots in our society is imminent, such as service robots. Robots are now capable of reliably manipulating objects in our daily lives but only when combined with artificial intelligence (AI) techniques for planning and decision-making, which allow a machine to determine how a task can be completed successfully. To perform decision making, AI planning methods use a set of planning operators to code the state changes in the environment produced by a robotic action. Given a specific goal, the planner then searches for the best sequence of planning operators, i. e. , the best plan that leads through the state space to satisfy the goal. In principle, planning operators can be hand-coded, but this is impractical for applications that involve many possible state transitions. An alternative is to learn them automatically from experience, which is most efficient when there is a human teacher. In this study, we propose a simple and efficient decision-making framework for this purpose. The robot executes its plan in a step-wise manner and any planning impasse produced by missing operators is resolved online by asking a human teacher for the next action to execute. Based on the observed state transitions, this approach rapidly generates the missing operators by evaluating the relevance of several cause–effect alternatives in parallel using a probability estimate, which compensates for the high uncertainty that is inherent when learning from a small number of samples. We evaluated the validity of our approach in simulated and real environments, where it was benchmarked against previous methods. Humans learn in the same incremental manner, so we consider that our approach may be a better alternative to existing learning paradigms, which require offline learning, a significant amount of previous knowledge, or a large number of samples.

IROS Conference 2015 Conference Paper

Simultaneously learning at different levels of abstraction

• Benjamin Quack
• Florentin Wörgötter
• Alejandro Agostini

Robotic applications in human environments are usually implemented using a cognitive architecture that integrates techniques of different levels of abstraction, ranging from artificial intelligence techniques for making decisions at a symbolic level to robotic techniques for grounding symbolic actions. In this work we address the problem of simultaneous learning at different levels of abstractions in such an architecture. This problem is important since human environments are highly variable, and many unexpected situations may arise during the execution of a task. The usual approach under this circumstance is to train each level individually to learn how to deal with the new situations. However, this approach is limited since it implies long task interruptions every time a new situation needs to be learned. We propose an architecture where learning takes place simultaneously at all the levels of abstraction. To achieve this, we devise a method that permits higher levels to guide the learning at the levels below for the correct execution of the task. The architecture is instantiated with a logic-based planner and an online planning operator learner, at the highest level, and with online reinforcement learning units that learn action policies for the grounding of the symbolic actions, at the lowest one. A human teacher is involved in the decision-making loop to facilitate learning. The framework is tested in a physically realistic simulation of the Sokoban game.

IROS Conference 2015 Conference Paper

Using structural bootstrapping for object substitution in robotic executions of human-like manipulation tasks

• Alejandro Agostini
• Mohamad Javad Aein
• Sándor Szedmák
• Eren Erdal Aksoy
• Justus H. Piater
• Florentin Wörgötter

In this work we address the problem of finding replacements of missing objects that are needed for the execution of human-like manipulation tasks. This is a usual problem that is easily solved by humans provided their natural knowledge to find object substitutions: using a knife as a screwdriver or a book as a cutting board. On the other hand, in robotic applications, objects required in the task should be included in advance in the problem definition. If any of these objects is missing from the scenario, the conventional approach is to manually redefine the problem according to the available objects in the scene. In this work we propose an automatic way of finding object substitutions for the execution of manipulation tasks. The approach uses a logic-based planner to generate a plan from a prototypical problem definition and searches for replacements in the scene when some of the objects involved in the plan are missing. This is done by means of a repository of objects and attributes with roles, which is used to identify the affordances of the unknown objects in the scene. Planning actions are grounded using a novel approach that encodes the semantic structure of manipulation actions. The system was evaluated in a KUKA arm platform for the task of preparing a salad with successful results.

EAAI Journal 2014 Journal Article

Learning weakly correlated cause–effects for gardening with a cognitive system

• Alejandro Agostini
• Carme Torras
• Florentin Wörgötter

We propose a cognitive system that combines artificial intelligence techniques for planning and learning to execute tasks involving delayed and variable correlations between the actions executed and their expected effects. The system is applied to the task of controlling the growth of plants, where the evolution of the plant attributes strongly depends on different events taking place in the temporally distant past history of the plant. The main problem to tackle is how to efficiently detect these past events. This is very challenging since the inclusion of time could make the dimensionality of the search space extremely large and the collected training instances may only provide very limited information about the relevant combinations of events. To address this problem we propose a learning method that progressively identifies those events that are more likely to produce a sequence of changes under a plant treatment. Since the number of experiences is very limited compared to the size of the event space, we use a probabilistic estimate that takes into account the lack of experience to prevent biased estimations. Planning operators are generated from most accurately predicted sequences of changes. Planning and learning are integrated in a decision-making framework that operates without task interruptions by allowing a human gardener to instruct the treatments when the knowledge acquired so far is not enough to make a decision.

IJCAI Conference 2011 Conference Paper

Integrating Task Planning and Interactive Learning for Robots to Work in Human Environments

• Alejandro Agostini
• Carme Torras
• Florentin W
• ouml; rg
• ouml; tter

Human environments are challenging for robots, which need to be trainable by lay people and learn new behaviours rapidly without disrupting much the ongoing activity. A system that integrates AI techniques for planning and learning is here proposed to satisfy these strong demands. The approach rapidly learns planning operators from few action experiences using a competitive strategy where many alternatives of cause-effect explanations are evaluated in parallel, and the most successful ones are used to generate the operators. The success of a cause-effect explanation is evaluated by a probabilistic estimate that compensates the lack of experience, producing more confident estimations and speeding up the learning in relation to other known estimates. The system operates without task interruption by integrating in the planning-learning loop a human teacher that supports the planner in making decisions. All the mechanisms are integrated and synchronized in the robot using a general decision-making framework. The feasibility and scalability of the architecture are evaluated in two different robot platforms: a Stä ubli arm, and the humanoid ARMAR III.

ECAI Conference 2004 Conference Paper

Learning Model Free Motor Control

• Alejandro Agostini
• Enric Celaya

IROS Conference 2004 Conference Paper

Trajectory tracking control of a rotational joint using feature-based categorization learning

• Alejandro Agostini
• Enric Celaya

Real world robot applications have to cope with large variations in the operating conditions due to the variability and unpredictability of the environment and its interaction with the robot. Performing an adequate control using conventional control techniques, that require the model of the plant and some knowledge about the influence of the environment, could be almost impossible. An alternative to traditional control techniques is to use an automatic learning system that uses previous experience to learn an adequate control policy. Learning by experience has been formalized in the field of reinforcement learning. But the application of reinforcement learning techniques in complex environments is only feasible when some generalization can be made in order to reduce the required amount of experience. This work presents an algorithm that performs a kind of generalization called categorization. This algorithm is able to perform efficient generalization of the observed situations, and learn accurate control policies in a short time without any previous knowledge of the plant and without the need of any kind of traditional control technique. Its performance is evaluated on the trajectory tracking control with simulated DC motors and compared with PID systems specifically tuned for the same problem.