Author name cluster

Tim Seyde

Possible papers associated with this exact author name in Arrow. This page groups case-insensitive exact name matches and is not a full identity disambiguation profile.

8 papers

2 author rows

IROS Conference 2024 Conference Paper

Competitive Multi-Team Behavior in Dynamic Flight Scenarios

Tim Seyde
Mathias Lechner
Joshua Rountree
Daniela Rus

Efficiently learning strategic multi-agent behavior remains a challenge for robotic systems deployed in real-world scenarios, especially when considering underactuated or dynamically unstable systems. Such systems demand an integrated approach that informs long-term strategic planning with constraints imposed by reactive control, and vice versa, to effectively accomplish task objectives in competitive scenarios. In this paper, we introduce a hierarchical control model to address this: a high-level controller synthesizes strategic guidance from aggregated team experiences, while a low-level controller formulates corresponding task-specific continuous controls. We apply this concept to coordination of competitive multi-team behavior in dynamic flight scenarios with F-16 aircraft. This work introduces a hierarchical reinforcement learning approach for multi-agent coordination, leveraging decoupled distributional value representations at the high-level together with goal-conditioned policy learning at the low-level, providing a control structure that integrates long-horizon strategic planning with short-horizon dynamic control. We further provide a parallel simulator for efficient learning with multi-agent F-16 dynamics.

Details

NeurIPS Conference 2023 Conference Paper

Gigastep - One Billion Steps per Second Multi-agent Reinforcement Learning

Mathias Lechner
lianhao yin
Tim Seyde
Tsun-Hsuan Johnson Wang
Wei Xiao
Ramin Hasani
Joshua Rountree
Daniela Rus

Multi-agent reinforcement learning (MARL) research is faced with a trade-off: it either uses complex environments requiring large compute resources, which makes it inaccessible to researchers with limited resources, or relies on simpler dynamics for faster execution, which makes the transferability of the results to more realistic tasks challenging. Motivated by these challenges, we present Gigastep, a fully vectorizable, MARL environment implemented in JAX, capable of executing up to one billion environment steps per second on consumer-grade hardware. Its design allows for comprehensive MARL experimentation, including a complex, high-dimensional space defined by 3D dynamics, stochasticity, and partial observations. Gigastep supports both collaborative and adversarial tasks, continuous and discrete action spaces, and provides RGB image and feature vector observations, allowing the evaluation of a wide range of MARL algorithms. We validate Gigastep's usability through an extensive set of experiments, underscoring its role in widening participation and promoting inclusivity in the MARL research community.

PDF Details

ICLR Conference 2023 Conference Paper

Solving Continuous Control via Q-learning

Tim Seyde
Peter Werner
Wilko Schwarting
Igor Gilitschenski
Martin A. Riedmiller
Daniela Rus
Markus Wulfmeier

While there has been substantial success for solving continuous control with actor-critic methods, simpler critic-only methods such as Q-learning find limited application in the associated high-dimensional action spaces. However, most actor-critic methods come at the cost of added complexity: heuristics for stabilisation, compute requirements and wider hyperparameter search spaces. We show that a simple modification of deep Q-learning largely alleviates these issues. By combining bang-bang action discretization with value decomposition, framing single-agent control as cooperative multi-agent reinforcement learning (MARL), this simple critic-only approach matches performance of state-of-the-art continuous actor-critic methods when learning from features or pixels. We extend classical bandit examples from cooperative MARL to provide intuition for how decoupled critics leverage state information to coordinate joint optimization, and demonstrate surprisingly strong performance across a variety of continuous control tasks.

Details

IROS Conference 2023 Conference Paper

Towards Cooperative Flight Control Using Visual-Attention

Lianhao Yin
Makram Chahine
Tsun-Hsuan Wang
Tim Seyde
Chao Liu 0021
Mathias Lechner
Ramin M. Hasani
Daniela Rus

The cooperation of a human pilot with an autonomous agent during flight control realizes parallel autonomy. We propose an air-guardian system that facilitates cooperation between a pilot with eye tracking and a parallel end-to-end neural control system. Our vision-based air-guardian system combines a causal continuous-depth neural network model with a cooperation layer to enable parallel autonomy between a pilot and a control system based on perceived differences in their attention profiles. The attention profiles for neural networks are obtained by computing the networks' saliency maps (feature importance) through the VisualBackProp algorithm, while the attention profiles for humans are either obtained by eye tracking of human pilots or saliency maps of networks trained to imitate human pilots. When the attention profile of the pilot and guardian agents align, the pilot makes control decisions. Otherwise, the air-guardian makes interventions and takes over the control of the aircraft. We show that our attention-based air-guardian system can balance the trade-off between its level of involvement in the flight and the pilot's expertise and attention. The guardian system is particularly effective in situations where the pilot was distracted due to information overload. We demonstrate the effectiveness of our method for navigating flight scenarios in simulation with a fixed-wing aircraft and on hardware with a quadrotor platform.

Details

AAMAS Conference 2022 Conference Paper

Autonomous Flight Arcade Challenge: Single- and Multi-Agent Learning Environments for Aerial Vehicles

Paul Tylkin
Tsun-Hsuan Wang
Tim Seyde
Kyle Palko
Ross Allen
Alexander Amini
Daniela Rus

The Autonomous Flight Arcade (AFA) is a novel suite of singleand multi-agent learning environments for control of aerial vehicles. These environments incorporate realistic physics using the Unity game engine with diverse objectives and levels of decisionmaking sophistication. In addition to the environments themselves, we introduce an interface for interacting with them, including the ability to vary key parameters, thereby both changing the difficulty and the core challenges. We also introduce a pipeline for collecting human gameplay within the environments. We demonstrate the performance of artificial agents in these environments trained using deep reinforcement learning, and also motivate these environments as a benchmark for designing non-learned classical control policies and agents trained using imitation learning from human demonstrations. Finally, we motivate the use of AFA environments as a testbed for training artificial agents capable of cooperative human-AI decision making, including parallel autonomy.

PDF

NeurIPS Conference 2021 Conference Paper

Is Bang-Bang Control All You Need? Solving Continuous Control with Bernoulli Policies

Tim Seyde
Igor Gilitschenski
Wilko Schwarting
Bartolomeo Stellato
Martin Riedmiller
Markus Wulfmeier
Daniela Rus

Reinforcement learning (RL) for continuous control typically employs distributions whose support covers the entire action space. In this work, we investigate the colloquially known phenomenon that trained agents often prefer actions at the boundaries of that space. We draw theoretical connections to the emergence of bang-bang behavior in optimal control, and provide extensive empirical evaluation across a variety of recent RL algorithms. We replace the normal Gaussian by a Bernoulli distribution that solely considers the extremes along each action dimension - a bang-bang controller. Surprisingly, this achieves state-of-the-art performance on several continuous control benchmarks - in contrast to robotic hardware, where energy and maintenance cost affect controller choices. Since exploration, learning, and the final solution are entangled in RL, we provide additional imitation learning experiments to reduce the impact of exploration on our analysis. Finally, we show that our observations generalize to environments that aim to model real-world challenges and evaluate factors to mitigate the emergence of bang-bang solutions. Our findings emphasise challenges for benchmarking continuous control algorithms, particularly in light of potential real-world applications.

PDF Details

ICRA Conference 2019 Conference Paper

Locomotion Planning through a Hybrid Bayesian Trajectory Optimization

Tim Seyde
Jan Carius
Ruben Grandia
Farbod Farshidian
Marco Hutter 0001

Locomotion planning for legged systems requires reasoning about suitable contact schedules. The contact sequence and timings constitute a hybrid dynamical system and prescribe a subset of achievable motions. State-of-the-art approaches cast motion planning as an optimal control problem. In order to decrease computational complexity, one common strategy separates footstep planning from motion optimization and plans contacts using heuristics. In this paper, we propose to learn contact schedule selection from high-level task descriptors using Bayesian Optimization. A bi-level optimization is defined in which a Gaussian Process model predicts the performance of trajectories generated by a motion planning nonlinear program. The agent, therefore, retains the ability to reason about suitable contact schedules, while explicit computation of the corresponding gradients is avoided. We delineate the algorithm in its general form and provide results for planning single-legged hopping. Our method is capable of learning contact schedule transitions that align with human intuition. It performs competitively against a heuristic baseline in predicting task appropriate contact schedules.

Details

ICRA Conference 2018 Conference Paper

Inclusion of Angular Momentum During Planning for Capture Point Based Walking

Tim Seyde
Apoorv Shrivastava
Johannes Englsberger
Sylvain Bertrand
Jerry E. Pratt
Robert J. Griffin

When walking at high speeds, the swing legs of robots produce a non-negligible angular momentum rate. To accommodate this, we provide a reference trajectory generator for bipedal walking that incorporates predicted centroidal angular momentum at the planning stage. This can be done efficiently as the Centroidal Moment Pivot (CMP), Instantaneous Capture Point (ICP) and the center of mass (CoM) all have closed-form trajectory solutions due to their linear dynamics. This is then used to produce smooth, continuous trajectories. We furthermore provide a lightweight model to estimate angular momentum as induced during leg swing of the gait cycle. Our proposed trajectory generator is tested thoroughly in simulation and has been shown to successfully operate on the real hardware.

Details