Author name cluster

Robert E. Mahony

Possible papers associated with this exact author name in Arrow. This page groups case-insensitive exact name matches and is not a full identity disambiguation profile.

55 papers

2 author rows

ICRA Conference 2025 Conference Paper

Asynchronous Multi-Object Tracking with an Event Camera

Angus Apps
Ziwei Wang 0002
Vladimir Perejogin
Timothy L. Molloy
Robert E. Mahony

Events cameras are ideal sensors for enabling robots to detect and track objects in highly dynamic environments due to their low latency output, high temporal resolution, and high dynamic range. In this paper, we present the Asynchronous Event Multi-Object Tracking (AEMOT) algorithm for detecting and tracking multiple objects by processing individual raw events asynchronously. AEMOT detects salient event blob features by identifying regions of consistent optical flow using a novel Field of Active Flow Directions built from the Surface of Active Events. Detected features are tracked as candidate objects using the recently proposed Asynchronous Event Blob (AEB) tracker in order to construct small intensity patches of each candidate object. A novel learnt validation stage promotes or discards candidate objects based on classification of their intensity patches, with promoted objects having their position, velocity, size, and orientation estimated at their event rate. We evaluate AEMOT on a new Bee Swarm Dataset, where it tracks dozens of small bees with precision and recall performance exceeding that of alternative event-based detection and tracking algorithms by over 37%. Source code and the labelled event Bee Swarm Dataset will be open sourced. 1 1 https://github.com/angus-apps/AEMOT

ICRA Conference 2025 Conference Paper

Equivariant Filter Design for Range-Only SLAM

Yixiao Ge
Arthur Pearce
Pieter van Goor
Robert E. Mahony

Range-only Simultaneous Localisation and Mapping (RO-SLAM) is of interest due to its practical applications in ultra-wideband (UWB) and Bluetooth Low Energy (BLE) localisation in terrestrial and aerial applications and acoustic beacon localisation in submarine applications. In this work, we consider a mobile robot equipped with an inertial measurement unit (IMU) and a range sensor that measures distances to a collection of fixed landmarks. We derive an equivariant filter (EqF) for the RO-SLAM problem based on a symmetry Lie group that is compatible with the range measurements. The proposed filter does not require bootstrapping or initialisation of landmark positions, and demonstrates robustness to the noprior situation. The filter is demonstrated on a real-world dataset, and it is shown to significantly outperform a state-of-the-art EKF alternative in terms of both accuracy and robustness.

ICRA Conference 2024 Conference Paper

An Equivariant Approach to Robust State Estimation for the ArduPilot Autopilot System

Alessandro Fornasier
Yixiao Ge
Pieter van Goor
Martin Scheiber
Andrew Tridgell
Robert E. Mahony
Stephan Weiss 0002

The majority of commercial and open-source autopilot software for uncrewed aerial vehicles rely on the tried and tested extended Kalman filter (EKF) to provide the state estimation solution for the inertial navigation system (INS). While modern implementations achieve remarkable robustness, it is often due to the careful implementation of exception code for a multitude of corner cases along with significant skilled tuning effort. In this paper, we use the data wealth of the ArduPilot community to identify and highlight the most common real-world challenges in INS state estimation, including sensor self-calibration, robustness in static conditions, global navigation satellite system (GNSS) outliers and shifts, and robustness to faulty inertial measurement units (IMUs). We propose a novel equivariant filter (EqF) formulation for the INS solution that exploits a Semi-Direct-Bias symmetry group for multi-sensor fusion with self-calibration capabilities and incorporates equivariant velocity-type measurements. We augment the filter with a simple innovation-covariance inflation strategy that seamlessly handles GNSS outliers and shifts without requiring coding of a whole set of exception cases. We use real-world data from the Ardupilot community to demonstrate the performance of the proposed filter on known cases where existing filters fail without careful exception handling or case-specific tuning and benchmark against the ArduPilot’s EKF3, the most sophisticated EKF implementation currently available.

ICRA Conference 2022 Conference Paper

A Linear Comb Filter for Event Flicker Removal

Ziwei Wang 0002
Dingran Yuan
Yonhon Ng
Robert E. Mahony

Event cameras are bio-inspired sensors that capture per-pixel asynchronous intensity change rather than the synchronous absolute intensity frames captured by a classical camera sensor. Such cameras are ideal for robotics applications since they have high temporal resolution, high dynamic range and low latency. However, due to their high temporal resolution, event cameras are particularly sensitive to flicker such as from fluorescent or LED lights. During every cycle from bright to dark, pixels that image a flickering light source generate many events that provide little or no useful information for a robot, swamping the useful data in the scene. In this paper, we propose a novel linear filter to preprocess event data to remove unwanted flicker events from an event stream. The proposed algorithm achieves over 4. 6 times relative improvement in the signal-to-noise ratio when compared to the raw event stream due to the effective removal of flicker from fluorescent lighting. Thus, it is ideally suited to robotics applications that operate in indoor settings or scenes illuminated by flickering light sources. Code, Datasets and Video: https://github.com/ziweiWWANG/EFR

ICRA Conference 2022 Conference Paper

Equivariant Filter Design for Inertial Navigation Systems with Input Measurement Biases

Alessandro Fornasier
Yonhon Ng
Robert E. Mahony
Stephan Weiss 0002

Inertial Navigation Systems (INS) are a key technology for autonomous vehicles applications. Recent advances in estimation and filter design for the INS problem have exploited geometry and symmetry to overcome limitations of the classical Extended Kalman Filter (EKF) approach that formed the mainstay of INS systems since the mid-twentieth century. The industry standard INS filter, the Multiplicative Extended Kalman Filter (MEKF), uses a geometric construction for attitude estimation coupled with classical Euclidean construction for position, velocity and bias estimation. The recent Invariant Extended Kalman Filter (IEKF) provides a geometric framework for the full navigation states, integrating attitude, position and velocity, but still uses the classical Euclidean construction to model the bias states. In this paper, we use the recently proposed Equivariant Filter (EqF) framework to derive a novel observer for biased inertial-based navigation in a fully geometric framework. The introduction of virtual velocity inputs with associated virtual bias leads to a full equivariant symmetry on the augmented system. The resulting filter performance is evaluated with both simulated and real-world data, and demonstrates increased robustness to a wide range of erroneous initial conditions, and improved accuracy when compared with the industry standard Multiplicative EKF (MEKF) approach.

IROS Conference 2022 Conference Paper

GoferBot: A Visual Guided Human-Robot Collaborative Assembly System

Zheyu Zhuang
Yizhak Ben-Shabat
Jiahao Zhang
Stephen Gould
Robert E. Mahony

The current transformation towards smart manufacturing has led to a growing demand for human-robot collaboration (HRC) in the manufacturing process. Perceiving and understanding the human co-worker's behaviour introduces challenges for collaborative robots to efficiently and effectively perform tasks in unstructured and dynamic environments. Integrating recent data-driven machine vision capabilities into HRC systems is a logical next step in addressing these challenges. However, in these cases, off-the-shelf components struggle due to generalisation limitations. Real-world evaluation is required in order to fully appreciate the maturity and robustness of these approaches. Furthermore, understanding the pure-vision aspects is a crucial first step before combining multiple modalities in order to understand the limitations. In this paper, we propose GoferBot, a novel vision-based semantic HRC system for a real-world assembly task. It is composed of a visual servoing module that reaches and grasps assembly parts in an unstructured multi-instance and dynamic environment, an action recognition module that performs human action prediction for implicit communication, and a visual handover module that uses the perceptual understanding of human behaviour to produce an intuitive and efficient collaborative assembly experience. GoferBot is a novel assembly system that seamlessly integrates all sub-modules by utilising implicit semantic information purely from visual perception.

IROS Conference 2022 Conference Paper

Smart Visual Beacons with Asynchronous Optical Communications using Event Cameras

Ziwei Wang 0002
Yonhon Ng
Jack Henderson
Robert E. Mahony

Event cameras are bio-inspired dynamic vision sensors that respond to changes in image intensity with a high temporal resolution, high dynamic range and low latency. These sensor characteristics are ideally suited to enable visual target tracking in concert with a broadcast visual communication channel for smart visual beacons with applications in distributed robotics. Visual beacons can be constructed by high-frequency modulation of Light Emitting Diodes (LEDs) such as vehicle headlights, Internet of Things (IoT) LEDs, smart building lights, etc. , that are already present in many real-world scenarios. The high temporal resolution characteristic of the event cameras allows them to capture visual signals at far higher data rates compared to classical frame-based cameras. In this paper, we propose a novel smart visual beacon architecture with both LED modulation and event camera demodulation algorithms. We quantitatively evaluate the relationship between LED transmission rate, communication distance and the message transmission accuracy for the smart visual beacon communication system that we prototyped. The proposed method achieves up to 4 kbps in an indoor environment and lossless transmission over a distance of 100 meters, at a transmission rate of 500 bps, in full sunlight, demonstrating the potential of the technology in an outdoor environment.

IROS Conference 2021 Conference Paper

A General Approach to State Refinement

Gerard Kennedy
Jin Gao
Zheyu Zhuang
Xin Yu 0002
Robert E. Mahony

Deep learning algorithms such as Convolutional Neural Networks (CNNs) are currently used to solve a range of robotics and computer vision problems. These networks typically estimate the desired representation in a single forward pass and must therefore learn to converge from a wide range of initial conditions to a precise result. This is challenging, and has led to increased interest in the development of separate refinement modules which learn to improve a given initial estimate, thus reducing the required search space. Such modules are usually developed ad-hoc for each given application, often requiring significant engineering investment. In this work we propose a generic innovation-based CNN. Our CNN is implemented along with a stochastic gradient descent (SGD) algorithm to iteratively refine a given initial estimate. The proposed approach provides a general framework for the development of refinement modules applicable to a wide range of robotics problems. We apply this framework to object pose estimation and depth estimation and demonstrate significant improvement over the initial estimates, in the range of 4. 2 - 8. 1%, for both applications.

ICRA Conference 2021 Conference Paper

An Equivariant Filter for Visual Inertial Odometry

Pieter van Goor
Robert E. Mahony

Visual Inertial Odometry (VIO) is of great interest due the ubiquity of devices equipped with both a monocular camera and Inertial Measurement Unit (IMU). Methods based on the extended Kalman Filter remain popular in VIO due to their low memory requirements, CPU usage, and processing time when compared to optimisation-based methods. In this paper, we analyse the VIO problem from a geometric perspective and propose a novel formulation on a smooth quotient manifold where the equivalence relationship is the well-known invariance of VIO to choice of reference frame. We propose a novel Lie group that acts transitively on this manifold and is compatible with the visual measurements. This structure allows for the application of Equivariant Filter (EqF) design leading to a novel filter for the VIO problem. Combined with a very simple vision processing front-end, the proposed filter demonstrates state-of-the-art performance on the EuRoC dataset compared to other EKF-based VIO algorithms.

ICRA Conference 2021 Conference Paper

End-to-end Multi-Instance Robotic Reaching from Monocular Vision

Zheyu Zhuang
Xin Yu 0002
Robert E. Mahony

Multi-instance scenes are especially challenging for end-to-end visuomotor (image-to-control) learning algorithms. "Pipeline" visual servo control algorithms use separate detection, selection and servo stages, allowing algorithms to focus on a single object instance during servo control. End-to-end systems do not have separate detection and selection stages and need to address the visual ambiguities introduced by the presence of an arbitrary number of visually identical or similar objects during servo control. However, end-to-end schemes avoid embedding errors from detection and selection stages in the servo control behaviour, are more dynamically robust to changing scenes and are algorithmically simpler. In this paper, we present a reactive real-time end-to-end visuomotor learning algorithm for multi-instance reaching. The proposed algorithm uses a monocular RGB image and the manipulator’s joint angles as the input to a light-weight fully-convolutional network (FCN) to generate control candidates. A key innovation of the proposed method is identifying the optimal control candidate by regressing a control-Lyapunov function (cLf) value. The multi-instance capability emerges naturally from the stability analysis associated with the cLf formulation. We demonstrate the proposed algorithm effectively reaching and grasping objects from different categories on a table-top amid other instances and distractors from an over-the-shoulder monocular RGB camera. The network is able to run up to ∼160 fps during inference on one GTX 1080 Ti GPU.

IROS Conference 2021 Conference Paper

Stereo Hybrid Event-Frame (SHEF) Cameras for 3D Perception

Ziwei Wang 0002
Liyuan Pan
Yonhon Ng
Zheyu Zhuang
Robert E. Mahony

Stereo camera systems play an important role in robotics applications to perceive the 3D world. However, conventional cameras have drawbacks such as low dynamic range, motion blur and latency due to the underlying frame- based mechanism. Event cameras address these limitations as they report the brightness changes of each pixel independently with a fine temporal resolution, but they are unable to acquire absolute intensity information directly. Although integrated hybrid event-frame sensors (e. g. , DAVIS) are available, the quality of data is compromised by coupling at the pixel level in the circuit fabrication of such cameras. This paper proposes a stereo hybrid event-frame (SHEF) camera system that offers a sensor modality with separate high-quality pure event and pure frame cameras, overcoming the limitations of each separate sensor and allowing for stereo depth estimation. We provide a SHEF dataset targeted at evaluating disparity estimation algorithms and introduce a stereo disparity estimation algorithm that uses edge information extracted from the event stream correlated with the edge detected in the frame data. Our disparity estimation outperforms the state-of-the-art stereo matching algorithm on the SHEF dataset.

ICRA Conference 2020 Conference Paper

Dynamic SLAM: The Need For Speed

Mina Henein
Jun Zhang
Robert E. Mahony
Viorela Ila

The static world assumption is standard in most simultaneous localisation and mapping (SLAM) algorithms. Increased deployment of autonomous systems to unstructured dynamic environments is driving a need to identify moving objects and estimate their velocity in real-time. Most existing SLAM based approaches rely on a database of 3D models of objects or impose significant motion constraints. In this paper, we propose a new feature-based, model-free, object-aware dynamic SLAM algorithm that exploits semantic segmentation to allow estimation of motion of rigid objects in a scene without the need to estimate the object poses or have any prior knowledge of their 3D models. The algorithm generates a map of dynamic and static structure and has the ability to extract velocities of rigid moving objects in the scene. Its performance is demonstrated on simulated, synthetic and real-world datasets.

ICRA Conference 2020 Conference Paper

LyRN (Lyapunov Reaching Network): A Real-Time Closed Loop approach from Monocular Vision

Zheyu Zhuang
Xin Yu 0002
Robert E. Mahony

We propose a closed-loop, multi-instance control algorithm for visually guided reaching based on novel learning principles. A control Lyapunov function methodology is used to design a reaching action for a complex multi-instance task in the case where full state information (poses of all potential reaching points) is available. The proposed algorithm uses monocular vision and manipulator joint angles as the input to a deep convolution neural network to predict the value of the control Lyapunov function (cLf) and corresponding velocity control. The resulting network output is used in real-time as visual control for the grasping task with the multi-instance capability emerging naturally from the design of the control Lyapunov function. We demonstrate the proposed algorithm grasping mugs (textureless and symmetric objects) on a table-top from an over-the-shoulder monocular RGB camera. The manipulator dynamically converges to the best-suited target among multiple identical instances from any random initial pose within the workspace. The system trained with only simulated data is able to achieve 90. 3% grasp success rate in the real-world experiments with up to 85Hz closed-loop control on one GTX 1080Ti GPU and significantly outperforms a Pose-Based-Visual-Servo (PBVS) grasping system adapted from a state-of-the-art single shot RGB 6D pose estimation algorithm. A key contribution of the paper is the inclusion of a first-order differential constraint associated with the cLf as a regularisation term during learning, and we provide evidence that this leads to more robust and reliable reaching/grasping performance than vanilla regression on general control inputs.

IROS Conference 2020 Conference Paper

Robust Ego and Object 6-DoF Motion Estimation and Tracking

Jun Zhang
Mina Henein
Robert E. Mahony
Viorela Ila

The problem of tracking self-motion as well as motion of objects in the scene using information from a camera is known as multi-body visual odometry and is a challenging task. This paper proposes a robust solution to achieve accurate estimation and consistent track-ability for dynamic multi-body visual odometry. A compact and effective framework is proposed leveraging recent advances in semantic instance-level segmentation and accurate optical flow estimation. A novel formulation, jointly optimizing SE(3) motion and optical flow is introduced that improves the quality of the tracked points and the motion estimation accuracy. The proposed approach is evaluated on the virtual KITTI Dataset and tested on the real KITTI Dataset, demonstrating its applicability to autonomous driving applications. For the benefit of the community, we make the source code public †.

IROS Conference 2019 Conference Paper

Learning Real-time Closed Loop Robotic Reaching from Monocular Vision by Exploiting A Control Lyapunov Function Structure

Zheyu Zhuang
Jürgen Leitner
Robert E. Mahony

Visual reaching and grasping is a fundamental problem in robotics research. This paper proposes a novel approach based on deep learning a control Lyapunov function and its derivatives by encouraging a differential constraint in addition to vanilla regression that directly regresses independent joint control inputs. A key advantage of the proposed approach is that an estimate of the value of the control Lyapunov function is available in real-time that can be used to monitor the system performance and provide a level of assurance concerning progress towards the goal. The results we obtain demonstrate that the proposed approach is more robust and more reliable than vanilla regression.

IROS Conference 2018 Conference Paper

Vision Based Forward Sensitive Reactive Control for a Quadrotor VTOL

Jean-Luc Stevens
Robert E. Mahony

Deployment of aerial robotic vehicles for real world tasks such as home deliveries, close range aerial inspection, etc. , require robotic vehicles to fly through complex and cluttered 3D environments such as forests, shrubbery or into balconies, garages, or sheds. Dense high-speed optical flow can provide real-time motion cues for obstacle avoidance that does not require 3D full reconstruction of the environment. However, classical reactive control does not `look ahead' and tends to bounce off obstacles rather than generating a smooth trajectory that anticipates and avoids upcoming obstacles. In this paper, we consider deriving a fully image based control criteria that forward predicts a cylinder of free space into the image flow representation of the environment and steers the vehicle by manoeuvering this cylinder through the upcoming environment. The length and radius of the cylinder provide a guarantee that the vehicle can indeed fly through the space identified and the fact that it is predicted forward into the environment leads to smooth anticipation of upcoming obstacles. Results are obtained for a quadrotor flying autonomously through a forest environment.

ICRA Conference 2017 Conference Paper

3D tracking of water hazards with polarized stereo cameras

Chuong V. Nguyen
Michael Milford
Robert E. Mahony

Current self-driving car systems operate well in sunny weather but struggle in adverse conditions. One of the most commonly encountered adverse conditions involves water on the road caused by rain, sleet, melting snow or flooding. While some advances have been made in using conventional RGB camera and LIDAR technology for detecting water hazards, other sources of information such as polarization offer a promising and potentially superior approach to this problem in terms of performance and cost. In this paper, we present a novel stereo-polarization system for detecting and tracking water hazards based on polarization and color variation of reflected light, with consideration of the effect of polarized light from sky as function of reflection and azimuth angles. To evaluate this system, we present a new large `water on road' datasets spanning approximately 2 km of driving in various on-road and off-road conditions and demonstrate for the first time reliable water detection and tracking over a wide range of realistic car driving water conditions using polarized vision as the primary sensing modality. Our system successfully detects water hazards up to more than 100m. Finally, we discuss several interesting challenges and propose future research directions for further improving robust autonomous car perception in hazardous wet conditions using polarization sensors.

ICRA Conference 2017 Conference Paper

A discrete-time attitude observer on SO(3) for vision and GPS fusion

Alireza Khosravian
Tat-Jun Chin
Ian D. Reid 0001
Robert E. Mahony

This paper proposes a discrete-time geometric attitude observer for fusing monocular vision with GPS velocity measurements. The observer takes the relative transformations obtained from processing monocular images with any visual odometry algorithm and fuses them with GPS velocity measurements. The objectives of this sensor fusion are twofold; first to mitigate the inherent drift of the attitude estimates of the visual odometry, and second, to estimate the orientation directly with respect to the North-East-Down frame. A key contribution of the paper is to present a rigorous stability analysis showing that the attitude estimates of the observer converge exponentially to the true attitude and to provide a lower bound for the convergence rate of the observer. Through experimental studies, we demonstrate that the observer effectively compensates for the inherent drift of the pure monocular vision based attitude estimation and is able to recover the North-East-Down orientation even if it is initialized with a very large attitude error.

IROS Conference 2017 Conference Paper

Exploring the effect of meta-structural information on the global consistency of SLAM

Mina Henein
Montiel Abello
Viorela Ila
Robert E. Mahony

Accurate online estimation of the environment structure simultaneously with the robot pose is a key capability for autonomous robotic vehicles. Classical simultaneous localization and mapping (SLAM) algorithms make no assumptions about the configuration of the points in the environment, however, real world scenes have significant structure (ground planes, buildings, walls, ceilings, etc.) that can be exploited. In this paper, we introduce meta-structural information associated with geometric primitives into the estimation problem and analyze their effect on the global structural consistency of the resulting map. Although we only consider the effect of adding planar and orthogonality information for the estimation of 3D points in a Manhattan-like world, this framework can be extended to any type of geometric, kinematic, dynamic or even semantic information. We evaluate our approach on a city-like simulated environment. We highlight the advantages of the proposed solution over SLAM formulation considering no prior knowledge about the configuration of 3D points in the environment.

ICRA Conference 2016 Conference Paper

Velocity aided attitude estimation for aerial robotic vehicles using latent rotation scaling

Guillaume Allibert
Robert E. Mahony
Moses Bangura

Flight performance of aerial robotic vehicles is critically dependent on the quality of the state estimates provided by onboard sensor systems. The attitude estimation problem has been extensively studied over the last ten years and the development of low complexity, high performance, robust non-linear observers for attitude has been one of the enabling technologies fueling the growth of small scale aerial robotic systems. The velocity aided attitude estimation problem, that is simultaneous estimation of attitude and linear velocity of an aerial platform, has only been tackled using the non-linear observer approach in the last few years. Prior contributions have lead to non-linear observers for which either there is no stability analysis or for which the analysis is extremely complex. In this paper, we propose a simple relaxation of the state space, allowing scaled rotation matrices R ∈ ℝ 3×3 such that RX T = uI where X = uR̂ and u > 0 is a positive scalar, along with additional observer dynamics to force u → 1 asymptotically. With this simple augmentation of the observer state space, we propose a non-linear observer with a straightforward Lyapunov stability analysis that demonstrates almost global asymptotic convergence along with local exponential convergence. Simulations as well as experimental results are provided to demonstrate the performance of the proposed observer.

ICRA Conference 2014 Conference Paper

Aerodynamic power control for multirotor aerial vehicles

Moses Bangura
Hyon Lim
H. Jin Kim
Robert E. Mahony

In this paper, a new motor control input and controller for small-scale electrically powered multirotor aerial vehicles is proposed. The proposed scheme is based on controlling aerodynamic power as opposed to the rotor speed of each motor-rotor system. Electrical properties of the brushless direct current motor are used to both estimate and control the mechanical power of the motor system which is coupled with aerodynamic power using momentum theory analysis. In comparison to current state-of-the-art motor control for multirotor aerial vehicles, the proposed approach is robust to unmodelled aerodynamic effects such as wind disturbances and ground effects. Theory and experimental results are presented to illustrate the performance of the proposed motor control.

ICRA Conference 2014 Conference Paper

An intuitive multimodal haptic interface for teleoperation of aerial robots

Xiaolei Hou
Robert E. Mahony

This paper presents a novel intuitive multi-modal force feedback interface for teleoperation of mobile robotic vehicles. Two different force feedback interfaces are considered: a force feedback joystick and a novel force feedback trackball. The joystick considered is based on the admittance user interface developed by the authors in earlier work and is configured to servo velocity of the vehicle. The force feedback trackball is configured to map vehicle velocity directly to trackball velocity, exploiting the effectively infinite workspace of the trackball to overcome the classical challenge of servo controlling a slave with infinite workspace using a master device with finite workspace. A key contribution of the paper is to provide a modeling framework, based on the bond graph formalism, that allows the energy consistent modeling of input from an admittance joystick as reference to an internal velocity regulation loop for the vehicle. Once this is implemented it is straightforward to interconnect multiple input devices, and in particular the trackball device, using standard interconnection rules in bond graphs. Experiments were performed, and the outcomes verify the feasibility and effectiveness of the proposed interface.

IROS Conference 2013 Conference Paper

Dynamic kinesthetic boundary for haptic teleoperation of aerial robotic vehicles

Xiaolei Hou
Robert E. Mahony

This paper introduces a novel dynamic kinesthetic boundary to aid a pilot to navigate an aerial robotic vehicle through a cluttered environment. Classical haptic teleoperation interfaces for aerial vehicles utilize force feedback to provide the pilot with a haptic feel of the robot's interaction with an environment. The proposed approach constructs a dynamic kinesthetic boundary on the master device that provides the pilot with hard boundaries in the haptic workspace to indicate approaching obstacles. An advantage of the proposed approach is that when the vehicle is flying free of obstacles, then the haptic feedback of the joystick can be used to provide a more natural feel of the vehicle dynamics. Furthermore, rather than a gradual onset of virtual potential forces that are felt in the classical approaches, a pilot encountering the dynamic kinesthetic boundary is immediately aware of the presence of the obstacle and can act accordingly. The approach is implemented on an admittance haptic joystick to ensure that the haptic boundaries are faithfully rendered. We prove that in the case of perfect velocity tracking, the proposed algorithm will ensure the vehicle never colliding with the environment. Experiments were conducted on a robotic platform and the results provide verification of the novel approach.

ICRA Conference 2013 Conference Paper

Image-based visual navigation for mobile robots

Liam O'Sullivan
Peter Corke
Robert E. Mahony

We introduce a new image-based visual navigation algorithm that allows the Cartesian velocity of a robot to be defined with respect to a set of visually observed features corresponding to previously unseen and unmapped world points. The technique is well suited to mobile robot tasks such as moving along a road or flying over the ground. We describe the algorithm in general form and present detailed simulation results for an aerial robot scenario using a spherical camera and a wide angle perspective camera, and present experimental results for a mobile ground robot.

IROS Conference 2013 Conference Paper

Intercontinental haptic teleoperation of a flying vehicle: A step towards real-time applications

Abeje Y. Mersha
Xiaolei Hou
Robert E. Mahony
Stefano Stramigioli
Peter Corke
Raffaella Carloni

This paper describes the theory and practice for a stable haptic teleoperation of a flying vehicle. It extends passivity-based control framework for haptic teleoperation of aerial vehicles in the longest intercontinental setting that presents great challenges. The practicality of the control architecture has been shown in maneuvering and obstacle-avoidance tasks over the internet with the presence of significant time-varying delays and packet losses. Experimental results are presented for teleoperation of a slave quadrotor in Australia from a master station in the Netherlands. The results show that the remote operator is able to safely maneuver the flying vehicle through a structure using haptic feedback of the state of the slave and the perceived obstacles.

ICRA Conference 2013 Conference Paper

Representation of vehicle dynamics in haptic teleoperation of aerial robots

Xiaolei Hou
Robert E. Mahony
Felix Schill

This paper considers the question of providing effective feedback of vehicle dynamic forces to a pilot in haptic teleoperation of aerial robots. We claim that the usual state-of-the-art haptic interface, based on research motivated by robotic manipulator slaves and virtual haptic environments, does a poor job of reflecting dynamic forces of a mobile robotic vehicle to the user. This leads us to propose a novel force feedback user interface for mobile robotic vehicles with dynamics. An analysis of the closed-loop force-displacement transfer functions experienced by the master joystick for the classical and the new approach clearly indicate the advantages of the proposed formulation. Both the classical and the proposed approach have been implemented in the teleoperation of a quadrotor vehicle and we present quantitative and cognitive performance data from a user study that corroborates the expected performance advantages.

IROS Conference 2013 Conference Paper

Visuo-inertial fusion for homography-based filtering and estimation

Alexandre Eudes
Pascal Morin
Robert E. Mahony
Tarek Hamel

The paper concerns visuo-inertial filtering and estimation based on homography and angular velocity measurements, i. e. data obtained from a mono-camera/IMU sensor. We extend recently developed nonlinear filters on the special linear group of homographies to the estimation of scene parameters and velocity of the sensor. A validation of the proposed solution and a comparative evaluation based on real data is presented.

ICRA Conference 2011 Conference Paper

A visual servoing model for generalised cameras: Case study of non-overlapping cameras

Andrew I. Comport
Robert E. Mahony
Fabien Spindler

This paper proposes an adaptation of classical image-based visual servoing to a generalised imaging model where cameras are modelled as sets of 3D viewing rays. This new model leads to a generalised visual servoing control formalism that can be applied to any type of imaging system whether it be multi-camera, catadioptric, non-central, etc. In this paper the generalised 3D viewing cones are parameterised geometrically via Plücker line coordinates. The new visual servoing model is tested on an a-typical stereo-camera system with non-overlapping cameras. In this case no 3D information is available from triangulation and the system is comparable to a 2D visual servoing system with non-central ray-based control law.

ICRA Conference 2011 Conference Paper

Aerial SLAM with a single camera using visual expectation

Michael Milford
Felix Schill
Peter Corke
Robert E. Mahony
Gordon F. Wyeth

Micro aerial vehicles (MAVs) are a rapidly growing area of research and development in robotics. For autonomous robot operations, localization has typically been calculated using GPS, external camera arrays, or onboard range or vision sensing. In cluttered indoor or outdoor environments, onboard sensing is the only viable option. In this paper we present an appearance-based approach to visual SLAM on a flying MAV using only low quality vision. Our approach consists of a visual place recognition algorithm that operates on 1000 pixel images, a lightweight visual odometry algorithm, and a visual expectation algorithm that improves the recall of place sequences and the precision with which they are recalled as the robot flies along a similar path. Using data gathered from outdoor datasets, we show that the system is able to perform visual recognition with low quality, intermittent visual sensory data. By combining the visual algorithms with the RatSLAM system, we also demonstrate how the algorithms enable successful SLAM.

ICRA Conference 2010 Conference Paper

A general optical flow based terrain-following strategy for a VTOL UAV using multiple views

Bruno Hérissé
Sophie Oustrieres
Tarek Hamel
Robert E. Mahony
François-Xavier Russotto

This paper presents a general approach for terrain following (including obstacle avoidance) of a vertical take-off and landing vehicle (VTOL) using multiple observation points. The VTOL vehicle is assumed to be a rigid body, equipped with a minimum sensor suite (camera, IMU and barometric altimeter), manoeuvering over a textured rough terrain made of a concatenation of planar surfaces. Assuming that the forward velocity is separately regulated to a desired non-zero value, the proposed control approach ensures terrain following and guarantees the vehicle does not collide with obstacles during the task. The proposed control acquires an optical flow from multiple spatially separate observation points, typically obtained via multiple cameras or non collinear directions in a unique camera. The proposed control algorithm has been tested extensively in simulation and then implemented on a quadrotor UAV to demonstrate the performance of the closed loop system.

ICRA Conference 2010 Conference Paper

A novel approach to haptic tele-operation of aerial robot vehicles

Stefano Stramigioli
Robert E. Mahony
Peter Corke

We present a novel, simple and effective approach for tele-operation of aerial robotic vehicles with haptic feedback. Such feedback provides the remote pilot with an intuitive feel of the robot's state and perceived local environment that will ensure simple and safe operation in cluttered 3D environments common in inspection and surveillance tasks. Our approach is based on energetic considerations and uses the concepts of network theory and port-Hamiltonian systems. We provide a general framework for addressing problems such as mapping the limited stroke of a `master' joystick to the infinite stroke of a `slave' vehicle, while preserving passivity of the closed-loop system in the face of potential time delays in communications links and limited sensor data.

IROS Conference 2010 Conference Paper

The landing problem of a VTOL Unmanned Aerial Vehicle on a moving platform using optical flow

Bruno Hérissé
Tarek Hamel
Robert E. Mahony
François-Xavier Russotto

This paper presents a nonlinear controller for hovering flight and landing control on a moving platform for a Vertical Take-off and Landing (VTOL) Unmanned Aerial Vehicle (UAV) by exploiting the measurement of the average optical flow. The VTOL vehicle is assumed to be equipped with a minimum sensor suite (a camera and an IMU), manoeuvring over a textured flat target plane. Two different tasks are considered in this paper: the first one concerns the stability of hovering flight and the second one concerns regulation of automatic vertical landing on a moving platform using the divergent optical flow as feedback information. Simulation and experimental results performed on a quad-rotor UAV demonstrate the performance of the proposed control strategy.

ICRA Conference 2009 Conference Paper

A new framework for force feedback teleoperation of robotic vehicles based on optical flow

Robert E. Mahony
Felix Schill
Peter Corke
Yoong Siang Oh

This paper proposes the use of optical flow from a moving robot to provide force feedback to an operator's joystick to facilitate collision free teleoperation. Optic flow is measured by wide angle cameras on board the vehicle and used to generate a virtual environmental force that is reflected to the user through the joystick, as well as feeding back into the control of the vehicle. The coupling between optical flow (velocity) and force is modelled as an impedance - in this case an optical impedance. We show that the proposed control is dissipative and prevents the vehicle colliding with the environment as well as providing the operator with a natural feel for the remote environment. The paper focuses on applications to aerial robotics vehicles, however, the ideas apply directly to other force actuated vehicles such as submersibles or space vehicles, and the authors believe the approach has potential for control of terrestrial vehicles and even teleoperation of manipulators. Experimental results are provided for a simulated aerial robot in a virtual environment controlled by a haptic joystick.

ICRA Conference 2009 Conference Paper

A nonlinear observer for 6 DOF pose estimation from inertial and bearing measurements

Grant Baldwin
Robert E. Mahony
Jochen Trumpf

This paper considers the problem of estimating pose from inertial and bearing-only vision measurements. We present a non-linear observer that evolves directly on the special Euclidean group SE(3) from inertial measurements and bearing measurements, such as provided by a visual system tracking known landmarks. Local asymptotic convergence of the observer is proved. The observer is computationally simple and its gains are easy to tune. Simulation results demonstrate robustness to measurement noise and initial conditions.

ICRA Conference 2009 Conference Paper

A nonlinear terrain-following controller for a VTOL unmanned aerial vehicle using translational optical flow

Bruno Hérissé
Tarek Hamel
Robert E. Mahony
François-Xavier Russotto

This paper presents a nonlinear controller for terrain following of a vertical take-off and landing vehicle (VTOL). The VTOL vehicle is assumed to be a rigid body, equipped with a minimum sensor suite (camera and IMU) along with a measure of the forward speed from another sensor such as global positioning system, maneuvering over a textured terrain made of planar surfaces. Assuming that the forward velocity is separately set to a desired value, the proposed control approach ensures terrain following and guaranties the vehicle does not collide with the ground during the task. The proposed control acquires an optical flow from three spatially separate observation points, typically obtained via three cameras or three non collinear directions in a unique camera. The proposed control algorithm has been tested extensively in simulation and then implemented on a quadrotor UAV to demonstrate the performance of the closed loop system.

ICRA Conference 2009 Conference Paper

Design principles of large quadrotors for practical applications

Pauline Pounds
Robert E. Mahony

Virtually all quadrotors used in research weigh less than 2 kg, and carry payload measured in hundreds of grams. To be useful platforms for expanded operations, these vehicles must be capable of carrying greater weight. Several obstacles in aerodynamics, design and control must be overcome to enable the construction of larger craft with payloads in excess of 1 kg. We report the key design considerations essential for the construction of heavy quadrotor MAVs and demonstrate a 4 kg quadrotor with 1 kg payload.

ICRA Conference 2009 Conference Paper

Dynamic estimation of homography transformations on the special linear group for visual servo control

Ezio Malis
Tarek Hamel
Robert E. Mahony
Pascal Morin

In the last decade, many vision-based robot controllers have been designed using Cartesian information encoded in the homography transformation that links two images of a planar object. For any approach, the performance of the closed-loop system depends on the quality of the homography estimates obtained. In this paper, we exploit the special linear Lie-group structure of the set of all homographies to develop a dynamic observer to estimate homographies online. The resulting estimates are effective and can be used to improve closed-loop response of several visual servoing algorithms.

IROS Conference 2008 Conference Paper

A complementary filter for attitude estimation of a fixed-wing UAV

Mark Euston
Paul William Coote
Robert E. Mahony
Jonghyuk Kim
Tarek Hamel

This paper considers the question of using a nonlinear complementary filter for attitude estimation of fixed-wing unmanned aerial vehicle (UAV) given only measurements from a low-cost inertial measurement unit. A nonlinear complementary filter is proposed that combines accelerometer output for low frequency attitude estimation with integrated gyrometer output for high frequency estimation. The raw accelerometer output includes a component corresponding to airframe acceleration, occurring primarily when the aircraft turns, as well as the gravitational acceleration that is required for the filter. The airframe acceleration is estimated using a simple centripetal force model (based on additional airspeed measurements), augmented by a first order dynamic model for angle-of-attack, and used to obtain estimates of the gravitational direction independent of the airplane manoeuvres. Experimental results are provided on a real-world data set and the performance of the filter is evaluated against the output from a full GPS/INS that was available for the data set.

IROS Conference 2008 Conference Paper

Hovering flight and vertical landing control of a VTOL Unmanned Aerial Vehicle using optical flow

Bruno Hérissé
François-Xavier Russotto
Tarek Hamel
Robert E. Mahony

This paper presents a nonlinear controller for hovering flight and touchdown control for a Vertical Take-off and Landing (VTOL) Unmanned Aerial Vehicle (UAV) using inertial optical flow. The VTOL vehicle is assumed to be a rigid body, equipped with a minimum sensor suite (camera and IMU), manoeuvring over a textured flat target plane. Two different tasks are considered in this paper: the first concerns the stability of hovering flight and the second one concerns regulation of automatic landing using the divergent optical flow as feedback information. Experimental results on a quad-rotor UAV demonstrate the performance of the proposed control strategy.

ICRA Conference 2007 Conference Paper

A practical Visual Servo Control for a Unmanned Aerial Vehicle

Nicolas Guenard
Tarek Hamel
Robert E. Mahony

An image-based visual servo control is presented for an unmanned aerial vehicle (UAV) capable of stationary or quasi-stationary flight. The proposed control design addresses visual servo of 'eye-in-hand' type systems. The control of the position and orientation dynamics are decoupled using a visual error based on a spherical centroid data, along with estimation of the gravitational inertial direction. The error used compensates for the poor conditioning of the Jacobian matrix seen in earlier work in this area by introducing a non-homogeneous gain term adapted to the visual sensitivity of the error measurements. A nonlinear controller is derived for the full dynamics of the system. Experimental results on an experimental UAV known as an X4-flyer made by the French Atomic Energy Commission (CEA) demonstrate the robustness and performances of the proposed control strategy.

ICRA Conference 2007 Conference Paper

Robust Nonlinear Fusion of Inertial and Visual Data for position, velocity and attitude estimation of UAV

Thibault Cheviron
Tarek Hamel
Robert E. Mahony
Grant Baldwin

This paper presents a coupled observer that uses accelerometer, gyrometer and vision sensors to provide estimates of pose and linear velocity for an aerial robotic vehicle. The observer is based on a non-linear complimentary filter framework and incorporates adaptive estimates of measurement bias in gyrometers and accelerometers commonly encountered in low-cost inertial measurement systems. Asymptotic stability of the observer estimates is proved as well as bounded energy of the observer error signals. Experimental data is provided for the proposed filter run on data obtained from an experiment involving a remotely controlled helicopter.

ICRA Conference 2006 Conference Paper

A Hierarchical Control Strategy for the Autonomous Navigation of a Ducted Fan Flying Robot

Jean Michel Pflimlin
Tarek Hamel
Philippe Souères
Robert E. Mahony

This paper describes a control strategy to stabilize the position of a vertical takeoff and landing (VTOL) unmanned aerial vehicle (UAV) in wind gusts. The proposed approach takes advantage of the cascade structure of the system to design a hierarchical controller. The idea is to separate the controller in a high level controller devoted to position control and a low level controller devoted to stabilization and attitude control. Both controllers are designed by means of backstepping techniques that allow the stabilization of the vehicle's position while on-line estimation of the unknown aerodynamic forces. The global stability of the connected system is proven, and simulations as well as experimental results are presented

ICRA Conference 2006 Conference Paper

Attitude Estimation on SO[3] based on Direct Inertial Measurements

Tarek Hamel
Robert E. Mahony

This paper considers the question of obtaining high quality attitude estimates from typical low cost inertial measurement units for applications in control of unmanned aerial vehicles. A nonlinear complimentary filter exploiting the structure of special orthogonal group S0(3) is proposed. The filter is expressed explicitly in terms of direct and untreated measurements. For a typical low cost inertial measurement where two inertial directions are measured (gravitational and magnetic fields) along with angular velocity, it is shown that the filter is well conditioned. If only a single direction is available (typically the gravitational field) along with angular velocity, it is shown that the full gyro bias vector is correctly estimated and that the estimated orientation converges to a set consistent with the measurements. Experimental results, for flight data from the HoverEyecopy UAV, demonstrate the efficiency of the proposed filter

ICRA Conference 2006 Conference Paper

Bounded Torque Control for Robot Manipulators Subject to Joint Velocity Constraints

Khoi B. Ngo
Robert E. Mahony

This paper presents a bounded torque control design to solve the set-point regulation problem for robot manipulators subject to joint velocity constraints. The control objectives are achieved by exploiting the passivity properties of the system and utilizing barrier function ideas to reshape the control Lyapunov function. The structure of the modified control Lyapunov function is reminiscent of those used in the artificial potential field method. The resulting controllers are modified proportional-derivative controllers which are simple, intuitive, and can easily be implemented in practice. In addition, asymptotic stability of the closed-loop system is guaranteed, all joint velocity constraints are strictly satisfied for all time, and the demanded torque input is bounded in norm, irrespective of the initial condition. The effectiveness of the proposed control design is demonstrated through simulations on a 2-link planar manipulator

IROS Conference 2006 Conference Paper

Stability and performance of image based visual servo control using first order spherical image moments

Odile Bourquardez
Robert E. Mahony
Tarek Hamel
François Chaumette

Image moments provide an important class of image features used for image-based visual servo control. Spherical image moments have the additional desirable property that they are invariant under rotation of the camera frame. For these features one can study the local and global stability and performance of the position control independently of the rotation control. In this paper we study a range of control algorithms including the classical approximately linearising control, a recently proposed robust control based on Lyapunov function design methodology, and modifications of these designs to improve global and asymptotic performance and robustness of the schemes. The comparison of performance demonstrates that the choice of image feature and control design for image-based visual servo are each equally important and highly coupled. We finally propose a control law using a modified image feature and a Lyapunov control design that ensures global asymptotic stability and good performance equally in image space as in task space

ICRA Conference 2004 Conference Paper

Pure 2D Visual Servo Control for a Class of Under-actuated Dynamic Systems

Tarek Hamel
Robert E. Mahony

A pure image-based strategy for visual servo control of a class of dynamic systems is proposed. The proposed design concerns the dynamics of unmanned aerial vehicles capable of quasi-stationary flight (hover and near hover flight). The visual servo control task considered is to locate the camera relative to a stationary target. The paper extends earlier work, by weakening the assumption on the construction of the visual error used. In prior work some inertial information was used in the error construction to guarantee the passivity properties of the control design. In this paper the visual error is defined purely in terms of the 2D image features derived from, the camera signal.

IROS Conference 2003 Conference Paper

A bearing-only control law for stable docking of unicycles

Ran Wei
Robert E. Mahony
David Austin

This paper proposes a new control method for stabilising control design for docking unicycle-like vehicles based on bearing-only information. An omni-directional panoramic camera is used to detect visual targets around the docking station and provides bearing (or heading) data for each observed landmark. The convergence of the controlled system is fully analysed and simulations are provided to demonstrate the ideal behaviour of the system. A robust and computationally cheap blob detection algorithm is proposed and results are provided to demonstrate its performance in extracting targets from cluttered scenes. Experimental results are presented demonstrating the performance of the algorithm on the ANU Nomadic Technologies Nomad XR4000 robot.

ICRA Conference 2002 Conference Paper

A Decoupled Image Space Approach to Visual Servo Control of a Robotic Manipulator

Robert E. Mahony
Tarek Hamel
François Chaumette

An image-based visual servo control is presented for a robotic manipulator. The proposed control design addresses visual servo of 'eye-in-hand' type systems. Using a novel representation of the visual error based on a spherical representation of target centroid information along with a measure of the rotation between the camera and the target, the control of the position and orientation is decoupled. A non-linear gain introduced into the orientation feedback kinematics prevents the target image from leaving the visual field. Semi-global convergence of the closed loop system is proved.

IROS Conference 2002 Conference Paper

Choice of image features for depth-axis control in image based visual servo control

Robert E. Mahony
Peter Corke
François Chaumette

This paper is concerned with choosing image features for image based visual servo control and how this choice influences the closed-loop dynamics of the system. In prior work, image features tend to be chosen on the basis of image processing simplicity and noise sensitivity. In this paper we show that the choice of feature directly influences the closed-loop dynamics in task-space. We focus on the depth axis control of a visual servo system and compare analytically various approaches that have been reported recently in the literature. The theoretical predictions axe verified by experiment.

ICRA Conference 2002 Conference Paper

Control of a Quadrotor Helicopter using Visual Feedback

Erdinç Altug
Jim Ostrowski 0001
Robert E. Mahony

We present control methods for an autonomous four-rotor helicopter, called a quadrotor, using visual feedback as the primary sensor. The vision system uses aground camera to estimate the pose (position and orientation) of the helicopter. Two methods of control are studied - one using a series of mode-based, feedback linearizing controllers, and the other using a backstepping-like control law. Various simulations of the model demonstrate the implementation of feedback linearization and the backstepping controllers. Finally, . we present initial flight experiments. where the helicopter is restricted to vertical and yaw motions.

ICRA Conference 2002 Conference Paper

Visual Servo Trajectory Tracking for a Four Rotor VTOL Aerial Vehicle

Tarek Hamel
Robert E. Mahony
Abdelhamid Chriette

An image-based visual servo control design is proposed for a four rotor vertical take-off and landing (VTOL) craft, known as an X4-flyer, accomplishing a trajectory tracking task. The approach taken is an image-based visual servo design that is applicable to under-actuated dynamic systems. The work is an extension of the authors earlier work (2000) to the trajectory tracking problem. Semiglobal stability of the closed loop system is proved for bounded trajectories.

JMLR Journal 2001 Journal Article

Prior Knowledge and Preferential Structures in Gradient Descent Learning Algorithms

Robert E. Mahony
Robert C. Williamson

A family of gradient descent algorithms for learning linear functions in an online setting is considered. The family includes the classical LMS algorithm as well as new variants such as the Exponentiated Gradient (EG) algorithm due to Kivinen and Warmuth. The algorithms are based on prior distributions defined on the weight space. Techniques from differential geometry are used to develop the algorithms as gradient descent iterations with respect to the natural gradient in the Riemannian structure induced by the prior distribution. The proposed framework subsumes the notion of "link-functions".

ICRA Conference 2001 Conference Paper

Visual Servoing For A Scale Model Autonomous Helicopter

Abdelhamid Chriette
Tarek Hamel
Robert E. Mahony

A control design to stabilise a reduced scale autonomous helicopter equipped with a camera is presented. The proposed algorithm is motivated by recent work in image-based visual servoing control for under-actuated dynamic systems. This work is extended by considering models that contain weakly nonminimum phase zero dynamics. This is an important class of systems since an offset between the camera and the centre of mass for a typical 'dynamic' autonomous vehicle will result in zero dynamics occurring in the image dynamics. In this paper we propose a simplified model of dynamics of the helicopter and show that by placing some constraints in the choice of the position of the camera we can minimise the effect of the zero dynamics.

IROS Conference 2001 Conference Paper

Visual servoing using linear features for under-actuated rigid body dynamics

Robert E. Mahony
Tarek Hamel

An image-based 'eye-in-hand' type visual servoing control design is presented for under-actuated rigid body dynamics. The dynamic model considered is often used to model the dynamics of unmanned aerial vehicles (UAV) such as helicopters and aeroplanes. The task considered is that of tracking parallel linear visual features. The proposed design exploits the geometry of the task considered and passivity-like properties of rigid body dynamics to derive a Lyapunov control algorithm using backstepping design techniques.

ICRA Conference 2000 Conference Paper

(Almost) Exact Path Tracking Control for an Autonomous Helicopter in Hover Manoeuvres

Robert E. Mahony
Rogelio Lozano

A simplified model of the dynamics of an autonomous model helicopter valid for manoeuvres close to hover is considered. The control task is to track a given path in the position co-ordinates as accurately as possible. This is made difficult since the model contains weakly non-minimum phase zero dynamics which cannot be stably inverted. By placing constraints on the physical weight distribution associated with the necessary sensor and computer components that must be carried by the autonomous unit a new representation the dynamics of the system in block pure feedback form is derived. The exact tracking problem is solved using input/output linearization techniques.