Author name cluster

Junaed Sattar

Possible papers associated with this exact author name in Arrow. This page groups case-insensitive exact name matches and is not a full identity disambiguation profile.

42 papers

1 author row

IROS Conference 2025 Conference Paper

Design and Development of the MeCO Open-Source Autonomous Underwater Vehicle

David Widhalm
Cory Ohnsted
Corey Knutson
Demetrious T. Kutzke
Sakshi Singh
Rishi Mukherjee
Grant Schwidder
Ying-Kun Wu

We present MeCO, the Medium Cost Open-source autonomous underwater vehicle (AUV), a versatile autonomous vehicle designed to support research and development in underwater human-robot interaction (UHRI) and marine robotics in general. An inexpensive platform to build compared to similarly-capable AUVs, the MeCO design and software are released under open-source licenses, making it a cost effective, extensible, and open platform. It is equipped with UHRI-focused systems, such as front and side facing displays, light-based communication devices, a transducer for acoustic interaction, and stereo vision, in addition to typical AUV sensing and actuation components. Additionally, MeCO is capable of real-time deep learning inference using the latest edge computing devices, while maintaining low-latency, closed-loop control through high-performance microcontrollers. MeCO is designed from the ground up for modularity in internal electronics, external payloads, and software architecture, exploiting open-source robotics and containerarization tools. We demonstrate the diverse capabilities of MeCO through simulated, closed-water, and open-water experiments. All resources necessary to build and run MeCO, including software and hardware design, have been made publicly available.

ICRA Conference 2025 Conference Paper

IBURD: Image Blending for Underwater Robotic Detection

Jungseok Hong
Sakshi Singh
Junaed Sattar

We present an image blending pipeline, IBURD, that creates realistic synthetic images to assist in the training of deep detectors for use on underwater autonomous vehicles (AUVs) for marine debris detection tasks. Specifically, IBURD generates both images of underwater debris and their pixel-level annotations, using source images of debris objects, their annotations, and target background images of marine environments. With Poisson editing and style transfer techniques, IBURD is even able to robustly blend transparent objects into arbitrary backgrounds and automatically adjust the style of blended images using the blurriness metric of target background images. These generated images of marine debris in actual underwater backgrounds address the data scarcity and data variety problems faced by deep-learned vision algorithms in challenging underwater conditions, and can enable the use of AUVs for environmental cleanup missions. Both quantitative and robotic evaluations of IBURD demonstrate the efficacy of the proposed approach for robotic detection of marine debris.

IROS Conference 2025 Conference Paper

One-Shot Gesture Recognition for Underwater Diver-To-Robot Communication *

Rishikesh Joshi
Junaed Sattar

Reliable human-robot communication is essential for effective underwater human-robot interaction (U-HRI), yet traditional methods such as acoustic signaling and predefined gesture-based models suffer from limitations in adaptability and robustness. In this work, we propose One-Shot Gesture Recognition (OSG), a novel method that enables real-time, pose-based, temporal gesture recognition underwater from a single demonstration, eliminating the need for extensive dataset collection or model retraining. OSG leverages shape-based classification techniques, including Hu moments, Zernike moments, and Fourier descriptors, to robustly recognize gestures in visually-challenging underwater environments. Our system achieves high accuracy on real-world underwater video data and operates efficiently on embedded hardware commonly found on autonomous underwater robots (AUVs), demonstrating its feasibility for deployment on-board robots. Compared to deep learning approaches, OSG is lightweight, computationally efficient, and highly adaptable, making it ideal for diver-to-robot communication. We evaluate OSG’s performance on an augmented gesture dataset and real-world underwater video data, comparing its accuracy against deep learning methods. Our results show OSG’s potential to enhance U-HRI by enabling the immediate deployment of user-defined gestures without the constraints of predefined gesture languages.

IROS Conference 2025 Conference Paper

Semantic Risk Assessment in Visual Scenes for AUV-Assisted Marine Debris Removal

Sakshi Singh
Junaed Sattar

Underwater debris is a significantly growing challenge that autonomous underwater vehicles (AUVs) can help alleviate, but robot-guided debris search and removal can also cause harm to the aquatic ecosystem or other humans engaged in cleanup missions if the AUVs are unable to assess the risks associated with its actions. We introduce a method for identifying such risks in an underwater scene in the context of AUV debris search and removal tasks. Our approach integrates a vision language model (VLM) with monocular depth estimation to effectively classify and localize objects in a marine scene, specifically submerged marine debris. We use the pixel distance and depth difference using the monocular depth map to identify entities that are sensitive to harm in proximity to the debris. We collect and annotate a custom dataset containing images in three different marine and aquatic environments containing debris and other such sensitive entities, and compare classification performance for different types of prompts. We observe that the prompts describing the debris properties (e. g. , "eroded trash") demonstrate a significant increase in accuracy compared to the use of object names directly as prompts. Our method successfully identifies debris that is safe to remove in complex scenes and turbid water conditions, highlighting the potential of using VLMs for risk assessment in AUV operations in the diverse underwater domain.

IROS Conference 2025 Conference Paper

The Common Objects Underwater (COU) Dataset for Robust Underwater Object Detection

Rishi Mukherjee
Sakshi Singh
Jack McWilliams
Junaed Sattar

We introduce COU: Common Objects Underwater, an instance-segmented image dataset of commonly found man-made objects in multiple aquatic and marine environments. COU contains approximately 10K segmented images, annotated from images collected during a number of underwater robot field trials in diverse locations. COU is created to address the lack of datasets with robust class coverage curated for underwater instance segmentation, which is particularly useful for training light-weight, real-time capable detectors for Autonomous Underwater Vehicles (AUVs). In addition, COU addresses the lack of diversity in object classes since the commonly available underwater image datasets focus only on marine life. Currently, COU contains images from both closed-water (pool) and open-water (lakes and oceans) environments, of 24 different classes of objects including marine debris, dive tools, and AUVs. To assess the efficacy of COU in training underwater object detectors, we use three state-of-the-art models to evaluate its performance and accuracy, using a combination of standard accuracy and efficiency metrics. The improved performance of COU-trained detectors over those solely trained on terrestrial data demonstrates the clear advantage of training with annotated underwater images. We make COU available for broad use under open-source licenses.

ICRA Conference 2024 Conference Paper

Adaptive Landmark Color for AUV Docking in Visually Dynamic Environments

Corey Knutson
Zhipeng Cao
Junaed Sattar

Autonomous Underwater Vehicles (AUVs) conduct missions underwater without the need for human intervention. A docking station (DS) can extend mission times of an AUV by providing a location for the AUV to recharge its batteries and receive updated mission information. Various methods for locating and tracking a DS exist, but most rely on expensive acoustic sensors, or are vision-based, which is significantly affected by water quality. In this paper, we present a vision-based method that utilizes adaptive color LED markers and dynamic color filtering to maximize landmark visibility in varying water conditions. Both AUV and DS utilize cameras to determine the water background color in order to calculate the desired marker color. No communication between AUV and DS is needed to determine marker color. Experiments conducted in a pool and lake show our method performs 10 times better than static color thresholding methods as background color varies. DS detection is possible at a range of 5 meters in clear water with minimal false positives.

ICRA Conference 2024 Conference Paper

Autonomous robotic re-alignment for face-to-face underwater human-robot interaction

Demetrious T. Kutzke
Ashwin Wariar
Junaed Sattar

The use of autonomous underwater vehicles (AUVs) to accomplish traditionally challenging and dangerous tasks has proliferated thanks to advances in sensing, navigation, manipulation, and on-board computing technologies. Utilizing AUVs in underwater human-robot interaction (UHRI) has witnessed comparatively smaller levels of growth due to limitations in bi-directional communication and significant technical hurdles to bridge the gap between analogies with terrestrial interaction strategies and those that are possible in the underwater domain. A necessary component to support UHRI is establishing a system for safe robotic-diver approach to establish face-to-face communication that considers nonstandard human body pose. In this work, we introduce a stereo vision system for enhancing UHRI that utilizes threedimensional reconstruction from stereo image pairs and machine learning for localizing human joint estimates. We then establish a convention for a coordinate system that encodes the direction the human is facing with respect to the camera coordinate frame. This allows automatic setpoint computation that preserves human body scale and can be used as input to an image-based visual servo control scheme. We show that our setpoint computations tend to agree both quantitatively and qualitatively with experimental setpoint baselines. The methodology introduced shows promise for enhancing UHRI by improving robotic perception of human orientation underwater.

ICRA Conference 2023 Conference Paper

Diver Interest via Pointing: Human-Directed Object Inspection for AUVs

Chelsey Edge
Junaed Sattar

In this paper, we present the Diver Interest via Pointing (DIP) algorithm, a highly modular method for conveying a diver's area of interest to an autonomous underwater vehicle (AUV) using pointing gestures for underwater humanrobot collaborative tasks. DIP uses a single monocular camera and exploits human body pose, even with complete dive gear, to extract underwater human pointing gesture poses and their directions. By extracting 2D scene geometry based on the human body pose and density of salient feature points along the direction of pointing, using a low-level feature detector, the DIP algorithm is able to locate objects of interest as indicated by the diver. DIP makes it possible for scuba divers and swimmers to use directional cues, through pointing, to an AUV for inspection, surveillance, manipulation, and navigation. We examine the elements that make up our method, provide quantitative and qualitative evaluation, and demonstrate AUV actuation based on diver pointing gestures in closed-water human-robot collaborative experiments. Our evaluations demonstrate the high efficacy of the DIP algorithm in correctly identifying the direction of a pointing gesture and locating an object within that region of interest. We also show that the findings of the algorithm qualitatively conform with human assessment of pointing gestures, directions, and targets.

ICRA Conference 2023 Conference Paper

HREyes: Design, Development, and Evaluation of a Novel Method for AUVs to Communicate Information and Gaze Direction *

Michael Fulton
Aditya Prabhu
Junaed Sattar

We present the design, development, and evaluation of HREyes: biomimetic communication devices which use light to communicate information and, for the first time, gaze direction from AUVs to humans. First, we introduce two types of information displays using the HREye devices: active lucemes and ocular lucemes. Active lucemes communicate information explicitly through animations, while ocular lucemes communicate gaze direction implicitly by mimicking human eyes. We present a human study in which our system is compared to the use of an embedded digital display that explicitly communicates information to a diver by displaying text. Our results demonstrate accurate recognition of active lucemes for trained interactants, limited intuitive understanding of these lucemes for untrained interactants, and relatively accurate perception of gaze direction for all interactants. The results on active luceme recognition demonstrate more accurate recognition than previous light-based communication systems for AUVs (albeit with different phrase sets). Additionally, the ocular lucemes we introduce in this work represent the first method for communicating gaze direction from an AUV, a critical aspect of nonverbal communication used in collabo-rative work. With readily available hardware as well as open-source and easily re-configurable programming, HREyes can be easily integrated into any AUV with the physical space for the devices and used to communicate effectively with divers in any underwater environment with appropriate visibility.

ICRA Conference 2022 Conference Paper

Continuous-Time Spline Visual-Inertial Odometry

Jiawei Mo
Junaed Sattar

We propose a continuous-time spline-based formulation for visual-inertial odometry (VIO). Specifically, we model the poses as a cubic spline, whose temporal derivatives are used to synthesize linear acceleration and angular velocity, which are compared to the measurements from the inertial measurement unit (IMU) for optimal state estimation. The spline boundary conditions create constraints between the camera and the IMU, with which we formulate VIO as a constrained nonlinear optimization problem. Continuous-time pose representation makes it possible to address many VIO challenges, e. g. , rolling shutter distortion and sensors that may lack synchronization. We conduct experiments on two publicly available datasets that demonstrate the state-of-the-art accuracy and real-time computational efficiency of our method.

IROS Conference 2022 Conference Paper

Robotic Detection of a Human-Comprehensible Gestural Language for Underwater Multi-Human-Robot Collaboration

Sadman Sakib Enan
Michael Fulton
Junaed Sattar

In this paper, we present a motion-based robotic communication framework that enables non-verbal communication among autonomous underwater vehicles (AUVs) and human divers. We design a gestural language for AUV-to-AUV communication which can be easily understood by divers observing the conversation - unlike typical radio frequency, light, or audio-based AUV communication. To allow AUVs to visually understand a gesture from another AUV, we propose a deep network (RRCommNet) which exploits a self-attention mechanism to learn to recognize each message by extracting maximally discriminative spatio-temporal features. We train this network on diverse simulated and real-world data. Our experimental evaluations, both in simulation and in closed-water robot trials, demonstrate that the proposed RRCommNet architecture is able to decipher gesture-based messages with an average accuracy of 88-94% on simulated data and 73-83% on real data (depending on the version of the model used). Further, by performing a message transcription study with human participants, we also show that the proposed language can be understood by humans with an overall transcription accuracy of 88 %. Finally, we discuss the inference runtime of RRCommNet on embedded GPU hardware, for real-time use on board AUVs in the field.

ICRA Conference 2022 Conference Paper

ROW-SLAM: Under-Canopy Cornfield Semantic SLAM

Jiacheng Yuan
Jungseok Hong
Junaed Sattar
Volkan Isler

We study a semantic SLAM problem where a robot is tasked with autonomous weeding under the corn canopy. The goal is to detect corn stalks and localize them in a global coordinate frame. This is a challenging scenario for existing algorithms because there is very little space between the camera and the plants, and the camera motion is primarily restricted to be along the row. To overcome these challenges, we present a multi-camera system where a side camera (facing the plants) is used for detection, whereas front and back cameras are used for motion estimation. Next, we show how semantic features in the environment (corn stalks, ground, and crop planes) can be used to develop a robust semantic SLAM solution and present results from field trials performed throughout the growing season across various cornfields.

ICRA Conference 2022 Conference Paper

Using Monocular Vision and Human Body Priors for AUVs to Autonomously Approach Divers

Michael Fulton
Jungseok Hong
Junaed Sattar

Direct communication between humans and autonomous underwater vehicles (AUVs) is a relatively under-explored area in human-robot interaction research, although many tasks (e. g. , surveillance, inspection, and search-and-rescue) require close diver-robot collaboration. Suboptimal AUV positioning relative to its human collaborators can lead to poor quality interaction and lead to excessive cognitive and physical load for divers. In this paper, we introduce a novel method for AUVs to autonomously navigate and achieve diver-relative positioning to begin interaction. Our method is based only on monocular vision, requires no global localization, and is computationally efficient. We present our algorithm and its implementation on board a physical AUV, performing extensive evaluations in the form of closed-water tests in a controlled pool. Our results show that the proposed monocular vision-based algorithm performs reliably and efficiently, operating entirely on-board the AUV.

IROS Conference 2021 Conference Paper

Predicting the Future Motion of Divers for Enhanced Underwater Human-Robot Collaboration

Tanmay Agarwal
Michael Fulton
Junaed Sattar

Autonomous Underwater Vehicles (AUVs) can be effective collaborators to human scuba divers in many applications, such as environmental surveying, mapping, or infrastructure repair. However, for these applications to be realized in the real world, it is essential that robots are able to both lead and follow their human collaborators. Current algorithms for diver following are not robust to non-uniform changes in the motion of the diver, and no framework currently exists for robots to lead divers. One method to improve the robustness of diver following and enable the capability of diver leading is to predict the future motion of a diver. In this paper, we present a vision-based approach for AUVs to predict the future motion trajectory of divers, utilizing the Vanilla-LSTM and Social-LSTM temporal deep neural networks. We also present a dense optical flow-based method to stabilize the input annotations from the dataset and reduce the effects of camera ego-motion. We analyze the results of these models on scenarios ranging from swimming pools to the open ocean and present the model’s accuracy at varying prediction lengths. We find that our LSTM models can generate predictions with significant accuracy 1. 5 seconds into the future and that stabilizing LSTM models significantly improves trajectory prediction performance.

ICRA Conference 2021 Conference Paper

Semantically-Aware Strategies for Stereo-Visual Robotic Obstacle Avoidance

Jungseok Hong
Karin de Langis
Cole Wyeth
Christopher Walaszek
Junaed Sattar

Mobile robots in unstructured, mapless environments must rely on an obstacle avoidance module to navigate safely. The standard avoidance techniques estimate the locations of obstacles with respect to the robot but are unaware of the obstacles’ identities. Consequently, the robot cannot take advantage of semantic information about obstacles when making decisions about how to navigate. We propose an obstacle avoidance module that combines visual instance segmentation with a depth map to classify and localize objects in the scene. The system avoids obstacles differentially, based on the identity of the objects: for example, the system is more cautious in response to unpredictable objects such as humans. The system can also navigate closer to harmless obstacles and ignore obstacles that pose no collision danger, enabling it to navigate more efficiently. We validate our approach in two simulated environments: one terrestrial and one underwater. Results indicate that our approach is feasible and can enable more efficient navigation strategies.

IROS Conference 2021 Conference Paper

Towards Robust Visual Diver Detection Onboard Autonomous Underwater Robots: Assessing the Effects of Models and Data

Karin de Langis
Michael Fulton
Junaed Sattar

Deep neural networks are the leading solution to the object detection problem. However, challenges arise when applying these networks to the kind of real-time, first-person video data that a robotic platform must process: specifically, detections may not be consistent from frame to frame, and objects may frequently appear at viewpoints that are particularly challenging for the model, resulting in inaccurate detections. In this paper, we present our approach for addressing these challenges for our particular vision problem: diver detection onboard autonomous underwater vehicles (AUVs). We begin by producing and releasing a dataset of approximately 105, 000 annotated images of divers sourced from videos in order to address the challenge of learning a wide variety of object rotations and translations. This is one of the largest and most varied diver detection datasets ever created, and we compare models trained and tested on both our dataset and a previous dataset to demonstrate that our dataset improves the state-of-the-art in diver detection. Then, in order to choose an object detection model that produces detections that are consistent from frame to frame, we evaluate several state-of-the-art object detection models on the temporal stability of their detections in addition to the typical accuracy and efficiency metrics, mean average precision (mAP) and frames per second. Importantly, our results showed that models with the highest mAP do not also have the highest temporal stability.

IROS Conference 2020 Conference Paper

A Fast and Robust Place Recognition Approach for Stereo Visual Odometry Using LiDAR Descriptors

Jiawei Mo
Junaed Sattar

Place recognition is a core component of Simultaneous Localization and Mapping (SLAM) algorithms. Particularly in visual SLAM systems, previously-visited places are recognized by measuring the appearance similarity between images representing these locations. However, such approaches are sensitive to visual appearance change and also can be computationally expensive. In this paper, we propose an alternative approach adapting LiDAR descriptors for 3D points obtained from stereo-visual odometry for place recognition. 3D points are potentially more reliable than 2D visual cues (e. g. , 2D features) against environmental changes (e. g. , variable illumination) and this may benefit visual SLAM systems in long-term deployment scenarios. Stereo-visual odometry generates 3D points with an absolute scale, which enables us to use LiDAR descriptors for place recognition with high computational efficiency. Through extensive evaluations on standard benchmark datasets, we demonstrate the accuracy, efficiency, and robustness of using 3D points for place recognition over 2D methods.

ICRA Conference 2020 Conference Paper

A Generative Approach Towards Improved Robotic Detection of Marine Litter

Jungseok Hong
Michael Fulton
Junaed Sattar

This paper presents an approach to address data scarcity problems in underwater image datasets for visual detection of marine debris. The proposed approach relies on a two-stage variational autoencoder (VAE) and a binary classifier to evaluate the generated imagery for quality and realism. From the images generated by the two-stage VAE, the binary classifier selects "good quality" images and augments the given dataset with them. Lastly, a multi-class classifier is used to evaluate the impact of the augmentation process by measuring the accuracy of an object detector trained on combinations of real and generated trash images. Our results show that the classifier trained with the augmented data outperforms the one trained only with the real data. This approach will not only be valid for the underwater trash classification problem presented in this paper, but it will also be useful for any data-dependent task for which collecting more images is challenging or infeasible.

IROS Conference 2020 Conference Paper

Design and Experiments with LoCO AUV: A Low Cost Open-Source Autonomous Underwater Vehicle

Chelsey Edge
Sadman Sakib Enan
Michael Fulton
Jungseok Hong
Jiawei Mo
Kimberly Barthelemy
Hunter Bashaw
Berik Kallevig

In this paper we present the LoCO AUV, a Low-Cost, Open Autonomous Underwater Vehicle. LoCO is a general-purpose, single-person-deployable, vision-guided AUV, rated to a depth of 100 meters. We discuss the open and expandable design of this underwater robot, as well as the design of a simulator in Gazebo. Additionally, we explore the platform's preliminary local motion control and state estimation abilities, which enable it to perform maneuvers autonomously. In order to demonstrate its usefulness for a variety of tasks, we implement a variety of our previously presented human-robot interaction capabilities on LoCO, including gestural control, diver following, and robot communication via motion. Finally, we discuss the practical concerns of deployment and our experiences in using this robot in pools, lakes, and the ocean. All design details, instructions on assembly, and code will be released under a permissive, open-source license.

ICRA Conference 2020 Conference Paper

Realtime Multi-Diver Tracking and Re-identification for Underwater Human-Robot Collaboration

Karin de Langis
Junaed Sattar

Autonomous underwater robots working with teams of human divers may need to distinguish between different divers, e. g. , to recognize a lead diver or to follow a specific team member. This paper describes a technique that enables autonomous underwater robots to track divers in real time as well as to reidentify them. The approach is an extension of Simple Online Realtime Tracking (SORT) with an appearance metric (deep SORT). Initial diver detection is performed with a custom CNN designed for realtime diver detection, and appearance features are subsequently extracted for each detected diver. Next, realtime tracking-by-detection is performed with an extension of the deep SORT algorithm. We evaluate this technique on a series of videos of divers performing human-robot collaborative tasks and show that our methods result in more divers being accurately identified during tracking. We also discuss the practical considerations of applying multi-person tracking to on-board autonomous robot operations, and we consider how failure cases can be addressed during on-board tracking.

IROS Conference 2020 Conference Paper

Semantic Segmentation of Underwater Imagery: Dataset and Benchmark

Md Jahidul Islam
Chelsey Edge
Yuyang Xiao
Peigen Luo
Muntaqim Mehtaz
Christopher Morse
Sadman Sakib Enan
Junaed Sattar

In this paper, we present the first large-scale dataset for semantic Segmentation of Underwater IMagery (SUIM). It contains over 1500 images with pixel annotations for eight object categories: fish (vertebrates), reefs (invertebrates), aquatic plants, wrecks/ruins, human divers, robots, and sea-floor. The images have been rigorously collected during oceanic explorations and human-robot collaborative experiments, and annotated by human participants. We also present a comprehensive benchmark evaluation of several state-of-the-art semantic segmentation approaches based on standard performance metrics. Additionally, we present SUIM-Net, a fully-convolutional deep residual model that balances the trade-off between performance and computational efficiency. It offers competitive performance while ensuring fast end-to-end inference, which is essential for its use in the autonomy pipeline by visually-guided underwater robots. In particular, we demonstrate its usability benefits for visual servoing, saliency prediction, and detailed scene understanding. With a variety of use cases, the proposed model and benchmark dataset open up promising opportunities for future research in underwater robot vision.

ICRA Conference 2020 Conference Paper

Underwater Image Super-Resolution using Deep Residual Multipliers

Md Jahidul Islam
Sadman Sakib Enan
Peigen Luo
Junaed Sattar

We present a deep residual network-based generative model for single image super-resolution (SISR) of underwater imagery for use by autonomous underwater robots. We also provide an adversarial training pipeline for learning SISR from paired data. In order to supervise the training, we formulate an objective function that evaluates the perceptual quality of an image based on its global content, color, and local style information. Additionally, we present USR-248, a large-scale dataset of three sets of underwater images of `high' (640×480) and `low' (80 × 60, 160 × 120, and 320×240) resolution. USR-248 contains paired instances for supervised training of 2×, 4×, or 8× SISR models. Furthermore, we validate the effectiveness of our proposed model through qualitative and quantitative experiments and compare the results with several state-of-the-art models' performances. We also analyze its practical feasibility for applications such as scene understanding and attention modeling in noisy visual conditions.

IROS Conference 2019 Conference Paper

Extending Monocular Visual Odometry to Stereo Camera Systems by Scale optimization

Jiawei Mo
Junaed Sattar

This paper proposes a novel approach for extending monocular visual odometry to a stereo camera system. The proposed method uses an additional camera to accurately estimate and optimize the scale of the monocular visual odometry, rather than triangulating 3D points from stereo matching. Specifically, the 3D points generated by the monocular visual odometry are projected onto the other camera of the stereo pair, and the scale is recovered and optimized by directly minimizing the photometric error. It is computationally efficient, adding minimal overhead to the stereo vision system compared to straightforward stereo matching, and is robust to repetitive texture. Additionally, direct scale optimization enables stereo visual odometry to be purely based on the direct method. Extensive evaluation on public datasets (e. g. , KITTI), and outdoor environments (both terrestrial and underwater) demonstrates the accuracy and efficiency of a stereo visual odometry approach extended by scale optimization, and its robustness in environments with challenging textures.

ICRA Conference 2019 Conference Paper

Robot Communication Via Motion: Closing the Underwater Human-Robot Interaction Loop

Michael Fulton
Chelsey Edge
Junaed Sattar

In this paper, we propose a novel method for underwater robot-to-human communication using the motion of the robot as “body language”. To evaluate this system, we develop simulated examples of the system's body language gestures, called kinemes, and compare them to a baseline system using flashing colored lights through a user study. Our work shows evidence that motion can be used as a successful communication vector which is accurate, easy to learn, and quick enough to be used, all without requiring any additional hardware to be added to our platform. We thus contribute to “closing the loop” for human-robot interaction underwater by proposing and testing this system, suggesting a library of possible body language gestures for underwater robots, and offering insight on the design of nonverbal robot-to-human communication methods.

ICRA Conference 2019 Conference Paper

Robotic Detection of Marine Litter Using Deep Visual Detection Models

Michael Fulton
Jungseok Hong
Md Jahidul Islam
Junaed Sattar

Trash deposits in aquatic environments have a destructive effect on marine ecosystems and pose a long-term economic and environmental threat. Autonomous underwater vehicles (AUVs) could very well contribute to the solution of this problem by finding and eventually removing trash. This paper evaluates a number of deep-learning algorithms performing the task of visually detecting trash in realistic underwater environments, with the eventual goal of exploration, mapping, and extraction of such debris by using AUVs. A large and publicly-available dataset of actual debris in open-water locations is annotated for training a number of convolutional neural network architectures for object detection. The trained networks are then evaluated on a set of images from other portions of that dataset, providing insight into approaches for developing the detection capabilities of an AUV for underwater trash removal. In addition, the evaluation is performed on three different platforms of varying processing power, which serves to assess these algorithms' fitness for real-time applications.

ICRA Conference 2019 Conference Paper

Visual Diver Recognition for Underwater Human-Robot Collaboration

Youya Xia
Junaed Sattar

This paper presents an approach for autonomous underwater robots to visually detect and identify divers. The proposed approach enables an autonomous underwater robot to detect multiple divers in a visual scene and distinguish between them. Such methods are useful for robots to identify a human leader, for example, in multi-human/robot teams where only designated individuals are allowed to command or lead a team of robots. Initial diver identification is performed using the Faster R-CNN algorithm with a region proposal network which produces bounding boxes around the divers' locations. Subsequently, a suite of spatial and frequency domain descriptors are extracted from the bounding boxes to create a feature vector. A K-Means clustering algorithm, with k set to the number of detected bounding boxes, thereafter identifies the detected divers based on these feature vectors. We evaluate the performance of the proposed approach on video footage of divers swimming in front of a mobile robot and demonstrate its accuracy.

ICRA Conference 2018 Conference Paper

Dynamic Reconfiguration of Mission Parameters in Underwater Human-Robot Collaboration

Md Jahidul Islam
Marc Ho
Junaed Sattar

This paper presents a real-time programming and parameter reconfiguration method for autonomous underwater robots in human-robot collaborative tasks. Using a set of intuitive and meaningful hand gestures, we develop a syntactically simple framework that is computationally more efficient than a complex, grammar-based approach. In the proposed framework, a convolutional neural network is trained to provide accurate hand gesture recognition; subsequently, a finite-state machine- based deterministic model performs efficient gesture-to-instruction mapping and further improves robustness of the interaction scheme. The key aspect of this framework is that it can be easily adopted by divers for communicating simple instructions to underwater robots without using artificial tags such as fiducial markers or requiring memorization of a potentially complex set of language rules. Extensive experiments are performed both on field-trial data and through simulation, which demonstrate the robustness, efficiency, and portability of this framework in a number of different scenarios. Finally, a user interaction study is presented that illustrates the gain in the ease of use of our proposed interaction framework compared to the existing methods for the underwater domain.

ICRA Conference 2018 Conference Paper

Enhancing Underwater Imagery Using Generative Adversarial Networks

Cameron Fabbri
Md Jahidul Islam
Junaed Sattar

Autonomous underwater vehicles (AUVs) rely on a variety of sensors - acoustic, inertial and visual - for intelligent decision making. Due to its non-intrusive, passive nature and high information content, vision is an attractive sensing modality, particularly at shallower depths. However, factors such as light refraction and absorption, suspended particles in the water, and color distortion affect the quality of visual data, resulting in noisy and distorted images. AUVs that rely on visual sensing thus face difficult challenges and consequently exhibit poor performance on vision-driven tasks. This paper proposes a method to improve the quality of visual underwater scenes using Generative Adversarial Networks (GANs), with the goal of improving input to vision-driven behaviors further down the autonomy pipeline. Furthermore, we show how recently proposed methods are able to generate a dataset for the purpose of such underwater image restoration. For any visually-guided underwater robots, this improvement can result in increased safety and reliability through robust visual perception. To that effect, we present quantitative and qualitative data which demonstrates that images corrected through the proposed approach generate more visually appealing images, and also provide increased accuracy for a diver tracking algorithm.

ICRA Conference 2017 Conference Paper

Mixed-domain biological motion tracking for underwater human-robot interaction

Md Jahidul Islam
Junaed Sattar

We present a novel algorithm for an autonomous underwater robot to visually detect and follow its companion human diver. Using both spatial-domain and frequency-domain features pertaining to human swimming patterns, we devise an algorithm to visually detect the position and swimming direction of the diver. Our algorithm is unique in the way that it allows detection of arbitrary motion directions, in addition to keeping track of a diver's position through the image sequence over time. A Hidden Markov Model (HMM)-based approach prunes the search-space of all potential trajectories relying on image intensities in the spatial-domain. A diver's motion signature is subsequently detected in a sequence of non-overlapping image subwindows exhibiting human swimming patterns. The pruning step ensures efficient computation by avoiding exponentially large search-spaces, whereas the frequency-domain detection allows us to detect the diver's position and motion direction accurately. Experimental validation of the proposed approach is presented on datasets collected from open-water and closed-water environments.

IROS Conference 2017 Conference Paper

Underwater multi-robot convoying using visual tracking by detection

Florian Shkurti
Wei-Di Chang
Peter Henderson 0002
Md Jahidul Islam
Juan Camilo Gamboa Higuera
Jimmy Li 0001
Travis Manderson
Anqi Xu 0003

We present a robust multi-robot convoying approach that relies on visual detection of the leading agent, thus enabling target following in unstructured 3-D environments. Our method is based on the idea of tracking-by-detection, which interleaves efficient model-based object detection with temporal filtering of image-based bounding box estimation. This approach has the important advantage of mitigating tracking drift (i. e. drifting away from the target object), which is a common symptom of model-free trackers and is detrimental to sustained convoying in practice. To illustrate our solution, we collected extensive footage of an underwater robot in ocean settings, and hand-annotated its location in each frame. Based on this dataset, we present an empirical comparison of multiple tracker variants, including the use of several convolutional neural networks, both with and without recurrent connections, as well as frequency-based model-free trackers. We also demonstrate the practicality of this tracking-by-detection strategy in real-world scenarios by successfully controlling a legged underwater robot in five degrees of freedom to follow another robot's independent motion.

IROS Conference 2014 Conference Paper

A risk assessment infrastructure for powered wheelchair motion commands without full sensor coverage

Pouria TalebiFard
Junaed Sattar
Ian M. Mitchell

Smart powered wheelchairs offer the possibility of enhanced mobility to a large and growing population-most notably older adults-and a key feature of such a chair is collision avoidance. Sensors are required to detect nearby obstacles; however, complete sensor coverage of the immediate neighbourhood is challenging for reasons including financial, computational, aesthetic, user identity and sensor reliability. It is also desirable to predict the future motion of the wheelchair based on potential input signals; however, direct modeling and control of commercial wheelchairs is not possible because of proprietary internals and interfaces. In this paper we design a dynamic egocentric occupancy map which maintains information about local obstacles even when they are outside the field of view of the sensor system, and we construct a neural network model of the mapping between joystick inputs and wheelchair motion. Using this map and model infrastructure, we can evaluate a variety of risk assessment metrics for collaborative control of a smart wheelchair. One such metric is demonstrated on a wheelchair with a single RGB-D camera in two scenarios: a doorway traversal where the near edge of the doorframe is no longer visible to the camera as the chair makes its turn, and a longer navigation through a typical cluttered office environment.

ICRA Conference 2014 Conference Paper

Ensuring safety in human-robot dialog - A cost-directed approach

Junaed Sattar
James J. Little

We present an approach for detecting potentially unsafe commands in human-robot dialog, where a robotic system evaluates task cost in input commands to ask input-specific, directed questions to ensure safe task execution. The goal is to reduce risk, both to the robot and the environment, by asking context-appropriate questions. Given an input program, (i. e. , a sequence of commands) the system evaluates a set of likely alternate programs along with their likelihood and cost, and these are given as input to a Decision Function to decide whether to execute the task or confirm the plan from the human partner. A process called token-risk grounding identifies the costly commands in the programs, and specifically asks the human user to clarify those commands. We evaluate our system in two simulated robot tasks, and also on-board the Willow Garage PR2 and TurtleBot robots in an indoor task setting. In both sets of evaluations, the results show that the system is able to identify specific commands that contribute to high task cost, and present users the option to either confirm or modify those commands. In addition to ensuring task safety, this results in an overall reduction in robot reprogramming time.

ICRA Conference 2011 Conference Paper

Towards quantitative modeling of task confirmations in human-robot dialog

Junaed Sattar
Gregory Dudek

We present a technique for robust human-robot interaction taking into consideration uncertainty in input and task execution costs incurred by the robot. Specifically, this research aims to quantitatively model confirmation feedback, as required by a robot while communicating with a human operator to perform a particular task. Our goal is to model human-robot interaction from the perspective of risk minimization, taking into account errors in communication, “risk” involved in performing the required task, and task execution costs. Given an input modality with non-trivial uncertainty, we calculate the cost associated with performing the task specified by the user, and if deemed necessary, ask the user for confirmation. The estimated task cost and the uncertainty measure are given as input to a Decision Function, the output of which is then used to decide whether to execute the task, or request clarification from the user. We test our system through human-interface experiments, based on a framework custom-designed for our family of amphibious robots, and demonstrate the utility of the framework in the presence of large task costs and uncertainties. We also present qualitative results of our algorithm from field trials of our robots in both open- and closed-water environments.

ICRA Conference 2010 Conference Paper

Graphical state-space programmability as a natural interface for robotic control

Junaed Sattar
Anqi Xu 0003
Gregory Dudek
Gabriel Charette

We present an interface for controlling mobile robots that combines aspects of graphical trajectory specification and state-based programming. This work is motivated by common tasks executed by our underwater vehicles, although we illustrate a mode of interaction that is applicable to mobile robotics in general. The key aspect of our approach is to provide an intuitive linkage between the graphical visualization of regions of interest in the environment, and activities relevant to these regions. In addition to introducing this novel programming paradigm, we also describe the associated system architecture developed on-board our amphibious robot. We then present a user interaction study that illustrates the benefits in usability of our graphical interface, compared to conventionally established programming techniques.

ICRA Conference 2009 Conference Paper

Robust servo-control for underwater robots using banks of visual filters

Junaed Sattar
Gregory Dudek

We present an application of machine learning to the semi-automatic synthesis of robust servo-trackers for underwater robotics. In particular, we investigate an approach based on the use of Boosting for robust visual tracking of color objects in an underwater environment. To this end, we use AdaBoost, the most common variant of the Boosting algorithm, to select a number of low-complexity but moderately accurate color feature trackers and we combine their outputs. The novelty of our approach lies in the design of this family of weak trackers, which enhances a straightforward color segmentation tracker in multiple ways. From a large and diverse family of possible filters, we select a small subset that optimizes the performance of our trackers. The tracking process applies these trackers on the input video frames, and the final tracker output is chosen based on the weights of the final array of trackers. By using computationally inexpensive, but somewhat accurate trackers as members of the ensemble, the system is able to run at quasi real-time, and thus, is deployable on-board our underwater robot. We present quantitative cross-validation results of our spatio-chromatic visual tracker, and conclude by pointing out some difficulties faced and subsequent shortcomings in the experiments we performed, along with directions of future research in the area of ensemble tracking in real-time.

ICRA Conference 2008 Conference Paper

A natural gesture interface for operating robotic systems

Anqi Xu 0003
Gregory Dudek
Junaed Sattar

A gesture-based interaction framework is presented for controlling mobile robots. This natural interaction paradigm has few physical requirements, and thus can be deployed in many restrictive and challenging environments. We present an implementation of this scheme in the control of an underwater robot by an on-site human operator. The operator performs discrete gestures using engineered visual targets, which are interpreted by the robot as parametrized actionable commands. By combining the symbolic alphabets resulting from several visual cues, a large vocabulary of statements can be produced. An iterative closest point algorithm is used to detect these observed motions, by comparing them with an established database of gestures. Finally, we present quantitative data collected from human participants indicating accuracy and performance of our proposed scheme.

IROS Conference 2008 Conference Paper

Enabling autonomous capabilities in underwater robotics

Junaed Sattar
Gregory Dudek
Olivia Chiu
Ioannis M. Rekleitis
Philippe Giguère
Alec Mills
Nicolas Plamondon
Chris Prahacs

Underwater operations present unique challenges and opportunities for robotic applications. These can be attributed in part to limited sensing capabilities, and to locomotion behaviours requiring control schemes adapted to specific tasks or changes in the environment. From enhancing teleoperation procedures, to providing high-level instruction, all the way to fully autonomous operations, enabling autonomous capabilities is fundamental for the successful deployment of underwater robots. This paper presents an overview of the approaches used during underwater sea trials in the coral reefs of Barbados, for two amphibious mobile robots and a set of underwater sensor nodes. We present control mechanisms used for maintaining a preset trajectory during enhanced teleoperations and discuss their experimental results. This is followed by a discussion on amphibious data gathering experiments conducted on the beach. We then present a tetherless underwater communication approach based on pure vision for high-level control of an underwater vehicle. Finally the construction details together with preliminary results from a set of distributed underwater sensor nodes are outlined.

ICRA Conference 2007 Conference Paper

A Visual Language for Robot Control and Programming: A Human-Interface Study

Gregory Dudek
Junaed Sattar
Anqi Xu 0003

We describe an interaction paradigm for controlling a robot using hand gestures. In particular, we are interested in the control of an underwater robot by an on-site human operator. Under this context, vision-based control is very attractive, and we propose a robot control and programming mechanism based on visual symbols. A human operator presents engineered visual targets to the robotic system, which recognizes and interprets them. This paper describes the approach and proposes a specific gesture language called "RoboChat". RoboChat allows an operator to control a robot and even express complex programming concepts, using a sequence of visually presented symbols, encoded into fiducial markers. We evaluate the efficiency and robustness of this symbolic communication scheme by comparing it to traditional gesture-based interaction involving a remote human operator

IROS Conference 2007 Conference Paper

Where is your dive buddy: tracking humans underwater using spatio-temporal features

Junaed Sattar
Gregory Dudek

We present an algorithm for underwater robots to track mobile targets, and specifically human divers, by detecting periodic motion. Periodic motion is typically associated with propulsion underwater and specifically with the kicking of human swimmers. By computing local amplitude spectra in a video sequence, we find the location of a diver in the robot's field of view. We use the Fourier transform to extract the responses of varying intensities in the image space over time to detect characteristic low frequency oscillations to identify an undulating flipper motion associated with typical gaits. In case of detecting multiple locations that exhibit large low-frequency energy responses, we combine the gait detector with other methods to eliminate false detections. We present results of our algorithm on open-ocean video footage of swimming divers, and also discuss possible extensions and enhancements of the proposed approach for tracking other objects that exhibit low- frequency oscillatory motion.

ICRA Conference 2006 Conference Paper

On the Performance of Color Tracking Algorithms for Underwater Robots under Varying Lighting and Visibility

Junaed Sattar
Gregory Dudek

We consider the use of visual target tracking for autonomous steering of an underwater robot. In this context, we consider a performance comparison for three key visual tracking algorithms used for servo control. We present a comparative study of the performance in underwater environments of three tracking algorithms that are widely used in vision applications. Variations in illumination, suspended particles and a resulting reduction in visibility hinders vision systems from performing satisfactorily in marine environments; at least not as well as they do in terrestrial (Le. non-underwater) surroundings. Our work focuses on quantitatively measuring the performance of three color-based tracking algorithms- color blob tracker, color histogram tracker and mean-shift tracker, in tracking objects underwater in different levels lighting and visibility. We also present results demonstrating the effect of suspended particles underwater, and in conclusion we summarize the three tracking algorithms by comparing their pros and cons

IROS Conference 2005 Conference Paper

A visual servoing system for an aquatic swimming robot

Junaed Sattar
Philippe Giguère
Gregory Dudek
Chris Prahacs

This paper describes a visual servoing system for an underwater legged robotic system named AQUA and initial experiments with the system performed in the open sea. A large class of significant applications can be leveraged by allowing such a robot to follow a diver or some other moving target. The robot uses a suite of sensing technologies, primarily based on computer vision, to allow it to navigate in shallow-water environments. The visual servoing system described here allows the robot to track and follow a given target underwater. The servo package is made up of two distinct parts: a tracker and a feedback controller. The system has been evaluated in the sea water and under natural lighting conditions. The servo system has been tested underwater, and with minor modifications, the system can be used while the robot is walking on the ground as well.

IROS Conference 2005 Conference Paper

A visually guided swimming robot

Gregory Dudek
Michael Jenkin
Chris Prahacs
Andrew Hogue
Junaed Sattar
Philippe Giguère
Andrew German
Hui Liu

We describe recent results obtained with AQUA, a mobile robot capable of swimming, walking and amphibious operation. Designed to rely primarily on visual sensors, the AQUA robot uses vision to navigate underwater using servo-based guidance, and also to obtain high-resolution range scans of its local environment. This paper describes some of the pragmatic and logistic obstacles encountered, and provides an overview of some of the basic capabilities of the vehicle and its associated sensors. Moreover, this paper presents the first ever amphibious transition from walking to swimming.